It's amazing how useful large context windows are in LLMs.

It's barely been a year since we had to deal with miniscule context windows of 4k tokens or so.

It was like living in the stone age, can you even imagine?

We had to do techniques like RAG (Retrieval Augmented Generation) to allow LLMs to work with large bases of knowledge.

We'd use RAG to do fuzzy semantic based matches for things that seemed related to the user's question and copy/paste as many into the context window as would fit.

This worked OK for factual questions, but it couldn't answer questions like "What are the major themes in this work", because the theme isn't a collection of small, semantically obvious facts; it's a high level vibe.

LLMs today with massive context windows are able to do quite good jobs at identifying themes when given the whole work.

And yet already it's easy to take large context windows for granted.

What progress we've made!