Contents
At a glance
- Category
- Agent concept - model plumbing
- Measured in
- Tokens, not words or characters
- What must fit
- System prompt, tool definitions, conversation, retrieved text, and the reply
- Grows fastest from
- Unfiltered tool results appended on every agent step
- Known weakness
- Attention to material in the middle of a long context is less reliable
- Billed as
- Input tokens on every call, partly offset by prompt caching of stable prefixes
What a context window is
The context window is the model's working memory for a single request, measured in tokens. Everything the model can see must fit: the system prompt, the tool definitions, the conversation so far, any documents you retrieved, and the space reserved for the reply. Exceed it and the request is rejected or silently truncated, which is worse.
The number is a headline spec and a poor one. A model advertising a million-token window will happily accept a huge document and still answer badly about the middle of it. Capacity and effective use are different properties, and the gap between them is where a lot of disappointing AI projects live.
What fills it, and how fast
In an agent the window fills by accumulation. Every step appends the model's reasoning, the tool call and the tool result, and the next step re-reads all of it. A tool that returns a full API response rather than the three fields you needed can add thousands of tokens per call, which is the most common cause of an agent that gets slower and more expensive as it runs and then stops making sense at step seven.
The fixes are unglamorous. Trim tool results to the fields the model actually needs before they enter the transcript. Summarise older turns instead of carrying them verbatim. Cap the number of retrieved chunks in your RAG step rather than sending twenty in the hope one helps. Set a hard step limit so the window cannot grow without bound.
Cost and the degradation curve
Context is billed as input tokens, so a large window is a direct cost multiplier on every step of every run. Prompt caching offsets some of it for the stable prefix, but only for the parts that do not change, which is why the ordering of your prompt matters.
Quality degrades before capacity does. Models reliably attend to the start and end of a long context and are less dependable about the middle - the effect is well enough known to have its own name, "lost in the middle". The practical implication is to put the instruction and the most important retrieved passage close to the model's attention, near the top and the bottom, rather than assuming that anything inside the window is equally read.
What you should know
Do not treat a big window as a substitute for retrieval. Pasting a whole knowledge base into every request costs more, answers worse and cannot be updated without re-sending everything. Retrieval selects the few passages that matter; the window is where they land.
Practical discipline: measure the token count of a typical run at each step and look at what is growing. Nine times out of ten it is unfiltered tool output. That measurement belongs in your observability traces from day one, because by the time a run is failing at step nine, guessing which step bloated it is a waste of an afternoon.
Current windows, and the one that is priced in bands
Published figures rather than generalities, checked August 2026. OpenAI's GPT-5.6 family - sol, terra and luna - all carry a 1,050,000-token context window with a 128,000-token maximum output, so the cheap tier is not the small-window tier. DeepSeek V4 Pro and V4 Flash both publish a 1M context length with 384K maximum output.
The important exception is gemini-2.5-pro, where the window is not just a capacity limit but a price boundary: Google prices prompts of 200,000 tokens or fewer at $1.25 input and $10.00 output per million, and prompts above that at $2.50 and $15.00. Crossing the line reprices the whole request rather than just the excess, which turns "how much context should I send" from an engineering question into a billing one.
Context window FAQ
What is a context window in an LLM?
The maximum amount of text a model can consider in one request, counted in tokens. It has to hold the system prompt, tool definitions, the conversation, any retrieved documents, and the space for the answer. Nothing outside it exists as far as the model is concerned.
Is a bigger context window always better?
No. Capacity and effective use are different. Models attend less reliably to material in the middle of very long contexts, and every extra token is billed on every call. A well-retrieved thousand tokens usually beats a hundred thousand tokens of hopeful pasting.
Why does my agent get worse after several steps?
Because the transcript has grown and the original instructions are now buried under pages of tool output. Trim tool results before they enter the context, summarise older turns, and cap the number of steps. Logging tokens per step shows you exactly where the bloat started.
Does a long context replace the need for RAG?
No. You could paste an entire handbook into the prompt, but you would pay for all of it on every call, update it by re-sending everything, and get worse answers than selecting the three relevant passages. Retrieval decides what goes in the window; the window is not a database.
How do I count tokens before sending?
Use your provider's tokenizer library, or the tokenizer utilities in the Hugging Face ecosystem for open models. A rough English rule of thumb is around four characters per token, but it is only a rule of thumb - code, names, numbers and other languages tokenize very differently.
What happens if I exceed the context window?
The API returns an error, or your framework silently truncates - usually from the oldest messages, which may include the system prompt. Silent truncation is the dangerous case because the run continues with the rules removed. Check what your framework does before you hit the limit in production.
Primary source
OpenAI - text generation and context guide
Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.
Want this built rather than explained?
Knowing what context window means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.
