In brief
- A token can be a word, word fragment, punctuation mark or another encoded unit.
- Token count depends on language, spelling and the model's tokenizer.
- A large context window allows more material but does not guarantee complete attention or correct answers.
- Long contexts usually increase cost and latency and still require deliberate selection of relevant content.
Why do models use tokens?
Models process numbers. A tokenizer translates text into numbered units the model can calculate with. Common words may form one token, while rare or compound words require several. Pages therefore cannot be converted into tokens with a reliable fixed ratio.
- Instructions and user input consume the token budget
- Document passages and tool results sent with the request also count
- Part of the budget must be reserved for the generated answer
- Different models can tokenize the same text differently
What is inside the context window?
The context window is the current workspace of a model request. It is not the same as persistent memory. An application decides which conversation turns, rules, sources and system information are resent or summarised for each request.
A simplified numerical example shows that every component consumes the same budget. The values are not a recommendation; they illustrate planning a request with a maximum of 32,000 tokens. The actual limit and overflow behaviour depend on the model, API and application.
| Budget component | Example budget | What it contains |
|---|---|---|
| System and safety instructions | 2,000 tokens | Role, permitted sources, output rules and safety boundaries |
| Question and conversation history | 4,000 tokens | Current input and selected or summarised previous messages |
| Sources and document excerpts | 17,000 tokens | Authorised results with sufficient context and citations |
| Tool results | 3,000 tokens | API responses, calculations or structured intermediate results |
| Response reserve | 5,000 tokens | Space for a complete, supported model output |
| Safety margin | 1,000 tokens | Buffer for variable tokenisation and unexpectedly large results |
What happens when the budget is insufficient?
Overflow must not remove the most important part at random. Depending on the interface, an oversized request is rejected or the application must shorten it before calling the model. A sound context manager protects system rules and the response reserve, prioritises current authorised sources and makes omissions visible.
- Summarise older conversation turns deliberately instead of silently truncating them
- Select sources by relevance, validity and permission
- Filter very large tool results or process them in multiple stages
- Reject the request or ask for clarification if essential context cannot fit
Why is a larger window not automatically better?
More context can help compare related documents. Unfiltered material can also hide relevant facts, combine conflicting versions and expose unnecessary data. Quality must be tested with realistic long cases even when inputs remain below the technical limit.
- More input raises cost and often response time
- Outdated or irrelevant content can reduce answer quality
- Permissions still apply to content sent only as context
- Maximum length is a capacity figure, not a quality promise
How should a company plan context?
The application should assemble the smallest sufficient context. Current facts are often retrieved through search or APIs, conversations are condensed and long documents split into relevant passages. Each task needs measures for quality, cost and latency.
- Step 1
Define the need
Identify which information the specific task genuinely requires.
- Step 2
Select context
Provide only authorised, relevant passages or live data.
- Step 3
Reserve budget
Leave enough room for tool results and a complete response.
- Step 4
Test limits
Evaluate very long, multilingual and contradictory cases deliberately.
Example: comparing contracts
An application must compare two long contracts. Instead of sending both documents with the entire chat history unfiltered, it extracts their structure, finds corresponding clauses and gives only those passages and citations to the model. Partial findings are then combined in a controlled way. Context, cost and traceability remain manageable.
What to remember
Treat context as a limited, sensitive working budget. Relevant and current information is more valuable than the largest possible volume of text.
Sources and further reading
These primary sources provide further detail on definitions, technical foundations or responsible use.
Content reviewed