LearnGrokEvery model has a maximum number of tokens it can hold at once, and that budget covers everything: the system message, the whole conversation history, any documents you pasted in, and the space reserved for the reply. Exceeding it is an immediate rejection, not a degraded answer.
Why it appears suddenly
The usual story is a chat interface that resends the entire history on every turn. Nothing changed about your code; the conversation simply got long enough. The same applies to a document-processing job that ran fine on ten files and failed on the eleventh.
What to cut: in order
- Duplicated content. The same document pasted twice across a conversation is the cheapest thing to remove and the most commonly present.
- Old turns in the middle. Keep the system message, keep the most recent exchanges, keep anything the current question depends on. Drop the rest. Middle turns are usually the least load-bearing.
- Raw material you have already extracted from. Once the model has produced a table from a document, carry the table forward and drop the document.
- The reply budget. If you have reserved a very large maximum output, you are spending window on space you rarely use.
What not to cut: the system message, and the instructions that define the output format. Removing those makes every subsequent answer worse in a way that is hard to attribute later.
For long documents
Chunk with overlap: split into parts that share a little text at the boundaries, so a sentence spanning a split is not lost, then process each part and combine your extracted results, not the raw parts. If the combination step is itself too large, you have a hierarchy: extract, merge in groups, merge the groups.
Count before you send
Estimating from character count is unreliable across languages and for code. If you are near the limit routinely, count tokens with the tokenizer for the model you are calling rather than guessing, and reject oversized inputs at your own boundary with a message a user can act on; that is a much better experience than a raw API error surfacing in your UI.
What changes
Context windows have grown steadily and differ per model; a limit you hit today may not exist on another model in the same family. Check the current window for your model at docs.x.ai models before you build an elaborate chunking pipeline you may not need.
