note · growing
Context is a budget
Stable prefix first, volatile last — otherwise you pay for it twice.
A large context window is a budget, not a bucket. Everything you put in it competes for the model's attention and for your latency target.
System prompt + tools4,000 tok
Retrieved chunks12,000 tok
Conversation history9,000 tok
Headroom for the answer7,000 tok
Rules I keep coming back to
- Put the stable part first — system prompt, tool definitions, long reference documents. Stable prefixes are what prompt caching can reuse.
- Put the volatile part last — the user turn, freshly retrieved chunks. A single edit near the top invalidates the cache for everything after it.
- Summarise history on a threshold, not every turn. Rolling summaries that rewrite themselves each turn destroy the cache and drift.
Measure before trimming. Half the "context is too big" problems are really one tool returning an unpaginated JSON blob.