note · growing
Chunking: boundary > size
Split on headings, keep 10–15% overlap, prepend a context line to every chunk.
Tuning chunk size is the cheapest measurable win — but the thing that matters is not size, it is the boundary. A 512-token chunk cut through the middle of an idea is worse than a clean 1024.
| Chunk | Gain | Loss |
|---|---|---|
| Small (128–256) | high precision | context is severed |
| Medium (512–1024) | the balance for most corpora | — |
| Large (2000+) | context intact | blurred embedding, noise |
Try in this order
- Split on headings, not on a fixed token count. Carry the heading chain into every chunk.
- 10–15% overlap. Keep the sentence that lands on a boundary on both sides.
- Contextual retrieval. Prepend a 50–100 token line — generated by a cheap model — saying where in which document this piece sits, then embed that.
There is no single global chunking strategy. A contract, a table and a chat log do not survive the same splitter — the real work is classifying document type at ingest.