A8gent
GlossaryAgent concepts

What is chunking?

Also called: text splitting, document chunking

Chunking is splitting documents into passages small enough to embed and retrieve individually, while keeping each passage self-contained enough to answer a question on its own.

Written by A8gent
Contents
  1. At a glance
  2. What chunking is and why it decides your results
  3. The strategies, from worst to best
  4. Sizes, overlap and cost
  5. What you should know
  6. Where chunk size shows up on an invoice
  7. FAQ
  8. Related terms

At a glance

Category
Agent concept - retrieval
Why it matters
The chunk is the unit of retrieval - if no chunk holds a full answer, none is possible
Worst strategy
Fixed character count, which cuts through tables and clauses
Best strategy
Split on the document's own structure - headings, sections, table units
Typical size
A few hundred tokens for prose, tuned against a real eval set
Overlap
Roughly 10-20% so boundary-spanning sentences survive intact somewhere

What chunking is and why it decides your results

Chunking is the step in a RAG pipeline where documents are cut into passages before being embedded. It is the least glamorous part of the pipeline and the one that most often decides whether the system works, because a chunk is the unit of retrieval: if no single chunk contains a complete answer, no amount of model quality will produce one.

The tension is simple. Small chunks embed precisely - a short passage has one clear meaning - but may lack the surrounding context that makes them useful. Large chunks carry context but blur into an average meaning that matches everything and answers nothing. Every strategy is a way of managing that trade.

The strategies, from worst to best

Fixed-size splitting cuts every N characters. It is the default in most libraries and the cause of most bad demos, because it cheerfully slices through the middle of a table, a price list or a clause. Recursive splitting improves on it by preferring to break at paragraph, then sentence, then word boundaries, and it is a reasonable baseline for prose.

Structure-aware splitting is what actually works: split on the document's own boundaries. Markdown headings, HTML sections, PDF headings, one row group per table, one question and its answer per FAQ entry. The rule of thumb is that a chunk should be something a human would consider a complete thought - a section, a policy clause, a whole procedure - and each one should carry its heading trail as a prefix so a passage from page forty still knows what document and section it came from.

Sizes, overlap and cost

Common practice lands chunks somewhere in the region of a few hundred tokens for prose, with an overlap of ten to twenty per cent so a sentence spanning a boundary appears in full in at least one chunk. Treat published numbers as starting points, not settings: the right size depends on your documents and your questions, and it is one of the few parameters genuinely worth tuning against an eval set.

Overlap costs storage and embedding proportionally - twenty per cent overlap means twenty per cent more vectors - which is minor at small scale and worth measuring at large. The bigger cost of getting chunking wrong is that you will not know: retrieval will return plausible passages and the model will answer from them, wrongly, with no error anywhere.

What you should know

Tables, forms and spreadsheets need their own handling. A table split across two chunks loses its header row, so the numbers arrive without column names. Either keep each table whole, or repeat the header in every chunk of it, or convert it to sentences at ingestion. The same goes for scanned PDFs, where extraction quality determines everything downstream and is worth checking manually before you index thousands of pages.

When retrieval underperforms, look at the chunks before you change models. Print the top five passages for ten failing questions and read them. In most cases the answer is visibly absent, cut in half, or missing its heading - which tells you exactly what to fix, and it is never the embedding model.

Where chunk size shows up on an invoice

Chunk size decides how many tokens ride along with every answer, and that is a billed quantity. Retrieving five 200-token chunks sends 1,000 tokens of context; retrieving five 2,000-token chunks sends 10,000. At gpt-5.6-terra's published $2.00 per million input tokens in August 2026 that difference is small per call and material at 100,000 calls a month.

It matters more where context length is a price band. On gemini-2.5-pro, prompts above 200,000 tokens are billed at double the input rate, so oversized chunks can push an otherwise cheap request across a threshold. And document ingestion can carry its own meter entirely: Microsoft bills Copilot Studio content processing tools at 8 Copilot Credits per page, which is charged whatever chunking strategy you then apply.

Chunking FAQ

What is chunking in RAG?

Splitting documents into passages that get embedded and retrieved individually. The chunk is the unit of retrieval, so it has to be small enough to have a clear meaning and complete enough to answer a question without the rest of the document.

What chunk size should I use?

A few hundred tokens is a common starting point for prose, but it depends entirely on your documents and questions. This is one of the few parameters worth tuning properly: build a small eval set of real questions and measure retrieval accuracy at two or three sizes.

Why do I need overlap between chunks?

So a sentence or idea that straddles a boundary appears complete in at least one chunk. Without overlap, the passage that answers the question can be split in half and neither half retrieves well. Ten to twenty per cent is typical and costs proportionally more storage.

How should I chunk tables and spreadsheets?

Not with a generic splitter. Keep each table whole where it fits, repeat the header row in every chunk of a long table, or convert rows to sentences at ingestion. A table split without its headers gives the model numbers with no idea what they measure.

Should chunks include their document heading?

Yes. Prefix each chunk with its document title and heading trail. It costs a few tokens and dramatically improves both retrieval, because the heading words are searchable, and generation, because the model can see which policy or section it is reading from.

My RAG answers are wrong - is chunking the problem?

Very often. Before changing models, print the top five retrieved passages for ten failing questions and read them. If the answer is absent, truncated or missing its heading, the fix is in chunking or extraction, not in the embedding model or the prompt.

Primary source

Pinecone - chunking strategies

Checked August 2026. Anything priced or rate-limited moves, so verify against the source before you budget.

Want this built rather than explained?

Knowing what chunking means is one thing; having it running in your business is another. We scope one workflow at a time, quote a fixed price in writing after a free consult, build it, and hand it over with documentation you own.