RAG Chunking Strategies: Semantic vs Fixed

Short answer: chunking sets the ceiling on everything downstream, and it is the cheapest thing to fix. If the passage that answers a question was split across two chunks, no embedding model, reranker or larger LLM recovers it. Get this right before touching anything else in the pipeline.

Most teams tune the model and leave text chunking on defaults. That is backwards.

Why it decides the ceiling

Retrieval returns chunks. If a chunk contains half the answer, the model composes over half the answer — usually confidently. Three failure modes come directly from bad splitting:

The clause and its qualifier land in different chunks. “Refunds are issued within 30 days” retrieves cleanly; “except for digital goods” sat in the next chunk and never came back.
A table is cut mid-row, so the header is in one chunk and the figures in another. Both are now meaningless.
The chunk is too large, so its embedding averages several topics and matches nothing precisely. Dilution is as damaging as truncation and much less obvious.

None of these look like chunking problems from the outside. They look like the model being wrong.

It is also why chunking is the first thing we audit in RAG development work, ahead of models and vector stores.

The four strategies

Fixed-size. Split every N tokens with some overlap. Trivial to implement, works acceptably on homogeneous prose, and cuts straight through tables, lists and code. It is the default in most frameworks, which is why so many pipelines underperform out of the box.

Recursive. Split on a hierarchy of separators — paragraphs, then sentences, then words — until chunks fit the target size. A meaningful improvement over fixed for almost no extra work, and the right minimum standard. If you are on fixed-size splitting, move here first.

Semantic chunking. Embed sentences, measure similarity between neighbours, and split where the topic shifts. Produces chunks that correspond to ideas rather than token counts. Costs an embedding pass over the corpus at ingest and genuinely helps on long unstructured prose — transcripts, reports, articles. It helps much less on documents that already have structure, because the structure was already telling you where the boundaries are.

Structure-aware. Split along the document’s own hierarchy: headings, sections, table boundaries, list items, code blocks. On structured content this beats everything else, and it is not close. A contract has clauses. A manual has sections. A spreadsheet has rows. Respect them and retrieval improves before you touch the model.

Agentic chunking — using an LLM to decide boundaries — is worth knowing about and rarely worth the ingest cost. It shines on genuinely messy input where structure is implied but not marked up. Measure it against structure-aware before paying for it.

Overlap is a patch, not a strategy

Overlap exists to stop a boundary landing mid-thought. It helps, and it is not free: overlapping chunks inflate index size, return near-duplicate results that crowd out genuinely different passages, and mask bad boundary logic rather than fixing it.

Ten to fifteen percent is a reasonable default. If you need substantially more to get acceptable results, the boundaries are wrong and the overlap is compensating.

Size: smaller than instinct suggests

The pull is toward large chunks so more context reaches the model. That damages retrieval, because the embedding of a large chunk is an average, and averages match nothing sharply.

The pattern that resolves this is retrieve small, return large: index small, precise chunks for matching, but pass the surrounding parent section to the model once a chunk matches. You get precise retrieval and complete context, which is the whole point. This is the single highest-leverage chunking change for most pipelines, and it is a storage-and-plumbing change rather than a model change.

Chunking is per document type

There is no single correct configuration, and looking for one is why teams stall. Classify documents at ingest and apply a strategy per type — the same discipline that makes document processing pipelines work.

Prose — recursive or semantic, moderate size.
Contracts and policies — structure-aware on clauses, never split a clause.
Tables — never split rows; carry the header into every chunk or the figures lose meaning.
Code — split on function or class boundaries, keep imports as context.
Transcripts — semantic, since speaker turns are not topic boundaries.
Slides and PDFs — split per page or per section, and preserve layout order; see multimodal RAG when the meaning lives in figures.

Measure, do not guess

Chunking is easy to evaluate and almost nobody does it. Build a set of a hundred real questions with the passage that should answer each, then measure retrieval recall at k for each chunking configuration. It is an afternoon and it settles arguments that otherwise run for weeks.

Re-measure when documents change shape. A configuration tuned on clean PDFs will quietly degrade when someone starts uploading scans. The full harness is in RAG evaluation.

The takeaway

Move off fixed-size splitting today; recursive is strictly better for the same effort. Go structure-aware wherever documents have structure, because that is where the largest gains are. Adopt retrieve-small-return-large. Then measure — and only then consider whether the embedding model is your problem.

EpochC provides custom RAG development services, including ingestion and chunking design tuned to your document types. See the graph + vector retrieval case study, or start a project.

RAG chunking strategies: semantic, fixed and structure-aware compared