An LLM can only read that much at once. Loading all info into one prompt is slow, pricey, and actually makes the answers worse. So before we can ask questions over a big pile of documents, we chop each one into bite-size pieces. Each piece is a chunk.
Chunking is the very first step of RAG (Retrieval Augmented Generation): slice the documents into chunks, embed each chunk into a vector, and store them. In this blog, I'll walk through the chunking techniques that I find genuinely practical — not the academic ones, just the stuff developers actually ship.
But before we get into the chunking techniques, I think there's a fair question to touch on at the point of 2026.
Lately you'll find no shortage of articles declaring that RAG is dead. The argument goes: agents are so capable now, and just letting one grep its way through the files is so flexible and powerful, that chunking and RAG feels like yesterday's plumbing.
My take: it's still very much needed — especially in the enterprise space. Agent-plus-grep works beautifully on a single codebase where you can afford to let it spend a while poking around. But point it at a large corpus — thousands or even millions of documents — where answers have to come back fast, and that approach falls apart. That's exactly the ground RAG owns, and RAG only works if the corpus was chunked and indexed ahead of time. So chunking isn't going anywhere.
Before the advanced stuff, here are the two ideas everyone starts with. They're simple, they work okay, and understanding their weaknesses is the whole reason the fancier techniques exist.
The most basic move: cut every N tokens and let neighbors overlap a little so a thought sitting right on a boundary isn't lost. Say you set a chunk size of 500 tokens with 50 tokens of overlap — chunk 1 is tokens 1–500, chunk 2 is 451–950, and so on. The overlap is a safety margin.
The problem is it's blind. It counts to 500 and cuts, so it'll happily split a sentence in half, or slice a table right down the middle: "Total revenue was $4.2" ends one chunk and "million, up 3% YoY" starts the next. Neither piece is much use on its own.
A bit smarter: split the document into sentences, and then turn each sentence into a vector, use cosine similarity to measure how closely neighboring sentences relate, and cut only where the topic clearly shifts — that is, where the similarity drops below a set threshold.
With this approach, the boundaries are more meaningful, but it's still blind to structure (a table or heading can get split), and the threshold is hard to tune — set it low and it rarely cuts, giving oversized chunks; set it high and it cuts too often, giving tiny ones. It's now often used as a baseline chunking method to start with.
Everything in this section is what I'd call simple, practical, and reasonably popular in developer communities — not theoretical or academic. Looking across the methods that have caught on, I noticed something: they fall cleanly into two camps, aiming at two very different problems.
Problem 1 — finding the optimal segmentation points: where should a document actually be cut, so each piece comes out clean and self-contained?
Problem 2 — providing more context to every chunk: how do we make sure a chunk still makes sense once it's pulled away from everything around it?
Let's take a look at them one camp at a time, starting with where to cut.
The big insight: most real documents already tell you where to cut — titles, headings, tables, lists. You just have to recover that structure first, and modern document parsers do exactly that. Docling has quite strong adoption among developers for parsing docx, pdf, pptx and the like into a clean structure, while Docstrange tends to work better on scanned documents that need OCR and visual information extraction. Either way, once the structure is back, you cut along it.
1Feed the raw file (PDF, DOCX, PPTX…) into a parser like Docling. Instead of one flat blob of text, you get clean markdown back — headings, tables and lists all preserved in the document's real structure.
2Split the markdown along its own headings, with a few simple rules. Every chunk lands on a real boundary — no sentence or table ever gets sliced through the middle.
This approach is fast and low-cost, and works very well when the material is well structured (e.g. legal, financial documents) — but it doesn't hold up when the source is messy with no real structure to follow, or when a single section is so long that splitting on structure alone still leaves the chunk far too big.
In short, it's highly dependent on document quality and not a universally applicable approach — but it works well for plenty of production systems in specialised domains.
The core idea is simple: load the whole document (or a very large part of it) into an LLM and just ask it to do the chunking for you. Because the model sees the full context, the boundaries it picks tend to be more sensible than anything a fixed rule could achieve. And since you drive it with a prompt, you get to steer how the segmentation happens — group by topic, by section, by whatever your use case needs — which makes it very flexible.
There are several variants of this approach; the one below is the version I've found most practical and robust in production. Watch how each numbered blocks (a paragraph) get grouped into clusters:
Welcome to Nimbus Cloud — what this guide covers.
Create an account on the Nimbus portal to begin.
Verify your email, then sign in to the dashboard.
From the dashboard, create your first project.
Billing is monthly and based on your usage.
View past invoices under the Billing tab.
Payment methods: credit card or bank transfer.
All data is encrypted at rest and in transit.
Support is available 24/7 via live chat.
You can also reach us at support@nimbus.io.
Use of the service is subject to our Terms.
1Load the original document and split it into paragraphs, giving every paragraph an ID. Here we get 11 short paragraphs — 11 IDs.
2Only needed if the document is bigger than the LLM's context window (say 200,000 tokens) — otherwise skip it. If it is, group the paragraphs into batches under a size limit N, with no overlapping paragraphs, so each fits in one LLM call.
3Feed each batch to the LLM. The prompt hands it every paragraph's ID and content and asks it to group related ones, returning JSON clusters — each with a short reason (same color = same cluster). The two batches go through as separate calls.
4Boundary handling. Only the seam — ¶5–8, in the dashed box — goes back to the LLM; every other paragraph stays exactly as it was. One more call, and ¶7 correctly joins billing while ¶8 is left on its own.
5Combine everything and validate that each ID landed in exactly one cluster (with fallback rules for anything the LLM dropped). Each color block is now a final chunk — grouped by meaning, not by length.
The big win is context: the LLM sees the full picture when it decides where the segmentation points should be, so it works across all kinds of documents — structured or not. The catch is cost and speed: it has to read the entire document at least once and generate the chunks, which gets slow and potentially expensive over a huge corpus. And because an LLM is probabilistic by nature, the results can be slightly unstable — the same document won't always chunk exactly the same way twice.
Unlike Camp 1 techniques that try to find the optimal segmentation points, Camp 2 techniques aim at enhancing the surrounding context for the chunks, regardless of how they are split. Let's take a look.
The idea starts from a trade-off: small chunks and big chunks are each good at only half the job. Small chunks are easy to find — a tight, specific match — but too thin on context for the model to answer well. Big chunks carry plenty of context but make search fuzzy. Parent–child chunking keeps both: you index the small child chunks for precise retrieval, but when one matches, you hand the model its bigger parent chunk to actually answer from.
Here's the flow with a simple example:
1Cut the document into big parent chunks — roughly a section each — then split every parent into small child chunks. Store both levels and the parent-child mapping.
2A question comes in. You match it against the small child chunks — small and focused, so the match is sharp and precise. Here it lands on one billing sentence.
3Here's the switch: instead of returning that one tiny child, you follow it up to its parent and hand back the whole section.
4The model answers from the roomy parent chunk — precise search, well-fed answer. You stop having to choose between small and big.
The appeal is its simplicity — both the concept and the implementation are straightforward, with no extra LLM calls, just a child-to-parent lookup at retrieval time — and it noticeably lifts answer quality when your chunks are individually too small to stand on their own. The catch is that it doesn't actually solve where to cut the parents. You still need structure-aware chunking (or some other Camp 1 technique) to split the document into good parent chunks in the first place — parent–child sits on top of that, it doesn't replace it.
A chunk pulled out of its document can be quietly meaningless. Take "The revenue grew 3% over the previous quarter." — whose revenue? which company? which quarter? On its own, it's nearly impossible to retrieve correctly.
Contextual Retrieval (popularized by Anthropic) fixes this: before storing each chunk, you have an LLM write a short "you are here" note — situating the chunk within the whole document — and prepend it to the chunk. The text now carries its own context. Here's the flow:
1Here's a chunk on its own. It's almost meaningless — and a search like "Nimbus cloud Q2 revenue" would never find it, since none of those words appear in it.
2Send the LLM the chunk plus the whole document, and ask it to situate the chunk. It writes a short context blurb pinning the chunk to its place in the bigger story.
3Glue the blurb onto the front of the chunk. The original text stays untouched below, with the added context on top — so the chunk now carries its own context wherever it goes.
4Embed and store the contextualized chunk. Retrieval gets dramatically more accurate — the text now literally spells out what it's about, so the query matches.
It works well and is easy to be added into an existing pipeline — it doesn't care how you split, it just enriches whatever chunks you already have. The real catch is cost: to contextualize every chunk, the LLM has to read through the whole document once per chunk, so each document gets processed many times over. Prompt caching softens the blow, but across a large corpus the total time and cost add up fast — it can end up more time-consuming and expensive than LLM chunking, which only reads each document once. For the full details, see Anthropic's engineering blog.
Same goal as contextual retrieval — give each chunk more context — but a different route that changes nothing about the text. The whole trick is in the order. Instead of cutting first and embedding each piece blind, you embed the whole document first, then cut. Because the embedding model reads the entire document in one pass, every token's vector already reflects the words around it; you just pool those token vectors into chunk vectors afterwards.
It's a technique from Jina AI, and the name says it all: cut late, not early. Here's the flow:
1The usual way: cut first, then embed each chunk on its own and mean-pool its tokens into a vector. Because each chunk is embedded in isolation, its tokens only ever see their own words, so chunk B's vector has no idea it's about Nimbus's cloud division.
2Late chunking flips the order: hand the entire document to the embedding model, uncut. It produces a vector for every token — and each one was computed while looking at the whole document, so context is baked in.
3Split with the very same boundaries and mean-pool each group — exactly as before. The only difference: these token vectors have already read the whole document, so the chunks come out context-rich. Cut late, not early.
It's elegant and cheap: no LLM calls and no bigger index — just one embedding pass over the document, and the text is left completely alone. The catch is that it only works with a mean-pooling embedding model, since the whole trick relies on averaging token vectors. Many of the newest embedding models have moved to a last-token approach instead, which late chunking can't use. And as of 2026, the largest-context mean-pooling models top out around 8k tokens — well short of the 32k that last-token models reach — so for long documents this approach ends up fairly limited.
That rounds out the practical toolkit. Before we wrap up, here are a few ideas I've come across that are clever and fun to think about, but that I wouldn't call practical or widely adopted yet — file them under "keep an eye on it." Notably, they all circle back to Camp 1: different takes on where to cut.
Treat sentences as dots in a graph, draw links between similar ones, then let tightly-linked groups (communities) fall out as segments. Boundaries emerge from the connection pattern instead of a fixed threshold.
// elegant, but heavier to run & tune — niche adoption
Borrowed from time-series analysis: read the document position by position as a "topic signal," and mathematically detect the exact spots where the signal shifts. Cut there.
// promising import from stats — not yet mainstream in RAG
Run a transformer over the text and, at each sentence boundary, have it classify whether that point is a topic break — much like BERT's next-sentence-prediction task. Cuts come from a learned classifier, not a hand-tuned threshold.
// accurate, but needs a trained model — not plug-and-play
Chunking isn't a solved, one-size-fits-all step — it's a small toolbox, and the two questions to keep asking are: am I cutting in the right places? and does each chunk know enough on its own? Here's the cheat sheet for the five practical techniques:
| Technique | Solves | Pros | Cons | Reach for it when… |
|---|---|---|---|---|
| Structure-aware | where | Fast, low-cost, no model calls. | Needs clean structure; long sections stay too big. | Your docs have real structure — headings, tables, contracts, manuals. |
| LLM chunking | where | Full-context boundaries; flexible via prompt; works on any doc. | Slow & costly at scale; slightly unstable. | Structure is messy or high one-time cost is acceptable, and you need good quality chunks. |
| Parent–child | what | Simple & cheap; no extra LLM calls. | Doesn't solve where to cut parents; stores two levels. | Small chunks search well but answers come out starved of context. |
| Contextual Retrieval | what | Big accuracy gains; split-agnostic; easy to add. | An LLM call per chunk; costly on huge corpora. | Chunks are ambiguous alone and you can afford an LLM pass while indexing. |
| Late chunking | what | Cheap; text untouched; no bigger index. | Needs a mean-pooling model; ~8k context limit. | You want whole-doc context in the vectors without rewriting any text. |
There's no single winner here — the right pick depends on your documents, your latency budget, and your scale. And these techniques aren't mutually exclusive: the strongest systems usually combine a Camp 1 technique to cut in the right places with a Camp 2 technique to give each chunk the context it needs. My advice is to start simple, measure your retrieval quality honestly, and only reach for a heavier technique once the numbers show you where it actually hurts.
Chunking rarely gets the spotlight, but it quietly decides how good your RAG can be. Get it right, and everything downstream has a much better shot.
Thanks for reading my notes, and hope you also learn something from it.