RAG Chunking Strategies: Fixed, Recursive, Semantic and More

Your friend missed a week of classes and wants your notes on "the refund rules". Photocopy the whole notebook and she digs through 40 pages. Cut out single lines and "within 10 working days" lands on a strip with no word about refunds.

So you cut along the topic breaks, keep the heading "Fee refunds" on top, and hand her two neat pages.

That is chunking. The way you cut decides whether the reader, or the retriever, can find and understand the right piece.

What is chunking in RAG?

Mapping the notes: your notebook is a source document, each cut-out is a chunk, the heading you keep on top is contextual metadata, and your friend searching through the cut-outs is the retriever.

Chunking is the step in a RAG pipeline that splits each document into smaller, self-contained passages, so each passage can be embedded, retrieved and placed in the LLM prompt on its own.

What is RAG? showed a fixed-size chunker with overlap. Here you get five strategies you can run and an experiment that measures their effect, on the same fictional Kshipra Institute of Technology (KIT) handbook stored in Vector Databases Explained.

Why does chunk size matter so much?

Every chunk becomes one vector that must stand for everything in it, so boundaries decide what a search can match.

• Too small: a chunk like "within 10 working days" has lost its subject. Its vector cannot match a question about refunds.
• Too large: a chunk covering fees, hostel and library rules gets a blurry vector that is a weak match for everything, and it wastes prompt tokens.
• Just right: one complete idea, with enough context that a human could understand it alone.

Typical production chunks are 200 to 800 tokens with 10 to 20 percent overlap, but the only reliable way to choose is to test, as we do below.

The five chunking strategies at a glance

The same handbook page cut five ways. Fixed-size ignores meaning; the others try to cut where the topic changes.

Sentence-based chunking

The cheapest upgrade over fixed windows: never cut inside a sentence. Split into sentences, then pack them into chunks up to a word budget.

import re

page = ("Semester fees are due by 15 July. A late fee of 100 rupees per day applies after that. "
        "The late fee is capped at 3,000 rupees. Students who withdraw before classes start get a "
        "full refund minus a 1,000 rupee processing charge. Withdrawal within the first 15 days of "
        "classes earns a 50 percent refund. Refunds are credited within 10 working days.")

def sentence_chunks(text, max_words=25):
    sentences = re.split(r"(?<=[.!?])\s+", text.strip())
    chunks, current = [], []
    for s in sentences:
        if current and len(" ".join(current + [s]).split()) > max_words:
            chunks.append(" ".join(current))
            current = []
        current.append(s)
    if current:
        chunks.append(" ".join(current))
    return chunks

for i, c in enumerate(sentence_chunks(page)):
    print(f"[{i}] ({len(c.split())} words) {c}")

Every chunk ends at a full stop, and none goes over 25 words. Notice chunk 1 mixes the late fee cap with the withdrawal refund, though: a word budget does not know where topics change.

Abbreviations like "Dr." or "Rs." fool this regex, so real pipelines use a proper sentence tokenizer.

Recursive chunking

Recursive chunking, the default in many RAG libraries, tries the biggest separator first (a blank line between paragraphs). Only if a piece is still too long does it fall back to a new line, then a sentence end, then a space. This hand-written version counts characters, as most libraries do.

def recursive_split(text, size=120, seps=("\n\n", "\n", ". ", " ")):
    if len(text) <= size:
        return [text]
    sep = next((s for s in seps if s in text), None)
    if sep is None:                                  # no separator left: hard cut
        return [text[i:i + size] for i in range(0, len(text), size)]
    chunks, buf = [], ""
    for p in text.split(sep):
        candidate = buf + sep + p if buf else p
        if len(candidate) <= size:
            buf = candidate
            continue
        if buf:
            chunks.append(buf)
        if len(p) > size:                            # still too long: next separator
            chunks.extend(recursive_split(p, size, seps[seps.index(sep) + 1:]))
            buf = ""
        else:
            buf = p
    if buf:
        chunks.append(buf)
    return chunks

handbook = """Library
Students may borrow 4 books for 14 days. Books can be renewed once online.

Fines
Late return costs 2 rupees per day per book. Lost books are charged at the printed price plus 20 percent. Fines must be cleared before exam hall tickets are issued."""

for i, c in enumerate(recursive_split(handbook)):
    print(f"[{i}] {len(c):>3} chars | {c!r}")

The Library paragraph fits, so it stays whole. The Fines paragraph is too long, so the splitter drops to the next separator. Look at chunk 1: the heading "Fines" is stranded as a five-character chunk, and the chunks below no longer say what they are about. That classic weakness motivates the next strategy.

In practice you would use a library splitter. This uses LangChain's text splitters package (pip install langchain-text-splitters); it is not runnable in the browser.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,        # characters, not tokens
    chunk_overlap=100,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(open("kit_handbook.txt", encoding="utf-8").read())
print(len(chunks), chunks[0][:200])

Document-structure-based chunking

Handbooks, manuals and web pages already have structure: headings, sections, lists and tables. Structure-based chunking splits along those boundaries and, crucially, keeps the heading path with each chunk. This is often called a contextual header.

handbook_md = """# KIT Student Handbook 2026
## Library
### Borrowing
Students may borrow 4 books for 14 days. Books can be renewed once online.
### Fines
Late return costs 2 rupees per day per book.
## Exams
### Revaluation
Apply within 7 days of the result. The fee is 300 rupees per paper."""

def split_by_headings(md):
    path, chunks, body = {}, [], []
    def flush():
        if body:
            header = " > ".join(path[k] for k in sorted(path))
            chunks.append(f"{header}\n{' '.join(body)}")
            body.clear()
    for line in md.splitlines():
        if line.startswith("#"):
            flush()
            level = len(line) - len(line.lstrip("#"))
            path = {k: v for k, v in path.items() if k < level}   # drop deeper headings
            path[level] = line.lstrip("# ").strip()
        elif line.strip():
            body.append(line.strip())
    flush()
    return chunks

for c in split_by_headings(handbook_md):
    print(c, end="\n---\n")

The fines chunk now reads "KIT Student Handbook 2026 > Library > Fines" followed by the rule. A question about "library fines" can match it even though the sentence itself never says "library". The revaluation chunk carries "Exams" in the same way.

The cost is uneven sizes, so production pipelines split by structure first, then run a recursive splitter on any section still too long.

Semantic chunking

When a document has no useful headings, such as a meeting transcript or a long email thread, you can let meaning decide. Semantic chunking embeds each sentence, compares each sentence with the next, and starts a new chunk wherever similarity drops below a threshold.

Semantic chunking cuts at the dips: where one sentence has little in common with the next, the topic has probably changed.

This version uses TF-IDF vectors so it runs in your browser. In production you would use sentence embeddings, which also catch synonyms.

import re
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

page = ("Hostel residents must be inside by 10 pm. Late entry to the hostel needs written "
        "approval from the warden. The hostel mess serves breakfast from 7:30 to 9 am. "
        "Merit scholarships give a 25 percent tuition waiver for a CGPA of 8.5 or more. "
        "A scholarship is renewed each year if the CGPA stays at 8.5. "
        "The placement cell accepts students with no active backlogs. "
        "Students register on the placement portal by 31 July.")
sentences = re.split(r"(?<=[.])\s+", page)

vecs = TfidfVectorizer(stop_words="english").fit_transform(sentences).toarray()
vecs /= np.linalg.norm(vecs, axis=1, keepdims=True)
sims = (vecs[:-1] * vecs[1:]).sum(axis=1)          # similarity of each neighbour pair
print("neighbour similarity:", np.round(sims, 2).tolist())

threshold = 0.05
chunks, current = [], [sentences[0]]
for sim, sent in zip(sims, sentences[1:]):
    if sim < threshold:                             # topic shift: start a new chunk
        chunks.append(" ".join(current))
        current = []
    current.append(sent)
chunks.append(" ".join(current))
for i, c in enumerate(chunks):
    print(f"[{i}] {c}")

The similarity list has two zeros, at the hostel-to-scholarship and scholarship-to-placement boundaries, so the page splits into three clean topics. Hard-coding 0.05 only works on this toy; real implementations usually cut at a percentile of the similarity values, such as the lowest 10 percent.

It costs an embedding per sentence at indexing time, and pipelines add minimum and maximum chunk sizes.

LLM-based chunking

The most expensive option asks an LLM to read a section and either mark the boundaries or rewrite it into standalone facts ("KIT refunds are credited within 10 working days of approval"). It handles messy scans and dense legal text well, but costs tokens for every page you index. Reserve it for small, high-value collections.

How to choose a chunking strategy: measure it

Opinions about chunk size are cheap; test them. This experiment chunks one page four ways, retrieves the top chunk for six questions, and checks whether it contains the answer.

import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

handbook = ("Semester fees are due by 15 July and a late fee of 100 rupees per day applies after the due date. "
    "If you withdraw before classes start, the institute refunds the full fee minus 1,000 rupees. "
    "Once approved, the refund money is credited to your bank within 10 working days. "
    "Each subject needs 75 percent attendance. A medical certificate must reach the office "
    "within 7 days of rejoining. The head of department can condone a shortage of up to 10 percent. "
    "Hostel residents must be inside by 10 pm and late entry needs warden approval. "
    "Students may borrow 4 books for 14 days, and late return costs 2 rupees per day per book. "
    "Revaluation of an exam paper must be requested within 7 days of the result and costs 300 rupees.")
tests = [("How long until the refund reaches my bank?", "10 working days"),
    ("What is the late fee for semester fees?", "100 rupees per day"),
    ("Deadline to submit a medical certificate?", "within 7 days of rejoining"),
    ("How much is the library fine?", "2 rupees per day"),
    ("How much attendance shortage can the HOD condone?", "up to 10 percent"),
    ("What does exam revaluation cost?", "300 rupees")]

fixed = lambda n: [" ".join(handbook.split()[i:i + n]) for i in range(0, len(handbook.split()), n)]
strategies = {"fixed 8 words": fixed(8), "fixed 30 words": fixed(30),
              "one sentence": re.split(r"(?<=[.])\s+", handbook), "whole page": [handbook]}
for name, chunks in strategies.items():
    vec = TfidfVectorizer(stop_words="english").fit(chunks)
    sims = cosine_similarity(vec.transform([q for q, _ in tests]), vec.transform(chunks))
    hits = sum(ans in chunks[row.argmax()] for row, (_, ans) in zip(sims, tests))
    avg = sum(len(c.split()) for c in chunks) / len(chunks)
    print(f"{name:<15} {len(chunks):>2} chunks, avg {avg:5.1f} words, hit@1 = {hits}/{len(tests)}")

Read the four lines of output carefully:

1. Fixed 8 words scores 0 out of 6. Every answer phrase got sliced across two chunks.
2. Fixed 30 words scores 3 out of 6, still cutting through answers.
3. One sentence per chunk scores 5 out of 6 with chunks of about 15 words.
4. The whole page scores 6 out of 6, but only because it is one chunk. Do that with a 200-page handbook and you send everything to the LLM, which is no longer retrieval.

The one sentence-based miss is the library fine question: its sentence never says "library". A contextual header like "Library > Fines" fixes exactly that, which is why structure plus headers is such a strong combination.

Common chunking mistakes

• Losing the subject. "It must be paid within 15 days" means nothing alone. Add contextual headers, or resolve pronouns when you clean the text.
• Splitting tables and lists mid-way. Keep table headers with every row group, and keep a numbered procedure together if it fits.
• Mixing token and character limits. Library splitters often count characters; embedding models limit tokens. Roughly four English characters make one token, and Hindi text usually uses more tokens per character.
• Forgetting metadata. Store the source file, section path and page number with each chunk for filtering and citations.
• Chunking once and never revisiting. When documents or question types change, rerun your retrieval test.

Chunking decides what can be found; the next lesson, hybrid search and re-ranking, improves how it is found.

Key takeaways

• Chunking decides what retrieval can find; a fact split across two chunks may never be retrieved.
• Never cut mid-sentence; sentence-based and recursive splitting are cheap upgrades over fixed windows.
• For structured documents, split by headings and prepend the heading path as a contextual header.
• Semantic chunking suits unstructured text and cuts where meaning shifts.
• Measure hit rate on real questions instead of guessing chunk size.

Want feedback on your own chunking experiments? The RAG module of Debugshala's GenAI course in Indore works through these trade-offs on real documents, and you can practise Python string-handling questions on Debugshala Test.