RAG Chunking Strategies: How to Split Documents for Better Retrieval

Chunking looks simple until retrieval starts failing.

You take a long document, split it into smaller pieces, embed those pieces, index them, and retrieve the chunks that appear most relevant to a query.

But a chunk can fail in two opposite ways.

It can be too small:

The maximum upload size is 5 GB.

The sentence may be correct, but without context you may not know which product, plan, version, region, or policy it belongs to.

Or it can be too large:

Pricing
Billing
Upload limits
Cancellation
Security
Enterprise plans
FAQ
Support policy
...

The relevant fact may exist, but it is surrounded by unrelated material that makes retrieval less specific and downstream reasoning noisier.

A good RAG chunk is small enough to retrieve precisely and complete enough to understand correctly.

This is why there is no universal best chunk size for Retrieval-Augmented Generation. The right strategy depends on source structure, query granularity, retrieval method, metadata, reranking, and what information must stay together for the answer to make sense.

This guide explains fixed-size chunking, overlap, recursive splitting, semantic and structure-aware chunking, parent-child retrieval, contextual retrieval, metadata enrichment, provider-specific controls, and how to evaluate chunking with real retrieval and answer-quality tests.

It builds directly on RAG Prompting. That guide focuses on what happens after evidence is retrieved. This article focuses on how documents should be divided so the right evidence can be found in the first place.

Quick Answer

A useful RAG chunk should preserve four qualities:

RETRIEVABILITY
Can the query find this chunk?

SELF-CONTAINED MEANING
Can it be understood without missing critical context?

LOW NOISE
Does it stay focused on one useful topic or unit?

SOURCE CONTEXT
Do we know where it came from and what it applies to?

Common strategies include fixed-size chunking, overlap, recursive splitting, semantic chunking, structure-aware chunking, parent-child retrieval, and contextual retrieval.

Chunk size is a parameter to evaluate, not a universal rule.

Key Takeaways

• There is no universal best RAG chunk size.
• Small chunks improve specificity but can lose qualifiers and context.
• Large chunks preserve context but can add noise and dilute relevance.
• Natural semantic and structural boundaries are often more useful than arbitrary token cuts.
• Overlap can preserve boundary context, but too much overlap creates redundancy and duplicate retrieval.
• Metadata can restore document context without repeating the entire source inside every chunk.
• Parent-child retrieval can match on small chunks while returning a larger parent section for generation.
• Contextual retrieval enriches chunks with document-level context before indexing.
• Tables, code, legal text, FAQs, PDFs, and transcripts benefit from content-aware boundaries.
• Chunking should be evaluated with retrieval metrics and downstream answer quality.
• PrompTessor does not split or index documents; it can help improve the RAG prompt that tells the model how to use retrieved chunks.

Table of Contents

• What Is RAG Chunking?
• What Makes a Good RAG Chunk?
• Too Small vs. Too Large vs. Well-Scoped
• Anatomy of a Good RAG Chunk
• Fixed-Size Chunking
• Chunk Overlap
• Recursive Chunking
• Semantic Chunking
• Structure-Aware Chunking
• Parent-Child Retrieval
• Contextual Retrieval
• Metadata Enrichment
• Chunking by Content Type
• How to Chunk Tables
• How to Chunk Code
• How to Chunk PDFs
• How to Chunk Transcripts
• How to Choose Chunk Size
• How to Choose Overlap
• How the Main Chunking Strategies Compare
• Chunking vs. Retrieval, Hybrid Search, and Reranking
• Chunking, Permissions, and Multi-Tenant Data
• Versions, Freshness, and Document Lifecycle
• Chunking, Cost, and Latency
• Current OpenAI and Gemini Chunking Controls
• Production Document-to-Retrieval Pipeline
• How to Evaluate RAG Chunking
• 18 RAG Chunking Examples
• Common Chunking Mistakes
• Where PrompTessor Fits
• RAG Chunking Checklist
• Related PrompTessor Guides
• Official Resources
• FAQ

What Is RAG Chunking?

RAG chunking is the process of dividing source material into smaller units that can be indexed and retrieved independently.

RAW DOCUMENT → PARSE → CHUNK → EMBED / INDEX → RETRIEVE → RAG PROMPT → MODEL

The chunk is usually the unit the retriever scores. If a critical relationship is split across a poor boundary, the retriever may find only half the meaning. If unrelated topics are bundled together, one matching phrase can make a noisy chunk rank highly.

Chunking therefore sits between parsing and retrieval and belongs to the broader information-selection problem described in Context Engineering.

What Makes a Good RAG Chunk?

A good chunk is not simply “N tokens long.” It should contain a coherent unit of meaning, align with a useful semantic boundary, preserve necessary local context, carry document context, minimize unrelated material, and remain traceable to the original source.

GOOD CHUNK = FOCUSED MEANING + ENOUGH CONTEXT + USEFUL METADATA + TRACEABLE SOURCE

Examples of critical context include a parent heading, product name, plan, effective date, clause number, API version, table header, function signature, or speaker identity.

Too Small vs. Too Large vs. Well-Scoped

Too Small

"The limit is 5 GB."

Missing context may include product, plan, unit, region, version, or effective date.

Too Large

Enterprise Plan + Upload Limits + API Limits + Billing + Security + FAQ + ...

The answer is present, but many unrelated topics share the same retrieval unit.

Well-Scoped

DOCUMENT: Product Limits
SECTION: Enterprise Upload Limits
CONTENT: Enterprise accounts can upload files up to 5 GB per file.
METADATA: plan=enterprise, topic=upload, status=active

Anatomy of a Good RAG Chunk

CHUNK ID: pricing-enterprise-upload-limit
DOCUMENT: Product Limits v4
SECTION: Enterprise Plan → Upload Limits
CONTENT: Enterprise accounts can upload files up to 5 GB per file.
METADATA: plan=enterprise, status=active, effective_date=2026-06-01
SOURCE: product-limits-v4.pdf, page 14

Not every corpus needs every field. Preserve the attributes your queries and filters actually depend on.

Fixed-Size Chunking

Fixed-size chunking divides text according to a predictable token or character length. It is simple, fast, easy to benchmark, and useful as a baseline.

Its main weakness is semantic blindness: it can split sentences, headings from paragraphs, table headers from rows, definitions from exceptions, and code in the middle of a function.

Fixed-size chunking is not inherently wrong. It becomes weak when the source contains meaningful structure that the splitter ignores.

Chunk Overlap

Overlap repeats part of one chunk in the next chunk so meaning crossing a boundary can remain available.

CHUNK 1: tokens 1–500
CHUNK 2: tokens 401–900
CHUNK 3: tokens 801–1300

Overlap can preserve setup, qualifiers, transitions, or nearby definitions. But more overlap increases index size, near-duplicate retrieval, and the risk that repeated text appears like independent evidence.

Use overlap to solve observed boundary failures, not because one percentage is fashionable.

Recursive Chunking

Recursive splitting tries larger natural boundaries first and falls back to smaller ones only when needed.

DOCUMENT → SECTIONS → PARAGRAPHS → SENTENCES → TOKEN LIMIT

For prose-heavy documents this is often a strong baseline because it preserves more structure than blind windows while still respecting maximum size.

Semantic Chunking

Semantic chunking places boundaries where the topic or meaning changes instead of splitting only by size.

REFUND ELIGIBILITY
────────────
REFUND EXCEPTIONS
────────────
REFUND PROCESS

This can improve topical coherence, but it adds preprocessing complexity and produces variable-size chunks. Treat it as a strategy to test rather than a guaranteed improvement.

Structure-Aware Chunking

Many documents already expose useful boundaries. Markdown has headings; HTML has sections and tables; API docs have endpoints; legal text has clauses; code has symbols; FAQs have question-answer pairs.

Structure-aware splitting often produces better units without a separate semantic model.

• Markdown: heading + governed paragraphs
• API docs: endpoint + parameters + example + errors
• Legal: clause/subclause + parent section
• FAQ: question + answer
• Code: function/method/class/module

Parent-Child Retrieval

Parent-child retrieval separates the matching unit from the generation unit. Small child chunks can be indexed for precise matching, while a larger parent section is returned after a child matches.

PARENT: Enterprise Upload Policy
  ├ CHILD A
  ├ CHILD B ← query match
  └ CHILD C

MATCH CHILD B → RETURN PARENT CONTEXT

This can combine retrieval precision with richer context for generation, especially when nearby exceptions or definitions matter.

Contextual Retrieval

Traditional chunking can remove document-level context that makes a passage retrievable. Anthropic's Contextual Retrieval approach adds chunk-specific context before indexing.

RAW CHUNK:
"The company's revenue grew 3%."

CONTEXTUALIZED:
"This passage is from ACME Corp's Q2 2026 earnings report and discusses year-over-year revenue. The company's revenue grew 3%."

Anthropic describes combining contextualized chunks with embeddings and BM25 and reports further gains from reranking in its own benchmark experiments. Those results support the technique, but they should not be treated as universal guarantees for every corpus.

Metadata Enrichment

Metadata gives retrieval systems context without forcing every attribute into the natural-language chunk. Useful fields can include document ID, title, section, product, version, effective date, status, region, language, author, page, entity ID, and access scope.

For a query like “What is the current refund policy for EU enterprise customers?”, metadata can filter region=EU, plan=enterprise, and status=active before semantic ranking.

Chunking by Content Type

Content	Useful Boundary	Important Context
Article	Heading + paragraphs	Title, section, date
Documentation	Concept / endpoint	Product, version, hierarchy
FAQ	Question + answer	Category, locale
Legal	Clause / subclause	Jurisdiction, effective date
Code	Function / class / module	File, symbol, language
PDF report	Section-aware block	Page, title, date
Table	Header + related rows	Table title, columns
Transcript	Speaker/topic window	Speaker, timestamp

How to Chunk Tables

Naive text splitting can separate values from the headers that give them meaning.

BAD:
Product | Free | Pro | Max
Monthly usage | 10

NEXT CHUNK:
1500 | 5000

For large tables, preserve table identity and column headers with each row or row group.

How to Chunk Code

Prefer semantic code units such as modules, classes, functions, methods, interfaces, and configuration blocks. Preserve metadata such as file path, language, symbol name, symbol type, parent class, and repository version.

Very large functions may still need sub-chunking, but each child should retain the signature and parent context needed to identify what the code does.

How to Chunk PDFs

PDFs are containers, not clean semantic structures. Before chunking, verify reading order, columns, headers, footers, tables, figures, footnotes, and scanned pages.

Useful metadata includes document title, page number, section heading, version, publication date, and figure/table references. Section-aware blocks are usually more useful than assuming one page equals one ideal chunk.

How to Chunk Transcripts

Single speaker turns can be too small because replies such as “Yes, that works” depend on prior turns. Use multi-turn windows, topic segments, or speaker-aware windows and preserve meeting ID, speaker, timestamp, and topic metadata.

How to Choose Chunk Size

Start with the questions users actually ask, then test candidate sizes under the same retrieval stack.

250 vs 500 vs 750 vs 1000 tokens
        ↓
SAME CORPUS + SAME QUERIES
        ↓
COMPARE
Recall@K
Relevance
Expected-source rank
Answer faithfulness
Citation accuracy
Latency / cost

The exact candidate sizes depend on your corpus. The important part is controlled comparison.

How to Choose Overlap

Inspect real examples where meaning is split across boundaries. Add enough overlap to preserve those relationships, then measure duplicate retrieval and index growth.

Use the smallest overlap that reliably preserves the context your boundary strategy tends to lose.

How the Main Chunking Strategies Compare

No single strategy dominates every source type. The practical question is which failure mode you are trying to avoid.

Strategy	How It Works	Strength	Main Limitation	Useful Fit
Fixed size	Split by token or character count.	Simple and predictable.	Ignores semantic structure.	Baseline experiments, uniform prose.
Overlap	Repeat a window across adjacent chunks.	Preserves boundary context.	Creates redundancy and near duplicates.	Long prose with boundary-sensitive meaning.
Recursive	Try large natural separators, then smaller ones.	Preserves common document structure.	Still depends on separator quality.	General prose and Markdown.
Semantic	Split when meaning or topic changes.	Produces coherent topical units.	More complex and variable in size.	Mixed-topic long documents.
Structure-aware	Use headings, clauses, endpoints, symbols, tables, or other source structure.	Matches author-defined semantics.	Requires source-specific parsers.	Docs, code, legal, HTML, structured content.
Parent-child	Match small children, return larger parent context.	Balances retrieval precision and generation context.	More indexing and retrieval logic.	Policies, documentation, long sections.
Contextual retrieval	Add document-level context to chunks before indexing.	Improves meaning for isolated passages.	Extra preprocessing and index text.	Chunks whose meaning depends heavily on document context.

These strategies can also be combined. A documentation pipeline might use heading-aware splitting, a maximum token limit, limited overlap, metadata enrichment, and parent-child retrieval at the same time.

A common mistake is to compare strategy names as if they were mutually exclusive products. In practice, production chunking is often a stack of decisions:

STRUCTURE-AWARE BOUNDARY
        +
MAXIMUM SIZE
        +
OPTIONAL OVERLAP
        +
DOCUMENT CONTEXT
        +
METADATA
        +
PARENT RELATIONSHIP

The best combination is the one that makes the expected evidence easy to retrieve without making each retrieved unit unnecessarily noisy.

Chunking vs. Retrieval, Hybrid Search, and Reranking

Chunking defines what can be retrieved. The retrieval stack determines which of those units are selected.

A poor chunk can still rank highly if it contains the query terms. A good chunk can still be missed if the retrieval method cannot find it.

CHUNKING
Defines candidate evidence units
        ↓
INDEX / EMBEDDINGS
Represents those units
        ↓
QUERY PROCESSING
Rewrites / expands / filters
        ↓
INITIAL RETRIEVAL
Vector, keyword, or hybrid
        ↓
RERANKING
Reorders candidates by usefulness
        ↓
FINAL EVIDENCE SET

This distinction helps diagnose failures correctly.

• If the relevant fact is split from the context required to understand it, improve chunking.
• If the correct chunk exists but never appears in candidates, investigate query formulation, embeddings, lexical retrieval, filters, and recall.
• If the correct chunk appears but ranks below weak matches, investigate reranking and scoring.
• If many near-duplicate chunks occupy the top results, investigate overlap, deduplication, and document versions.

Hybrid search can be particularly useful when exact identifiers and semantic meaning both matter. Product IDs, error codes, API names, clause numbers, and acronyms may benefit from lexical matching even when semantic vector search is also used.

Chunking, Permissions, and Multi-Tenant Data

Private RAG systems should not treat access control as a generation-time prompt instruction.

If a user is not authorized to read a document, its chunks should normally be excluded before those chunks reach the model.

Useful retrieval attributes can include:

tenant_id
workspace_id
team_id
visibility
owner_id
classification
region
access_group

Then retrieval can apply deterministic filters before semantic search or before returning candidates.

This matters because chunking can accidentally weaken source-level boundaries if a pipeline copies text into a shared index without preserving document ownership or access metadata.

The same principle applies to prompt injection. Retrieved chunks remain untrusted content even when the user is authorized to read them. Access authorization answers “may the user see this source?” It does not answer “should instructions written inside this source control the model?” See Prompt Injection for that separate trust boundary.

Versions, Freshness, and Document Lifecycle

Chunk quality is not only about text boundaries. The corpus also needs lifecycle information.

Suppose three policy versions are indexed:

refund-policy-v2.pdf  status=archived
refund-policy-v3.pdf  status=archived
refund-policy-v4.pdf  status=active

Without version metadata, retrieval may surface all three because the wording is highly similar.

That creates a conflict the model should never have needed to resolve.

Useful lifecycle fields include:

• version,
• effective date,
• expiration date,
• active / archived status,
• supersedes / superseded-by relationships,
• last updated time,
• source owner.

When documents change, decide whether old chunks should remain searchable for historical questions or be removed from the active retrieval path. Historical retention and current-answer retrieval are different requirements.

Chunking, Cost, and Latency

Chunking strategy also affects system cost.

Smaller chunks usually create more index entries. Heavy overlap increases the amount of text embedded and stored. Large top-k values send more context to the generation model. Reranking more candidates can improve quality but adds another runtime step.

SMALLER CHUNKS
→ more chunks
→ potentially larger index
→ more candidate matches

MORE OVERLAP
→ more embedded text
→ more near duplicates

LARGER CHUNKS
→ fewer chunks
→ more tokens per retrieved result
→ potentially noisier generation context

The cheapest index is not automatically the best system, and the highest-recall configuration is not automatically the best production configuration.

Evaluate quality together with:

• ingestion time,
• embedding cost,
• index/storage size,
• retrieval latency,
• reranking latency,
• tokens sent to the model,
• and end-to-end answer latency.

The goal is a quality-cost tradeoff appropriate for the application, not maximum retrieval complexity.

Current OpenAI and Gemini Chunking Controls

OpenAI Vector Stores

OpenAI's current vector-store file API supports automatic and static chunking. The current documented automatic strategy uses a maximum chunk size of 800 tokens with 400 overlapping tokens.

The static strategy allows a documented maximum chunk size from 100 to 4096 tokens, and overlap cannot exceed half of the maximum chunk size. These are product defaults and API limits, not universal RAG recommendations.

OpenAI vector-store search also supports file-attribute filters, ranking options, result-count controls, and optional query rewriting, so chunking is only one part of retrieval.

Gemini File Search

Gemini File Search automatically chunks, embeds, and indexes imported files. Current documentation also exposes max_tokens_per_chunk and max_overlap_tokens through custom chunking configuration.

Documentation examples are examples, not a universal best setting. Test custom values against your source types and query distribution.

Production Document-to-Retrieval Pipeline

RAW SOURCE → PARSE → STRUCTURE DETECTION → CHUNK → ADD CONTEXT → ATTACH METADATA → EMBED / INDEX → FILTER / PERMISSION CHECK → RETRIEVE → RERANK / DEDUPLICATE → SELECT EVIDENCE → RAG PROMPT

Versions, freshness, source identity, access control, and evaluation are cross-cutting concerns. Chunking should not be optimized in isolation.

How to Evaluate RAG Chunking

For each test query, define the evidence that should be retrieved. Then ask whether the expected source appeared, at what rank, whether the chunk contained enough context, and whether irrelevant material came with it.

Useful retrieval metrics include Recall@K, relevance rate, expected-source rank or MRR, duplicate-result rate, and latency.

Then measure downstream correctness, faithfulness, citation accuracy, completeness, and context utilization. The broader AI Prompt Evaluation framework can be used to turn retrieval failures into regression tests.

18 RAG Chunking Examples

The right boundary depends on what users ask and what context is required to answer correctly.

Example 1: SaaS Product Documentation

Weak chunking: One entire documentation page per chunk.

Better boundary: Split by task or concept under headings.

Useful metadata: product, version, section, status

Why it helps: Queries target one feature or setup task, not the entire page.

Example 2: API Documentation

Weak chunking: Arbitrary token blocks split endpoint examples.

Better boundary: One endpoint with description, parameters, example, and errors.

Useful metadata: api_version, endpoint, method

Why it helps: Keeps the callable contract together.

Example 3: Help Center

Weak chunking: Several FAQ answers merged together.

Better boundary: Question + answer pair.

Useful metadata: category, product, locale

Why it helps: Matches common user support queries.

Example 4: Pricing Documentation

Weak chunking: Whole pricing page.

Better boundary: Plan or policy topic blocks.

Useful metadata: plan, currency, effective_date

Why it helps: Improves plan-specific retrieval.

Example 5: Legal Contract

Weak chunking: One page per chunk.

Better boundary: Clause or subclause with parent heading.

Useful metadata: contract_id, clause, jurisdiction

Why it helps: Preserves legal relationships better than page boundaries.

Example 6: HR Handbook

Weak chunking: Fixed slices across policies.

Better boundary: Policy topic with applicability context.

Useful metadata: country, employment_type, effective_date

Why it helps: Avoids applying one region policy to another.

Example 7: Research Paper

Weak chunking: One page at a time.

Better boundary: Method, result, or discussion subsection.

Useful metadata: paper_id, section, year

Why it helps: Aligns with scientific question types.

Example 8: Financial Report

Weak chunking: Pages without section labels.

Better boundary: Section-aware chunks with period/company context.

Useful metadata: company, quarter, year, filing_type

Why it helps: Reduces ambiguity around financial figures.

Example 9: Product Catalog

Weak chunking: Multiple products in one chunk.

Better boundary: One product/entity per chunk.

Useful metadata: sku, category, availability

Why it helps: Entity-specific retrieval is more precise.

Example 10: E-commerce Specifications

Weak chunking: Attributes separated from product identity.

Better boundary: Product identity + grouped specifications.

Useful metadata: sku, brand, model

Why it helps: Prevents specs from losing entity context.

Example 11: Codebase

Weak chunking: Token windows split functions.

Better boundary: Function/method/class units with symbol context.

Useful metadata: file_path, language, symbol

Why it helps: Matches developer questions to code semantics.

Example 12: Markdown Wiki

Weak chunking: Raw token windows.

Better boundary: Heading hierarchy plus paragraphs.

Useful metadata: page, heading_path, updated_at

Why it helps: Uses structure already present.

Example 13: HTML Knowledge Base

Weak chunking: Plain text without DOM structure.

Better boundary: Article sections, lists, tables, callouts.

Useful metadata: url, heading, content_type

Why it helps: Preserves useful page semantics.

Example 14: PDF Report

Weak chunking: One page per chunk regardless of reading order.

Better boundary: Parsed section-aware blocks with page metadata.

Useful metadata: document, page, section

Why it helps: Keeps traceability without arbitrary page boundaries.

Example 15: Data Table

Weak chunking: Rows split from headers.

Better boundary: Header context + related row or row group.

Useful metadata: table_id, entity, period

Why it helps: Keeps values interpretable.

Example 16: Meeting Transcript

Weak chunking: Single speaker utterance.

Better boundary: Multi-turn or topic window.

Useful metadata: meeting_id, speakers, timestamps

Why it helps: Preserves what short replies refer to.

Example 17: Support Tickets

Weak chunking: Every message indexed independently.

Better boundary: Case-level issue/resolution or parent-child messages.

Useful metadata: ticket_id, product, status, error_code

Why it helps: Keeps troubleshooting context while allowing focused matches.

Example 18: Policy Library

Weak chunking: All versions indexed identically.

Better boundary: Section chunks with version and active-status metadata.

Useful metadata: policy_id, version, status, effective_date

Why it helps: Supports freshness filtering and avoids archived-policy confusion.

Common Chunking Mistakes

1. Choosing One Magic Token Count

Different corpora and query types need different granularity.

2. Ignoring Document Structure

Headings, clauses, symbols, and table boundaries already encode meaning.

3. Using Overlap Without Measuring Duplication

Overlap can preserve context but also flood retrieval with near duplicates.

4. Removing Too Much Context

A fact without product, date, plan, or section can become ambiguous.

5. Keeping Too Much Context

One relevant phrase can cause a giant noisy chunk to rank highly.

6. Losing Source Identity

Chunks should remain traceable for citations, debugging, permissions, and updates.

7. Treating Metadata as an Afterthought

Filters for version, region, status, or access scope can be as important as similarity.

8. Splitting Tables Like Plain Text

Values need the headers that define them.

9. Splitting Code Like Prose

Functions and classes are better units than arbitrary windows.

10. Ignoring Duplicate Document Versions

Chunking cannot fix a corpus full of unlabeled stale duplicates.

11. Evaluating Only Retrieval Similarity

Measure whether retrieved chunks support correct downstream answers.

12. Changing Chunking Without Re-Evaluating

Chunking changes the retrieval corpus itself, so regression tests should be rerun.

Where PrompTessor Fits

PrompTessor is not a document chunking or indexing system.

DOCUMENT CORPUS
→ PARSE / CHUNK
→ EMBED / INDEX
→ RETRIEVE
→ SELECTED CHUNKS
→ RAG PROMPT
→ PrompTessor
  Analyze / Optimize / Refine
→ CLEARER EVIDENCE INSTRUCTIONS
→ MODEL

PrompTessor can help improve the instruction layer that tells a model how to use retrieved chunks: source boundaries, relevance rules, conflict handling, missing-information behavior, citations, and output structure.

PrompTessor does not split documents, choose chunk sizes, create embeddings, operate a vector database, perform retrieval, or rerank chunks.

For the next stage after chunking, see RAG Prompting.

RAG Chunking Checklist

• What types of questions will users ask?
• What is the smallest natural unit that can answer them correctly?
• Which context must stay attached?
• Are headings or structural boundaries available?
• Are chunks too small to preserve qualifiers?
• Are chunks too large and topically noisy?
• Is overlap solving a measured boundary problem?
• How much duplicate retrieval does overlap create?
• Would recursive or structure-aware splitting preserve better boundaries?
• Would parent-child retrieval improve precision plus context?
• Would contextual enrichment help ambiguous chunks?
• Are source IDs preserved?
• Do you need version, date, region, product, or status metadata?
• Are access-control attributes applied before retrieval?
• Are tables preserving headers?
• Is code chunked around symbols?
• Is PDF reading order correct before chunking?
• Are duplicate document versions removed or labeled?
• Have you created a retrieval test set?
• Do you measure retrieval and downstream answer quality?
• Do you re-evaluate after changing embeddings, reranking, or chunking?

Related PrompTessor Guides

• RAG Prompting: How to Write Better Prompts for Retrieval-Augmented Generation — what the model should do with chunks after retrieval.
• Context Engineering — the broader architecture for selecting and organizing model context.
• Long-Context Prompting — when direct large-context input may be preferable to retrieval.
• Prompt Injection — why retrieved chunks remain untrusted content.
• AI Prompt Evaluation — how to build regression tests for retrieval and downstream answers.
• Structured Outputs — useful for machine-readable grounded answer pipelines.

Official Resources

• OpenAI — Vector store file chunking strategy
• OpenAI — Search vector store
• Google AI for Developers — Gemini File Search
• Anthropic — Contextual Retrieval

FAQ

What is RAG chunking?

RAG chunking is the process of dividing source documents into smaller retrievable units before embedding, indexing, and retrieval.

Why does chunk size matter in RAG?

Chunk size changes the tradeoff between retrieval specificity and contextual completeness. Small chunks can lose meaning, while large chunks can add irrelevant information.

What is the best chunk size for RAG?

There is no universal best chunk size. Test multiple sizes using representative queries, retrieval metrics, and downstream answer-quality metrics.

Is 500 tokens a good RAG chunk size?

It can be a useful experiment or baseline for some corpora, but it should not be treated as a universal recommendation.

What happens when chunks are too small?

They may lose headings, qualifiers, entity identity, dates, exceptions, or relationships required to interpret the text.

What happens when chunks are too large?

They may contain several unrelated topics, reducing retrieval precision and adding unnecessary context.

What is chunk overlap?

Chunk overlap repeats some content between adjacent chunks to preserve context that crosses a boundary.

How much overlap should RAG chunks use?

Use enough overlap to fix measured boundary problems without creating excessive redundancy.

Can too much overlap hurt retrieval?

Yes. Heavy overlap increases index size and can cause near-duplicate chunks to dominate results.

What is recursive chunking?

Recursive chunking tries larger natural separators first, then falls back to smaller separators until content fits the target size.

What is semantic chunking?

Semantic chunking places boundaries around changes in meaning or topic rather than relying only on a fixed token count.

What is structure-aware chunking?

It uses document structure such as headings, HTML sections, legal clauses, endpoints, or code symbols to define boundaries.

What is parent-child retrieval?

It indexes smaller child chunks for precise matching but can return a larger parent section for generation.

What is contextual retrieval?

It enriches chunks with document-level information before indexing so isolated passages retain more retrieval context.

What metadata should RAG chunks contain?

Useful metadata may include document ID, title, section, version, effective date, region, product, entity, language, page, owner, and access scope.

Should metadata be embedded into the text?

Sometimes contextual text helps retrieval, while structured metadata is better for filters and permissions. The right mix depends on the system.

How should FAQs be chunked?

Keeping each question paired with its answer is often a useful natural boundary.

How should legal documents be chunked?

Use clause or subclause boundaries while preserving parent headings, document identity, jurisdiction, version, and effective date.

How should tables be chunked?

Keep column headers and table identity attached to the rows they describe.

How should code be chunked?

Prefer semantic units such as functions, methods, classes, modules, or interfaces with symbol metadata.

How should PDFs be chunked?

First ensure correct reading order, then use section-aware blocks with page and document metadata.

How should transcripts be chunked?

Use multi-turn, topic-based, or speaker-aware windows so short replies remain connected to what they reference.

Does better chunking replace reranking?

No. Chunking defines retrieval units; reranking chooses among retrieved candidates.

Does better chunking replace metadata filters?

No. Metadata filters can remove wrong versions, regions, products, or unauthorized sources before semantic ranking.

What chunking does OpenAI Vector Store use by default?

OpenAI currently documents an automatic strategy with a maximum chunk size of 800 tokens and 400 overlapping tokens. Those are product defaults, not universal RAG guidance.

Can OpenAI vector stores use custom chunk sizes?

Yes. The current API supports a static chunking strategy with configurable maximum chunk size and overlap within documented limits.

Can Gemini File Search customize chunking?

Yes. Current Gemini File Search documentation exposes maximum tokens per chunk and maximum overlapping tokens.

What is Anthropic Contextual Retrieval?

It is a retrieval approach that adds chunk-specific context before indexing and combines contextualized retrieval with techniques such as embeddings, BM25, and reranking.

How do you evaluate a chunking strategy?

Run representative queries with known expected evidence, measure retrieval and ranking, then measure downstream correctness, faithfulness, citation accuracy, and completeness.

How can PrompTessor help with RAG chunking?

PrompTessor does not perform chunking or retrieval. It can help improve the RAG prompt that tells the model how to use chunks after retrieval.

Conclusion

Chunking is not a preprocessing detail you can optimize with one universal token number. It defines the units your retrieval system can find.

The most useful chunks preserve coherent meaning, enough local and document context, low topical noise, source identity, and metadata required for filtering and interpretation.

Different corpora benefit from different approaches: fixed windows, overlap, recursive splitting, semantic boundaries, document structure, parent-child retrieval, or contextual enrichment.

A good chunk is small enough to retrieve precisely and complete enough to understand correctly.

Once the right evidence is retrieved, the problem moves to the next layer: telling the model how to use that evidence safely and faithfully. That is where RAG prompting begins.