Document Processing in Retrieval-Augmented Generation

Lessons in this section

1. Document Ingestion

   Turning PDFs, Word files, Markdown and HTML into clean, structured text: parsing, keeping headings, tables and footnotes, multi-column and OCR, cleaning and normalization, boilerplate removal and section trees.

2. Metadata Tagging

   Extracting and attaching metadata to chunks: title, section path, author, dates, versions and access tags; header injection, filter labels and metadata-aware assembly; indexed fields, path tags and time and place scoping.

3. Chunking Strategies

   The main ways to split documents for RAG: fixed-length by characters or tokens with sentence fallbacks, sentence, paragraph, section and header-aware chunking, and semantic chunking at topic shifts.

4. Chunk Optimization

   Tuning chunks for retrieval: sliding-window overlap and ratios, over- vs under-chunking, sizing to queries, standalone completeness, orphans, pronoun resolution, benchmarking, contextual headers and parent-child chunks.