Document Ingestion

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Turning PDFs, Word files, Markdown and HTML into clean, structured text: parsing, keeping headings, tables and footnotes, multi-column and OCR, cleaning and normalization, boilerplate removal and section trees.

Imagine collecting recipes from different sources. Some recipes are in books, some are in tables, and others are handwritten on cards.

Before storing them in one searchable collection, you need to copy the recipe text carefully. You must preserve headings, ingredient lists, and instructions while leaving out unrelated details such as page numbers and repeated headers.

A RAG system performs a similar task through document ingestion. Document ingestion is the process of collecting information from sources such as PDFs, Word files, web pages, and scanned documents, then converting it into clean, organised content that the system can process and search.

The main idea: parse each file type with the right tool, keep the structure (headings, tables, footnotes, reading order), clean the text (spaces, encoding, noise, repeated headers and footers), and segment it into a tree of sections, so every paragraph knows where it belongs.

A Real-World Example

Brightgrid collected information from its help-centre web pages and PDF product manuals. However, the first version of the assistant often found confusing or unrelated information.

The problem came from how the documents were processed:

Website menus and navigation text were copied into every article. As a result, a search for "sign in" matched many unrelated pages. Some PDF manuals used two columns. The text from both columns was mixed together, making the sentences difficult to understand. Information from tables lost its rows and headings. For example, plan names, prices, and limits were combined into one unclear line.

The team improved the document-ingestion process. It removed repeated menus and footers, preserved the correct reading order in PDF files, and converted table rows into clear sentences.

In simple words: garbage in, garbage out. Clean text in, good answers out.

What You Will Learn

In this lesson, you will learn:

1. How to extract text from PDF, Word, Markdown and HTML while keeping headings, tables, comments and footnotes.
2. How to handle multi-column layouts and scanned pages (OCR).
3. How to clean text: whitespace, UTF-8 and Unicode, noise, and boilerplate such as headers and footers.
4. How to segment a document into a tree of sections and sub-sections.

Document Parsing

PDF, Word, Markdown and HTML Text Extraction

Each format stores text differently, so each needs its own parser:

PDF is the hardest. It was designed for printing, not for reading text back. A PDF knows "put the letter R at this spot", not "this is a heading". Parsers have to guess paragraphs and headings from font sizes and positions. Common tools include pdf.js and pdfplumber for PDFs, mammoth for Word files, and any HTML parser for web pages.

Preserving Structural Formatting & Headings

Headings are valuable: they tell you what each paragraph is about. A good habit is to convert every format into Markdown-style text. It's simple, readable, and keeps headings as #, ## and ###.

HTML to text, keeping headings

// Extract text from HTML, but keep headings as Markdown so the structure survives.
const html = `<h1>Lamp is flickering</h1>
<p>Brightgrid lamps must not be plugged into a <b>wall dimmer</b>.</p>
<h2>Firmware</h2>
<p>Update the lamp from <a href="/settings">Device settings</a>.</p>`;

const text = html
  .replace(/<h1>(.*?)<\/h1>/g, "# $1")
  .replace(/<h2>(.*?)<\/h2>/g, "## $1")
  .replace(/<[^>]+>/g, "") // drop every other tag, keep its text
  .trim();

console.log(text);

Expected output:

# Lamp is flickering
Brightgrid lamps must not be plugged into a wall dimmer.
## Firmware
Update the lamp from Device settings.

Handling Tables, Comments, and Footnotes

Tables break easily. Flattened carelessly, a table becomes a jumble of words. Two good options:

• Keep it as a Markdown table if it's small.
• Turn each row into a sentence that repeats the column names. Then any single row still makes sense if it ends up in a chunk by itself.

Footnotes should be attached to the sentence they explain. Otherwise "Or $39 a year." floats alone at the bottom of the page. Comments and tracked changes in Word files are usually not content. Drop them, or keep them only if reviewers' notes are part of what you want to search.

Flatten a table and attach a footnote

// Tables and footnotes: flatten them so each piece still makes sense on its own.
const table = {
  headers: ["Plan", "Price", "Schedules"],
  rows: [
    ["Free", "$0", "5"],
    ["Plus", "$3.99 a month*", "unlimited"],
  ],
};
const footnotes = { "*": "Or $39 a year." };

for (const row of table.rows) {
  let line = table.headers.map((h, i) => `${h}: ${row[i]}`).join(", ") + ".";
  for (const [mark, note] of Object.entries(footnotes)) {
    if (line.includes(mark)) {
      line = line.replace(mark, "") + ` (Note: ${note})`;
    }
  }
  console.log(line);
}

Expected output:

Plan: Free, Price: $0, Schedules: 5.
Plan: Plus, Price: $3.99 a month, Schedules: unlimited. (Note: Or $39 a year.)

Multi-Column Layout & OCR Text Extraction

Multi-column pages must be read in reading order: down the left column, then down the right one. A naive extractor reads straight across the page and mixes both columns.

Scanned documents are just pictures of text. OCR (optical character recognition) turns them back into text, and gives each word a confidence score. OCR makes typical mistakes, like reading "30" as "3O". Low-confidence words should be flagged for review instead of trusted, especially numbers, prices and dates.

Reading order and OCR confidence

// Multi-column PDFs and OCR: put text blocks in reading order, drop low-confidence words.
const blocks = [
  { x: 320, y: 100, text: "Refunds take 5 business days." },
  { x: 40, y: 100, text: "You can return any product" },
  { x: 40, y: 120, text: "within 30 days of delivery." },
  { x: 320, y: 120, text: "Shipping is refunded if faulty." },
];

// A naive extractor reads straight across the page, mixing the two columns.
const naive = [...blocks].sort((a, b) => a.y - b.y || a.x - b.x).map((b) => b.text);
console.log("naive:  ", naive.join(" "));

// Reading order: left column top to bottom, then the right column.
const column = (b) => (b.x < 300 ? 0 : 1);
const ordered = [...blocks]
  .sort((a, b) => column(a) - column(b) || a.y - b.y)
  .map((b) => b.text);
console.log("ordered:", ordered.join(" "));

// OCR gives each word a confidence. Flag words below 0.8 for review instead of trusting them.
const ocr = [
  { w: "Return", c: 0.98 },
  { w: "within", c: 0.97 },
  { w: "3O", c: 0.55 },
  { w: "days", c: 0.96 },
];
console.log("ocr:    ", ocr.map((t) => (t.c < 0.8 ? `[?${t.w}]` : t.w)).join(" "));

Expected output:

naive:   You can return any product Refunds take 5 business days. within 30 days of delivery. Shipping is refunded if faulty.
ordered: You can return any product within 30 days of delivery. Refunds take 5 business days. Shipping is refunded if faulty.
ocr:     Return within [?3O] days

Text Cleaning & Normalization

Stripping Unnecessary Whitespace & Artifacts

Extracted text is full of artifacts: double spaces, broken lines in the middle of sentences, words split with hyphens at line ends ("pro-\nduct"), and stray symbols. They waste tokens and break keyword matches. Collapse repeated spaces, join broken lines, and re-join hyphenated words.

Character Encoding Normalization (UTF-8)

Text is stored as numbers, and an encoding says which number means which character. UTF-8 is the standard today. Problems appear when one file uses another encoding, or when the same character can be written in two ways. "é" can be one character or "e" plus an accent mark. They look the same but don't match in a search. Unicode normalization (NFC) makes them identical. Also replace special spaces, like the non-breaking space, with normal ones.

Noise Reduction in Raw Extracted Text

Noise is text that isn't content: cookie notices, "Was this article helpful? Yes / No", share buttons, page numbers, empty table cells. Remove it with rules for your sources, such as dropping elements with certain HTML tags or classes.

Boilerplate, Header & Footer Removal

Boilerplate is text repeated on every page: running headers, footers, copyright lines, navigation. A simple, strong rule: a line that appears on every page of a document is boilerplate.

Remove repeated lines and clean up

// Cleaning: fix spaces, odd characters, broken words and repeated page furniture.
const pages = [
  "Brightgrid Help Center   Page 1\nYou can return any pro-\nduct within 30 days.\n© 2026 Brightgrid",
  "Brightgrid Help Center   Page 2\nRefunds take 5 business days.\n© 2026 Brightgrid",
];

// Lines that appear on every page are boilerplate (headers and footers).
const count = new Map();
for (const p of pages) {
  for (const line of new Set(
    p.split("\n").map((l) => l.replace(/Page \d+/, "").trim()),
  )) {
    count.set(line, (count.get(line) ?? 0) + 1);
  }
}
const boilerplate = new Set(
  [...count].filter(([, n]) => n === pages.length).map(([l]) => l),
);

const cleaned = pages
  .map((p) =>
    p
      .split("\n")
      .filter((l) => !boilerplate.has(l.replace(/Page \d+/, "").trim()))
      .join("\n"),
  )
  .join("\n")
  .replace(/ /g, " ") // non-breaking space -> normal space
  .replace(/-\n(?=[a-z])/g, "") // join words split across lines
  .replace(/\n/g, " ")
  .replace(/\s+/g, " ")
  .normalize("NFC")
  .trim();

console.log("boilerplate:", [...boilerplate].join(" | "));
console.log("cleaned:    ", cleaned);

Expected output:

boilerplate: Brightgrid Help Center | © 2026 Brightgrid
cleaned:     You can return any product within 30 days. Refunds take 5 business days.

Try every cleaning step in the lab and watch the extracted text change:

Document Segmentation

Structural Document Tree Parsing

After cleaning, rebuild the document's structure as a tree. The title is the root, each ## heading is a branch, and each ### heading is a smaller branch. Every paragraph hangs from the heading above it.

Logical Hierarchy Identification

Sometimes headings aren't marked. A PDF only has big bold text, and a Word file may use plain bold paragraphs. Then you guess the hierarchy from clues: font size, bold text, numbering like "2.1", or short lines with no full stop. Mark guessed headings so you can check them later.

Section & Sub-Section Mapping

Once you have the tree, every paragraph gets a section path, like Brightgrid Hub > Setup > Pairing. That path is very useful later: it becomes metadata (next lesson), it can be added to chunks so they make sense alone, and it helps chunking split at natural boundaries.

Build the section path for each paragraph

// Turn headings into a document tree, so every paragraph knows its section path.
const markdown = `# Brightgrid Hub
## Setup
### Pairing
Hold the button for 5 seconds.
### Wi-Fi
Use a 2.4 GHz network.
## Troubleshooting
Restart the hub.`;

const path = [];
for (const line of markdown.split("\n")) {
  const h = line.match(/^(#+) (.*)/);
  if (h) {
    const level = h[1].length;
    path.length = level - 1; // drop deeper levels
    path[level - 1] = h[2];
  } else {
    console.log(`${path.join(" > ").padEnd(36)} | ${line}`);
  }
}

Expected output:

Brightgrid Hub > Setup > Pairing     | Hold the button for 5 seconds.
Brightgrid Hub > Setup > Wi-Fi       | Use a 2.4 GHz network.
Brightgrid Hub > Troubleshooting     | Restart the hub.

Common mistakes

• Indexing navigation and footers. Every chunk then matches words like "sign in" or "privacy".
• Losing table structure. Prices and limits in tables are often exactly what users ask about.
• Trusting OCR numbers. "3O days" or "$3.9g" quietly produce wrong answers.
• Throwing away headings. Without them, chunks lose context and section paths can't be built.
• Never looking at the output. Most ingestion bugs are obvious when a person reads a sample.

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• Use the right parser for each format. PDF needs the most care.
• Keep structure: headings as #, tables as rows with column names, footnotes next to their sentence.
• Read multi-column pages in reading order, and flag low-confidence OCR words.
• Clean: collapse whitespace, join broken words, normalize to UTF-8 and NFC, remove noise and boilerplate that repeats on every page.
• Segment the document into a tree, and give every paragraph a section path.

Next: Metadata Tagging, attaching titles, dates, permissions and labels to every chunk.