Metadata Filtering

By the Syncfusion DevTutorials team · Last updated October 20, 2026

Restricting search with metadata: pre-retrieval hard filters, scoping and partitioning, tenant isolation, post-filtering, missing metadata, conflict rules, soft penalties, AND/OR expressions, bitmaps and cardinality.

On a shopping website, you tick "size 44", "in stock" and "under $100" before you look at the results. Imagine the shop did it the other way round: it showed you its ten most popular shoes, and then removed the ones that are not size 44 or in stock. You might be left with one shoe, or none, even though the shop has dozens that fit.

Metadata filtering decides which chunks are allowed in the search: by region, date, category, tenant or permission. When you filter matters as much as what you filter.

The main idea: pre-retrieval (hard) filters remove chunks before vector search, which guarantees correct scope, isolation and security. Post-retrieval filters and soft penalties adjust candidates after search, to handle missing metadata and conflicts. Compound AND/OR expressions, bitmap indexes and attention to cardinality keep filtering fast.

A Real-World Example

Brightgrid's EU customers kept getting answers in dollars. The search code did this:

1. Find the top 5 chunks for "how much is Plus?".
2. Then remove chunks whose region isn't EU.

The top 5 were all US articles: there are more of them, so they score slightly higher. After the filter, nothing was left, and the assistant fell back to a general answer with the US price from its memory.

The fix was to filter first: search only among chunks with region EU or "any". The EU price article now comes first every time.

A bigger lesson came with a business customer. Brightgrid started hosting help centers for retail partners, each partner a tenant. The team made the tenant filter mandatory: the server adds it to every search from the signed-in session, so one partner's documents can never appear in another partner's answers.

In simple words: narrow the shelf first, then search it.

What You Will Learn

In this lesson, you will learn:

1. Pre-retrieval filtering: hard filters, scoping by region, date or category, payload partitioning and tenant isolation.
2. Post-retrieval filtering: filtering candidates, missing metadata, conflict rules and soft penalty scoring.
3. Filter optimization: AND/OR expressions, bitmap indexes, reducing overhead and managing cardinality.

Pre-Retrieval Filtering

Applying Hard Filters Before Vector Search

A hard filter is a rule a chunk must pass, or it cannot be returned at all: status = current, region in (EU, any), audience includes customer. Apply hard filters before or during the nearest-neighbour search, so the top k are chosen only among allowed chunks.

Post-filter vs pre-filter

// Pre-filter vs post-filter. Post-filtering after top-k can leave you with almost nothing.
const chunks = [
  { id: "plus-price-us", region: "US", score: 0.91 },
  { id: "plus-trial-us", region: "US", score: 0.88 },
  { id: "plus-cancel-us", region: "US", score: 0.86 },
  { id: "plus-price-eu", region: "EU", score: 0.84 },
  { id: "plus-trial-eu", region: "EU", score: 0.8 },
];
const k = 3;
const byScore = (a, b) => b.score - a.score;

const post = [...chunks]
  .sort(byScore)
  .slice(0, k)
  .filter((c) => c.region === "EU");
const pre = chunks
  .filter((c) => c.region === "EU")
  .sort(byScore)
  .slice(0, k);

console.log(
  "post-filter (search, then filter):",
  post.map((c) => c.id).join(", ") || "(nothing left)",
);
console.log("pre-filter  (filter, then search):", pre.map((c) => c.id).join(", "));

Expected output:

post-filter (search, then filter): (nothing left)
pre-filter  (filter, then search): plus-price-eu, plus-trial-eu

Restricting Scope by Region, Date, or Category

Common hard filters:

• Region or language: only content for the user's market.
• Date: only documents in force today, or updated after a date.
• Category or product: only billing articles for a billing question (often chosen by query routing, Lesson 14).
• Status: never archived or draft content.

Index Payload Partitioning

When a filter is used on almost every query and splits the data into big groups, like region, language or tenant, consider storing each group in its own partition: a separate collection, index or shard. Searches then run in a smaller index, which is faster and never mixes groups by mistake.

Multi-Tenant Data Isolation Filters

In a multi-tenant system, many customers (tenants) share one application. Isolation rules:

• The tenant id comes from the signed-in session, never from the user's question or request body.
• The server always adds the tenant filter. There's no code path that searches without it.
• For strict needs, use a separate index per tenant.

Tenant id comes from the session

// Multi-tenant isolation: every query is forced to the caller's tenant. The filter is
// added by the server, never taken from the user's request.
const index = [
  { id: "acme-policy#1", tenant: "acme", text: "Acme staff get 25 days of leave." },
  { id: "globex-policy#1", tenant: "globex", text: "Globex staff get 20 days of leave." },
];

function search(question, session) {
  const tenant = session.tenantId; // from the signed-in session, not from the question
  return index.filter((c) => c.tenant === tenant).map((c) => c.id);
}

console.log(
  "acme user:",
  search("how many days of leave?", { tenantId: "acme" }).join(", "),
);
console.log(
  "globex user:",
  search("show me acme's leave policy", { tenantId: "globex" }).join(", "),
);

Expected output:

acme user: acme-policy#1
globex user: globex-policy#1

Post-Retrieval Filtering

Filtering Candidate Chunks Post-Search

Post-filtering still has uses: checks that the index cannot express, such as a call to a permission service, or rules that need the full text. To avoid running out of results, over-retrieve: search for 3–5 times more candidates than you need, then filter.

Handling Missing or Omitted Metadata

Real data is messy: some chunks have no region or no date. Decide on purpose what happens:

• For security fields like access tags: missing means deny.
• For helpful fields like region or date: missing can mean "keep, with a small penalty".

Conflict Resolution via Metadata Rules

When retrieved chunks disagree, metadata can decide. Write rules down, for example:

1. Current beats archived.
2. Article beats ticket.
3. The newer validFrom wins.
4. Region-specific beats "any".

Apply them when packing the context (Lesson 16), and tell the model the rule in its instructions.

Dynamic Soft Filter Penalty Scoring

A soft filter lowers a chunk's score instead of removing it. It's good for preferences ("prefer the user's region, prefer newer"), because a slightly older or more general chunk can still win if it is much more relevant.

Soft penalties instead of removal

// Soft filters: instead of removing chunks, lower their score. Handle missing metadata too.
const user = { region: "EU" };
const hits = [
  { id: "plus-price-us", region: "US", updated: "2026-07-01", score: 0.9 },
  { id: "plus-price-eu", region: "EU", updated: "2026-07-01", score: 0.84 },
  { id: "plus-faq", region: undefined, updated: "2025-01-10", score: 0.82 },
];

function adjusted(h) {
  let s = h.score;
  if (h.region === undefined) {
    s -= 0.05; // unknown region: small penalty, keep it
  } else if (h.region !== user.region) {
    s -= 0.3; // wrong region: big penalty
  }
  if (h.updated < "2026-01-01") {
    s -= 0.05; // older than this year: small penalty
  }
  return s;
}

hits
  .map((h) => ({ ...h, final: adjusted(h) }))
  .sort((a, b) => b.final - a.final)
  .forEach((h) =>
    console.log(`${h.id.padEnd(14)} ${h.score.toFixed(2)} -> ${h.final.toFixed(2)}`),
  );

Expected output:

plus-price-eu  0.84 -> 0.84
plus-faq       0.82 -> 0.72
plus-price-us  0.90 -> 0.60

Filter Optimization

Compound Logical Filter Expressions (AND/OR)

Real filters combine conditions: "current AND (region is EU OR any) AND updated after 2026-01-01". Represent them as data (a small tree of and/or/eq/gte), not as strings glued together. Data can be checked, logged and safely translated into your database's filter language.

Evaluate an AND/OR filter tree

// Compound filters with AND / OR, written as data so the server can check them safely.
const filter = {
  and: [
    { field: "status", eq: "current" },
    {
      or: [
        { field: "region", eq: "EU" },
        { field: "region", eq: "any" },
      ],
    },
    { field: "updated", gte: "2026-01-01" },
  ],
};

function matches(doc, f) {
  if (f.and) {
    return f.and.every((x) => matches(doc, x));
  }
  if (f.or) {
    return f.or.some((x) => matches(doc, x));
  }
  if ("eq" in f) {
    return doc[f.field] === f.eq;
  }
  if ("gte" in f) {
    return doc[f.field] >= f.gte;
  }
  return false;
}

const docs = [
  { id: "a", status: "current", region: "EU", updated: "2026-03-01" },
  { id: "b", status: "current", region: "any", updated: "2026-05-01" },
  { id: "c", status: "archived", region: "EU", updated: "2026-02-01" },
  { id: "d", status: "current", region: "US", updated: "2026-04-01" },
  { id: "e", status: "current", region: "EU", updated: "2025-12-01" },
];
console.log(
  "matching:",
  docs
    .filter((d) => matches(d, filter))
    .map((d) => d.id)
    .join(", "),
);

Expected output:

matching: a, b

In-Memory Boolean Index Bitmaps

A bitmap index stores one bit per chunk for each field value: a 1 if the chunk has that value. Combining conditions becomes simple bit operations, which computers do extremely fast: AND is &, OR is |.

Bitmaps for region and status

// Bitmap indexes: one bit per chunk for each field value. AND/OR become fast bit operations.
const chunks = [
  { region: "EU", status: "current" },
  { region: "US", status: "current" },
  { region: "EU", status: "archived" },
  { region: "EU", status: "current" },
  { region: "US", status: "archived" },
];

function bitmap(field, value) {
  return chunks.reduce((bits, c, i) => (c[field] === value ? bits | (1 << i) : bits), 0);
}
const show = (bits) =>
  [...Array(chunks.length)].map((_v, i) => ((bits >> i) & 1 ? "1" : "0")).join("");

const eu = bitmap("region", "EU");
const current = bitmap("status", "current");
console.log("region=EU      ", show(eu));
console.log("status=current ", show(current));
console.log("EU AND current ", show(eu & current));
console.log("EU OR current  ", show(eu | current));

// Cardinality: how many different values a field has. Few values suit bitmaps well.
for (const field of ["region", "status"]) {
  console.log(`${field}: ${new Set(chunks.map((c) => c[field])).size} distinct values`);
}

Expected output:

region=EU       10110
status=current  11010
EU AND current  10010
EU OR current   11110
region: 2 distinct values
status: 2 distinct values

Filtering Overhead Reduction

• Index the fields you filter on (Lesson 5).
• Put the most selective condition first, the one that removes the most chunks.
• Cache the bitmaps for common filters, like "status = current".
• Partition by the filters used on every query, like tenant.

Metadata Index Cardinality Management

Cardinality is how many distinct values a field has. Low cardinality fields (status, region, language) suit bitmaps and partitions. High cardinality fields (user id, exact timestamps, document id) do not. For those, use ranges (dates) or a normal database index instead. Avoid inventing new values freely ("EU", "Europe", "eu"). That both breaks filters and raises cardinality.

Common mistakes

• Post-filtering a small top k, which returns nothing for minority groups.
• Filters chosen by the model, which can be manipulated.
• Missing metadata treated as a match for security fields.
• Inconsistent values that make filters silently miss.
• No over-retrieval when post-filtering is unavoidable.

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• Pre-retrieval hard filters choose which chunks may be searched. Use them for scope, status, region, tenant and access.
• Post-filtering a small top k can leave nothing. If you must post-filter, over-retrieve.
• Take tenant and access from the session, never from the user or the model. For security fields, missing means deny.
• Use soft penalties for preferences, and written metadata rules to resolve conflicts.
• Build filters as AND/OR data trees. Bitmaps make low-cardinality filters fast. Partition by filters used on every query.

Next: Query Transformation, rewriting, expanding and splitting questions before searching.