# Metadata Filtering

Course: [Retrieval-Augmented Generation](https://www.devtutorials.ai/learn/rag) · Section: Retrieval Engine
Source: https://www.devtutorials.ai/learn/rag/retrieval-engine/metadata-filtering

> Restricting search with metadata: pre-retrieval hard filters, scoping and partitioning, tenant isolation, post-filtering, missing metadata, conflict rules, soft penalties, AND/OR expressions, bitmaps and cardinality.

On a shopping website, you tick "size 44", "in stock" and "under $100" before you look at the results. Imagine the shop did it the other way round: it showed you its ten most popular shoes, and **then** removed the ones that are not size 44 or in stock. You might be left with one shoe, or none, even though the shop has dozens that fit.

**Metadata filtering** decides which chunks are allowed in the search: by region, date, category, tenant or permission. **When** you filter matters as much as **what** you filter.

**The main idea:** **pre-retrieval** (hard) filters remove chunks before vector search, which guarantees correct scope, isolation and security. **Post-retrieval** filters and **soft penalties** adjust candidates after search, to handle missing metadata and conflicts. Compound **AND/OR** expressions, **bitmap indexes** and attention to **cardinality** keep filtering fast.

## A Real-World Example

Brightgrid's EU customers kept getting answers in dollars. The search code did this:

1. Find the top 5 chunks for "how much is Plus?".
2. Then remove chunks whose region isn't EU.

The top 5 were all US articles: there are more of them, so they score slightly higher. After the filter, **nothing was left**, and the assistant fell back to a general answer with the US price from its memory.

The fix was to filter **first**: search only among chunks with region EU or "any". The EU price article now comes first every time.

A bigger lesson came with a business customer. Brightgrid started hosting help centers for retail partners, each partner a **tenant**. The team made the tenant filter **mandatory**: the server adds it to every search from the signed-in session, so one partner's documents can never appear in another partner's answers.

**In simple words:** narrow the shelf first, then search it.

## What You Will Learn

In this lesson, you will learn:

1. Pre-retrieval filtering: hard filters, scoping by region, date or category, payload partitioning and tenant isolation.
2. Post-retrieval filtering: filtering candidates, missing metadata, conflict rules and soft penalty scoring.
3. Filter optimization: AND/OR expressions, bitmap indexes, reducing overhead and managing cardinality.

In the RAG Lab's **Retrieval settings**, choose a **Category** or an **Updated after** date. The lab filters before searching, and the **Trace** tab shows the filter that was applied. The lab already applies two hard filters by default: archived articles and support tickets are skipped.

## Pre-Retrieval Filtering

### Applying Hard Filters Before Vector Search

A **hard filter** is a rule a chunk must pass, or it cannot be returned at all: `status = current`, `region in (EU, any)`, `audience includes customer`. Apply hard filters **before or during** the nearest-neighbour search, so the top k are chosen only among allowed chunks.

```js runnable title="Post-filter vs pre-filter"
// Pre-filter vs post-filter. Post-filtering after top-k can leave you with almost nothing.
const chunks = [
  { id: "plus-price-us", region: "US", score: 0.91 },
  { id: "plus-trial-us", region: "US", score: 0.88 },
  { id: "plus-cancel-us", region: "US", score: 0.86 },
  { id: "plus-price-eu", region: "EU", score: 0.84 },
  { id: "plus-trial-eu", region: "EU", score: 0.8 },
];
const k = 3;
const byScore = (a, b) => b.score - a.score;

const post = [...chunks]
  .sort(byScore)
  .slice(0, k)
  .filter((c) => c.region === "EU");
const pre = chunks
  .filter((c) => c.region === "EU")
  .sort(byScore)
  .slice(0, k);

console.log(
  "post-filter (search, then filter):",
  post.map((c) => c.id).join(", ") || "(nothing left)",
);
console.log("pre-filter  (filter, then search):", pre.map((c) => c.id).join(", "));
```

Expected output:

```text
post-filter (search, then filter): (nothing left)
pre-filter  (filter, then search): plus-price-eu, plus-trial-eu
```

### Restricting Scope by Region, Date, or Category

Common hard filters:

- **Region or language:** only content for the user's market.
- **Date:** only documents in force today, or updated after a date.
- **Category or product:** only billing articles for a billing question (often chosen by query routing, Lesson 14).
- **Status:** never archived or draft content.

### Index Payload Partitioning

When a filter is used on almost every query and splits the data into big groups, like region, language or tenant, consider storing each group in its own **partition**: a separate collection, index or shard. Searches then run in a smaller index, which is faster and never mixes groups by mistake.

### Multi-Tenant Data Isolation Filters

In a **multi-tenant** system, many customers (tenants) share one application. Isolation rules:

- The **tenant id comes from the signed-in session**, never from the user's question or request body.
- The server **always** adds the tenant filter. There's no code path that searches without it.
- For strict needs, use a **separate index per tenant**.

```js runnable title="Tenant id comes from the session"
// Multi-tenant isolation: every query is forced to the caller's tenant. The filter is
// added by the server, never taken from the user's request.
const index = [
  { id: "acme-policy#1", tenant: "acme", text: "Acme staff get 25 days of leave." },
  { id: "globex-policy#1", tenant: "globex", text: "Globex staff get 20 days of leave." },
];

function search(question, session) {
  const tenant = session.tenantId; // from the signed-in session, not from the question
  return index.filter((c) => c.tenant === tenant).map((c) => c.id);
}

console.log(
  "acme user:",
  search("how many days of leave?", { tenantId: "acme" }).join(", "),
);
console.log(
  "globex user:",
  search("show me acme's leave policy", { tenantId: "globex" }).join(", "),
);
```

Expected output:

```text
acme user: acme-policy#1
globex user: globex-policy#1
```

Never let the language model choose security filters. A question like "show me Acme's policy" must not be able to widen the search. Access and tenant filters are set by your server code, from who the user is.

## Post-Retrieval Filtering

### Filtering Candidate Chunks Post-Search

Post-filtering still has uses: checks that the index cannot express, such as a call to a permission service, or rules that need the full text. To avoid running out of results, **over-retrieve**: search for 3–5 times more candidates than you need, then filter.

### Handling Missing or Omitted Metadata

Real data is messy: some chunks have no region or no date. Decide **on purpose** what happens:

- For **security** fields like access tags: missing means **deny**.
- For **helpful** fields like region or date: missing can mean "keep, with a small penalty".

### Conflict Resolution via Metadata Rules

When retrieved chunks disagree, metadata can decide. Write rules down, for example:

1. Current beats archived.
2. Article beats ticket.
3. The newer `validFrom` wins.
4. Region-specific beats "any".

Apply them when packing the context (Lesson 16), and tell the model the rule in its instructions.

### Dynamic Soft Filter Penalty Scoring

A **soft filter** lowers a chunk's score instead of removing it. It's good for preferences ("prefer the user's region, prefer newer"), because a slightly older or more general chunk can still win if it is much more relevant.

```js runnable title="Soft penalties instead of removal"
// Soft filters: instead of removing chunks, lower their score. Handle missing metadata too.
const user = { region: "EU" };
const hits = [
  { id: "plus-price-us", region: "US", updated: "2026-07-01", score: 0.9 },
  { id: "plus-price-eu", region: "EU", updated: "2026-07-01", score: 0.84 },
  { id: "plus-faq", region: undefined, updated: "2025-01-10", score: 0.82 },
];

function adjusted(h) {
  let s = h.score;
  if (h.region === undefined) {
    s -= 0.05; // unknown region: small penalty, keep it
  } else if (h.region !== user.region) {
    s -= 0.3; // wrong region: big penalty
  }
  if (h.updated < "2026-01-01") {
    s -= 0.05; // older than this year: small penalty
  }
  return s;
}

hits
  .map((h) => ({ ...h, final: adjusted(h) }))
  .sort((a, b) => b.final - a.final)
  .forEach((h) =>
    console.log(`${h.id.padEnd(14)} ${h.score.toFixed(2)} -> ${h.final.toFixed(2)}`),
  );
```

Expected output:

```text
plus-price-eu  0.84 -> 0.84
plus-faq       0.82 -> 0.72
plus-price-us  0.90 -> 0.60
```

## Filter Optimization

### Compound Logical Filter Expressions (AND/OR)

Real filters combine conditions: "current **AND** (region is EU **OR** any) **AND** updated after 2026-01-01". Represent them as **data** (a small tree of and/or/eq/gte), not as strings glued together. Data can be checked, logged and safely translated into your database's filter language.

```js runnable title="Evaluate an AND/OR filter tree"
// Compound filters with AND / OR, written as data so the server can check them safely.
const filter = {
  and: [
    { field: "status", eq: "current" },
    {
      or: [
        { field: "region", eq: "EU" },
        { field: "region", eq: "any" },
      ],
    },
    { field: "updated", gte: "2026-01-01" },
  ],
};

function matches(doc, f) {
  if (f.and) {
    return f.and.every((x) => matches(doc, x));
  }
  if (f.or) {
    return f.or.some((x) => matches(doc, x));
  }
  if ("eq" in f) {
    return doc[f.field] === f.eq;
  }
  if ("gte" in f) {
    return doc[f.field] >= f.gte;
  }
  return false;
}

const docs = [
  { id: "a", status: "current", region: "EU", updated: "2026-03-01" },
  { id: "b", status: "current", region: "any", updated: "2026-05-01" },
  { id: "c", status: "archived", region: "EU", updated: "2026-02-01" },
  { id: "d", status: "current", region: "US", updated: "2026-04-01" },
  { id: "e", status: "current", region: "EU", updated: "2025-12-01" },
];
console.log(
  "matching:",
  docs
    .filter((d) => matches(d, filter))
    .map((d) => d.id)
    .join(", "),
);
```

Expected output:

```text
matching: a, b
```

### In-Memory Boolean Index Bitmaps

A **bitmap index** stores one bit per chunk for each field value: a 1 if the chunk has that value. Combining conditions becomes simple bit operations, which computers do extremely fast: AND is `&`, OR is `|`.

```js runnable title="Bitmaps for region and status"
// Bitmap indexes: one bit per chunk for each field value. AND/OR become fast bit operations.
const chunks = [
  { region: "EU", status: "current" },
  { region: "US", status: "current" },
  { region: "EU", status: "archived" },
  { region: "EU", status: "current" },
  { region: "US", status: "archived" },
];

function bitmap(field, value) {
  return chunks.reduce((bits, c, i) => (c[field] === value ? bits | (1 << i) : bits), 0);
}
const show = (bits) =>
  [...Array(chunks.length)].map((_v, i) => ((bits >> i) & 1 ? "1" : "0")).join("");

const eu = bitmap("region", "EU");
const current = bitmap("status", "current");
console.log("region=EU      ", show(eu));
console.log("status=current ", show(current));
console.log("EU AND current ", show(eu & current));
console.log("EU OR current  ", show(eu | current));

// Cardinality: how many different values a field has. Few values suit bitmaps well.
for (const field of ["region", "status"]) {
  console.log(`${field}: ${new Set(chunks.map((c) => c[field])).size} distinct values`);
}
```

Expected output:

```text
region=EU       10110
status=current  11010
EU AND current  10010
EU OR current   11110
region: 2 distinct values
status: 2 distinct values
```

### Filtering Overhead Reduction

- **Index** the fields you filter on (Lesson 5).
- Put the **most selective** condition first, the one that removes the most chunks.
- **Cache** the bitmaps for common filters, like "status = current".
- **Partition** by the filters used on every query, like tenant.

### Metadata Index Cardinality Management

**Cardinality** is how many distinct values a field has. **Low** cardinality fields (status, region, language) suit bitmaps and partitions. **High** cardinality fields (user id, exact timestamps, document id) do not. For those, use ranges (dates) or a normal database index instead. Avoid inventing new values freely ("EU", "Europe", "eu"). That both breaks filters and raises cardinality.

Log every filter applied to a search, next to the results. When someone asks "why didn't I see article X?", the filter log usually answers it in seconds.

## Common mistakes

- **Post-filtering a small top k**, which returns nothing for minority groups.
- **Filters chosen by the model**, which can be manipulated.
- **Missing metadata treated as a match** for security fields.
- **Inconsistent values** that make filters silently miss.
- **No over-retrieval** when post-filtering is unavoidable.

## Exercises

Each exercise ends with a value on the last line. Press **Check** to compare it with the expected answer.

## What to Remember

- **Pre-retrieval** hard filters choose which chunks may be searched. Use them for scope, status, region, **tenant** and **access**.
- **Post-filtering** a small top k can leave nothing. If you must post-filter, **over-retrieve**.
- Take tenant and access from the **session**, never from the user or the model. For security fields, **missing means deny**.
- Use **soft penalties** for preferences, and written **metadata rules** to resolve conflicts.
- Build filters as **AND/OR data trees**. **Bitmaps** make low-**cardinality** filters fast. **Partition** by filters used on every query.

Next: **Query Transformation**, rewriting, expanding and splitting questions before searching.