Vector Embeddings

By the Syncfusion DevTutorials team · Last updated October 20, 2026

How text becomes vectors that capture meaning: dense vectors and dimensions, embedding models and how to choose them, L2 normalization, mean vs CLS pooling, and running small quantized models in the browser.

Think about a large map where sentences are placed according to their meaning.

Sentences with similar meanings appear close together. For example:

“I cannot sign in” “I forgot my password”

These sentences use different words, but they describe related problems. They would appear close together on the map.

A sentence such as “Where is my parcel?” has a different meaning, so it would appear farther away.

An embedding represents the position of text on this meaning-based map. Instead of using a visible location, it represents the text as a list of numbers.

An embedding model creates these number lists. Text with similar meaning usually receives similar embeddings. A RAG system can compare the question’s embedding with document embeddings to find content that is close in meaning.

The main idea: Embeddings help a RAG system find related information even when the question and document use different words.

A Real-World Example

A Brightgrid customer asks:

My light will not connect to the internet.

The related help article uses different wording:

Lamp will not connect to Wi-Fi.

A keyword search may rank this article poorly because the question and the article use different words.

Embedding-based search compares meaning instead of relying only on matching words. It can recognise that:

light and lamp are related connect to the internet and connect to Wi-Fi describe a similar problem

This helps the system retrieve the correct troubleshooting article.

Brightgrid runs a small embedding model inside the browser. The model converts the question and help articles into embeddings without sending the text to an external model service. After the required model files have been downloaded, the browser may also be able to reuse them for later searches.

In simple words: Embeddings help search find related meanings, even when the question and document use different words.

What You Will Learn

In this lesson, you will learn:

1. Vector space basics: dense vectors, semantic encoding, dimensions.
2. Embedding models: text-to-vector mapping, semantic distance, how to choose a model, and domain vs general models.
3. Normalization and pooling: L2 normalization, mean vs CLS pooling, and why cosine works.
4. Local execution: running small models in the browser with Transformers.js, WebGPU and WASM, quantization, caching, batching, workers and cold starts.

Vector Space Fundamentals

Dense Vector Representations

A vector is just a list of numbers, like [0.12, -0.53, 0.88, …]. A dense vector uses every position. Each number carries a little bit of meaning. That's different from sparse vectors like keyword counts, where almost every position is zero because each position stands for one word from a big vocabulary.

Toy dense vectors

// A dense vector: every number carries a little meaning. (Toy 4-number vectors;
// real models use 384 to 3,072 numbers, and no single number has a human name.)
const dims = ["account", "lighting", "money", "shipping"];
const vectors = {
  "I can't sign in": [0.9, 0.1, 0.0, 0.0],
  "I forgot my password": [0.85, 0.05, 0.05, 0.0],
  "My lamp keeps flickering": [0.05, 0.95, 0.0, 0.05],
  "How much is Plus?": [0.2, 0.1, 0.9, 0.0],
  "Where is my parcel?": [0.0, 0.05, 0.1, 0.95],
};

for (const [text, v] of Object.entries(vectors)) {
  const top = dims[v.indexOf(Math.max(...v))];
  console.log(
    `${text.padEnd(26)} [${v.map((x) => x.toFixed(2)).join(", ")}]  mostly: ${top}`,
  );
}

Expected output:

I can't sign in            [0.90, 0.10, 0.00, 0.00]  mostly: account
I forgot my password       [0.85, 0.05, 0.05, 0.00]  mostly: account
My lamp keeps flickering   [0.05, 0.95, 0.00, 0.05]  mostly: lighting
How much is Plus?          [0.20, 0.10, 0.90, 0.00]  mostly: money
Where is my parcel?        [0.00, 0.05, 0.10, 0.95]  mostly: shipping

In this toy example, each of the 4 numbers has a name. In real models, no single number means anything on its own. Meaning is spread across all of them together.

Semantic Encoding Concepts

Semantic means "about meaning". An embedding model is trained on millions of pairs of texts that mean the same thing: a question and its answer, a title and its article, two ways of saying one thing. Training pulls the vectors of matching pairs together and pushes unrelated ones apart. After training, similar meaning gives nearby vectors, even for sentences the model never saw.

Vector Dimensionality & Space Structure

The dimension is how many numbers each vector has. Common sizes are 384, 768, 1,024, 1,536 and 3,072. More dimensions can store finer differences in meaning, but cost more memory and slower search. A useful mental picture is a map with hundreds of directions instead of two. Similar texts form clusters, like neighbourhoods on the map.

Embedding Model Characteristics

Text-to-Vector Mapping

You give the model text, and it gives back one vector per text. Two rules matter:

• Use the same model for chunks and questions. Vectors from different models live on different maps and can't be compared.
• Respect the input limit. Models read a maximum number of tokens (often 256 or 512). Longer text is cut off silently, which is another reason to keep chunks a sensible size.

Semantic Distance Relationships

Two vectors are compared with cosine similarity: the cosine of the angle between them. It runs from -1 (opposite) through 0 (unrelated) to 1 (same direction). In practice, related texts often score 0.5–0.9 and unrelated ones 0–0.3, but the exact numbers differ by model, so never reuse a threshold from one model with another.

Close in meaning, close in space

// Semantic distance: similar meanings point in similar directions.
const v = {
  "I can't sign in": [0.9, 0.1, 0.0, 0.0],
  "I forgot my password": [0.85, 0.05, 0.05, 0.0],
  "My lamp keeps flickering": [0.05, 0.95, 0.0, 0.05],
  "Where is my parcel?": [0.0, 0.05, 0.1, 0.95],
};

const dot = (a, b) => a.reduce((s, x, i) => s + x * b[i], 0);
const norm = (a) => Math.sqrt(dot(a, a));
const cosine = (a, b) => dot(a, b) / (norm(a) * norm(b));

const query = "I can't sign in";
for (const [text, vec] of Object.entries(v)) {
  if (text !== query) {
    console.log(`cosine("${query}", "${text}") = ${cosine(v[query], vec).toFixed(3)}`);
  }
}
console.log('No shared key words with "I forgot my password", yet it is the closest.');

Expected output:

cosine("I can't sign in", "I forgot my password") = 0.997
cosine("I can't sign in", "My lamp keeps flickering") = 0.162
cosine("I can't sign in", "Where is my parcel?") = 0.006
No shared key words with "I forgot my password", yet it is the closest.

Model Selection Parameters

Fine-Tuned vs Domain-General Models

General models are trained on broad web text and work well for everyday language. Domain models are trained or fine-tuned on specific text such as law, medicine or code, and understand its terms better. Fine-tuning a model on your own question-answer pairs can raise retrieval quality further, but you need good training pairs and must re-embed your whole corpus whenever the model changes. Start with a strong general model, measure it with your gold set, and only fine-tune if the numbers say you need to.

Vector Normalization & Pooling

L2 Normalization of Vector Magnitudes

The length (magnitude) of a vector is the square root of the sum of its squared numbers. L2 normalization divides every number by that length, so each vector has length 1. After that, only the direction matters, and the simple dot product equals the cosine similarity, which is faster to compute.

Normalize, then dot product equals cosine

// L2 normalization: scale each vector to length 1, so only its direction matters.
const a = [3, 4]; // length 5
const b = [0.6, 0.8]; // same direction, length 1

const length = (v) => Math.hypot(...v);
const normalize = (v) => v.map((x) => x / length(v));
const dot = (x, y) => x.reduce((s, xi, i) => s + xi * y[i], 0);

console.log("lengths:", length(a), length(b));
console.log("dot before:", dot(a, b).toFixed(2), " (depends on length)");
const na = normalize(a);
console.log(
  "a normalized:",
  na.map((x) => x.toFixed(2)).join(", "),
  "length",
  length(na).toFixed(2),
);
console.log("dot after:", dot(na, b).toFixed(2), " (now equals the cosine)");

Expected output:

lengths: 5 1
dot before: 5.00  (depends on length)
a normalized: 0.60, 0.80 length 1.00
dot after: 1.00  (now equals the cosine)

Mean vs CLS Token Pooling Strategies

A transformer model first produces one vector per token. To get one vector for the whole text, you pool them:

• Mean pooling: average all the token vectors.
• CLS pooling: use the vector of a special first token, [CLS], which some models are trained to fill with a summary.

Mean pooling vs CLS pooling

// Pooling: a model gives one vector per token. Combine them into one sentence vector.
const tokens = ["[CLS]", "lamp", "keeps", "flickering"];
const tokenVectors = [
  [0.3, 0.5, 0.1], // [CLS]: a special first token
  [0.1, 0.9, 0.0],
  [0.2, 0.2, 0.2],
  [0.0, 0.8, 0.1],
];

// Mean pooling: average every token's vector.
const mean = tokenVectors[0].map(
  (_x, j) => tokenVectors.reduce((s, v) => s + v[j], 0) / tokenVectors.length,
);
// CLS pooling: use the first token's vector, which the model was trained to fill with a summary.
const cls = tokenVectors[0];

console.log("tokens:", tokens.join(" "));
console.log("mean pooling:", mean.map((x) => x.toFixed(3)).join(", "));
console.log("CLS pooling: ", cls.map((x) => x.toFixed(3)).join(", "));
console.log(
  "Use the pooling your model was trained with. all-MiniLM-L6-v2 uses mean pooling.",
);

Expected output:

tokens: [CLS] lamp keeps flickering
mean pooling: 0.150, 0.600, 0.100
CLS pooling:  0.300, 0.500, 0.100
Use the pooling your model was trained with. all-MiniLM-L6-v2 uses mean pooling.

Always use the pooling the model was trained with. The wrong pooling gives vectors that still look normal, but search gets quietly worse.

Cosine Distance Optimization

Search engines often use distance (smaller means closer) instead of similarity. Cosine distance is 1 minus the cosine similarity. With normalized vectors, many databases skip the division entirely and just use the dot product. Normalize once when you store vectors, and normalize each query once. Then every comparison is a single, fast dot product.

Local Embedding Execution

Small Footprint Local Models (Transformers.js)

Transformers.js is a JavaScript library that runs transformer models directly in the browser or in Node.js. The RAG Lab uses it like this:

import { pipeline } from "@huggingface/transformers";

const extractor = await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2", { dtype: "q8" });
const output = await extractor(["My lamp won't connect to Wi-Fi"], { pooling: "mean", normalize: true });
console.log(output.tolist()[0].length); // 384

Notice the two options: mean pooling and normalize, exactly as above.

WebGPU & WASM Execution Runtimes

The model's maths can run on two engines in the browser:

• WASM (WebAssembly) runs on the CPU and works in every modern browser.
• WebGPU runs on the graphics card and is much faster for big batches, but needs a recent browser and a supported device.

A good app tries WebGPU first and falls back to WASM.

Memory Caching & Quick Model Loading

The first time, the model files are downloaded. After that, the browser keeps them in its Cache Storage, and Transformers.js loads them from there with no download. The RAG Lab checks the cache first: if the model is already there, it loads it straight away, and the label says so.

Quantized ONNX Model Embeddings

Models are usually shipped in the ONNX format, and often quantized: each weight is stored in 8 bits (int8) instead of 32 bits (float32). That makes the model about four times smaller and faster, with a very small loss in quality. The same idea can shrink stored vectors too:

Quantize a vector to 8 bits

// Quantization: store each number in 8 bits instead of 32. Four times smaller, a tiny error.
const v = [0.12, -0.53, 0.88, -0.07, 0.31, -0.94];

const scale = Math.max(...v.map(Math.abs)) / 127;
const q = v.map((x) => Math.round(x / scale)); // whole numbers from -127 to 127
const back = q.map((x) => x * scale);

console.log("original: ", v.join(", "));
console.log("int8:     ", q.join(", "));
console.log("restored: ", back.map((x) => x.toFixed(3)).join(", "));
const maxErr = Math.max(...v.map((x, i) => Math.abs(x - back[i])));
console.log("largest error:", maxErr.toFixed(4));
console.log(`size for 384 numbers: ${384 * 4} bytes as float32, ${384} bytes as int8`);

Expected output:

original:  0.12, -0.53, 0.88, -0.07, 0.31, -0.94
int8:      16, -72, 119, -9, 42, -127
restored:  0.118, -0.533, 0.881, -0.067, 0.311, -0.940
largest error: 0.0034
size for 384 numbers: 1536 bytes as float32, 384 bytes as int8

In-Memory, Batch and Cached Computation

Embedding is the slow part of indexing. Three habits make it much faster:

• Batch: send many texts in one call. Models process a batch almost as fast as one text.
• Cache: never embed the same text twice. Keep vectors in memory and save them (for example in IndexedDB) so a page reload doesn't redo the work.
• Embed only what changed: when one article is edited, re-embed only its chunks.

Batch and cache embeddings

// Batching and caching: embed many texts in a few calls, and never embed the same text twice.
const cache = new Map();
let modelCalls = 0;

// Stand-in for the real model call, which takes a batch of texts.
function fakeModel(batch) {
  modelCalls += 1;
  return batch.map((t) => [t.length % 7, t.split(" ").length]);
}

function embedAll(texts, batchSize) {
  const todo = [...new Set(texts.filter((t) => !cache.has(t)))];
  for (let i = 0; i < todo.length; i += batchSize) {
    const batch = todo.slice(i, i + batchSize);
    fakeModel(batch).forEach((vec, j) => cache.set(batch[j], vec));
  }
  return texts.map((t) => cache.get(t));
}

const chunks = Array.from({ length: 50 }, (_v, i) => `chunk number ${i % 40}`); // 40 unique texts
embedAll(chunks, 16);
console.log(
  `first run: ${chunks.length} texts, ${cache.size} unique, ${modelCalls} model calls`,
);
embedAll(chunks, 16);
console.log(
  `second run: still ${modelCalls} model calls (everything came from the cache)`,
);

Expected output:

first run: 50 texts, 40 unique, 3 model calls
second run: still 3 model calls (everything came from the cache)

Asynchronous Worker Thread Processing

Running a model on the page's main thread freezes the page while it works. Put the model in a Web Worker, a background thread. The page sends texts to the worker as messages and gets vectors back, and stays responsive the whole time.

Embedded Performance Tuning

• Latency: a short question embeds in a few milliseconds once the model is loaded. Indexing many chunks is what takes time, so do it in the background, in batches.
• Memory: a small model needs tens of MB. Quantized weights and int8 vectors cut memory by about four times.
• Cold start: the first load is the slowest. Start loading the model early, for example when the page opens, show progress, and fall back to keyword search until it's ready. The RAG Lab does exactly this.
• Parallel encoding: several workers can embed different batches at once on multi-core CPUs. On a GPU, one larger batch is usually better than many workers.

Compare word-based and meaning-based vectors for the same question. Load the embedding model first, then switch the vectors setting:

Common mistakes

• Chunks longer than the model's input limit, so the end is silently ignored.
• Reusing similarity thresholds from another model. Score ranges differ between models.
• Re-embedding the whole corpus on every change instead of only changed chunks.
• No fallback during cold start, so search is broken until the model loads.
• Forgetting to re-embed after switching models.

Exercises

Each exercise ends with a value on the last line. Press Check to compare it with the expected answer.

What to Remember

• An embedding is a dense vector: hundreds of numbers where similar meanings get similar numbers.
• Compare vectors with cosine similarity. Score ranges depend on the model.
• Choose a model by quality on your gold set, dimensions, input limit, size, speed, language and licence. Start general; fine-tune only if needed.
• L2-normalize vectors so dot product equals cosine. Use the pooling (mean or CLS) the model was trained with.
• Small quantized ONNX models run in the browser with Transformers.js on WebGPU or WASM. Cache the model and the vectors, batch the work, use a Web Worker, and plan for cold start.
• Changing models means re-embedding everything.

Next: Similarity Search, finding the nearest vectors quickly, and seeing them on a map.