# Speech to Text

Course: [Multimodal AI](https://www.devtutorials.ai/learn/multimodal-ai) · Section: Listening, Speaking and Generating
Source: https://www.devtutorials.ai/learn/multimodal-ai/listening-and-generating/speech-to-text

> Transcribe voice notes the app can trust: vocabulary hints for product names, number normalisation, word error rate, and timestamps.

## What Speech to Text Is

**Speech to text** (also called ASR, automatic speech recognition) = a model that turns audio into written words.

In this lesson, you will add voice notes to the claims app. Customers record a short message about their claim, and the app turns it into text it can check.

Here is the problem. A customer says: "My Nimbus Mini stopped working." The model writes: "my nimbus many stopped working." The app searches for the product "nimbus many", finds nothing, and the claim stalls.

Press **Play**, or use **Next** to go one step at a time.

- **Frame 1:** what the customer said.
- **Frame 2:** what the model wrote. The product name is wrong.
- **Frame 3:** with a list of product names as hints, the model gets it right.

**Simple lesson:** a speech model only knows the words it has heard before. Your product names are new to it.

### A Real-World Example

Think about a new call-centre worker taking notes on the phone.

- They do not know the product names yet, so they write what they hear.
- Someone gives them a list of product names. Now they spell them right.
- They write "£49", not "forty-nine pounds", so the form can add it up.
- They note the time in the call when the customer named the product.

The speech pipeline does the same:

- **The product list** = vocabulary hints (a prompt or word list for the model).
- **"£49", not "forty-nine"** = number normalisation.
- **The time in the call** = word timestamps.

**Simple lesson:** give the model your words, and turn its words into values your app can use.

## What You Will Learn

By the end of this lesson, you will be able to:

- explain the common errors of speech models
- measure **word error rate** (WER)
- add vocabulary hints for product names
- normalise numbers before you store or compare them
- add word timestamps so a reviewer can jump to the right moment
- label who is speaking in a recorded call (diarization)

This lesson uses the same pipeline stages with audio: the speech model is **Perceive**, and normalisation is **Structure**.

## Measuring It: Word Error Rate

**Word error rate (WER)** = the number of word changes needed to turn the transcript into what was really said, divided by the number of words said.

A change is a wrong word, a missing word, or an extra word.

Step through the frames.

- **The two lines:** what was said, and what the model wrote.
- **The marks:** each word change.
- **The score:** changes divided by words.

| WER | What it feels like |
|---|---|
| Under 5% | Good enough to search and check |
| 5–15% | Readable, but names and numbers break |
| Over 15% | Someone must listen again |

**WER counts every word the same.** A wrong "the" and a wrong product name both count as one error. That is why the lab has a second target just for product names: at least 95% right.

**Simple lesson:** measure WER, but also measure the words your app depends on.

## Numbers: Words or Digits?

Speech models often write numbers as words. "Forty nine" will never match the 49 on a receipt.

Step through both frames.

- **Frame 1:** the price, as the model wrote it.
- **Frame 2:** after normalisation, the amount is 49 and can be checked.

**Normalise** = make text consistent before you compare or store it: lower case, no punctuation, number words turned into digits.

**Normalise both sides before you measure WER.** If you normalise only the transcript, "49" and "forty-nine" count as errors that are not really errors. Compare like with like.

**Simple lesson:** apps need values, not words. Turn number words into digits.

## Four Fixes, One at a Time

The lab has 12 voice notes, some in quiet rooms and some in noisy kitchens. Here is what each fix does.

Step through all five frames.

- **Naive:** the smallest model, nothing else. Every product name is wrong.
- **Vocabulary hints:** product names come right.
- **Normalise numbers:** every amount is digits.
- **A bigger model:** fewer errors in noisy clips.
- **Word timestamps:** a reviewer can jump to the moment.

| Model | Size | Where it runs |
|---|---|---|
| Whisper tiny | about 40 MB | On the device, fast |
| Whisper base | about 75 MB | On the device |
| Whisper small | about 250 MB | On the device, needs WebGPU to be quick |
| Cloud speech service | none to download | The cloud; the audio leaves the device |

**Easy way to remember:** hints for names, rules for numbers, size for noise. Try the cheaper fixes before you pay for a bigger model.

## Who Said It?

Some audio has more than one speaker: a recorded support call has the agent and the customer. A plain transcript runs them together, so nobody can tell who promised the refund.

**Diarization** = splitting a recording by speaker, and labelling each part (Agent, Customer). Some speech services do it for you; on the device, a small speaker model can do it next to Whisper.

| Without speaker labels | With speaker labels |
|---|---|
| the kettlemax lid is cracked I can refund 24.50 today great please do | **Customer:** the KettleMax lid is cracked · **Agent:** I can refund 24.50 today · **Customer:** great please do |

Speech models also handle many languages and accents. Whisper base and small are multilingual; the tiny English model is not. Test with the accents and languages your customers really use.

Click the button and look at the **Pipeline** tab.

- **The waveform:** the voice note, with noise shown in grey.
- **The transcript:** errors are marked red.
- **Settings:** open the Settings tab to see which Whisper size your device can run.

### Lab Exercise: Transcribe 12 Voice Notes

Your task: change the **Perceive** and **Structure** settings, then press **Check**.

Click the button and open the **Pipeline** tab.

- **Model:** tiny, base, small or cloud.
- **Vocabulary hints, normalise numbers, timestamps, speaker labels:** turn each on or off.
- **To pass:** WER at most 5%, product names at least 95% right, every number as digits, timestamps on, and speaker labels on the two calls.

## Build It in TypeScript

`normalizeNumbers()` turns number words into digits.

```ts
const ONES = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine", "ten", "eleven", "twelve",
  "thirteen", "fourteen", "fifteen", "sixteen", "seventeen", "eighteen", "nineteen"];
const TENS = ["", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety"];

export function normalizeNumbers(text: string): string {
  const words = text.toLowerCase().replace(/[^\w\s-]/g, "").split(/\s+/).filter(Boolean);
  const out: string[] = [];
  for (let i = 0; i < words.length; i++) {
    const tens = TENS.indexOf(words[i]), ones = ONES.indexOf(words[i]);
    if (tens >= 2) {                                   // "forty nine" → 49
      const next = ONES.indexOf(words[i + 1] ?? "");
      if (next > 0 && next < 10) { out.push(String(tens * 10 + next)); i++; } else out.push(String(tens * 10));
    } else if (ones >= 0) out.push(String(ones));
    else out.push(words[i]);
  }
  return out.join(" ");
}
```

Click the button and open the **Code** tab.

- **The starter:** only lower-cases the text.
- **Run on the media bank:** your function runs on every test sentence.

## TL;DR

1. **Speech to text** turns audio into words. It does not know your product names.
2. **WER** = word changes ÷ words said. Aim for 5% or less.
3. **Vocabulary hints** fix product names.
4. **Normalise** numbers to digits before you store or compare.
5. A **bigger model** helps in noise, but costs more and is slower.
6. **Timestamps** let a reviewer jump to the right moment.
7. **Speaker labels** say who said what in a call.

## Common mistakes

| Mistake | Wrong idea | Correct idea | What goes wrong |
|---|---|---|---|
| Only measuring WER | "4% WER is great." | Also measure product names and numbers. | The one word that matters is wrong. |
| Skipping hints | "A bigger model will know the names." | Give the model your product list. | "Nimbus many" again, at higher cost. |
| Storing number words | "The text is right." | Normalise to digits. | The amount never matches the receipt. |
| Normalising one side | "Just clean the transcript." | Normalise both sides before you compare. | WER looks worse than it is. |
| No timestamps | "The text is enough." | Keep word times for review. | Reviewers replay the whole note. |
| One block of text for a call | "The words are all there." | Label each speaker. | Nobody can tell who promised the refund. |

## Check Your Understanding

## Exercises

## What to Remember

- A speech model writes what it hears, not what you meant.
- Give it your product names.
- Normalise numbers to digits.
- Measure WER and the words your app depends on.
- Keep timestamps for review.
- Label speakers in calls.

## Further Reading

- [Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), by OpenAI](https://arxiv.org/abs/2212.04356): the paper behind the Whisper models.
- [Transformers.js documentation, by Hugging Face](https://huggingface.co/docs/transformers.js/index): run Whisper in the browser.

Next: **Voice Agents and Turn-Taking**, where the app talks back, and must know when the customer has finished speaking.