Voice Agents and Turn-Taking

By the Syncfusion DevTutorials team · Last updated October 27, 2026

Build a voice agent that answers fast without talking over people: streaming, endpointing, barge-in, noise handling, holding lines during tool calls, and hand-over to a person.

What a Voice Agent Is

Voice agent = an app that listens to a person, decides what to say, and answers out loud, in a back-and-forth conversation.

In this lesson, you will build the voice line for the claims app. A customer calls, explains the problem, and the agent asks for what it needs.

The hard part is not the words. It is the timing. The first version waits too long to answer, then talks over the customer when they pause to think.

Press Play, or use Next to go one step at a time.

• The timeline: the customer's speech on top, the agent's below.
• The pause: the customer stops for 650 ms in the middle of a sentence.
• The red mark: the agent decides the turn is over and starts talking.

Simple lesson: a voice agent must know when the person has really finished.

A Real-World Example

Think about a good waiter taking an order.

• They answer quickly once you finish, so you do not wonder if they heard.
• They wait when you say "I'll have the… um…", because you are not done.
• They stop talking at once if you cut in with "Sorry, no onions".
• They ignore the clatter from the kitchen.

A voice agent needs the same four skills:

• Answer quickly = a latency budget: first audio within 1.5 s.
• Wait for "um" = endpointing: deciding when the turn is over.
• Stop when cut off = barge-in.
• Ignore the kitchen = robust voice activity detection (VAD).

Simple lesson: good turn-taking feels like a polite person, not a fast machine.

What You Will Learn

By the end of this lesson, you will be able to:

• break down where the waiting comes from
• compare a cascade agent with a speech-to-speech agent
• choose an endpointing rule
• handle barge-in without stopping for every cough
• keep the line alive while a tool runs
• hand over to a person when asked
• meet a 1.5 s reply budget

Where the Waiting Comes From

A cascade agent has three models in a row: speech to text, an LLM, then text to speech. Each one adds time.

Step through the frames.

• The bar: time for each step, after the agent decides the customer has finished.
• Without streaming: the LLM writes the whole answer before speech starts. 1.9 s.
• With streaming: speech starts on the first sentence. 0.52 s.

Simple lesson: stream every step, so the first words come out before the last ones are written.

When Has the User Finished?

Endpointing = deciding that the user's turn is over.

Step through the frames.

• Silence rule, 500 ms: fast, but cuts the customer off at a 650 ms pause.
• Silence rule, 1,000 ms: never cuts off, but every answer is slow.
• Semantic rule: waits 350 ms after a finished sentence, and 1,100 ms after an unfinished one.

A semantic endpoint looks at the words so far. "My order number is…" is clearly not finished. "The lid is cracked." is.

Simple lesson: wait longer when the sentence is not finished, and shorter when it is.

Interruptions and Noise

Barge-in = the agent stops talking as soon as the user starts. Without it, the agent talks over the customer who is correcting the address.

But a sensitive barge-in also stops for a cough.

Step through the frames.

• Sensitive VAD: the agent stops mid-sentence for a cough in the kitchen.
• Robust VAD: the cough is ignored; a real interruption still stops the agent within 180 ms.

Tools and People

A real voice agent does more than talk. It looks things up, and sometimes it must pass the call to a person.

• Tool calls: "Has order A-1042 shipped?" needs an order lookup that takes 1.8 s. Silence that long feels like a dropped call. Say a short holding line first ("Let me check that for you…"), then answer when the result comes back.
• Hand-over: when someone asks for a person, transfer them, with a short summary of the call so they do not repeat themselves. Never keep them talking to the bot.

Click the button and look at the Pipeline tab.

• The timeline: customer and agent, with each event marked.
• Red marks: a cut-off, a slow answer, a stop for noise, or a trapped caller.
• Hear the conversation: your browser's voices read both sides, with what the agent really says.

Lab Exercise: Reply Fast, Never Trap the Caller

Your task: change the Ingest, Perceive and Use settings, then press Check.

Click the button and open the Pipeline tab.

• Design: cascade or speech-to-speech; streaming on or off.
• Endpoint: silence (with a time) or semantic.
• Barge-in and VAD: on or off; sensitive or robust.
• Holding line and hand-over: on or off.
• To pass: first audio within 1.5 s on every turn (even during the order lookup), no one cut off, barge-in stops within 0.3 s, no stop for a cough, and a hand-over when someone asks for a person.

Build It in TypeScript

Turn-taking is a small function you can test without a model.

// A sentence is finished when it ends with . ? or !
export function shouldEndTurn({ silenceMs, words }: { silenceMs: number; words: string }): boolean {
  const finished = /[.?!]$/.test(words.trim());
  return silenceMs >= (finished ? 350 : 1100);
}

The lab uses this rule when you pick Semantic. Real apps often use a small model instead of punctuation, but the idea is the same.

Click the button and open the Code tab.

• The starter: ends the turn after 500 ms, whatever was said.
• Run on the media bank: your function decides 19 moments from the lab's conversations.

TL;DR

1. A voice agent listens, decides and speaks, in turns.
2. Latency adds up across steps: stream every step.
3. Cascade = checkable text in the middle; speech-to-speech = faster.
4. Endpointing: wait longer for unfinished sentences.
5. Barge-in: stop within 0.3 s when the user talks.
6. Robust VAD: do not stop for a cough.
7. Holding line during a tool call; hand over to a person when asked.

Common mistakes

Check Your Understanding

Exercises

What to Remember

• Timing matters as much as words.
• Stream every step.
• Wait longer when a sentence is unfinished.
• Stop at once when the user talks.
• Ignore short noises.
• Never leave the line silent; never trap a caller.

Further Reading

• Web Audio API, by MDN: capture and play audio in the browser.
• Transformers.js documentation, by Hugging Face: on-device speech and voice activity models.

Next: Image Generation with Checks, where the app makes pictures, and checks every one against the brief before it is used.