---
theme: seriph
title: "Lesson 35 — Why Does a Voice Agent Feel Slow?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Expand · Chapter 9 · Multimodal Interaction
# Why Does a Voice Agent Feel Slow?
Cascaded pipelines, latency waterfalls, streaming, and turn detection
Lesson 35 of 42 · 18 minutes · Voice; Cascaded Pipeline; Full-Chain Streaming; Streaming Voice Perception
---
Expand · Chapter 9 · Multimodal Interaction
# Problems this chapter will solve
Lesson 35
Why Does a Voice Agent Feel Slow?
Lesson 36
When Should Voice Stop Taking Turns?
Lesson 37
How Does an Agent Act Through Pixels?
Lesson 38
How Does an Agent Turn Plans into Physical Actions?
---
# Why this problem matters
Turn detection
VAD waits for silence and can cut off a thinking pause.
Serial work
ASR, LLM, and TTS latency accumulate when stages wait.
Queueing
High utilization amplifies latency nonlinearly.
---
# Three ideas to keep in view
Cascaded
VAD → ASR → LLM → TTS
Streaming
Emit partial transcripts, tokens, and audio chunks early
Convergence
Early recognition is fast but may change as context arrives
---
# The book's visual model
Latency waterfall for a serial voice pipeline
---
# Wait for completion vs. Stream the chain
Wait for completion
- Stable transcript
- Simple control
- Every stage adds delay
Stream the chain
- Earlier first audio
- Overlapped work
- Corrections and cancellation required
Streaming changes when information becomes available—not the component boundaries.
---
# Pipeline stages should overlap
~~~python
async for partial_text in asr.stream(audio):
llm.update(partial_text)
async for sentence in llm.sentences():
tts.enqueue(sentence)
if user_interrupts(): cancel_output()
~~~
---
# Test the claim
9-11 min
Preflight a cascaded voice Agent
Observe: VAD model, ASR/LLM/TTS provider configuration, and missing runtime prerequisites
9-22 min
Generate controlled streaming-ASR scenarios
Observe: Normal speech, a 900 ms pause, and background noise under identical source content
Demo budget: 3 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ cd chapter9/live-audio/backend && npm run check
$ cd chapter9/streaming-speech && python prepare_scenarios.py audio/sentence.wav validation/course-scenarios
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
The silence threshold is both a latency control and a turn-taking assumption.
Finding 2
Streaming hides work behind speech but introduces unstable partial hypotheses.
Finding 3
Time to first useful audio matters more than full-response completion time.
---
layout: center
---
Where the claim stops
# Boundary condition
A setup check or generated audio scenario verifies wiring and controls—not human conversational quality.
---
layout: center
---
Engineering takeaway
# Design rule
Instrument the latency of every boundary, then stream and overlap only where cancellation and correction are designed.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
Which latency number would best predict whether a user interrupts or abandons your voice Agent?
---
layout: center
class: text-center
---
Next · Lesson 36
Remove more boundaries—and decide what fast interaction should do while slow reasoning continues.
→
