---
theme: seriph
title: "Lesson 36 — When Should Voice Stop Taking Turns?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Expand · Chapter 9 · Multimodal Interaction
# When Should Voice Stop Taking Turns?
Omni, full-duplex interaction, fast-slow thinking, and controllable speech
Lesson 36 of 42 · 19 minutes · End-to-End Omnimodal Models; Full-Duplex Models; Thinking Architectures; Human-like Speech
---
layout: center
class: text-center
---
The central question
How can an Agent listen, speak, interrupt, and think deeply without making conversation stall?
---
# Why this problem matters
Omni
Preserve prosody and emotion across one end-to-end model.
Full duplex
Choose listen/speak/stop actions many times per second.
Fast + slow
Keep interaction alive while a strategist works in the background.
---
# Three ideas to keep in view
Turn-based Omni
End-to-end audio but still waits for a turn boundary
Interactive model
Concurrent input/output with barge-in and backchannels
Latent bridge
Exchange richer internal state than plain text
---
# The book's visual model
Fast and slow thinking architecture alternatives
---
# Modular cascade vs. End-to-end/full duplex
Modular cascade
- Easy to debug
- Providers interchangeable
- Prosody lost through text
End-to-end/full duplex
- Lower boundary latency
- Preserves acoustic cues
- Harder to observe and control
The newer architecture buys interaction quality with reduced modularity.
---
# Fast interaction can delegate
~~~python
intent = realtime_model.listen(audio_frame)
if intent.needs_deep_work:
job = strategist.start(intent.context)
realtime_model.respond(interaction_state)
if job.ready: realtime_model.integrate(job.result)
~~~
---
# Test the claim
9-32 min
Test the end-to-end speech contract offline
Observe: Exact model contract, audio response handling, and fail-closed behavior without an endpoint
9-42 min
Audit controllable-speech media and listening evidence
Observe: 24 reference profiles, routed controls, media hashes, and the distinction from human MOS
Demo budget: 4 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ python -m pytest chapter9/end-to-end-speech/test_step_audio.py chapter9/end-to-end-speech/test_none_content.py -q
$ cd chapter9/controllable-tts && python validate_artifacts.py
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Full-duplex models replace discrete turns with continuous interaction decisions.
Finding 2
Fast-slow separation preserves responsiveness without forcing every answer to be shallow.
Finding 3
Voice quality must be judged from audio—not configuration labels or text transcripts.
---
layout: center
---
Where the claim stops
# Boundary condition
End-to-end behavior is harder to attribute to ASR, reasoning, timing, or synthesis when a failure occurs.
---
layout: center
---
Engineering takeaway
# Design rule
Choose the simplest voice architecture that meets the interaction target, and preserve modality-native observability at every boundary you remove.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
What should your fast interaction model be allowed to say before the slow model has verified the answer?
---
layout: center
class: text-center
---
Next · Lesson 37
Carry perception and action into visual interfaces, where every click changes the next observation.
→
