---
theme: seriph
title: "Lesson 38 — How Does an Agent Turn Plans into Physical Actions?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Expand · Chapter 9 · Multimodal Interaction
# How Does an Agent Turn Plans into Physical Actions?
Planning-control separation, VLA control, safety gates, and Sim2Real
Lesson 38 of 42 · 18 minutes · Robot Manipulation; Planning and Control; VLA Control; Sim2Real Transfer
---
layout: center
class: text-center
---
The central question
How can slow semantic planning drive fast physical control without losing safety?
---
# Why this problem matters
Planning
A vision-language model selects goals and interprets the scene.
Control
A fast policy turns the current observation into motor commands.
Safety
External gates must constrain forces, motion, workspace, and authority.
---
# Three ideas to keep in view
Two-layer loop
Slow planning chooses subgoals; fast control executes motion
Action chunking
Predict several future controls per expensive inference
Sim2Real
Train across calibrated visual and physical variation
---
# The book's visual model
Vision-Language-Action model architecture
---
# Open-loop plan vs. Guarded feedback loop
Open-loop plan
- Commit to a long motion
- Assume the world stays fixed
- Detect errors late
Guarded feedback loop
- Short action horizon
- Re-observe continuously
- Interrupt on state change
Physical autonomy depends on feedback frequency and authority boundaries.
---
# Slow plan, fast guarded control
~~~python
subgoal = planner.choose(observation, task)
chunk = controller.predict(observation, subgoal)
for action in safety_filter(chunk):
robot.execute(action)
observation = robot.observe()
if world_changed(observation): break
~~~
---
# Test the claim
9-9 dry configuration1 min
Inspect a fail-closed robot navigation contract
Observe: Exact model ID, task, camera, three motion tools, decision frequency, and no actuation
Robot safety gates2 min
Run evidence-validator regressions for physical experiments
Observe: Why dry runs, mock artifacts, and unverified motion cannot satisfy completion
Demo budget: 3 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ python chapter9/gemini-xlerobot-navigation/navigation.py
$ python chapter9/xlerobot-teleoperation/test_validator.py && python chapter9/gemini-xlerobot-navigation/test_validator.py && python chapter9/rgb-sim2real-grasping/test_validator.py
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Planning and control operate at different semantic and temporal scales.
Finding 2
Action chunks reduce inference pressure but delay response to unexpected change.
Finding 3
Physical completion requires calibrated hardware, authorization, measurements, and direct artifacts.
---
layout: center
---
Where the claim stops
# Boundary condition
A source audit, preflight, validator test, or dry configuration demonstrates architecture and blockers—not a successful robot run.
---
layout: center
---
Engineering takeaway
# Design rule
Keep physical actions behind external safety gates, short feedback horizons, and measurements the model cannot fabricate.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
How quickly must a physical controller reconsider an action when the world changes unexpectedly?
---
layout: center
class: text-center
---
Chapter 9 complete · Next · Lesson 39
Scale from one Agent loop to several loops that exchange context, artifacts, and control.
→
