ai-agent-book/slides/lesson-38.md

theme, title, info, author, transition, mdc, lineNumbers, monaco, aspectRatio, canvasWidth, layout, class

theme	title	info	author	transition	mdc	lineNumbers	monaco	aspectRatio	canvasWidth	layout	class
seriph	Lesson 38 — How Does an Agent Turn Plans into Physical Actions?	English video course for AI Agents in Depth	Bojie Li	slide-left	true	false	false	16/9	980	cover	cover

Expand · Chapter 9 · Multimodal Interaction

How Does an Agent Turn Plans into Physical Actions?

Planning-control separation, VLA control, safety gates, and Sim2Real

Lesson 38 of 42 · 18 minutes · Robot Manipulation; Planning and Control; VLA Control; Sim2Real Transfer

---

layout: center class: text-center

The central question

How can slow semantic planning drive fast physical control without losing safety?

---

Why this problem matters

Planning

A vision-language model selects goals and interprets the scene.

Control

A fast policy turns the current observation into motor commands.

Safety

External gates must constrain forces, motion, workspace, and authority.

---

Three ideas to keep in view

Two-layer loop

Slow planning chooses subgoals; fast control executes motion

Action chunking

Predict several future controls per expensive inference

Sim2Real

Train across calibrated visual and physical variation

---

The book's visual model

Vision-Language-Action model architecture

---

Open-loop plan vs. Guarded feedback loop

Open-loop plan

• Commit to a long motion
• Assume the world stays fixed
• Detect errors late

Guarded feedback loop

• Short action horizon
• Re-observe continuously
• Interrupt on state change

Physical autonomy depends on feedback frequency and authority boundaries.

---

Slow plan, fast guarded control

subgoal = planner.choose(observation, task)
chunk = controller.predict(observation, subgoal)
for action in safety_filter(chunk):
    robot.execute(action)
    observation = robot.observe()
    if world_changed(observation): break

---

Test the claim

9-9 dry configuration1 min

Inspect a fail-closed robot navigation contract

Observe: Exact model ID, task, camera, three motion tools, decision frequency, and no actuation

Robot safety gates2 min

Run evidence-validator regressions for physical experiments

Observe: Why dry runs, mock artifacts, and unverified motion cannot satisfy completion

Demo budget: 3 minutes · one contiguous terminal block

---

class: course-terminal

Live demo

Switching to the terminal

$ python chapter9/gemini-xlerobot-navigation/navigation.py

$ python chapter9/xlerobot-teleoperation/test_validator.py && python chapter9/gemini-xlerobot-navigation/test_validator.py && python chapter9/rgb-sim2real-grasping/test_validator.py

Run the command(s), narrate decisions, and point to the observation—not just the output.

---

What the evidence supports

Finding 1

Planning and control operate at different semantic and temporal scales.

Finding 2

Action chunks reduce inference pressure but delay response to unexpected change.

Finding 3

Physical completion requires calibrated hardware, authorization, measurements, and direct artifacts.

---

layout: center

Where the claim stops

Boundary condition

A source audit, preflight, validator test, or dry configuration demonstrates architecture and blockers—not a successful robot run.

---

layout: center

Engineering takeaway

Design rule

Keep physical actions behind external safety gates, short feedback horizons, and measurements the model cannot fabricate.

---

Continue the experiment

Experiment 9-7: XLeRobot teleoperation chapter9/xlerobot-teleoperation/ Experiment 9-9: robot navigation chapter9/gemini-xlerobot-navigation/ Experiment 9-11: RGB Sim2Real grasping chapter9/rgb-sim2real-grasping/ Sim2Real pipeline book-en/images/fig9-13.svg

---

layout: center class: text-center

Pause and apply

Your turn

How quickly must a physical controller reconsider an action when the world changes unexpectedly?

---

layout: center class: text-center

Chapter 9 complete · Next · Lesson 39

Scale from one Agent loop to several loops that exchange context, artifacts, and control.

→