ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter8/SimpleVLA-RL/robotwin2_tasks_description.md

@@ -0,0 +1,702 @@
+# RoboTwin 2.0 Tasks: Experimental Setup
+
+**Tasks Used in Experiments**: `beat_block_hammer` and `move_can_pot`
+
+**Based on**: [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://arxiv.org/pdf/2509.09674) (2025)
+
+---
+
+## Table of Contents
+1. [Overview](#overview)
+2. [Task 1: beat_block_hammer](#task-1-beat_block_hammer)
+3. [Task 2: move_can_pot](#task-2-move_can_pot)
+4. [Environment Randomization](#environment-randomization)
+5. [Language Instruction System](#language-instruction-system)
+6. [Training Configuration](#training-configuration)
+
+---
+
+## Overview
+
+Both tasks are part of the RoboTwin 2.0 benchmark, designed for dual-arm manipulation with realistic physics simulation using SAPIEN. These tasks test different manipulation capabilities:
+
+- **beat_block_hammer**: Tool use and contact-based manipulation
+- **move_can_pot**: Pick-and-place with spatial reasoning
+
+### Common Characteristics
+
+| Property | Value |
+|----------|-------|
+| **Simulator** | SAPIEN (CPU-based physics) |
+| **Robot Platform** | ALOHA (dual-arm, 7 DOF per arm) |
+| **Action Space** | 14-dimensional continuous (7 DOF × 2 arms) |
+| **Observation** | RGB images (3 cameras) + Proprioception (14-dim joint states) |
+| **Camera Views** | Head camera + Left wrist + Right wrist (224×224×3 each) |
+| **Max Environment Steps** | 200 steps per episode |
+| **Action Chunking** | 25 action chunks per VLA inference |
+| **VLA Calls per Episode** | ~8 (200 ÷ 25) |
+| **Traj Mini-Batch Size** | 8 |
+
+---
+
+## Task 1: beat_block_hammer
+
+### Task Description
+
+**Objective**: Use the robot arm to grasp a hammer and strike a block on the table.
+
+**Natural Language Description**: "There is a hammer and a block on the table, use the arm to grab the hammer and beat the block"
+
+### Scene Setup
+
+The environment contains two objects on a table:
+
+#### Hammer
+- **Model**: Standard hammer (model ID: 020_hammer)
+- **Initial Position**: Fixed location at table center
+  - X-coordinate: 0 meters (table center)
+  - Y-coordinate: -0.06 meters (slightly back from center)
+  - Z-coordinate: 0.783 meters (on table surface)
+- **Orientation**: Fixed (slightly angled)
+- **Physical Properties**: Very light mass (0.001 kg) for easy manipulation
+- **Functional Points**: 
+  - Point 0: Hammer head (for striking)
+  - Base point: Handle (for grasping)
+
+#### Block
+- **Model**: Static red cube (doesn't move when hit)
+- **Size**: 2.5 cm × 2.5 cm × 2.5 cm
+- **Initial Position**: **Randomized** (see randomization section)
+- **Color**: Red (for visibility)
+- **Physical Properties**: Static (fixed to table, doesn't react to physics)
+- **Functional Points**:
+  - Point 0: Block center
+  - Point 1: Block top surface (target for hammer)
+
+### Success Criteria
+
+The task is considered successful when **ALL** of the following conditions are met:
+
+1. **Position Accuracy**: The hammer head is within 2 cm of the block position (in X-Y plane)
+   - Measured as Euclidean distance in horizontal plane
+   - Tolerance: ±2 cm in both X and Y directions
+
+2. **Contact Detection**: The hammer and block are in physical contact
+   - Verified through SAPIEN's collision detection system
+   - Must be simultaneous with position accuracy
+
+**Mathematical Formulation**:
+```
+Success = (|hammer_x - block_x| < 0.02 m) AND 
+          (|hammer_y - block_y| < 0.02 m) AND 
+          (hammer_contacts_block = True)
+```
+
+### Expert Demonstration Strategy
+
+Human demonstrations follow a four-step procedure:
+
+**Step 1: Arm Selection**
+- Automatically select arm based on block position
+- If block X-coordinate < 0: use left arm
+- If block X-coordinate ≥ 0: use right arm
+- This creates symmetric training data for both arms
+
+**Step 2: Grasp Hammer**
+- Approach hammer from 12 cm away (pre-grasp phase)
+- Move gripper to handle position
+- Close gripper at 1 cm distance (secure grasp)
+- Gripper fully closes around handle
+
+**Step 3: Lift Hammer**
+- Raise hammer 7 cm upward (Z-axis displacement)
+- Creates clearance from table surface
+- Prepares for striking motion
+
+**Step 4: Strike Block**
+- Move hammer toward block's top surface
+- Approach from 6 cm above (pre-placement phase)
+- Lower hammer until contact with block
+- Maintain closed gripper throughout strike
+- Stop when contact is detected
+
+### Challenges & Learning Opportunities
+
+**Key Challenges**:
+1. **Spatial Reasoning**: Block position varies significantly (50×20 cm range)
+2. **Tool Use**: Must use hammer as extension of arm, not direct manipulation
+3. **Contact Control**: Requires precise positioning for contact detection
+4. **Arm Coordination**: Single-arm task but must avoid self-collision
+
+**RL Discovery Potential**:
+- SFT model learns: grasp → lift → move → strike (following demonstrations)
+- RL may discover: more efficient striking trajectories, improved contact timing
+- Demonstrates RL's ability to refine precision and timing in tool-use scenarios
+
+---
+
+## Task 2: move_can_pot
+
+### Task Description
+
+**Objective**: Pick up a can from the table and place it beside a pot.
+
+**Natural Language Description**: "There is a can and a pot on the table, use one arm to pick up the can and move it to beside the pot"
+
+### Scene Setup
+
+The environment contains two objects on a table:
+
+#### Pot (Kitchen Pot)
+- **Model**: Kitchen pot with handles (model family: 060_kitchenpot)
+- **Model Variants**: 7 different pot models (IDs 0-6)
+  - Different sizes, shapes, colors
+  - Randomly selected each episode
+- **Initial Position**: Near table center
+  - X-coordinate: 0 meters (center)
+  - Y-coordinate: 0 meters (center)
+  - Z-coordinate: On table surface
+- **Orientation**: Random rotation ±22.5° around vertical axis
+- **Physical Properties**: Normal mass, can be moved if pushed hard
+- **Functional Points**: Center point for distance measurement
+
+#### Can (Sauce Can)
+- **Model**: Cylindrical sauce can (model family: 105_sauce-can)
+- **Model Variants**: 5 different can models (IDs: 0, 2, 4, 5, 6)
+  - Different labels, sizes
+  - Randomly selected from subset
+- **Initial Position**: **Randomized** (see randomization section)
+- **Orientation**: Random rotation around vertical axis
+- **Physical Properties**: Standard mass, cylindrical shape
+- **Functional Points**: Center point and top surface
+
+### Success Criteria
+
+The task is considered successful when **ALL** of the following conditions are met:
+
+1. **Horizontal Distance**: Can is 18 cm (±20 cm tolerance) from pot horizontally
+   - Measured on the correct side (left arm → left of pot, right arm → right of pot)
+   - Must be on the correct side: cannot be on opposite side of pot
+
+2. **Lateral Alignment**: Can Y-position is within 3.5 cm of pot Y-position
+   - Ensures can is "beside" pot, not in front or behind
+
+3. **Can Orientation (Upright)**:
+   - X-axis rotation: 90° ± 15° (upright, not tilted forward/backward)
+   - Y-axis rotation: 0° ± 15° (upright, not tilted left/right)
+   - Can must maintain stable upright position
+
+4. **On Table Surface**: Can Z-position ≤ original pot height + 0.1 cm
+   - Ensures can is resting on table, not floating or dropped
+
+5. **Released**: Both robot grippers are open
+   - Confirms can has been released successfully
+   - Not being held by robot
+
+**Mathematical Formulation**:
+```
+Success = (0 < |can_x - pot_x| < 0.2 m) AND          # Distance tolerance
+          (|can_y - pot_y| < 0.035 m) AND             # Lateral alignment
+          (|can_rotation_x - 90°| < 15°) AND          # Upright X
+          (|can_rotation_y - 0°| < 15°) AND           # Upright Y
+          (can_z ≤ table_surface + 0.001 m) AND       # On surface
+          (can_on_correct_side = True) AND            # Correct side of pot
+          (left_gripper_open = True) AND              # Released
+          (right_gripper_open = True)                 # Released
+```
+
+### Expert Demonstration Strategy
+
+Human demonstrations follow a five-step procedure:
+
+**Step 1: Arm Selection**
+- Automatically select arm based on can position
+- If can X-coordinate > 0: use right arm
+- If can X-coordinate ≤ 0: use left arm
+- This determines which side of pot to place can
+
+**Step 2: Grasp Can**
+- Approach can from 5 cm away (pre-grasp phase)
+- Move gripper to can center
+- Close gripper around can body
+- Secure cylindrical grasp
+
+**Step 3: Lift and Retract**
+- Move 10 cm backward (away from pot, Y-axis: -0.1 m)
+- Simultaneously lift 10 cm upward (Z-axis: +0.1 m)
+- Creates clearance from pot and other obstacles
+- Prevents collision during transport
+
+**Step 4: Transport to Target**
+- Calculate target position beside pot:
+  - If left arm: target_x = pot_x - 0.18 m (18 cm to left)
+  - If right arm: target_x = pot_x + 0.18 m (18 cm to right)
+  - target_y = pot_y (same lateral position)
+  - target_z = table_surface
+- Move gripper with can to target position
+- Maintain upright orientation throughout
+
+**Step 5: Place Can**
+- Approach target from 5 cm above (pre-placement phase)
+- Lower can smoothly to table surface
+- Ensure stable contact with table
+- Open gripper to release can
+- Retract gripper away from can
+
+### Challenges & Learning Opportunities
+
+**Key Challenges**:
+1. **Spatial Reasoning**: Understanding "beside" relationship (specific distance)
+2. **Dual Randomization**: Both pot and can positions/models vary
+3. **Orientation Maintenance**: Must keep can upright throughout manipulation
+4. **Precision Placement**: Tight tolerance on final can orientation (±15°)
+5. **Side Selection**: Must place on correct side based on arm used
+
+**RL Discovery Potential - "Pushcut" Phenomenon Observed**:
+- SFT learns: grasp → lift high → move → lower carefully (grasp-move-place strategy)
+- **RL discovers: PUSH the can directly to target position** (pushcut strategy)
+- This is a novel behavior NOT present in any demonstration data
+- Pushcut strategy is faster and more robust than the demonstrated approach
+- Paper Section 6.1: "the RL-trained model instead learns to accomplish the task by simply pushing Object A into position"
+- Demonstrates emergence of completely new manipulation strategies through RL exploration
+
+---
+
+## Environment Randomization
+
+### Purpose of Randomization
+
+Randomization serves multiple purposes in robot learning:
+
+1. **Prevent Overfitting**: Ensures policy learns general strategies, not memorizing specific positions
+2. **Improve Generalization**: Policy must work across diverse configurations
+3. **Test Robustness**: Validates policy can handle variability
+4. **Enable Transfer**: Prepares policy for real-world deployment where exact positions cannot be controlled
+
+### beat_block_hammer Randomization
+
+#### Block Position (Fully Randomized)
+
+The block position is randomly sampled each episode:
+
+**X-Coordinate (Left-Right)**:
+- Range: -0.25 to +0.25 meters (50 cm span)
+- Distribution: Uniform random
+- Constraints: Must not be too close to center (|x| ≥ 0.05 m)
+- Purpose: Tests both left and right arm capabilities
+
+**Y-Coordinate (Front-Back)**:
+- Range: -0.05 to +0.15 meters (20 cm span)
+- Distribution: Uniform random
+- Constraints: None specific
+- Purpose: Tests depth perception and reaching
+
+**Z-Coordinate (Height)**:
+- Fixed: 0.76 meters (on table surface)
+- No variation: Block always at same height
+
+**Orientation (Rotation)**:
+- Rotation around Z-axis: 0 to 0.5 radians (0° to ~29°)
+- Distribution: Uniform random
+- Purpose: Tests rotational invariance
+- Note: Block is square, so rotation matters less
+
+**Additional Constraints**:
+- Block must be at least 5 cm from table center (|x| ≥ 0.05)
+- Block must be at least 0.1 cm from origin (x² + y² ≥ 0.001)
+- These prevent trivial configurations
+
+**Prohibited Areas**:
+- 10 cm padding around hammer position
+- 5 cm padding around block position
+- Robot must plan path avoiding these zones
+
+#### Hammer Position (Fixed)
+
+The hammer position is **not randomized**:
+- Always at same position: [0, -0.06, 0.783] meters
+- Fixed orientation
+- Rationale: Simplifies task to focus on block positioning challenge
+
+**Randomization Summary for beat_block_hammer**:
+```
+Total Randomized Parameters: 3
+- Block X: 50 cm range
+- Block Y: 20 cm range  
+- Block Rotation: ~29° range
+
+Configuration Space: ~50 × 20 × 29 = 29,000 possible discrete configurations
+Continuous Space: Infinite (continuous sampling)
+```
+
+### move_can_pot Randomization
+
+#### Pot Randomization (Partial)
+
+**Model Variant**:
+- Randomly select from 7 pot models (IDs 0-6)
+- Different visual appearances (colors, sizes, shapes)
+- Tests visual generalization
+
+**Position**:
+- X-coordinate: 0 meters (fixed at center)
+- Y-coordinate: 0 meters (fixed at center)
+- Z-coordinate: On table surface (fixed)
+
+**Orientation**:
+- Rotation around Z-axis: ±22.5° (±π/8 radians)
+- Distribution: Uniform random
+- Purpose: Tests rotation invariance for target reference
+
+#### Can Randomization (Full)
+
+**Model Variant**:
+- Randomly select from 5 can models (IDs: 0, 2, 4, 5, 6)
+- Different labels and appearances
+- Note: IDs 1 and 3 are excluded (possibly unstable or problematic)
+
+**Position**:
+- X-coordinate: -0.3 to +0.3 meters (60 cm span)
+  - Constraint: |x| ≥ 0.2 m (must not be too close to center)
+  - Prevents can from being directly in front of robot
+- Y-coordinate: +0.05 to +0.15 meters (10 cm span, front of table)
+  - Always on front side of table
+  - Easier for robot to reach
+- Z-coordinate: On table surface (computed based on can model)
+
+**Orientation**:
+- Rotation around Z-axis: ±45° (±π/4 radians)
+- Distribution: Uniform random
+- Purpose: Tests grasping from different angles
+
+**Spatial Constraints**:
+- Can must be at least 30 cm from pot center
+  - Formula: (can_x - pot_x)² + (can_y - pot_y)² ≥ 0.09 m²
+  - Prevents can from starting too close to target
+- If constraint violated, resample position
+
+**Prohibited Areas**:
+- 3 cm padding around pot position
+- 10 cm padding around can position
+- 15 cm × 20 cm zone on approach side of pot (arm-dependent)
+  - If left arm: zone extends 15 cm to left of pot
+  - If right arm: zone extends 15 cm to right of pot
+  - Prevents trivial straight-line motions
+
+**Randomization Summary for move_can_pot**:
+```
+Total Randomized Parameters: 7
+- Pot Model: 7 variants
+- Pot Rotation: ±22.5°
+- Can Model: 5 variants
+- Can X: 60 cm range (with constraints)
+- Can Y: 10 cm range
+- Can Rotation: ±45°
+
+Configuration Space: 7 × 45° × 5 × 60 × 10 × 90° ≈ 8.5M possible discrete configurations
+Continuous Space: Infinite (continuous sampling)
+```
+
+### Randomization Process
+
+Each episode follows this randomization procedure:
+
+**Episode Start**:
+1. Sample all random parameters from their respective distributions
+2. Check spatial constraints (distance requirements, prohibited zones)
+3. If constraints violated: resample violating parameters
+4. Repeat until valid configuration found
+5. Initialize SAPIEN scene with sampled configuration
+6. Generate task instruction (random selection from template bank)
+7. Compute initial observation (RGB images + proprioception)
+8. Begin episode execution
+
+**Key Design Principles**:
+- **Constraint Validation**: Ensures physically feasible configurations
+- **Iterative Sampling**: Rejection sampling until valid
+- **Symmetric Design**: Equal probability for left/right arm usage
+- **Difficulty Calibration**: Constraints prevent too-easy or impossible scenarios
+
+### Impact on RL Training
+
+**Dynamic Sampling Interaction**:
+- Randomization creates natural difficulty distribution
+- Dynamic Sampling (paper Section 3.3) further filters:
+  - Excludes episodes where all 8 samples succeed (too easy)
+  - Excludes episodes where all 8 samples fail (too hard)
+  - Keeps only episodes with mixed outcomes (learnable)
+- Combined effect: RL focuses on appropriate difficulty frontier
+
+**Curriculum Learning Effect**:
+- Early training: High failure rate, mostly hard configurations kept
+- Mid training: More mixed outcomes, broader configuration range
+- Late training: High success rate, need new challenging configurations
+- This naturally implements curriculum without manual intervention
+
+---
+
+## Language Instruction System
+
+### Instruction Generation Pipeline
+
+Both tasks use a sophisticated language instruction system to test generalization:
+
+**Template-Based Generation**:
+1. Define full task description (detailed)
+2. Specify instruction preferences (word count, style)
+3. Define object schema (placeholders for specific objects)
+4. Use LLM to generate diverse paraphrases
+5. Manually curate into "seen" and "unseen" sets
+
+**Schema Placeholders**:
+- `{A}`: Primary object (hammer in task 1, pot in task 2)
+- `{B}`: Secondary object (can in task 2)
+- `{a}`: Arm designation (left or right)
+
+### beat_block_hammer Instructions
+
+**Full Description**: "there is a hammer and a block on the table, use the arm to grab the hammer and beat the block"
+
+**Schema**: 
+- `{A}` → hammer identifier (e.g., "020_hammer/base0")
+- `{a}` → arm used (e.g., "left" or "right")
+
+**Instruction Preferences**:
+- Word count: ≤10 words
+- Must mention: grasping action, striking action
+- Can mention: arm designation (optional)
+
+**Seen Instructions (50+ variants)**:
+Examples used during training:
+- "Pick {A} and strike the block."
+- "Lift {A} using {a} to hit the block."
+- "Grab {A} with {a} and beat the block."
+- "Use {A} to hammer the block."
+- "Take {A} and smash the block."
+- "Hold {A} and pound the block."
+
+**Unseen Instructions (10 variants)**:
+Examples reserved for evaluation:
+- "Grab {A} and beat the block."
+- "Use {a} to pick up {A}."
+- "Grab {A} and strike the block."
+- "Use {a} to grab {A} then beat block"
+
+**Instruction Diversity**:
+- Verbs: grab, pick, lift, take, hold, use, employ
+- Actions: beat, strike, hit, hammer, pound, smash
+- Styles: imperative, sequential, compound sentences
+- Tests: synonym understanding, sentence structure variations
+
+### move_can_pot Instructions
+
+**Full Description**: "there is a can and a pot on the table, use one arm to pick up the can and move it to beside the pot"
+
+**Schema**:
+- `{A}` → pot identifier (e.g., "060_kitchenpot/base3")
+- `{B}` → can identifier (e.g., "105_sauce-can/base2")
+- `{a}` → arm used (e.g., "left" or "right")
+
+**Instruction Preferences**:
+- Word count: ≤10 words
+- Must mention: picking action, placement location
+- Must specify: relative position to pot
+
+**Seen Instructions (50+ variants)**:
+Examples used during training:
+- "Use {a} to grab {B} and move it next to {A}"
+- "Pick {B} up with {a} then place near {A}"
+- "Grab {B} with {a} and set it near {A}"
+- "Lift {B} and move it near {A}"
+- "Take {B} to {A} using {a}"
+- "Set {B} right next to {A}"
+
+**Unseen Instructions (10 variants)**:
+Examples reserved for evaluation:
+- "Pick up {B} and move it near {A}"
+- "Grab {B} and set it beside {A}"
+- "Use {a} to pick up {B}, move it near {A}"
+- "Grab {B} and place it beside {A}"
+
+**Instruction Diversity**:
+- Verbs: grab, pick, lift, take, move, transfer, relocate
+- Spatial terms: near, beside, next to, by, close to
+- Styles: with/without arm mention, sequential vs compound
+- Tests: spatial relationship understanding, action sequence parsing
+
+### Episode-Level Instruction Assignment
+
+**During Training** (Seen Instructions):
+- Each rollout: randomly select one instruction from seen set
+- 50+ instructions → low probability of repeating same instruction
+- Each instruction has equal selection probability
+- Ensures model doesn't overfit to specific phrasings
+
+**During Validation** (Mixed):
+- IID validation: use seen instructions (same distribution as training)
+- OOD validation: use unseen instructions (test generalization)
+- Both sets used to measure different aspects of performance
+
+**Runtime Substitution**:
+When instruction is selected, placeholders are replaced:
+- `{A}` → actual object identifier from episode
+- `{B}` → actual object identifier from episode  
+- `{a}` → "left" or "right" based on arm selection algorithm
+
+**Example Runtime Process**:
+```
+Template: "Use {a} to grab {B} and move it next to {A}"
+
+Episode Configuration:
+- Pot model: 060_kitchenpot/base3
+- Can model: 105_sauce-can/base5
+- Can X-position: +0.22 (right side)
+- Selected arm: right
+
+Final Instruction: "Use right to grab 105_sauce-can/base5 and move it next to 060_kitchenpot/base3"
+
+Simplified for VLA: "Use right arm to grab the sauce can and move it next to the kitchen pot"
+```
+
+### Instruction System Benefits
+
+**For Training**:
+1. **Language Generalization**: 50+ variants prevent language overfitting
+2. **Robustness**: Model must understand task concept, not memorize phrases
+3. **Diversity**: Different instruction styles cover various language patterns
+4. **Grounding**: Schema system links language to specific objects in scene
+
+**For Evaluation**:
+1. **Unseen Instructions**: Tests true language understanding
+2. **IID vs OOD**: Measures both in-distribution and out-of-distribution performance
+3. **Zero-Shot Transfer**: Unseen instructions never appeared in training
+4. **Generalization Metric**: Success rate on unseen instructions indicates robustness
+
+---
+
+## Training Configuration
+
+### Dataset Configuration
+
+**Training Data Source**:
+- Pre-collected feasible seeds: 1000 per task
+- Seeds validated through simulation to ensure solvability
+- Stored in: `verl/utils/envs/robotwin2/seeds/robotwin2_train_seeds.json`
+
+**Validation Data**:
+- IID Validation: 128 seeds from training distribution
+- OOD Validation: 128 seeds from held-out distribution
+- Total validation: 256 episodes per task
+
+**Data Loading**:
+- Batch size: 64 task instances per training step
+- Samples per task: 8 (for GRPO grouping)
+- Total rollouts per step: 512 (64 × 8)
+- Shuffle: Enabled (random task ordering each epoch)
+
+### Rollout Configuration
+
+**Action Space**:
+- Continuous 14-dimensional actions
+- 7 DOF per arm (position + orientation + gripper)
+- Action normalization: [-1, 1] range
+- Denormalization statistics: Task-specific (learned from demonstrations)
+
+**Action Chunking**:
+- Chunks per VLA inference: 25
+- Action tokens per chunk: 14 (one per dimension)
+- Total tokens per inference: 350 (25 × 14)
+- VLA calls per episode: ~8 (200 steps ÷ 25)
+
+**Rollout Parameters**:
+- Temperature: 1.6 (Higher Rollout Temperature - paper enhancement)
+- Sampling: Enabled (do_sample=True)
+- Max environment steps: 200
+- Micro-batch size: 1 (sequential processing)
+- Episode timeout: 200 steps = success or failure
+
+### Success Rate Targets
+
+Based on paper results and dynamic sampling:
+
+**beat_block_hammer**:
+- Initial SFT performance: ~25-35% (estimated)
+- Target after 300 steps: ~50-70% (paper-level)
+- Dynamic sampling keeps: 10-90% success rate episodes
+- Expected at convergence: ~60-80% success
+
+**move_can_pot**:
+- Initial SFT performance: ~20-30% (estimated)
+- Target after 300 steps: ~45-65% (paper-level)
+- Dynamic sampling keeps: 10-90% success rate episodes
+- Expected at convergence: ~55-75% success
+
+**Training Progress Indicators**:
+- Epoch 0-5: Mostly failures, learning basic grasping
+- Epoch 5-10: Increasing success, learning task structure
+- Epoch 10-15: Rapid improvement, discovering strategies
+- Epoch 15-20: Convergence, refining precision
+- Beyond 20: Marginal gains, potential overfitting
+
+### Computational Requirements
+
+**Per Training Step** (~20 minutes):
+- Rollout phase: ~18 minutes (87.8% of time)
+  - Environment initialization: ~2 minutes
+  - VLA inference: ~2 minutes (8 calls × 300ms × 512 rollouts / 8 GPUs)
+  - Physics simulation: ~14 minutes (25 steps × 60ms × 512 rollouts / 8 GPUs)
+- PPO update phase: ~2.4 minutes (11.8% of time)
+- Overhead: ~0.2 minutes (0.4% of time)
+
+**Full Training** (300 steps as per paper):
+- Total time: ~300 steps × 20 minutes = 6000 minutes ≈ 100 hours ≈ 4.3 days
+- GPU usage: 8 × A100 GPUs (or equivalent)
+- Total rollouts: 300 × 512 = 153,600 episodes
+- Total VLA inferences: ~153,600 × 8 = ~1.2M forward passes
+- Total environment steps: ~153,600 × 100 = ~15M physics steps (average)
+
+**Resource Distribution**:
+- GPU compute: ~20% utilization (during VLA inference only)
+- CPU compute: ~80% utilization (physics simulation dominates)
+- GPU memory: ~6 GB per GPU (FSDP sharding)
+- System memory: ~64 GB (environment states, buffers)
+
+---
+
+## Summary
+
+### Task Comparison
+
+| Aspect | beat_block_hammer | move_can_pot |
+|--------|------------------|--------------|
+| **Skill Type** | Tool use + Contact | Pick & Place + Spatial |
+| **Complexity** | Medium | Medium-High |
+| **Randomization** | 1 object (block) | 2 objects (pot + can) |
+| **Success Tolerance** | Tight (±2 cm) | Moderate (±20 cm horizontal) |
+| **Key Challenge** | Precise contact | Spatial reasoning |
+| **Expected Success** | 60-80% after RL | 55-75% after RL |
+| **Novel Behaviors** | "Pushcut" (push vs lift) | Efficient low trajectories |
+
+### Key Experimental Insights
+
+**Why These Tasks?**
+1. **Diverse Skills**: Test different manipulation capabilities (tool use vs spatial reasoning)
+2. **Realistic Complexity**: Neither too easy nor impossible, appropriate for RL learning
+3. **Measurable Success**: Clear binary success criteria, no ambiguity
+4. **Rich Randomization**: Sufficient variability to test generalization
+5. **Dual-Arm Platform**: Tests coordination and arm selection logic
+
+**What RL Learns**:
+1. **Beyond Demonstrations**: Discovers strategies not in expert data ("pushcut")
+2. **Robustness**: Handles high variability through 153k diverse rollouts
+3. **Efficiency**: Finds shorter, more direct paths than demonstrations
+4. **Generalization**: Works on unseen instructions and configurations
+5. **Exploration**: Higher temperature (1.6) enables diverse strategy discovery
+
+**Expected Outcomes**:
+- Both tasks show improvement with RL over SFT baseline
+- "Pushcut" phenomenon may emerge in beat_block_hammer
+- move_can_pot benefits from optimized trajectories
+- Success rates reach 60-75% range after 300 training steps
+- Unseen instructions show strong generalization (within 5-10% of seen)
+