# veRL Training Process: Complete System Analysis **Based on**: [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://arxiv.org/pdf/2509.09674) (2025) ## Table of Contents 1. [Overall Architecture](#overall-architecture) 2. [Training Configuration](#training-configuration) 3. [Complete Training Loop](#complete-training-loop) 4. [Phase 1: Rollout with Dynamic Sampling](#phase-1-rollout-with-dynamic-sampling) 5. [Phase 2: Outcome Reward Modeling](#phase-2-outcome-reward-modeling) 6. [Phase 3: Advantage Estimation (GRPO)](#phase-3-advantage-estimation-grpo) 7. [Phase 4: Actor Update (PPO with Clip Higher)](#phase-4-actor-update-ppo-with-clip-higher) 8. [Phase 5: Validation](#phase-5-validation) 9. [Data Flow and Synchronization](#data-flow-and-synchronization) 10. [Performance Analysis](#performance-analysis) 11. [Key Paper Insights](#key-paper-insights) --- ## Overall Architecture ### System Components ``` ┌─────────────────────────────────────────────────────────────────┐ │ Ray Framework │ │ ┌────────────────┐ ┌────────────────┐ ┌────────────────┐ │ │ │ ActorRollout │ │ Critic │ │ RefPolicy │ │ │ │ Worker │ │ Worker │ │ Worker │ │ │ │ (8 GPUs) │ │ (8 GPUs) │ │ (8 GPUs) │ │ │ └────────────────┘ └────────────────┘ └────────────────┘ │ │ ↓ ↓ ↓ │ │ ┌──────────────────────────────────────────────────────────┐ │ │ │ Shared GPU Resource Pool │ │ │ │ [GPU 0, GPU 1, ..., GPU 7] │ │ │ └──────────────────────────────────────────────────────────┘ │ └─────────────────────────────────────────────────────────────────┘ ↓ ┌─────────────────┐ │ RayTrainer │ │ (Main Process) │ └─────────────────┘ ``` **Key Components** (from `main_ppo.py:159-173`): ```python role_worker_mapping = { Role.ActorRollout: ray.remote(RobActorRolloutRefWorker), # VLA model for rollout & training Role.Critic: ray.remote(CriticWorker), # Value network (GAE only) Role.RefPolicy: ray.remote(RobActorRolloutRefWorker) # Frozen reference policy (KL penalty) } resource_pool_spec = { 'global_pool': [8] * 1, # 8 GPUs on 1 node } mapping = { Role.ActorRollout: 'global_pool', Role.Critic: 'global_pool', Role.RefPolicy: 'global_pool', } ``` **Worker Responsibilities**: 1. **ActorRollout Worker**: - Generate rollouts in environments - Compute log probabilities - Update actor policy (backpropagation) - Compute entropy for exploration 2. **Critic Worker** (optional, only for GAE): - Estimate value function V(s) - Used for advantage estimation 3. **RefPolicy Worker** (optional, only if `kl_coef > 0`): - Compute reference log probabilities from frozen SFT model - Used for KL penalty regularization --- ## Training Configuration From `run_openvla_oft_rl_twin2.sh`: ### Dataset Configuration ```bash data.task_suite_name=robotwin2_beat_block_hammer data.num_trials_per_task=1000 # 1000 task instances data.n_samples=8 # 8 rollouts per task instance data.train_batch_size=64 # 64 task instances per training step data.val_batch_size=256 # 256 for validation data.oversample_factor=1 # No oversampling ``` ### Model Configuration ```bash actor_rollout_ref.model.path=$SFT_MODEL_PATH # OpenVLA-7B SFT model actor_rollout_ref.model.vla=openvla-oft actor_rollout_ref.model.action_token_len=14 # 14 tokens per action actor_rollout_ref.model.action_chunks_len=25 # 25 action chunks per VLA call ``` ### Training Configuration ```bash # Optimization actor_rollout_ref.actor.optim.lr=5e-6 actor_rollout_ref.actor.ppo_mini_batch_size=128 # Mini-batch size for PPO updates actor_rollout_ref.actor.ppo_micro_batch_size=8 # Micro-batch size (= num_gpus) # PPO Hyperparameters actor_rollout_ref.actor.clip_ratio_high=0.28 # PPO clip ratio upper bound actor_rollout_ref.actor.clip_ratio_low=0.2 # PPO clip ratio lower bound actor_rollout_ref.actor.entropy_coeff=0.0 # No entropy bonus actor_rollout_ref.actor.grad_clip=1 # Gradient clipping # Advantage Estimation algorithm.adv_estimator=grpo # GRPO (Group Relative Policy Optimization) algorithm.kl_ctrl.kl_coef=0.00 # No KL penalty (RefPolicy disabled) # Training Schedule trainer.total_epochs=100 # 100 epochs trainer.save_freq=20 # Save checkpoint every 20 steps trainer.test_freq=4 # Validate every 4 steps trainer.val_before_train=True # Run validation before training ``` ### Computed Training Parameters **IMPORTANT: Terminology Clarification** There are THREE different types of "steps" in this system: 1. **Training Steps (global_steps)**: Main training loop iterations - This is what we mean by "1500 steps" - One training step = collect rollouts + update policy 2. **Environment Steps**: Physical actions executed in simulation - Maximum 200 per rollout for beat_block_hammer task - These are the actual robot control steps 3. **VLA Inference Steps**: Number of VLA model forward passes - 200 env steps ÷ 25 action_chunks_len = 8 VLA calls per rollout - Each VLA call generates 25 action chunks **Dataset Size** (from `rob_dataset.py:171-197`): ```python # For RoboTwin 2.0, loads pre-collected feasible seeds train_dataset_size = num_trials_per_task = 1000 task instances # Training dataloader train_batch_size = 64 * oversample_factor = 64 shuffle = True drop_last = True # Training steps per epoch (NOT environment steps!) steps_per_epoch = train_dataset_size // train_batch_size = 1000 // 64 = 15 training steps (last 40 samples dropped) ``` **Total Training Steps** (from `ray_trainer.py:329-334`): ```python total_training_steps = len(train_dataloader) * total_epochs = 15 * 100 = 1500 training steps (per config) ``` **⚠️ INCONSISTENCY ALERT**: The paper ([SimpleVLA-RL](https://arxiv.org/pdf/2509.09674), Figure 3) shows training curves up to **300 RL Training Steps**, not 1500! This means either: 1. **Paper uses fewer epochs**: `300 / 15 = 20 epochs` instead of 100 2. **Config is for extended training**: The provided config may be for experiments beyond the paper 3. **Partial results shown**: Paper may show early results from longer runs **Most likely**: The paper experiments used `trainer.total_epochs=20` (300 steps), while the provided config has `trainer.total_epochs=100` for extended training. --- ## Complete Training Loop ### High-Level Structure (from `ray_trainer.py:469-717`) ```python def fit(self): global_steps = 0 # ========== Initial Validation (Optional) ========== if val_before_train: val_metrics = self._validate(global_steps=0) # Runs 256 rollouts without training # ========== Main Training Loop ========== for epoch in range(total_epochs): # 100 epochs self.train_dataloader.start_new_epoch() while True: # Inner loop: process all batches in epoch # ------------------ Step 1: Rollout ------------------ # Collect batch_size * n_samples = 64 * 8 = 512 rollouts valid_batch = [] while len(valid_batch) < batch_size * n_samples: batch_dict = train_dataloader.get_next_batch() # Get 64 task instances # Generate 8 rollouts per task instance gen_batch_output = actor_rollout_wg.generate_sequences(batch_dict) # Verify success and compute raw scores scores_tensor = reward_fn.verify(gen_batch_output) # Filter by accuracy if enabled filtered_batch = self.filter(scores_tensor, gen_batch_output, n_samples=8) valid_batch.append(filtered_batch) # ------------------ Step 2: Reference Policy (Optional) ------------------ if use_reference_policy: ref_log_prob = ref_policy_wg.compute_ref_log_prob(valid_batch) valid_batch.union(ref_log_prob) # ------------------ Step 3: Reward Computation ------------------ reward_tensor_dict, reward_metrics = reward_fn(valid_batch) valid_batch['token_level_scores'] = reward_tensor_dict['all'] # Apply KL penalty if using reference policy valid_batch, kl_metrics = apply_kl_penalty(valid_batch, kl_ctrl) # ------------------ Step 4: Advantage Estimation ------------------ valid_batch = compute_advantage(valid_batch, gamma, lam, adv_estimator='grpo') # ------------------ Step 5: Actor Update (Backpropagation) ------------------ if global_steps >= critic_warmup: actor_output = actor_rollout_wg.update_actor(valid_batch) entropy_output = actor_rollout_wg.compute_entropy(valid_batch) # ------------------ Step 6: Validation ------------------ if (global_steps + 1) % test_freq == 0: # Every 4 steps val_metrics = self._validate(global_steps) # ------------------ Step 7: Checkpoint Saving ------------------ if (global_steps + 1) % save_freq == 0: # Every 20 steps actor_rollout_wg.save_checkpoint(checkpoint_path) global_steps += 1 if len(valid_batch) == 0: # Epoch exhausted break # ========== Final Validation ========== val_metrics = self._validate(global_steps=global_steps) ``` ### Training Summary | Metric | Value | Calculation | Notes | |--------|-------|-------------|-------| | **Epochs (Config File)** | 100 | `trainer.total_epochs=100` | Extended training | | **Epochs (Paper)** | **20** | Inferred from Figure 3 | **Actual paper experiments** | | **Training Steps per Epoch** | 15 | `1000 tasks // 64 batch_size` | Main loop iterations | | **Total Training Steps (Config)** | 1,500 | `15 × 100 epochs` | If using config as-is | | **Total Training Steps (Paper)** | **300** | `15 × 20 epochs` | **Shown in Figure 3** | | **Rollouts per Training Step** | 512 | `64 tasks × 8 samples` | - | | **Total Rollouts (Paper)** | **153,600** | `300 × 512 rollouts` | **For 300 steps** | | **Env Steps per Rollout** | ~50-200 | Varies by task success | Physical robot actions | | **VLA Calls per Rollout** | ~2-8 | `env_steps / 25` | Model forward passes | | **Total Env Steps (Paper)** | ~15M | `153.6k × 100 avg` | **For 300 steps** | | **Total VLA Inferences (Paper)** | ~600k | `153.6k × 4 avg` | **For 300 steps** | | **Validations (Paper)** | **75** | `300 // 4 test_freq` | Every 4 training steps | | **Checkpoints (Paper)** | **15** | `300 // 20 save_freq` | Every 20 training steps | --- ## Phase 1: Rollout ### Rollout Process (from `ray_trainer.py:513-556`) ```python # 1. Load task batch from dataset batch_dict = train_dataloader.get_next_batch() # Returns: { # 'task_suite_name': ['robotwin2_beat_block_hammer'] * 64, # 'task_id': [-1] * 64, # 'trial_id': [seed_1, seed_2, ..., seed_64], # From pre-collected feasible seeds # 'trial_seed': [seed_1, seed_2, ..., seed_64], # } # 2. Prepare generation batch gen_batch = DataProto.from_single_dict(batch_dict) gen_batch.select(batch_keys=['task_id', 'trial_id', 'trial_seed']) # 3. Expand to n_samples copies (for GRPO/RLOO) batch_lst = [[batch[i] for _ in range(n_samples)] for i in range(len(batch))] # Creates 64 × 8 = 512 copies # 4. Set generation config gen_batch.meta_info = { 'eos_token_id': tokenizer.eos_token_id, 'n_samples': 8, # Number of samples per task 'pad_token_id': tokenizer.pad_token_id, } # 5. Generate rollouts (distributed across 8 GPUs) gen_batch_output = actor_rollout_wg.generate_sequences(gen_batch) ``` ### Inside `generate_sequences` (from `rob_rollout.py:495-507`) ```python def generate_sequences(self, prompts): batch_size = prompts.batch.batch_size[0] # 512 # Use different batch sizes for training vs validation if prompts.meta_info.get('n_samples') is None: micro_batch_size = val_micro_batch_size # 8 (validation) else: micro_batch_size = config.micro_batch_size # 1 (training) num_chunks = max(batch_size // micro_batch_size, 1) # 512 // 1 = 512 chunks batch_prompts = prompts.chunk(chunks=num_chunks) # Process each chunk sequentially (but distributed across GPUs by Ray) output = [self._generate_minibatch(p) for p in batch_prompts] output = DataProto.concat(output) return output ``` ### Detailed Rollout Execution (from `rob_rollout.py:609-775`) For each minibatch (1 task instance with n_samples=8): ```python def _generate_minibatch_robotwin(prompts): # Extract task information batch_size = 8 # n_samples task_id = [-1] * 8 trial_id = [same_seed] * 8 # All 8 samples use same seed trial_seed = [same_seed] * 8 task_suite_name = ['robotwin2_beat_block_hammer'] * 8 # ========== 1. Initialize Environments (Parallel) ========== env_wrappers = [] for idx in range(8): wrapper = RobotwinEnvWrapper(task_name, trial_id[idx], trial_seed[idx], ...) env_wrappers.append(wrapper) # Initialize all 8 environments in parallel using ThreadPoolExecutor init_futures = [thread_pool.submit(w.initialize) for w in env_wrappers] wait_for_completion(init_futures) # ~5-15 seconds (SAPIEN scene creation) # ========== 2. Collect Initial Observations ========== inputs = [] task_descriptions = [] for wrapper in env_wrappers: obs = wrapper.get_obs() # Get RGB + proprioception instruction = wrapper.get_instruction() inputs.append(obs_to_input(obs)) task_descriptions.append(instruction) # ========== 3. Main Rollout Loop ========== max_steps = 200 # Maximum ENVIRONMENT steps for beat_block_hammer # This is NOT the same as training steps! # These are physical robot control steps step = 0 # Current environment step counter vla_history = [] while step < max_steps: active_indices = [i for i, w in enumerate(env_wrappers) if w.active] if len(active_indices) == 0: break # All environments done # --- 3.1 Process VLA Input --- vla_input = process_input(inputs, task_descriptions) # Converts images to tensors, tokenizes text, normalizes proprio # Transfers to GPU, pads sequences # Shape: { # 'input_ids': (8, seq_len), # 'pixel_values': (8, num_images*patches, vision_dim), # 'proprio': (8, 14), # 'attention_mask': (8, seq_len), # } # --- 3.2 VLA Inference (GPU) --- vla_output = _generate_one_step(vla_input) # Forward pass through OpenVLA model: # - Vision encoder: Extract image features # - LLM: Generate action tokens (14 tokens × 25 chunks = 350 tokens) # - Action decoder: Convert tokens to continuous actions # Output: actions shape (8, 25, 14) # - 25 action chunks # - 14 dimensions (7 DOF × 2 arms) # --- 3.3 Execute Actions (CPU, Parallel) --- step_futures = [] for idx in active_indices: future = thread_pool.submit(env_wrappers[idx].step, vla_output['action'][idx]) step_futures.append((idx, future)) # Wait for all environments to finish executing 25 action chunks new_inputs = [] for idx, future in step_futures: obs, done = future.result() # ~1500ms for 25 steps new_inputs.append(obs_to_input(obs)) if done: env_wrappers[idx].active = False # --- 3.4 Store Trajectory Data --- vla_history.append({ 'responses': vla_output['responses'], # (8, 25*14) action tokens 'input_ids': vla_output['input_ids'], # (8, seq_len) input tokens 'attention_mask': vla_output['attention_mask'], 'pixel_values': vla_output['pixel_values'], 'action': vla_output['action'], # (8, 25, 14) continuous actions 'proprio': vla_output['proprio'], # (8, 14) proprioception 'step': step }) inputs = new_inputs step += 25 # Advance by action_chunks_len (environment steps, not training steps!) # After this loop, step will be at most 200 (max_steps) # ========== 4. Environment Cleanup ========== cleanup_futures = [thread_pool.submit(w.close) for w in env_wrappers] wait_for_completion(cleanup_futures) torch.cuda.empty_cache() # ========== 5. Prepare Output Batch ========== output_batch = prepare_output_batch(vla_history, task_records, batch_size=8) # Concatenates all timesteps: shape (8, num_steps, ...) # Adds metadata: complete (bool), finish_step (int) return output_batch ``` ### Rollout Output Format ```python output_batch = { 'responses': Tensor(8, num_steps, 350), # Action tokens (25*14) 'input_ids': Tensor(8, num_steps, seq_len), # Input token IDs 'attention_mask': Tensor(8, num_steps, seq_len), 'pixel_values': Tensor(8, num_steps, patches, dim), 'proprio': Tensor(8, num_steps, 14), # Proprioception 'complete': Tensor(8), # Success flags 'finish_step': Tensor(8), # Number of steps executed } ``` ### Rollout Timing Breakdown For 1 task instance with 8 samples: | Phase | Duration | Notes | |-------|----------|-------| | Environment Init | 5-15s | SAPIEN scene creation (parallel) | | VLA Inference (×8 steps) | ~2.4s | 300ms × 8, GPU 100% | | Action Execution (×8 steps) | ~12s | 1500ms × 8, CPU-bound | | Environment Cleanup | 1-3s | Close scenes, save videos | | **Total** | **20-30s** | Per minibatch | For full training step (512 rollouts = 64 tasks × 8 samples): - With 8 GPUs processing in parallel: **~30-40 seconds** - Ray schedules 64 tasks across 8 workers dynamically --- ## Phase 2: Reward Computation ### 2.1 Verification (from `ray_trainer.py:559-570`) ```python # Called during rollout collection scores_tensor, reward_metrics, format_metrics, _ = reward_fn.verify(roll_batch) ``` ### Verification Logic (from `main_ppo.py:42-60`) ```python def verify(self, data): completes = data.batch['complete'].tolist() # Success flags from environments batch_size = data.batch['responses'].size(0) # 512 # Convert boolean success to float rewards score = [float(item) for item in completes] # [0.0, 1.0, 0.0, ...] # Format correctness (always 1.0 for VLA) format = [1.0] * batch_size # Store in batch data.batch['acc'] = torch.tensor(score, dtype=torch.float32) data.batch['format_correctness'] = torch.tensor(format, dtype=torch.float32) # Compute metrics reward_metrics = {'all': data.batch['acc'].mean().item()} # Average success rate format_metrics = {'all': 1.0} return score, reward_metrics, format_metrics, reward_format_metrics ``` ### 2.2 Filtering (from `ray_trainer.py:573-578`) ```python if config.data.filter_accuracy: # Filter rollouts by average success rate per task # Keeps tasks with success rate in [accuracy_lower_bound, accuracy_upper_bound] # Default: [0.1, 0.9] filtered_batch = self.filter(scores_tensor, roll_batch, n_samples=8) ``` ### Dynamic Sampling (Paper Section 3.3) **Key Insight from Paper**: This implements "Dynamic Sampling" from the paper, which ensures non-zero advantage estimates and stable gradients by filtering out groups where all trajectories either succeed or fail. **Mathematical Formulation** (Equation 10 from paper): ``` 0 < |{traj_i(a_i, s_i) | is_successful(traj_i(a_i, s_i))}| < G ``` Where G is the group size (n_samples = 8). **Implementation** (from `ray_trainer.py:755-816`): ```python def filter(reward_tensor, batch, n_samples=8): # Reshape to (num_tasks, n_samples) reward_matrix = reward_tensor.sum(-1).reshape(-1, n_samples) # (64, 8) # Compute average success rate per task group acc_tensor = torch.mean(reward_matrix, dim=-1) # (64,) # Example: [0.0, 0.25, 0.5, 0.75, 1.0, ...] # Dynamic Sampling: Filter by accuracy bounds # Keeps only groups with MIXED outcomes (not all success or all failure) acc_mask = (acc_tensor >= 0.1) & (acc_tensor <= 0.9) # 0.1 threshold: excludes all-failure groups (0.0) # 0.9 threshold: excludes all-success groups (1.0) # Expand mask to cover all samples in kept groups final_mask = acc_mask.repeat_interleave(n_samples) # (64*8,) # Apply mask filtered_batch = batch.slice(final_mask) return filtered_batch ``` **Dynamic Sampling Benefits** (from paper): 1. **Non-zero advantages**: Groups with all success/failure have zero variance, leading to zero advantages 2. **Stable gradients**: Mixed outcomes provide meaningful learning signal 3. **Curriculum learning**: Naturally focuses on tasks of appropriate difficulty 4. **Proven effective in LLM RL**: Based on recent LLM RL research (Yu et al., 2025; Cui et al., 2025a) **Filtering Effect**: - Original batch: 64 tasks × 8 samples = 512 rollouts - After Dynamic Sampling: ~30-50 tasks × 8 samples = ~240-400 rollouts (varies) - Excluded: ~20% all-failure groups + ~10% all-success groups - Remaining: ~70% mixed-outcome groups for effective training ### 2.3 Outcome Reward Modeling (Paper Section 3.2) **Key Insight from Paper**: SimpleVLA-RL uses **outcome-only rewards** (success/failure) without hand-crafted process rewards. This is inspired by LLM RL success (DeepSeek-R1) and addresses the scalability challenges of designing dense rewards for every task. ### 2.3.1 Reward Shaping (from `ray_trainer.py:631-647`) ```python def reward_fn(batch): # Initialize reward tensors reward_tensor = torch.zeros_like(batch['responses'], dtype=torch.float32) # (N, num_steps, 350) verifier_reward = torch.zeros_like(batch['responses'], dtype=torch.float32) # Reshape to (N, num_steps * 350) reward_tensor = reward_tensor.reshape((reward_tensor.shape[0], -1)) verifier_reward = verifier_reward.reshape((verifier_reward.shape[0], -1)) # Compute valid response length valid_response_length = batch['finish_step'] * action_token_len # finish_step × 14 # Place reward at the end of trajectory (sparse reward) verifier_score = batch['acc'].cpu().numpy().tolist() # [0.0, 1.0, 0.0, ...] for i in range(verifier_reward.shape[0]): verifier_reward[i, valid_response_length[i] - 1] += verifier_score[i] reward_tensor_dict = {'gt_scores': verifier_reward} # Apply reward coefficient if config.verifier.reward_coef != 0: reward_tensor += config.verifier.reward_coef * reward_tensor_dict['gt_scores'] reward_tensor_dict['all'] = reward_tensor return reward_tensor_dict, reward_metrics ``` **Reward Structure**: ``` Trajectory: [a_0, a_1, a_2, ..., a_T] Rewards: [0.0, 0.0, 0.0, ..., r_T] # Sparse reward at end ↑ Success (1.0) or Failure (0.0) ``` --- ## Phase 3: Advantage Estimation (GRPO) **Algorithm**: Group Relative Policy Optimization (GRPO) **Key Insight from Paper**: GRPO is specifically designed for outcome-based rewards in RL. Instead of traditional GAE which requires value functions and dense rewards, GRPO uses relative comparisons within groups of trajectories from the same task. ### 3.1 KL Penalty (Optional) (from `ray_trainer.py:642-647`) ```python batch, kl_metrics = apply_kl_penalty( batch, kl_ctrl=self.kl_ctrl, kl_penalty='kl', action_token_len=14, action_chunks_len=25 ) ``` ### KL Penalty Logic (from `ray_trainer.py:85-120`) ```python def apply_kl_penalty(data, kl_ctrl, kl_penalty='kl', action_token_len=14, action_chunks_len=25): responses = data.batch['responses'] # (N, num_steps, 350) # Compute trajectory length traj_length = responses.size(1) * action_chunks_len # num_steps × 25 token_level_scores = data.batch['token_level_scores'] # Raw rewards # Compute response mask (valid tokens) finish_step = data.batch['finish_step'] * action_token_len # finish_step × 14 steps = torch.arange(traj_length * action_token_len) steps_expanded = steps.unsqueeze(0).expand(batch_size, -1) response_mask = steps_expanded < finish_step.unsqueeze(1) # Compute KL divergence if reference policy is used if 'ref_log_prob' in data.batch.keys(): # KL = log π(a|s) - log π_ref(a|s) kld = old_log_probs - ref_log_prob # (N, response_length) kld = kld * response_mask beta = kl_ctrl.value # KL coefficient (adaptive or fixed) else: beta = 0 kld = torch.zeros_like(response_mask, dtype=torch.float32) # Apply KL penalty to rewards token_level_rewards = token_level_scores - beta * kld # Update KL controller current_kl = masked_mean(kld, mask=response_mask, axis=-1).mean().item() kl_ctrl.update(current_kl=current_kl, n_steps=batch_size) data.batch['token_level_rewards'] = token_level_rewards metrics = {'critic/kl': current_kl, 'critic/kl_coeff': beta} return data, metrics ``` ### 3.2 Advantage Computation (from `ray_trainer.py:649-655`) ```python batch = compute_advantage( batch, gamma=1.0, # config.algorithm.gamma (not shown in script) lam=0.95, # config.algorithm.lam (not shown in script) adv_estimator='grpo', # Group Relative Policy Optimization config=config ) ``` ### GRPO Advantage Estimation (from `ray_trainer.py:160-173`) ```python def compute_advantage(data, gamma, lam, adv_estimator='grpo', config): responses = data.batch['responses'] response_length = responses.size(1) * responses.size(2) # num_steps × 350 # Compute response mask finish_step = data.batch['finish_step'] * action_token_len steps = torch.arange(response_length) steps_expanded = steps.unsqueeze(0).expand(batch_size, -1) response_mask = steps_expanded < finish_step.unsqueeze(1) token_level_rewards = data.batch['token_level_rewards'] index = data.non_tensor_batch['uid'] # Unique ID per task # GRPO: Group samples by task, use mean as baseline advantages, returns = core_algos.compute_grpo_outcome_advantage( token_level_rewards=token_level_rewards, eos_mask=response_mask, index=index # Groups samples from same task ) data.batch['advantages'] = advantages data.batch['returns'] = returns return data ``` ### GRPO Algorithm (Paper Section 2.3 & `verl/trainer/ppo/core_algos.py`) **Paper Definition**: GRPO is an outcome-based advantage estimation method that groups trajectories from the same prompt/task and uses the group mean as a baseline. **Why GRPO for VLA?** 1. **No value function needed**: Unlike GAE, doesn't require training a separate critic 2. **Works with sparse rewards**: Designed for outcome-only rewards (success/failure) 3. **Relative learning**: Encourages better actions relative to other attempts at the same task 4. **Variance reduction**: Group mean provides better baseline than zero ```python def compute_grpo_outcome_advantage(token_level_rewards, eos_mask, index): """ Group Relative Policy Optimization (GRPO) Paper Formula (Section 2.3): A_i = R_i - (1/N) Σ_{j=1}^N R_j Where: - R_i: Total return for trajectory i - N: Number of samples per task group (n_samples = 8) - A_i: Advantage (how much better than group average) Process: 1. Compute total return for each sample: R_i = Σ rewards 2. Group samples by task (using unique ID) 3. Compute group baseline: b = mean(R_1, ..., R_N) 4. Compute advantages: A_i = R_i - b This encourages relative improvement within each task group. """ batch_size = token_level_rewards.size(0) # Compute total returns (sum of rewards over trajectory) returns = (token_level_rewards * eos_mask).sum(-1) # (batch_size,) # Group by task index (uid) unique_indices = torch.unique(index) advantages = torch.zeros_like(returns) for idx in unique_indices: mask = (index == idx) # All 8 samples from same task group_returns = returns[mask] # N=8 samples # Compute group baseline (mean return across 8 samples) baseline = group_returns.mean() # Compute relative advantages (how much better than average) group_advantages = group_returns - baseline advantages[mask] = group_advantages # Broadcast advantages to all timesteps advantages_broadcast = advantages.unsqueeze(-1).expand_as(token_level_rewards) advantages_broadcast = advantages_broadcast * eos_mask returns_broadcast = returns.unsqueeze(-1).expand_as(token_level_rewards) returns_broadcast = returns_broadcast * eos_mask return advantages_broadcast, returns_broadcast ``` **GRPO Example**: ``` Task 1 (8 samples): - Returns: [0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 1.0, 0.0] - Baseline: mean = 0.5 - Advantages: [-0.5, 0.5, -0.5, 0.5, -0.5, 0.5, 0.5, -0.5] Task 2 (8 samples): - Returns: [0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 0.0, 0.0] - Baseline: mean = 0.25 - Advantages: [-0.25, -0.25, -0.25, -0.25, 0.75, 0.75, -0.25, -0.25] ``` --- ## Phase 4: Actor Update (PPO with Clip Higher) **Algorithm**: Proximal Policy Optimization (PPO) with modified clipping range **Key Insight from Paper** (Section 3.3 - "Clipping Higher"): - Standard PPO clips ratio to `[1-ε, 1+ε]` typically `[0.8, 1.2]` - SimpleVLA-RL increases upper bound to 1.28: `[0.8, 1.28]` - **Rationale**: Higher upper bound allows more exploration by not restricting probability increase of low-probability actions - **Inspired by**: DAPO (Yu et al., 2025) for LLM RL - **Result**: Shown in Figure 3(b) - ~10% improvement over baseline by step 300 ### 4.1 Update Trigger (from `ray_trainer.py:660-671`) ```python if global_steps >= critic_warmup: # Default: critic_warmup = 0 # Update actor using PPO batch.meta_info['is_filtered'] = True batch.meta_info['train_mode'] = False actor_output = actor_rollout_wg.update_actor(batch) # Compute entropy for exploration entropy_output = actor_rollout_wg.compute_entropy(data=batch) ``` ### 4.2 PPO Update (in ActorRolloutRefWorker) ```python def update_actor(self, batch): """ Proximal Policy Optimization (PPO) update Batch contains: - responses: (N, num_steps, 350) action tokens - old_log_probs: log π_old(a|s) - advantages: A(s, a) - returns: R(s, a) - attention_mask: valid token mask """ # ========== PPO Configuration ========== ppo_mini_batch_size = 128 # Mini-batch size for PPO updates ppo_micro_batch_size = 8 # Micro-batch size (= num_gpus) clip_ratio_low = 0.2 # PPO clip lower bound (1 - 0.2 = 0.8) clip_ratio_high = 0.28 # PPO clip upper bound (1 + 0.28 = 1.28) ← CLIP HIGHER! num_ppo_epochs = 1 # Number of PPO epochs (default: 1) # Note: clip_ratio_high = 0.28 means clipping at 1.28 (not 1.2) # This is the "Clip Higher" enhancement from the paper # ========== Prepare Data ========== batch_size = len(batch) # After filtering: ~240-400 rollouts # Shuffle data for mini-batch SGD indices = torch.randperm(batch_size) shuffled_batch = batch.slice(indices) # ========== Mini-Batch Loop ========== num_mini_batches = batch_size // ppo_mini_batch_size # ~2-3 mini-batches metrics = defaultdict(list) for mini_batch_idx in range(num_mini_batches): start = mini_batch_idx * ppo_mini_batch_size end = start + ppo_mini_batch_size mini_batch = shuffled_batch.slice(range(start, end)) # ========== Micro-Batch Loop (for memory efficiency) ========== num_micro_batches = ppo_mini_batch_size // ppo_micro_batch_size # 128 // 8 = 16 accumulated_loss = 0.0 for micro_batch_idx in range(num_micro_batches): micro_start = micro_batch_idx * ppo_micro_batch_size micro_end = micro_start + ppo_micro_batch_size micro_batch = mini_batch.slice(range(micro_start, micro_end)) # --- Forward Pass --- with torch.autocast(device_type='cuda', dtype=torch.bfloat16): output = model( input_ids=micro_batch['input_ids'], pixel_values=micro_batch['pixel_values'], proprio=micro_batch['proprio'], attention_mask=micro_batch['attention_mask'], labels=micro_batch['responses'], # Teacher forcing ) logits = output.logits # (micro_batch_size, seq_len, vocab_size) # Compute log probabilities log_probs = F.log_softmax(logits, dim=-1) # Gather log probs for actual actions action_log_probs = torch.gather( log_probs, dim=-1, index=micro_batch['responses'].unsqueeze(-1) ).squeeze(-1) # (micro_batch_size, seq_len) # Get old log probs from rollout old_log_probs = micro_batch['old_log_probs'] # Compute probability ratio: π(a|s) / π_old(a|s) ratio = torch.exp(action_log_probs - old_log_probs) # Get advantages advantages = micro_batch['advantages'] # Normalize advantages (per micro-batch) advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-8) # --- PPO Clipped Objective with "Clip Higher" --- # Standard PPO: L_CLIP = min(ratio * A, clip(ratio, 0.8, 1.2) * A) # SimpleVLA-RL: L_CLIP = min(ratio * A, clip(ratio, 0.8, 1.28) * A) # The 1.28 upper bound allows more exploration! clipped_ratio = torch.clamp(ratio, 1 - clip_ratio_low, 1 + clip_ratio_high) # = torch.clamp(ratio, 0.8, 1.28) loss_unclipped = ratio * advantages loss_clipped = clipped_ratio * advantages policy_loss = -torch.min(loss_unclipped, loss_clipped) # Apply mask to only valid tokens response_mask = micro_batch['attention_mask'][:, -action_log_probs.size(1):] policy_loss = (policy_loss * response_mask).sum() / response_mask.sum() # Value loss (if using critic) # value_loss = F.mse_loss(values, returns) # Total loss loss = policy_loss # + value_coef * value_loss # Normalize by num_micro_batches for gradient accumulation loss = loss / num_micro_batches # --- Backward Pass --- loss.backward() accumulated_loss += loss.item() # Store metrics metrics['actor/policy_loss'].append(policy_loss.item()) metrics['actor/ratio_mean'].append(ratio.mean().item()) metrics['actor/ratio_max'].append(ratio.max().item()) metrics['actor/ratio_min'].append(ratio.min().item()) metrics['actor/clip_fraction'].append( ((ratio < 1 - clip_ratio_low) | (ratio > 1 + clip_ratio_high)).float().mean().item() ) # --- Gradient Clipping and Optimizer Step (after accumulating micro-batches) --- torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step() optimizer.zero_grad() # Average metrics for key in metrics: metrics[key] = np.mean(metrics[key]) return DataProto(meta_info={'metrics': metrics}) ``` ### PPO Update Summary | Parameter | Value | Notes | |-----------|-------|-------| | **Batch Size** | ~240-400 | After accuracy filtering | | **Mini-Batch Size** | 128 | For PPO updates | | **Micro-Batch Size** | 8 | For memory efficiency | | **Num Mini-Batches** | ~2-3 | 240-400 // 128 | | **Num Micro-Batches** | 16 | 128 // 8 | | **Clip Ratio** | [0.2, 0.28] | PPO clipping range | | **Learning Rate** | 5e-6 | Adam optimizer | | **Gradient Clipping** | 1.0 | Max norm | **Timing**: - Forward + Backward per micro-batch: ~500ms - Total per mini-batch: 16 × 500ms = 8s - Total per step: 2-3 × 8s = ~16-24s ### 4.3 Entropy Computation (from `ray_trainer.py:666`) ```python entropy_output = actor_rollout_wg.compute_entropy(data=batch) ``` **Purpose**: Measure policy entropy for exploration monitoring ```python def compute_entropy(data): # Forward pass to get logits logits = model(...) # Compute entropy: H = -Σ p(a) log p(a) probs = F.softmax(logits, dim=-1) log_probs = F.log_softmax(logits, dim=-1) entropy = -(probs * log_probs).sum(-1) # (batch_size, seq_len) # Average over valid tokens entropy_mean = (entropy * response_mask).sum() / response_mask.sum() return DataProto(meta_info={'metrics': {'actor/entropy': entropy_mean.item()}}) ``` --- ## Phase 5: Validation ### 5.1 Validation Trigger (from `ray_trainer.py:673-679`) ```python if (global_steps + 1) % test_freq == 0: # Every 4 steps val_metrics = self._validate(global_steps=global_steps+1) val_metrics = {f'val/{key}': val for key, val in val_metrics.items()} logger.log(data=val_metrics, step=global_steps) ``` ### 5.2 Validation Process (from `ray_trainer.py:336-391`) ```python def _validate(self, global_steps=0): reward_tensor_lst = [] data_source_lst = [] # Loop over validation dataloader for test_data in val_dataloader: # 256 / 256 = 1 batch test_batch = DataProto.from_single_dict(test_data) # Set validation config test_batch.meta_info = { 'eos_token_id': tokenizer.eos_token_id, 'pad_token_id': tokenizer.pad_token_id, 'recompute_log_prob': False, 'do_sample': False, # Greedy sampling for validation 'validate': True, # Validation mode flag 'global_steps': global_steps # For video naming } # Generate rollouts (no n_samples, single rollout per task) test_output_gen_batch = actor_rollout_wg.generate_sequences(test_batch) # Uses val_micro_batch_size = 8 (larger than training micro_batch_size = 1) test_batch = test_batch.union(test_output_gen_batch) # Evaluate using reward function verifier_score, reward_metrics, format_metrics, _ = val_reward_fn.verify(test_batch) reward_tensor = torch.tensor(verifier_score, dtype=torch.float32).unsqueeze(-1) # Log metrics for k, v in reward_metrics.items(): metric_dict['test_reward/' + k] = v reward_tensor_lst.append(reward_tensor) data_source_lst.append(test_batch.non_tensor_batch['data_source']) # Aggregate results reward_tensor = torch.cat(reward_tensor_lst, dim=0).sum(-1).cpu() # (256,) data_sources = np.concatenate(data_source_lst, axis=0) # Compute metrics by data source data_source_reward = {} for i in range(reward_tensor.shape[0]): data_source = data_sources[i] if data_source not in data_source_reward: data_source_reward[data_source] = [] data_source_reward[data_source].append(reward_tensor[i].item()) metric_dict = {} for data_source, rewards in data_source_reward.items(): metric_dict[f'test_score/{data_source}'] = np.mean(rewards) metric_dict[f'test_score/all'] = reward_tensor.mean().item() return metric_dict ``` ### Validation Dataset (from `rob_dataset.py:136-209`) ```python # For validation val_dataset = Robotwin_Dataset( task_suite_name='robotwin2_beat_block_hammer', num_trials_per_task=128, # Overridden for validation train_val='valid' ) # Loads two types of seeds: # 1. IID seeds: From training distribution (128 seeds) # 2. OOD seeds: From eval distribution (128 seeds) # Total: 256 task instances # Data source labels: # - 'robotwin2_beat_block_hammer_train_iid' # - 'robotwin2_beat_block_hammer_eval_ood' ``` ### Validation Metrics ```python { 'val/test_score/robotwin2_beat_block_hammer_train_iid': 0.65, # IID success rate 'val/test_score/robotwin2_beat_block_hammer_eval_ood': 0.42, # OOD success rate 'val/test_score/all': 0.535, # Overall success rate } ``` **Validation Timing**: - 256 rollouts with val_micro_batch_size=8 - Distributed across 8 GPUs: 256 / 8 = 32 rollouts per GPU - Per rollout: ~20-30s - Total: ~30-40s (parallel execution) --- ## Data Flow and Synchronization ### Overall Data Flow ``` ┌──────────────────────────────────────────────────────────────────┐ │ Training Step i │ └──────────────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 1. Dataset Sampling │ │ - Sample 64 task instances from dataset │ │ - Each with unique feasible seed │ │ - Expand to 64 × 8 = 512 copies for n_samples │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 2. Rollout (Distributed across 8 GPUs via Ray) │ │ - Each GPU processes ~64 rollouts (8 tasks × 8) │ │ - Initialize environments │ │ - VLA inference + action execution loop │ │ - Collect trajectory data │ │ - Output: (512, num_steps, ...) │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 3. Verification & Filtering (Main Process) │ │ - Compute success flags │ │ - Filter by accuracy: keep 10%-90% success rate │ │ - Output: (~300, num_steps, ...) │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 4. Reference Policy (Optional, Distributed) │ │ - Compute log π_ref(a|s) for KL penalty │ │ - Only if kl_coef > 0 │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 5. Reward Computation (Main Process) │ │ - Apply reward shaping (sparse reward at end) │ │ - Apply KL penalty if using ref policy │ │ - Output: token_level_rewards │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 6. Advantage Estimation (Main Process) │ │ - GRPO: Group samples by task │ │ - Compute advantages = returns - group_mean │ │ - Output: advantages, returns │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 7. Actor Update (Distributed across 8 GPUs) │ │ - PPO mini-batch loop (2-3 mini-batches) │ │ - Micro-batch loop (16 micro-batches per mini-batch)│ │ - Forward + backward + gradient accumulation │ │ - Gradient clipping + optimizer step │ │ - Output: actor_metrics │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 8. Entropy Computation (Distributed) │ │ - Measure policy entropy for exploration monitoring │ │ - Output: entropy_metrics │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 9. Validation (Every 4 steps, Distributed) │ │ - 256 rollouts (greedy sampling) │ │ - Compute success rates (IID and OOD) │ │ - Output: val_metrics │ └──────────────────────────────────────────────────────────┘ │ ↓ ┌──────────────────────────────────────────────────────────┐ │ 10. Checkpoint Saving (Every 20 steps) │ │ - Save actor model weights │ │ - Save critic model weights (if GAE) │ │ - Save optimizer state │ └──────────────────────────────────────────────────────────┘ │ ↓ global_steps += 1 ``` ### Ray Distributed Execution ```python # Worker allocation (8 GPUs) actor_rollout_wg.world_size = 8 # 8 workers, 1 per GPU actor_rollout_wg.local_size = 8 # All on same node # When calling actor_rollout_wg.generate_sequences(batch) # Ray automatically distributes workload: batch_size = 512 # 64 tasks × 8 samples world_size = 8 micro_batch_size = 1 # Each GPU processes: local_batch_size = batch_size // world_size = 512 // 8 = 64 rollouts # Ray scheduling: # GPU 0: processes rollouts [0:64] # GPU 1: processes rollouts [64:128] # GPU 2: processes rollouts [128:192] # ... # GPU 7: processes rollouts [448:512] # All GPUs run in parallel, synchronization at end ``` ### FSDP Model Sharding ```bash actor_rollout_ref.actor.fsdp_config.param_offload=False actor_rollout_ref.actor.fsdp_config.grad_offload=True actor_rollout_ref.actor.fsdp_config.optimizer_offload=True ``` **FSDP Strategy**: - **Param Offload**: Disabled → Parameters stay on GPU - **Grad Offload**: Enabled → Gradients offloaded to CPU after backward - **Optimizer Offload**: Enabled → Optimizer states kept on CPU **Model Sharding**: ``` OpenVLA-7B Model: ├── Vision Backbone (400M-700M params) → Sharded across 8 GPUs ├── Projector (8M params) → Sharded across 8 GPUs ├── LLM (7B params) → Sharded across 8 GPUs └── Total: ~7.5B params → ~940MB per GPU (in bfloat16) ``` --- ## Performance Analysis ### Training Step Timing Breakdown **ORIGINAL ESTIMATE (WRONG!)**: | Phase | Estimated Duration | GPU Util | Notes | |-------|----------|----------|-------| | **1. Rollout** | 30-40s | 15% | Bottleneck: CPU physics simulation | | **2. Verification & Filtering** | 1-2s | 0% | CPU operations | | **3. Reference Policy** | 2-3s | 80% | If kl_coef > 0 (disabled in config) | | **4. Reward Computation** | <1s | 0% | CPU tensor operations | | **5. Advantage Estimation** | <1s | 0% | CPU tensor operations | | **6. Actor Update** | 16-24s | 90% | PPO mini-batch loop | | **7. Entropy Computation** | 1-2s | 80% | Additional forward pass | | **8. Logging** | <1s | 0% | Write metrics to WandB | | **Total per step** | **50-70s** | **30-40%** | Average GPU utilization | **ACTUAL REALITY (from user logs)**: | Phase | Actual Duration | GPU Util | Notes | |-------|----------|----------|-------| | **1. Rollout** | 1070-1100s (~18 min) | 15-20% | Collects ~896 rollouts (to get 512 after filtering) | | **2. Verification & Filtering** | 4-5s | 0% | Filters 896 → 512 rollouts | | **3. Reference Policy** | 0s | 0% | Disabled (kl_coef=0) | | **4. Reward Computation** | <1s | 0% | CPU tensor operations | | **5. Advantage Estimation** | 0.14-0.20s | 0% | GRPO computation | | **6. Actor Update (PPO)** | 140-145s (~2.4 min) | 90% | 4 mini-batches, dataloader_length=4 | | **7. Entropy Computation** | <1s | 80% | Included in actor update | | **8. Logging** | <1s | 0% | Write metrics to WandB | | **Total per training step** | **1220-1300s (~20-22 min)** | **20%** | Average GPU utilization | **Additional Operations**: - **Validation** (every 4 training steps): +30-40s (256 rollouts) - **Checkpoint Saving** (every 20 training steps): +10-20s **Why Rollout is So Slow**: - Each rollout takes ~1.2 seconds (includes env init, 8 VLA calls, 200 env steps, cleanup) - Need to collect 896 rollouts to get 512 valid ones after filtering (75% efficiency) - 896 × 1.2s = 1075 seconds ≈ 18 minutes ### Full Training Timeline **PAPER EXPERIMENTS (300 steps, 20 epochs)**: ``` Total Training Duration: = 300 training_steps × 1230 seconds/step = 369,000 seconds = 102.5 hours ≈ 4.3 days (with 8 GPUs running continuously) ``` **IF USING FULL CONFIG (1500 steps, 100 epochs)**: ``` Total Training Duration: = 1500 training_steps × 1230 seconds/step = 1,845,000 seconds = 512.5 hours ≈ 21.4 days (with 8 GPUs running continuously) ``` ### Rollout Statistics **For Paper (300 steps)**: | Metric | Value | |--------|-------| | **Total Rollouts** | 153,600 | | **Successful Rollouts** | ~60,000-80,000 (depends on learning) | | **Failed Rollouts** | ~70,000-90,000 | | **Filtered Out** | ~40,000 (accuracy filtering) | | **Used for Training** | ~110,000-120,000 | **For Full Config (1500 steps)**: | Metric | Value | |--------|-------| | **Total Rollouts** | 768,000 | | **Successful Rollouts** | ~300,000-400,000 (depends on learning) | | **Failed Rollouts** | ~400,000-500,000 | | **Filtered Out** | ~200,000 (accuracy filtering) | | **Used for Training** | ~500,000-600,000 | ### GPU Memory Usage ``` Per GPU (Rollout Phase): ├── Model Parameters (sharded): ~940MB ├── Activation Memory: ~1.5GB ├── Environment Buffers: ~500MB ├── KV Cache: ~2GB └── Total: ~5GB / 40GB (12.5%) Per GPU (Training Phase): ├── Model Parameters (sharded): ~940MB ├── Gradients: ~940MB ├── Activation Memory: ~3GB ├── Optimizer States (offloaded to CPU): 0MB └── Total: ~5GB / 40GB (12.5%) Peak Memory: ~6GB / 40GB (15%) ``` --- --- ## Key Paper Insights ### What Makes SimpleVLA-RL Work? Based on [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://arxiv.org/pdf/2509.09674), the framework achieves state-of-the-art performance through **three key enhancements**: #### 1. **Dynamic Sampling** (Section 3.3, Equation 10) **Problem**: Groups with all success or all failure have zero advantage variance, leading to unstable gradients. **Solution**: During rollout collection, exclude groups where all trajectories either succeed or fail: ``` 0 < |{traj_i | is_successful(traj_i)}| < G ``` **Implementation**: `data.accuracy_lower_bound=0.1` and `data.accuracy_upper_bound=0.9` **Result**: ~15% improvement over baseline (Figure 3a) **Benefit**: - Non-zero advantages ensure meaningful learning signal - Natural curriculum learning (focuses on appropriate difficulty) - Proven effective in LLM RL (Yu et al., 2025; Cui et al., 2025a) --- #### 2. **Clipping Higher** (Section 3.3) **Problem**: Standard PPO's symmetric clipping `[0.8, 1.2]` restricts exploration by limiting probability increases. **Solution**: Modify clipping range to `[0.8, 1.28]` (asymmetric): ```python clip_ratio_high = 0.28 # Upper bound at 1.28 (not 1.2) clip_ratio_low = 0.2 # Lower bound at 0.8 ``` **Implementation**: `actor_rollout_ref.actor.clip_ratio_high=0.28` **Result**: ~10% improvement over baseline (Figure 3b) **Benefit**: - Allows larger probability increases for low-probability actions - Encourages exploration of new action patterns - Inspired by DAPO (Yu et al., 2025) for LLM RL --- #### 3. **Higher Rollout Temperature** (Section 3.3) **Problem**: Low temperature (1.0) during rollout leads to deterministic, repetitive trajectories. **Solution**: Increase sampling temperature from 1.0 to 1.6 during rollout phase: ```bash actor_rollout_ref.rollout.temperature=1.6 ``` **Implementation**: Used during `generate_action_verl()` in rollout, not during training **Result**: ~15% improvement over baseline (Figure 3c) **Benefit**: - Generates diverse trajectories for better exploration - Critical for discovering new successful strategies - Widely used in recent LLM RL work (Liu et al., 2025c; An et al., 2025) --- ### Novel Phenomenon: "Pushcut" **Discovery** (Section 6.1): During RL training, the policy discovered a previously unseen action pattern called "pushcut": - SFT model only learns to grasp-and-lift objects - After RL, model learns to **push** objects toward target while moving - This "sliding" strategy is more efficient than pure lifting - **Not seen in any training demonstrations!** **Significance**: - Shows RL enables emergence of novel, optimal strategies - Goes beyond imitation of human demonstrations - Similar to AlphaGo discovering unconventional moves --- ### Performance Results (from Paper) **LIBERO Benchmark** (Figure 3): - Baseline SFT: ~60% success rate (Long) - With all enhancements: ~90% success rate (Long) at 300 steps - **+30 percentage points improvement** **RoboTwin 2.0 Benchmark** (Table 1): - Outperforms π₀ (previous SOTA) on multiple tasks - Example: beat_block_hammer task success rate improvement **Real-World Experiments** (Section 5.3): - Successfully generalizes to real robot hardware - Demonstrates sim-to-real transfer capability --- ### Why RL Works for VLA (Paper's Main Argument) **Parallel to LLM RL Success**: 1. **DeepSeek-R1** showed RL dramatically improves step-by-step reasoning 2. **VLA = step-by-step action planning** (analogous to CoT reasoning) 3. RL enables learning from **outcome rewards only** (no hand-crafted process rewards) **Key Advantages**: 1. **Addresses Data Scarcity**: No need for massive human demonstration data 2. **Improves Generalization**: RL naturally explores beyond training distribution 3. **Discovers Novel Patterns**: Like "pushcut" - goes beyond imitation 4. **Scalable**: Only requires environment simulator, not human operators --- ## Summary ### Key Characteristics of veRL Training 1. **GRPO + PPO Pipeline**: Separates advantage estimation (GRPO) from policy update (PPO) - GRPO works with sparse outcome rewards - PPO provides stable policy updates 2. **Three Critical Enhancements**: Dynamic Sampling + Clip Higher + Higher Temperature - Each provides ~10-15% improvement independently - Combined effect: ~30% improvement over baseline SFT 3. **Outcome-Only Rewards**: No hand-crafted process rewards - Success/failure at trajectory end - Inspired by LLM RL (DeepSeek-R1) - Scalable to new tasks 4. **Exploration-Focused**: Multiple mechanisms encourage exploration - Dynamic sampling focuses on learnable tasks - Clip higher allows larger policy updates - High temperature generates diverse rollouts 5. **300-Step Training**: Paper shows convergence at 300 training steps (~4.3 days) - Much faster than expected for robotic RL - Config has 1500 steps for extended experiments ### System Bottlenecks 1. **Rollout is 87.8% of time**: Physics simulation dominates - Each training step: ~18 minutes rollout, ~2.4 minutes training - Need ~896 rollouts to get 512 valid (after filtering) 2. **Dynamic Sampling reduces effective batch size**: ~30% overhead - Required for stable training - Worthwhile tradeoff for better learning 3. **Low GPU utilization during rollout**: ~15-20% - VLA inference only 300ms vs 1500ms physics - Fundamental limitation of CPU-based simulators ### Recommendations for Improvement **Based on Paper**: 1. **Tune the three key hyperparameters**: - `data.accuracy_lower_bound` and `data.accuracy_upper_bound` (Dynamic Sampling) - `actor_rollout_ref.actor.clip_ratio_high` (Clip Higher) - `actor_rollout_ref.rollout.temperature` (Exploration) 2. **Task-Specific Tuning**: - Harder tasks may need wider accuracy bounds (0.05-0.95) - Easier tasks may need higher temperature (1.8-2.0) **Based on Implementation Analysis**: 1. **Increase Rollout Throughput**: - Use GPU-accelerated simulator (Isaac Gym/Sim) - Reduce action chunk length (25 → 10-15) - Pipeline CPU/GPU work 2. **Optimize Training Efficiency**: - Increase n_samples for better GRPO baselines - Use multiple PPO epochs if overfitting is not a concern - Adjust filtering to keep more data if collection is expensive 3. **Monitor Key Metrics**: - `actor/pg_clipfrac`: Should be 2-5% (too high = policy changing too fast) - `critic/advantages/mean`: Should be near 0 (GRPO working correctly) - Dynamic sampling efficiency: Aim for 70-80% retention rate