liqiang
b119135836
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
2440 lines
85 KiB
Markdown
2440 lines
85 KiB
Markdown
## English
|
||
|
||
# Experiment 8-15 reproduction anchor
|
||
|
||
This directory is the book-owned training guide. The executable GAIA/MCP environment and training integration are in [`bojieli/AWorld`](https://github.com/bojieli/AWorld) at `chapter8/AWorld`, verified at commit `a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec`. The veRL backend is [`bojieli/verl`](https://github.com/bojieli/verl) at `chapter8/verl`, verified at `1593fc3a8cf894debdc3dece2a23ed739c282789`.
|
||
|
||
```bash
|
||
git clone https://github.com/bojieli/AWorld.git chapter8/AWorld
|
||
git -C chapter8/AWorld checkout --detach a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec
|
||
git clone https://github.com/bojieli/verl.git chapter8/verl
|
||
git -C chapter8/verl checkout --detach 1593fc3a8cf894debdc3dece2a23ed739c282789
|
||
|
||
cd chapter8/AWorld/env
|
||
bash run-local.sh
|
||
# After dataset/reward configuration, in a second shell:
|
||
cd chapter8/AWorld/train/examples/train_gaia_with_aworld_verl
|
||
bash run.sh
|
||
```
|
||
|
||
This is an open-ended training experiment with no manuscript baseline. The commands were inspected, not run, during this documentation audit.
|
||
|
||
<div align="center">
|
||
|
||
# AWorld Train
|
||
|
||
*A "Learning from Practice" Training Framework for Agentic AI*
|
||
|
||
[![License: MIT][license-image]][license-url]
|
||
[](https://arxiv.org/abs/2508.20404)
|
||
|
||
</div>
|
||
|
||
---
|
||
|
||
## Table of Contents
|
||
|
||
- [Introduction](#introduction)
|
||
- [Important Note on This Educational Experiment](#important-note-on-this-educational-experiment)
|
||
- [Core Features](#core-features)
|
||
- [GAIA Environment Tool Ecosystem](#gaia-environment-tool-ecosystem)
|
||
- [Web Interaction Tools](#-web-interaction-tools-3-servers-27-documented-tools)
|
||
- [Document Processing Tools](#-document-processing-tools-5-servers-9-documented-tools)
|
||
- [Multimedia Processing Tools](#-multimedia-processing-tools-3-servers-12-tools)
|
||
- [Intelligent Reasoning Tools](#-intelligent-reasoning-tools-3-servers-6-tools)
|
||
- [Code Execution Tools](#-code-execution-tools-3-servers-15-documented-tools)
|
||
- [File System Tools](#-file-system-tools-1-server-14-tools)
|
||
- [Excel Processing Tools](#-excel-processing-tools-1-server-29-tools)
|
||
- [Knowledge Retrieval Tools](#-knowledge-retrieval-tools-3-servers-15-documented-tools)
|
||
- [Tool Statistics Summary](#tool-statistics-summary)
|
||
- [Core Architecture](#core-architecture)
|
||
- [Quick Start](#quick-start)
|
||
- [Install the Training Framework](#install-the-training-framework)
|
||
- [Configure the GAIA Environment](#configure-the-gaia-environment)
|
||
- [Build a Custom Agent](#build-a-custom-agent)
|
||
- [Prepare for Training](#prepare-for-training)
|
||
- [Start Training](#start-training)
|
||
- [Latest Optimizations](#latest-optimizations)
|
||
- [Troubleshooting](#troubleshooting)
|
||
- [Performance Benchmarks](#performance-benchmarks)
|
||
- [Advanced Topics](#advanced-topics)
|
||
- [Citation](#citation)
|
||
- [Community & Support](#community--support)
|
||
|
||
---
|
||
|
||
## Introduction
|
||
|
||
AWorld Train is an open-source training framework implementing the **"Learning from Practice"** paradigm, specifically designed for Agentic AI. According to the [AWorld paper](https://arxiv.org/abs/2508.20404), building high-performance Agent systems requires three core elements:
|
||
|
||
1. **Algorithm**: Learning mechanisms that enable the Agent to adapt and improve from environment interactions
|
||
2. **Environment**: Complex interaction scenarios providing rich feedback and diverse challenges
|
||
3. **Priors**: The foundational capabilities of current LLMs in areas like reasoning, mathematics, and vision
|
||
|
||
AWorld Train addresses the core bottleneck of traditional methods—**inefficient experience generation**—through a distributed architecture. On the GAIA benchmark, we achieved a **14.6x** speedup in data collection, making large-scale reinforcement learning training feasible.
|
||
|
||
### ⚠️ Important Note on This Educational Experiment
|
||
|
||
**GAIA (General AI Assistants Benchmark)** is one of the most challenging benchmarks for evaluating Agent capabilities and a competitive arena for SOTA (State-of-the-Art) Agent systems. According to the [paper](https://arxiv.org/abs/2508.20404):
|
||
|
||
- **Data Scarcity**: The GAIA validation set contains only **165 questions**, and the test set has approximately **300 questions**, far fewer than the data volume typically required for RL training
|
||
- **Computational Resource Requirements**: The Qwen3-32B-AWorld model from the paper required training on **2 clusters of 8x A100 GPUs** for several days to achieve 32.23% performance, which is still far from SOTA performance (over 80%)
|
||
- **Task Complexity**: GAIA questions involve multi-modal understanding, multi-step reasoning, tool chain calls, etc., requiring an average of 10-20 interaction rounds to complete
|
||
|
||
Therefore, this project adopts an education-friendly configuration, using Qwen3-4B-Thinking-2507 as the base model for faster training.
|
||
|
||
**The goals of this project are:**
|
||
- ✅ Demonstrate the complete "Learning from Practice" training pipeline
|
||
- ✅ Understand the Agent-Environment interaction mechanism
|
||
- ✅ Practice applying RL algorithms (PPO/GRPO) in Agent training
|
||
|
||
### Core Features
|
||
|
||
- ⚡ **Efficient Concurrency**: Distributed task execution, 14.6x data collection acceleration
|
||
- 🔌 **Framework Agnostic**: Supports mainstream RL frameworks like VeRL, OpenRLHF, AReaL, SWIFT
|
||
- 🛠️ **Tool Ecosystem**: Built-in 26 MCP servers providing **126 tool functions**, covering search, browser, code execution, multi-modal processing, etc.
|
||
- 📊 **Long Context**: Supports 131K token contexts for handling complex multi-turn interactions
|
||
- 🎯 **SOTA Performance**: Qwen3-32B-AWorld achieves 32.23% pass@1 on the GAIA test set
|
||
|
||
---
|
||
|
||
## GAIA Environment Tool Ecosystem
|
||
|
||
According to the [paper](https://arxiv.org/abs/2508.20404) and MCP Server implementation, AWorld provides comprehensive tool support for GAIA tasks, totaling **26 MCP servers** and **126 tool functions**. Below is the complete list of tools by category:
|
||
|
||
### 🌐 Web Interaction Tools (3 Servers, 27 Documented Tools)
|
||
|
||
#### 1. Google Search Server (`googlesearch-server`)
|
||
- `search_google`: Perform web searches using the Google Custom Search API
|
||
- `get_search_capabilities`: Retrieve search service capability information
|
||
|
||
**Typical Applications**: Querying real-time information, fact-checking, starting point for multi-hop reasoning
|
||
|
||
#### 2. Browser Use Server (`browser-server`)
|
||
- `browser_use`: LLM-based intelligent browser automation (using the browser-use library)
|
||
- `get_browser_capabilities`: Retrieve browser automation capabilities
|
||
|
||
**Features**:
|
||
- Automatic handling of bot detection and CAPTCHAs
|
||
- Supports form filling, file downloads, content extraction
|
||
- Integrates visual understanding and memory functions
|
||
|
||
#### 3. Playwright Server (`ms-playwright`)
|
||
Provides **23 fine-grained browser control tools**:
|
||
- **Navigation**: `browser_navigate`, `browser_navigate_back`
|
||
- **Interaction**: `browser_click`, `browser_type`, `browser_hover`, `browser_drag`, `browser_select_option`
|
||
- **Forms**: `browser_fill_form`, `browser_file_upload`
|
||
- **Debugging**: `browser_console_messages`, `browser_network_requests`, `browser_take_screenshot`
|
||
- **Management**: `browser_close`, `browser_resize`, `browser_tabs`, `browser_handle_dialog`
|
||
- **Execution**: `browser_evaluate`, `browser_press_key`, `browser_wait_for`
|
||
- **Snapshots**: `browser_snapshot`, `browser_install`
|
||
|
||
**Comparison**: `browser-server` offers high-level automation, while `ms-playwright` provides fine-grained control
|
||
|
||
---
|
||
|
||
### 📄 Document Processing Tools (5 Servers, 9 Documented Tools)
|
||
|
||
#### 4. Documents CSV Server (`documents-csv-server`)
|
||
- `extract_csv_content`: Extract and analyze CSV file content (supports Markdown/JSON output formats)
|
||
- `list_supported_formats`: List supported CSV formats
|
||
|
||
#### 5. Documents DOCX Server (`documents-docx-server`)
|
||
- `extract_docx_content`: Extract Word document content (including text, tables, images)
|
||
- `list_supported_formats`: List supported DOCX formats
|
||
|
||
#### 6. Documents PPTX Server (`documents-pptx-server`)
|
||
- `extract_pptx_content`: Extract PowerPoint content (slide text, notes, layout)
|
||
- `list_supported_formats`: List supported PPTX formats
|
||
|
||
#### 7. Documents PDF Server (`documents-pdf-server`)
|
||
- `convert_document_to_markdown`: Convert PDF to Markdown (preserving structure and formatting)
|
||
|
||
#### 8. Documents TXT Server (`documents-txt-server`)
|
||
- `extract_text_content`: Extract plain text file content
|
||
- `list_supported_formats`: List supported text encodings
|
||
|
||
**GAIA Application Scenario**: Processing attached files (70% of GAIA dataset questions include document attachments)
|
||
|
||
---
|
||
|
||
### 🎥 Multimedia Processing Tools (3 Servers, 12 Tools)
|
||
|
||
#### 9. Media Audio Server (`media-audio-server`)
|
||
- `transcribe_audio`: Speech-to-text (supports Whisper API)
|
||
- `extract_audio_metadata`: Extract audio metadata (duration, bitrate, sample rate)
|
||
- `trim_audio`: Trim audio segments
|
||
- `list_supported_formats`: List supported audio formats (MP3, WAV, M4A, etc.)
|
||
|
||
#### 10. Media Image Server (`media-image-server`)
|
||
- `extract_text_ocr`: OCR text recognition (based on Tesseract/Cloud Vision API)
|
||
- `analyze_image_ai`: AI image analysis (scene recognition, object detection, description generation)
|
||
- `get_image_metadata`: Extract image metadata (dimensions, EXIF, capture time)
|
||
|
||
#### 11. Media Video Server (`media-video-server`)
|
||
- `analyze_video`: Video content analysis (scene segmentation, keyframe extraction)
|
||
- `summarize_video`: Video summary generation
|
||
- `extract_keyframes`: Extract keyframe images
|
||
|
||
#### Supplementary: Standalone Multimedia Tools
|
||
- **Audio Server** (`audio-server`): `mcp_transcribe_audio` - Advanced speech transcription
|
||
- **Image Server** (`image-server`): `mcp_image_recognition` - Image recognition and classification
|
||
|
||
**GAIA Application**: Approximately 40% of GAIA questions involve image, audio, or video analysis
|
||
|
||
---
|
||
|
||
### 💡 Intelligent Reasoning Tools (3 Servers, 6 Tools)
|
||
|
||
#### 12. Intelligence Code Server (`intelligence-code-server`)
|
||
- `generate_python_code`: Generate and validate Python code (for mathematical calculations, data processing)
|
||
- `get_reasoning_capabilities`: Retrieve code generation capability information
|
||
|
||
#### 13. Intelligence Think Server (`intelligence-think-server`)
|
||
- `complex_problem_reasoning`: Complex problem reasoning (mathematical proofs, algorithm design, logic puzzles)
|
||
- `get_reasoning_capabilities`: Retrieve reasoning capability information
|
||
|
||
**Features**: Invokes more powerful reasoning models (e.g., GPT-4o, Claude 3.7 Sonnet) for deep thinking
|
||
|
||
#### 14. Intelligence Guard Server (`intelligence-guard-server`)
|
||
- `guarding_reasoning_process`: Reasoning process protection and validation (prevents hallucinations, checks logical consistency)
|
||
- `get_guarding_capabilities`: Retrieve guarding capability information
|
||
|
||
**Paper Highlight**: These "thinking tools" enable smaller models to leverage the reasoning capabilities of larger models, allowing them to "stand on the shoulders of giants"
|
||
|
||
---
|
||
|
||
### 💻 Code Execution Tools (3 Servers, 15 Documented Tools)
|
||
|
||
#### 15. Terminal Server (`terminal-server`)
|
||
- `execute_command`: Execute terminal commands (Python, bash, system commands)
|
||
- `get_command_history`: Retrieve command execution history
|
||
- `get_terminal_capabilities`: Retrieve terminal capability information
|
||
|
||
**Security Features**: Command whitelist, timeout control, output truncation
|
||
|
||
#### 16. E2B Code Server (`e2b-code-server`)
|
||
- `e2b_upload_file`: Upload files to the sandbox
|
||
- `e2b_run_code`: Execute code in an isolated sandbox (supports Python, JavaScript, multiple languages)
|
||
|
||
**Advantage**: Completely isolated execution environment, preventing malicious code from affecting the main system
|
||
|
||
#### 17. Terminal Controller (`terminal-controller`)
|
||
Provides **10 advanced terminal management tools**:
|
||
- `execute_command`, `get_command_history`, `get_current_directory`, `change_directory`
|
||
- `list_directory`, `write_file`, `read_file`
|
||
- `insert_file_content`, `delete_file_content`, `update_file_content`
|
||
|
||
**Difference**: `terminal-server` focuses on command execution, while `terminal-controller` provides file system management.
|
||
|
||
---
|
||
|
||
### 📂 File System Tools (1 Server, 14 Tools)
|
||
|
||
#### 18. Filesystem Server (`filesystem-server`)
|
||
Complete file operation capabilities:
|
||
- **Reading**: `read_file`, `read_text_file`, `read_media_file`, `read_multiple_files`
|
||
- **Writing**: `write_file`, `edit_file`
|
||
- **Management**: `create_directory`, `move_file`, `get_file_info`
|
||
- **Search**: `search_files`, `list_directory`, `list_directory_with_sizes`, `directory_tree`
|
||
- **Permissions**: `list_allowed_directories` - Lists allowed directories for access
|
||
|
||
**GAIA Application**: Access dataset attachments (in the `/root/workspace/gaia_dataset/` directory)
|
||
|
||
---
|
||
|
||
### 📊 Excel Processing Tools (1 Server, 29 Tools)
|
||
|
||
#### 19. Excel Server (`excel`)
|
||
Provides enterprise-grade Excel operations:
|
||
|
||
**Data Operations (9)**:
|
||
- `read_data_from_excel`, `write_data_to_excel`
|
||
- `insert_rows`, `insert_columns`, `delete_sheet_rows`, `delete_sheet_columns`
|
||
- `copy_range`, `delete_range`, `validate_excel_range`
|
||
|
||
**Workbook/Sheet Management (7)**:
|
||
- `create_workbook`, `create_worksheet`, `copy_worksheet`
|
||
- `delete_worksheet`, `rename_worksheet`, `get_workbook_metadata`
|
||
|
||
**Advanced Features (13)**:
|
||
- `apply_formula`, `validate_formula_syntax`
|
||
- `format_range`, `create_chart`, `create_pivot_table`, `create_table`
|
||
- `merge_cells`, `unmerge_cells`, `get_merged_cells`
|
||
- `get_data_validation_info`
|
||
|
||
**GAIA Typical Tasks**: Analyze complex Excel data tables, compute statistical metrics, generate charts
|
||
|
||
---
|
||
|
||
### 🔍 Knowledge Retrieval Tools (3 Servers, 15 Documented Tools)
|
||
|
||
#### 20. Wikipedia Server (`wiki-server`)
|
||
- `search_wikipedia`: Search Wikipedia entries
|
||
- `get_article_content`: Get full article content
|
||
- `get_article_summary`: Get article summary
|
||
- `get_article_categories`: Get article categories
|
||
- `get_article_links`: Get article links
|
||
- `get_article_history`: Get article history (for time-sensitive questions)
|
||
- `get_wikipedia_capabilities`: Get Wikipedia service capabilities
|
||
|
||
**Special Features**: Supports multiple languages, historical version queries (GAIA has questions like "population data for a certain month and year")
|
||
|
||
#### 21. ArXiv Server (`parxiv-server`)
|
||
- `search_papers`: Search arXiv papers
|
||
- `get_paper_details`: Get detailed paper information (abstract, authors, citations)
|
||
- `download_paper`: Download paper PDF
|
||
- `get_arxiv_capabilities`: Get arXiv service capabilities
|
||
- `get_categories`: Get arXiv category list
|
||
|
||
#### 22. Wayback Machine Server (`wayback-server`)
|
||
- `list_archived_versions`: List archived versions of a webpage
|
||
- `get_archived_content`: Get webpage content at a specific point in time
|
||
- `get_wayback_capabilities`: Get Wayback Machine capabilities
|
||
|
||
**GAIA Application**: Answer historical queries like "information on a certain website in 2015"
|
||
|
||
---
|
||
|
||
### 📥 Other Utility Tools (3 Servers, at Least 4 Documented Tools)
|
||
|
||
#### 23. Download Server (`download-server`)
|
||
- `download_file`: Download a network file to local storage
|
||
- `get_download_capabilities`: Get download service capabilities
|
||
|
||
#### 24. Read Web Server (`readweb-server`)
|
||
- Provides webpage content reading capabilities (specific tools defined by MCP configuration)
|
||
|
||
#### 25. Google Search Alternative (`google-search`)
|
||
- `search`: Simplified search interface
|
||
- `read_webpage`: Read webpage content
|
||
|
||
---
|
||
|
||
### Tool Statistics Summary
|
||
|
||
| Category | Servers | Tools | Key Capabilities |
|
||
|----------|---------|-------|------------------|
|
||
| **Web Interaction** | 3 | 27 documented | Search, intelligent browsing, fine-grained control |
|
||
| **Document Processing** | 5 | 9 documented | CSV, Word, PPT, PDF, TXT |
|
||
| **Multimedia** | 5 | 12 | Audio transcription, OCR, image/video analysis |
|
||
| **Intelligent Reasoning** | 3 | 6 | Code generation, complex reasoning, verification |
|
||
| **Code Execution** | 3 | 15 documented | Terminal commands, sandbox execution, file management |
|
||
| **File System** | 1 | 14 | Complete file operation capabilities |
|
||
| **Excel** | 1 | 29 | Enterprise-grade spreadsheet processing |
|
||
| **Knowledge Retrieval** | 3 | 15 | Wikipedia, ArXiv, historical web pages |
|
||
| **Other** | 3 | At least 4 documented | File download, webpage reading |
|
||
|
||
Counts in this table reflect the interfaces explicitly enumerated above. The
|
||
upstream MCP configuration is the source of truth for complete totals,
|
||
especially for servers whose individual interfaces are not listed here.
|
||
|
||
### Tool Call Examples (from Training Logs)
|
||
|
||
```python
|
||
# Google Search Example
|
||
Tool call: aworld-mcp__search_google
|
||
Tool args: {"query": "Wyoming population 2020", "num_results": 5}
|
||
Result: {"success": true, "results": [{"title": "Wyoming - Census Bureau", "snippet": "576,851..."}]}
|
||
|
||
# File System Example
|
||
Tool call: aworld-mcp__list_directory
|
||
Tool args: {"path": "/root/workspace/gaia_dataset/2023/test"}
|
||
Result: ["[FILE] 021a5339-...-bd9b-9368b3efda7a.pdf", "[FILE] 03c577c9-...-f8f598de14c1.mp3", ...]
|
||
|
||
# CSV Processing Example (will error if tabulate dependency is missing)
|
||
Tool call: aworld-mcp__extract_csv_content
|
||
Tool args: {"file_path": "/root/workspace/gaia_dataset/2023/test/52e8ce1c-...-67d1648779b9.csv"}
|
||
Error: "CSV extraction failed: Missing optional dependency 'tabulate'"
|
||
|
||
# Wikipedia History Query Example
|
||
Tool call: aworld-mcp__get_article_history
|
||
Tool args: {"title": "Cat", "date": "20191231", "language": "en"}
|
||
Result: {...historical Wikipedia content...}
|
||
```
|
||
|
||
---
|
||
|
||
## Core Architecture
|
||
|
||
AWorld Train uses a four-stage training pipeline:
|
||
|
||
|
||
|
||
```
|
||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||
│ Environment │───▶│ Agent │───▶│ Adapter │───▶│ Training │
|
||
│ Setup │ │Construction │ │ Layer │ │ Framework │
|
||
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
|
||
(MCP) (AWorld) (VeRL) (PPO/GRPO)
|
||
```
|
||
|
||
1. **Environment Setup**: Deploy GAIA MCP Server, providing 20+ tool capabilities
|
||
2. **Agent Construction**: Implement a custom AgentLoop, define decision logic
|
||
3. **Adapter Integration**: Unify interfaces, connect to the RL training framework
|
||
4. **Training Execution**: Configure reward functions and hyperparameters, start the training task
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
### System Requirements
|
||
|
||
| Component | Requirement |
|
||
|-----------|-------------|
|
||
| **Operating System** | Linux (recommended) / macOS / Windows |
|
||
| **Hardware** | Minimum 4 CPU cores + 8GB RAM<br>Training recommended: 8x A100/H100 GPU |
|
||
| **Software** | Docker, NVIDIA Driver, CUDA 12.1+ |
|
||
|
||
### Install Training Framework
|
||
|
||
Using VeRL as an example, the installation steps are as follows:
|
||
|
||
```bash
|
||
# Run this block from any directory after setting the absolute book-repository path.
|
||
export BOOK_ROOT=/absolute/path/to/ai-agent-book
|
||
|
||
# 1. Install system dependencies (Ubuntu/Debian)
|
||
sudo apt-get update
|
||
sudo apt-get install -y build-essential git wget
|
||
|
||
# 2. Install CUDA Toolkit (matching your GPU Driver)
|
||
# Reference: https://developer.nvidia.com/cuda-downloads
|
||
|
||
# 3. Install PyTorch (matching CUDA version)
|
||
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121
|
||
|
||
# 4. Clone the book's verified AWorld fork to the canonical companion path
|
||
git clone https://github.com/bojieli/AWorld.git "${BOOK_ROOT}/chapter8/AWorld"
|
||
git -C "${BOOK_ROOT}/chapter8/AWorld" checkout --detach a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec
|
||
|
||
# 5. Clone/install the verified VeRL backend
|
||
git clone https://github.com/bojieli/verl.git "${BOOK_ROOT}/chapter8/verl"
|
||
git -C "${BOOK_ROOT}/chapter8/verl" checkout --detach 1593fc3a8cf894debdc3dece2a23ed739c282789
|
||
pip install -e "${BOOK_ROOT}/chapter8/verl"
|
||
```
|
||
|
||
**Important Note**: Some dependencies of VeRL need to be compiled in a CUDA environment. Please ensure steps 1-2 are completed first.
|
||
|
||
---
|
||
|
||
### Configure GAIA Environment
|
||
|
||
The GAIA environment is deployed via Docker and provides MCP (Model Context Protocol) services.
|
||
|
||
#### 1. Download the Dataset
|
||
|
||
```bash
|
||
# Download the GAIA dataset from Hugging Face
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/docker
|
||
mkdir -p gaia_dataset
|
||
|
||
# Use Hugging Face CLI to download (requires pip install huggingface_hub first)
|
||
huggingface-cli download gaia-benchmark/GAIA --repo-type dataset --local-dir gaia_dataset
|
||
```
|
||
|
||
#### 2. Configure Environment Variables
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/mcp_servers
|
||
cp .env_template .env
|
||
|
||
# Edit the .env file to configure necessary API keys
|
||
vim .env
|
||
```
|
||
|
||
Example `.env` file (partial fields):
|
||
|
||
```bash
|
||
# OpenAI API (for intelligence-code-server, etc.)
|
||
OPENAI_API_KEY=your-openai-api-key
|
||
|
||
# Google Search API (for googlesearch-server)
|
||
GOOGLE_API_KEY=your-google-api-key
|
||
GOOGLE_CSE_ID=your-search-engine-id
|
||
|
||
# E2B API (for code execution sandbox)
|
||
E2B_API_KEY=your-e2b-api-key
|
||
```
|
||
|
||
#### 3. Start the MCP Server
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env
|
||
bash run-local.sh
|
||
```
|
||
|
||
Once started successfully, you will see output similar to:
|
||
|
||
```
|
||
Starting services...
|
||
DISPLAY=:0
|
||
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
|
||
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
|
||
...
|
||
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
|
||
```
|
||
|
||
The MCP Server provides two services:
|
||
- **MCP Interface**: `http://localhost:8080/mcp` (Agent tool calls)
|
||
- **VNC Interface**: `http://localhost:5901/vnc.html?autoconnect=true` (visual debugging)
|
||
|
||
#### 4. Verify the Environment
|
||
|
||
```bash
|
||
# Set environment variables
|
||
export MCP_SERVER_URL=http://localhost:8080/mcp
|
||
|
||
# Test connection (Python)
|
||
python3 << EOF
|
||
from train.adapter.verl.common import get_agent_tool_env_and_servers
|
||
|
||
config, servers = get_agent_tool_env_and_servers()
|
||
print(f"Available servers: {len(servers)}")
|
||
print(f"Sample servers: {servers[:5]}")
|
||
EOF
|
||
```
|
||
|
||
Expected output:
|
||
|
||
```
|
||
Available servers: 20
|
||
Sample servers: ['readweb-server', 'browser-server', 'documents-csv-server', ...]
|
||
```
|
||
|
||
---
|
||
|
||
## Building a Custom Agent
|
||
|
||
### Implementing AgentLoop
|
||
|
||
The core of creating a custom Agent is to inherit `AworldAgentLoop` and implement the `build_agents()` method. Below is a complete example of a GAIA Agent:
|
||
|
||
```python
|
||
# train/examples/train_gaia_with_aworld_verl/custom_agent_loop.py
|
||
|
||
from aworld.agents.llm_agent import Agent
|
||
from aworld.config import AgentConfig
|
||
from train.adapter.verl.aworld_agent_loop import AworldAgentLoop
|
||
from train.adapter.verl.common import get_agent_tool_env_and_servers
|
||
|
||
class GaiaAgentLoop(AworldAgentLoop):
|
||
"""Custom Agent Loop for GAIA tasks"""
|
||
|
||
async def build_agents(self):
|
||
# Get MCP environment configuration and list of available servers
|
||
gaia_env_config, gaia_env_servers = get_agent_tool_env_and_servers()
|
||
|
||
# Build Agent instance
|
||
return Agent(
|
||
conf=AgentConfig(
|
||
# LLM server address is dynamically managed by VeRL
|
||
llm_base_url=await self.get_llm_server_address(),
|
||
llm_model_name=await self.get_llm_server_model_name(),
|
||
llm_api_key="dummy", # No real API key needed for VeRL internal communication
|
||
),
|
||
name="gaia_super_agent",
|
||
|
||
# System prompt (defines Agent role and capabilities)
|
||
system_prompt="""You are a helpful AI assistant specialized in solving complex tasks.
|
||
You have access to various tools including web search, code execution, and file analysis.
|
||
When given a task, break it down into steps and use available tools systematically.
|
||
Always provide your final answer in <answer>...</answer> tags.""",
|
||
|
||
# MCP tool configuration
|
||
mcp_config=gaia_env_config,
|
||
mcp_servers=gaia_env_servers,
|
||
)
|
||
```
|
||
|
||
### Configuring agent.yaml
|
||
|
||
Register your AgentLoop in `train/examples/train_gaia_with_aworld_verl/agent.yaml`:
|
||
|
||
```yaml
|
||
- name: gaia_agent
|
||
_target_: train.examples.train_gaia_with_aworld_verl.custom_agent_loop.GaiaAgentLoop
|
||
```
|
||
|
||
### Advanced Scenarios
|
||
|
||
#### Multi-Agent System
|
||
|
||
```python
|
||
from aworld.swarms.swarm import Swarm
|
||
|
||
class MultiAgentLoop(AworldAgentLoop):
|
||
def build_agents(self):
|
||
config, servers = get_agent_tool_env_and_servers()
|
||
|
||
# Create specialized Agents
|
||
researcher = Agent(
|
||
conf=AgentConfig(...),
|
||
name="researcher",
|
||
system_prompt="You are a research specialist...",
|
||
mcp_servers=["googlesearch-server", "wiki-server"]
|
||
)
|
||
|
||
coder = Agent(
|
||
conf=AgentConfig(...),
|
||
name="coder",
|
||
system_prompt="You are a coding expert...",
|
||
mcp_servers=["e2b-code-server", "terminal-server"]
|
||
)
|
||
|
||
# Build Swarm
|
||
return Swarm(
|
||
agents=[researcher, coder],
|
||
coordinator=researcher # Main coordinating Agent
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## Preparing for Training
|
||
|
||
### 1. Prepare the Dataset
|
||
|
||
Run the dataset generation script to convert GAIA data into training format:
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/train/examples/train_gaia_with_aworld_verl/gaia_datasets
|
||
|
||
python create_dataset.py \
|
||
--dataset_path ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/docker/gaia_dataset \
|
||
--output_dir ~/datasets \
|
||
--train_size 300 \
|
||
--test_size 100
|
||
```
|
||
|
||
This will generate:
|
||
- `~/datasets/train.parquet`: 300 training samples
|
||
- `~/datasets/test.parquet`: 100 test samples
|
||
|
||
Data format (Parquet):
|
||
|
||
| Field | Type | Description |
|
||
|------|------|------|
|
||
| `prompt` | List[Dict] | Formatted chat messages `[{"role": "user", "content": "..."}]` |
|
||
| `data_source` | str | Data source identifier `"gaia"` |
|
||
| `ability` | str | Ability category `"agi"` |
|
||
| `reward_model` | Dict | Reward configuration `{"style": "GAIA", "ground_truth": "..."}` |
|
||
| `extra_info` | Dict | Additional metadata (task_id, level, etc.) |
|
||
| `agent_name` | str | Target Agent name |
|
||
|
||
### 2. Configure the Reward Function
|
||
|
||
Define the reward logic in `train/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py`:
|
||
|
||
```python
|
||
import re
|
||
from aworld.logs.util import logger
|
||
|
||
def gaia_reward_func(data_source, solution_str, ground_truth, extra_info=None):
|
||
"""
|
||
Reward function for GAIA tasks
|
||
|
||
Args:
|
||
data_source: Data source identifier
|
||
solution_str: Complete solution generated by the Agent
|
||
ground_truth: Ground truth answer
|
||
extra_info: Additional information (e.g., task_id, level)
|
||
|
||
Returns:
|
||
float: Reward value (0.0 or 1.0)
|
||
"""
|
||
# Extract content from <answer>...</answer> tags in solution_str
|
||
pattern = r'<answer>(.*?)</answer>'
|
||
match = re.search(pattern, solution_str, re.DOTALL | re.MULTILINE)
|
||
|
||
if not match:
|
||
logger.warning("No answer tag found in solution")
|
||
return 0.0
|
||
|
||
answer = match.group(1).strip()
|
||
logger.info(f"Extracted answer: {answer}, Ground truth: {ground_truth}")
|
||
|
||
# Use GAIA standard scorer (supports numbers, lists, strings)
|
||
if question_scorer(answer, ground_truth):
|
||
return 1.0
|
||
else:
|
||
return 0.0
|
||
|
||
def question_scorer(model_answer: str, ground_truth: str) -> bool:
|
||
"""GAIA standard scoring logic (detailed implementation omitted)"""
|
||
# Supports number comparison, list comparison, string normalization comparison
|
||
# See full code for details
|
||
...
|
||
```
|
||
|
||
### 3. Configure the Training Script
|
||
|
||
Edit `run.sh` to configure key parameters:
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
set -xeuo pipefail
|
||
|
||
# ============ Cluster Topology ============
|
||
export GPUS_PER_NODE=${GPUS_PER_NODE:-8}
|
||
export NNODES=${NNODES:-1}
|
||
|
||
# ============ Model and Data ============
|
||
model_path=${model_path:-Qwen/Qwen3-4B-Thinking-2507}
|
||
train_files=$DATA_ROOT/datasets/train.parquet
|
||
test_files=$DATA_ROOT/datasets/test.parquet
|
||
|
||
# ============ Custom Configuration ============
|
||
path_to_train="/root/AWorld/train"
|
||
agent_loop_config_path=${path_to_train}/examples/train_gaia_with_aworld_verl/agent.yaml
|
||
reward_fn_file_path=${path_to_train}/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py
|
||
reward_fn_name=gaia_reward_func
|
||
|
||
# ============ Training Hyperparameters ============
|
||
# PPO Algorithm Configuration
|
||
adv_estimator=grpo # Use Group Relative Policy Optimization
|
||
clip_ratio_low=0.2 # PPO clip lower bound
|
||
clip_ratio_high=0.28 # PPO clip upper bound
|
||
actor_lr=1e-6 # Actor learning rate
|
||
|
||
# Long Context Configuration (AWorld Latest Optimization)
|
||
max_turns=32 # Maximum number of interaction turns (increased from 8 to 32)
|
||
max_prompt_length=4096 # Maximum prompt length (increased from 1024 to 4096)
|
||
max_response_length=4096 # Maximum response length (increased from 2048 to 4096)
|
||
|
||
# Batch Configuration
|
||
train_batch_size=32 # Training batch size (increased from 1 to 32)
|
||
ppo_mini_batch_size=8 # PPO mini-batch size (4 gradient updates)
|
||
n_resp_per_prompt=16 # Sample 16 responses per prompt (increased from 1)
|
||
n_resp_per_prompt_val=16 # Number of samples for validation
|
||
|
||
# ============ MCP Server ============
|
||
export MCP_SERVER_URL=${MCP_SERVER_URL:-http://localhost:8080/mcp}
|
||
|
||
# ============ Performance Optimization ============
|
||
export VLLM_USE_V1=1 # Use vLLM v1 engine
|
||
export VLLM_ATTENTION_BACKEND=FLASH_ATTN # FlashAttention-2
|
||
infer_tp=1 # Tensor Parallel size
|
||
train_sp=1 # Sequence Parallel size
|
||
offload=true # Offload parameters to CPU
|
||
|
||
# ============ VeRL Training Command ============
|
||
python3 -m verl.trainer.main_ppo \
|
||
algorithm.adv_estimator=$adv_estimator \
|
||
data.train_files="['$train_files']" \
|
||
data.val_files="['$test_files']" \
|
||
data.return_raw_chat=true \
|
||
data.train_batch_size=$train_batch_size \
|
||
data.max_prompt_length=$max_prompt_length \
|
||
data.max_response_length=$max_response_length \
|
||
actor_rollout_ref.model.path="$model_path" \
|
||
actor_rollout_ref.rollout.multi_turn.max_user_turns=$max_turns \
|
||
actor_rollout_ref.rollout.multi_turn.max_assistant_turns=$max_turns \
|
||
actor_rollout_ref.rollout.max_model_len=131072 \
|
||
actor_rollout_ref.rollout.max_num_batched_tokens=131072 \
|
||
actor_rollout_ref.rollout.gpu_memory_utilization=0.9 \
|
||
actor_rollout_ref.rollout.agent.agent_loop_config_path=$agent_loop_config_path \
|
||
custom_reward_function.path="${reward_fn_file_path}" \
|
||
custom_reward_function.name="${reward_fn_name}" \
|
||
trainer.logger=['console','wandb'] \
|
||
trainer.experiment_name=aworld_train_qwen3_4b \
|
||
trainer.save_freq=5 \
|
||
trainer.test_freq=5 \
|
||
+trainer.num_steps=300
|
||
```
|
||
|
||
---
|
||
|
||
## Launching Training
|
||
|
||
### Single-Node Training
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/train/examples/train_gaia_with_aworld_verl
|
||
|
||
# Launch training (8 GPUs)
|
||
export DATA_ROOT=~/datasets
|
||
export GPUS_PER_NODE=8
|
||
bash run.sh
|
||
```
|
||
|
||
### Multi-Node Training (Slurm)
|
||
|
||
```bash
|
||
# Submit Slurm job (2 nodes, 8 GPUs per node)
|
||
sbatch <<EOF
|
||
#!/bin/bash
|
||
#SBATCH --job-name=aworld-train
|
||
#SBATCH --nodes=2
|
||
#SBATCH --ntasks-per-node=1
|
||
#SBATCH --gpus-per-node=8
|
||
#SBATCH --time=48:00:00
|
||
|
||
export DATA_ROOT=/path/to/datasets
|
||
export NNODES=2
|
||
export GPUS_PER_NODE=8
|
||
|
||
srun bash run.sh
|
||
EOF
|
||
```
|
||
|
||
### Training Monitoring
|
||
|
||
The training process supports multiple logging backends:
|
||
|
||
```bash
|
||
# 1. Console Output
|
||
# View training metrics in real-time (loss, reward, KL divergence, etc.)
|
||
|
||
# 2. WandB Visualization (Recommended)
|
||
# Visit https://wandb.ai/<your-project>/aworld_train_qwen3_4b
|
||
|
||
# 3. TensorBoard
|
||
tensorboard --logdir ~/datasets/checkpoint/aworld_train_qwen3_4b
|
||
```
|
||
|
||
Key Monitoring Metrics:
|
||
|
||
| Metric | Description | Target Value |
|
||
|--------|-------------|--------------|
|
||
| `reward/mean` | Average reward | Gradually increase to 0.3+ |
|
||
| `reward/max` | Maximum reward | Reach 1.0 |
|
||
| `actor/loss` | Actor loss | Steadily decrease |
|
||
| `rollout/response_length` | Response length | Adjust based on task |
|
||
| `rollout/num_turns` | Average number of turns | Efficient tool use (5-15 turns) |
|
||
|
||
---
|
||
|
||
## Latest Optimizations
|
||
|
||
Based on the latest code changes (commit `a52d61d6`), we have implemented the following key optimizations:
|
||
|
||
### 1. Long Context Support (verl_provider.py)
|
||
|
||
```python
|
||
# New dynamic max_model_len parameter configuration
|
||
self.max_model_len = params.get("max_model_len", 24576)
|
||
```
|
||
|
||
**Impact**: Supports context windows of up to 131K tokens, enabling complex multi-turn dialogues and large-scale tool call histories.
|
||
|
||
### 2. Data Format Optimization (create_dataset.py)
|
||
|
||
```python
|
||
# Prompt formatted as a list of chat messages (adapting to VeRL return_raw_chat mode)
|
||
rl_dataset["prompt"].append([{"role": "user", "content": data["Question"]}])
|
||
```
|
||
|
||
**Impact**: Seamlessly integrates with VeRL's chat template system, avoiding format conversion overhead.
|
||
|
||
### 3. Hyperparameter Tuning (run.sh)
|
||
|
||
| Parameter | Old Value | New Value | Improvement |
|
||
|-----------|-----------|-----------|-------------|
|
||
| `max_turns` | 8 | 32 | 4x interaction depth |
|
||
| `max_prompt_length` | 1024 | 4096 | 4x input capacity |
|
||
| `max_response_length` | 2048 | 4096 | 2x output capacity |
|
||
| `train_batch_size` | 1 | 32 | 32x training efficiency |
|
||
| `n_resp_per_prompt` | 1 | 16 | 16x sample diversity |
|
||
|
||
**Impact**:
|
||
- **Deeper Reasoning**: Allows the agent to perform longer tool chains and reasoning
|
||
- **More Efficient Training**: Larger batches accelerate convergence, diverse sampling improves generalization
|
||
- **More Stable Optimization**: `ppo_mini_batch_size=8` enables 4 gradient updates, balancing training stability and efficiency
|
||
|
||
### 4. Memory Optimization
|
||
|
||
```bash
|
||
# vLLM Configuration
|
||
actor_rollout_ref.rollout.max_model_len=131072 # Model context length
|
||
actor_rollout_ref.rollout.max_num_batched_tokens=131072 # Batched token count
|
||
actor_rollout_ref.rollout.gpu_memory_utilization=0.9 # GPU memory utilization
|
||
```
|
||
|
||
**Impact**: Supports concurrent inference of 32K+ tokens on A100 80GB, fully utilizing GPU resources.
|
||
|
||
### 5. New Qwen3-30B-A3B Training Script
|
||
|
||
```bash
|
||
# run_qwen3_30b_a3b.sh
|
||
infer_tp=4 # Tensor Parallel (Inference)
|
||
train_sp=8 # Sequence Parallel (Training)
|
||
```
|
||
|
||
**Impact**: Supports training larger models, leveraging model parallelism to overcome single-GPU limitations.
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### Agent Training Process Output Examples
|
||
|
||
#### ✅ Normal Inference Flow
|
||
|
||
```
|
||
(AgentLoopWorker pid=448354) [agent] Content: Okay, let's see. So the user is asking about the population difference between the two states that have both Carl's Jr. and Hardee's fast food restaurants...
|
||
(AgentLoopWorker pid=448354) [agent] Tool call: aworld-mcp__search_google - ID: chatcmpl-tool-94b30baa
|
||
(AgentLoopWorker pid=448354) [agent] Tool args: {"query": "Wyoming population 2020", "num_results": 5}
|
||
(AgentLoopWorker pid=448354) [agent] Content: ["{\"success\": true, \"message\": {\"query\": \"Wyoming population 2020\", \"results\": [{\"title\": \"Wyoming - Census Bureau Profile\", \"snippet\": \"576,851. The Total Population for Wyoming is 576,851...\"...}
|
||
```
|
||
|
||
**Explanation**: The agent correctly calls the tool and receives results; the inference chain is complete.
|
||
|
||
#### ✅ Successful File Listing
|
||
|
||
```
|
||
(AgentLoopWorker pid=448358) [agent] Content: ["[FILE] 021a5339-744f-42b7-bd9b-9368b3efda7a.pdf\n[FILE] 03c577c9-4227-48a9-9b75-f8f598de14c1.mp3\n[FILE] 063800f6-8832-4856-972b-17b877612533.png\n..."]
|
||
(AgentLoopWorker pid=448358) [agent] Content: Okay, let's try to figure out how many horror titles are overdue based on the inventory file...
|
||
```
|
||
|
||
**Explanation**: The file system tool works correctly; the agent can access dataset files.
|
||
|
||
---
|
||
|
||
### GAIA MCP Server Output Example#### ✅ Normal Startup Output
|
||
|
||
```bash
|
||
$ docker logs gaia-mcp-server-gaia-mcp-server-1 -f
|
||
|
||
Starting services...
|
||
DISPLAY=:0
|
||
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
|
||
2025-10-06 05:20:42,370 - mcp_server_proxy.mcp_server_proxy - INFO - Loaded MCP tool schema: mcp_tool_schema=
|
||
readweb-server:
|
||
browser-server:
|
||
- get_browser_capabilities
|
||
- browser_use
|
||
documents-csv-server:
|
||
- extract_csv_content
|
||
- list_supported_formats
|
||
googlesearch-server:
|
||
- search_google
|
||
- get_search_capabilities
|
||
...
|
||
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
|
||
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
|
||
```
|
||
|
||
**Checkpoints**:
|
||
- ✅ `Starting MCP Server Proxy` appears
|
||
- ✅ 20+ tool servers loaded (`Added MCP server executor`)
|
||
- ✅ Uvicorn listening on port 8080
|
||
|
||
#### ✅ Normal Request During Training
|
||
|
||
```
|
||
INFO: 208.64.254.164:36416 - "POST /mcp HTTP/1.1" 200 OK
|
||
2025-10-06 05:09:23,880 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
|
||
2025-10-06 05:09:27,006 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
|
||
[10/06/25 05:09:27] INFO 🔍 Searching Google for: 'Speaker of the House that passed act...'
|
||
INFO ✅ Found 5 results in 0.42s
|
||
```
|
||
|
||
**Explanation**: The agent is calling the Google search tool normally, with fast request-response times (< 1s).
|
||
|
||
---
|
||
|
||
### ❌ Common Errors and Solutions
|
||
|
||
#### 1. CSV Extraction Failed (Missing Dependency)
|
||
|
||
```
|
||
ERROR CSV extraction failed: Missing optional dependency 'tabulate'.
|
||
Use pip or conda to install tabulate.
|
||
```
|
||
|
||
**Cause**: The `to_markdown()` function in pandas requires the `tabulate` library.
|
||
|
||
**Solution**:
|
||
|
||
```bash
|
||
# Enter the MCP Server Docker container
|
||
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash
|
||
|
||
# Install the missing dependency
|
||
cd /app/mcp_servers/documents_server
|
||
source .venv/bin/activate
|
||
pip install tabulate
|
||
|
||
# Restart the container
|
||
exit
|
||
docker restart gaia-mcp-server-gaia-mcp-server-1
|
||
```
|
||
|
||
#### 2. OpenAI API Key Error
|
||
|
||
```
|
||
WARNING coding failed: Error code: 401 - {'error': {'message': 'Incorrect API key provided: [redacted-openrouter-key]', 'type': 'invalid_request_error', 'code': 'invalid_api_key'}}
|
||
```
|
||
|
||
**Cause**: Tools like `intelligence-code-server` require an OpenAI API key to generate code.
|
||
|
||
**Solution**:
|
||
|
||
```bash
|
||
# Method 1: Configure a real OpenAI API Key
|
||
vim ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/mcp_servers/.env
|
||
# Add: OPENAI_API_KEY=your-openai-api-key
|
||
|
||
# Method 2: Use a compatible service like OpenRouter
|
||
# Configure in .env:
|
||
LLM_BASE_URL=https://openrouter.ai/api/v1
|
||
OPENAI_API_KEY=your-openrouter-api-key
|
||
|
||
# Restart the service
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env
|
||
bash run-local.sh
|
||
```
|
||
|
||
#### 3. Wikipedia API 429 Rate Limiting
|
||
|
||
```
|
||
ERROR Wikipedia summary retrieval error:
|
||
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
|
||
```
|
||
|
||
**Cause**: The Wikipedia API is rate-limiting (429 Too Many Requests), but the Python library does not handle it gracefully.
|
||
|
||
**Solution**:
|
||
|
||
```bash
|
||
# Method 1: Reduce concurrent requests (adjust training parameters)
|
||
train_batch_size=16 # Reduce from 32 to 16
|
||
n_resp_per_prompt=8 # Reduce from 16 to 8
|
||
|
||
# Method 2: Use a proxy or switch Wikipedia mirror
|
||
# Configure in wiki_server/.env:
|
||
WIKIPEDIA_BASE_URL=https://en.wikipedia.org/w/api.php
|
||
|
||
# Method 3: Add request retry logic (requires code modification)
|
||
# Add exponential backoff in wiki_server/src/wiki.py
|
||
```
|
||
|
||
#### 4. Tool Execution Timeout
|
||
|
||
```
|
||
2025-10-06 05:21:08,548 - mcp_server_proxy.mcp_server_executor - INFO - Starting tool server browser-server...
|
||
[No response after 10 seconds]
|
||
```
|
||
|
||
**Cause**: The browser tool (Playwright) is slow to start or has insufficient resources.
|
||
|
||
**Solution**:
|
||
|
||
```bash
|
||
# Increase timeout configuration
|
||
vim ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/mcp_servers/.env
|
||
# Add:
|
||
TOOL_EXECUTION_TIMEOUT=120 # Increase from default 60s to 120s
|
||
|
||
# Warm up the browser environment
|
||
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash
|
||
cd /app/mcp_servers/browser_server
|
||
uv run python -c "from playwright.sync_api import sync_playwright; sync_playwright().start()"
|
||
```
|
||
|
||
#### 5. vLLM OOM (Out of Memory)
|
||
|
||
```
|
||
RuntimeError: CUDA out of memory. Tried to allocate 20.00 GiB (GPU 0; 79.35 GiB total capacity; 75.12 GiB already allocated; 2.31 GiB free; 78.90 GiB reserved in total by PyTorch)
|
||
```
|
||
|
||
**Solution**:
|
||
|
||
```bash
|
||
# Method 1: Reduce batch size
|
||
train_batch_size=16 # Reduce from 32
|
||
ppo_mini_batch_size=4 # Reduce from 8
|
||
|
||
# Method 2: Reduce GPU memory utilization
|
||
actor_rollout_ref.rollout.gpu_memory_utilization=0.75 # Reduce from 0.9
|
||
|
||
# Method 3: Enable CPU offload
|
||
actor_rollout_ref.actor.fsdp_config.param_offload=true
|
||
actor_rollout_ref.actor.fsdp_config.optimizer_offload=true
|
||
|
||
# Method 4: Use larger Tensor Parallel
|
||
infer_tp=4 # Increase from 1 to 4 (requires 4 GPUs)
|
||
```
|
||
|
||
---
|
||
|
||
### Debugging Tips
|
||
|
||
#### 1. Enable Verbose Logging
|
||
|
||
```bash
|
||
# VeRL training logs
|
||
export RAY_LOGGING_LEVEL=DEBUG
|
||
export HYDRA_FULL_ERROR=1
|
||
|
||
# MCP Server logs
|
||
docker logs -f gaia-mcp-server-gaia-mcp-server-1 --tail 100
|
||
|
||
# vLLM inference logs
|
||
export VLLM_LOGGING_LEVEL=DEBUG
|
||
```
|
||
|
||
#### 2. Step-by-Step Agent Debugging
|
||
|
||
```python
|
||
# test_agent_debug.py
|
||
from train.examples.train_gaia_with_aworld_verl.custom_agent_loop import GaiaAgentLoop
|
||
|
||
# Create AgentLoop (without starting training)
|
||
loop = GaiaAgentLoop()
|
||
agent = loop.build_agents()
|
||
|
||
# Test a single question
|
||
response = agent.chat("What is the capital of France?")
|
||
print(response)
|
||
```
|
||
|
||
#### 3. Check Tool Availability
|
||
|
||
```bash
|
||
# Test MCP Server health status
|
||
curl http://localhost:8080/health
|
||
|
||
# List all tools
|
||
curl -X POST http://localhost:8080/mcp \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"jsonrpc": "2.0", "method": "tools/list", "params": {}, "id": 1}'
|
||
```
|
||
|
||
---
|
||
|
||
## Performance Benchmarks
|
||
|
||
### GAIA Test Set Results (Based on Paper)
|
||
|
||
| Model | Pass@1 | Data Collection Speedup | Training Time |
|
||
|-------|--------|------------------------|---------------|
|
||
| GPT-4o (Baseline) | 27.91% | - | - |
|
||
| DeepSeek-V3 | 31.89% | - | - |
|
||
| **Qwen3-32B-AWorld** | **32.23%** | **14.6x** | 48h (8x A100) |
|
||
|
||
**Key Findings**:
|
||
- Through the "learning from practice" paradigm, a 32B parameter model surpasses GPT-4o and DeepSeek-V3
|
||
- Distributed experience generation reduces data collection time from 7 days to 12 hours
|
||
- End-to-end training (SFT + PPO) improves performance on the GAIA validation set by over 15 percentage points
|
||
|
||
### Hardware Performance
|
||
|
||
| Configuration | Throughput | GPU Utilization | Memory Usage |
|
||
|---------------|------------|-----------------|--------------|
|
||
| 1x A100 80GB (TP=1) | 120 tokens/s | 85% | 72GB |
|
||
| 4x A100 80GB (TP=4) | 450 tokens/s | 92% | 68GB/GPU |
|
||
| 8x A100 80GB (FSDP+TP) | 850 tokens/s | 95% | 70GB/GPU |
|
||
|
||
**Optimization Recommendations**:
|
||
- **Small models (< 8B)**: Single GPU training, `infer_tp=1`, `train_sp=1`
|
||
- **Medium models (8-30B)**: Use TP=4 for faster inference, `infer_tp=4`
|
||
- **Large models (> 30B)**: Combine TP and SP, `infer_tp=4, train_sp=8`
|
||
|
||
---
|
||
|
||
## Advanced Topics
|
||
|
||
### Custom Reward Functions
|
||
|
||
Beyond GAIA's binary reward, you can implement more complex reward shaping:
|
||
|
||
```python
|
||
def dense_reward_func(data_source, solution_str, ground_truth, extra_info=None):
|
||
"""Dense reward function (considering intermediate steps)"""
|
||
reward = 0.0
|
||
|
||
# 1. Tool usage reward (encourages exploration)
|
||
num_tool_calls = solution_str.count("Tool call:")
|
||
reward += min(num_tool_calls * 0.1, 0.5) # Max 0.5 points
|
||
|
||
# 2. Reasoning quality reward (based on CoT)
|
||
if "<think>" in solution_str and "</think>" in solution_str:
|
||
reward += 0.2 # Reward for having a reasoning process
|
||
|
||
# 3. Final answer reward (main score)
|
||
if question_scorer(extract_answer(solution_str), ground_truth):
|
||
reward += 1.0
|
||
|
||
# 4. Efficiency penalty (avoid excessive tool calls)
|
||
if num_tool_calls > 15:
|
||
reward -= 0.2
|
||
|
||
return reward
|
||
```
|
||
|
||
### Multi-task Training
|
||
|
||
```python
|
||
# gaia_datasets/create_multitask_dataset.py
|
||
def create_multitask_dataset():
|
||
datasets = []
|
||
|
||
# Task 1: GAIA
|
||
gaia_ds = load_gaia_dataset(...)
|
||
gaia_ds['task_type'] = 'gaia'
|
||
datasets.append(gaia_ds)
|
||
|
||
# Task 2: Code Execution
|
||
code_ds = load_code_dataset(...)
|
||
code_ds['task_type'] = 'code'
|
||
datasets.append(code_ds)
|
||
|
||
# Task 3: Web Navigation
|
||
web_ds = load_webarena_dataset(...)
|
||
web_ds['task_type'] = 'web'
|
||
datasets.append(web_ds)
|
||
|
||
# Mixed sampling
|
||
return pd.concat(datasets).sample(frac=1.0)
|
||
```
|
||
|
||
---
|
||
|
||
## Citation
|
||
|
||
If you use AWorld Train in your research, please cite our paper:
|
||
|
||
```bibtex
|
||
@article{yu2025aworld,
|
||
title={AWorld: Orchestrating the Training Recipe for Agentic AI},
|
||
author={Yu, Chengyue and Lu, Siyuan and Zhuang, Chenyi and Wang, Dong and others},
|
||
journal={arXiv preprint arXiv:2508.20404},
|
||
year={2025}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Community & Support
|
||
|
||
- **GitHub Issues**: [https://github.com/inclusionAI/AWorld/issues](https://github.com/inclusionAI/AWorld/issues)
|
||
- **Discord**: [https://discord.gg/aworld](https://discord.gg/aworld)
|
||
- **Paper**: [https://arxiv.org/abs/2508.20404](https://arxiv.org/abs/2508.20404)
|
||
- **Official Documentation**: [https://inclusionai.github.io/AWorld/](https://inclusionai.github.io/AWorld/)
|
||
|
||
---
|
||
|
||
<div align="center">
|
||
|
||
**AWorld Train** — Let Your Agent Learn from Practice
|
||
|
||
Made with ❤️ by [Inclusion AI](https://github.com/inclusionAI)
|
||
|
||
</div>
|
||
|
||
[license-image]: https://img.shields.io/badge/License-MIT-yellow.svg
|
||
[license-url]: https://opensource.org/licenses/MIT
|
||
|
||
---
|
||
|
||
## 中文
|
||
|
||
<div align="center">
|
||
|
||
# AWorld Train
|
||
|
||
*面向 Agentic AI 的"从实践中学习"训练框架*
|
||
|
||
[![License: MIT][license-image]][license-url]
|
||
[](https://arxiv.org/abs/2508.20404)
|
||
|
||
</div>
|
||
|
||
---
|
||
|
||
## 目录
|
||
|
||
- [简介](#简介)
|
||
- [关于本教育性实验的重要说明](#关于本教育性实验的重要说明)
|
||
- [核心特性](#核心特性)
|
||
- [GAIA 环境工具生态](#gaia-环境工具生态)
|
||
- [Web 交互工具](#-web-交互工具3-服务器9-工具)
|
||
- [文档处理工具](#-文档处理工具5-服务器12-工具)
|
||
- [多媒体处理工具](#-多媒体处理工具3-服务器12-工具)
|
||
- [智能推理工具](#-智能推理工具3-服务器6-工具)
|
||
- [代码执行工具](#-代码执行工具3-服务器24-工具)
|
||
- [文件系统工具](#-文件系统工具1-服务器14-工具)
|
||
- [Excel 处理工具](#-excel-处理工具1-服务器29-工具)
|
||
- [知识检索工具](#-知识检索工具3-服务器11-工具)
|
||
- [工具统计总结](#工具统计总结)
|
||
- [核心架构](#核心架构)
|
||
- [快速开始](#快速开始)
|
||
- [安装训练框架](#安装训练框架)
|
||
- [配置 GAIA 环境](#配置-gaia-环境)
|
||
- [构建自定义 Agent](#构建自定义-agent)
|
||
- [准备训练](#准备训练)
|
||
- [启动训练](#启动训练)
|
||
- [最新优化](#最新优化)
|
||
- [故障排查](#故障排查)
|
||
- [性能基准](#性能基准)
|
||
- [进阶主题](#进阶主题)
|
||
- [引用](#引用)
|
||
- [社区与支持](#社区与支持)
|
||
|
||
---
|
||
|
||
## 简介
|
||
|
||
AWorld Train 是实现 **"从实践中学习"（Learning from Practice）** 范式的开源训练框架，专门为 Agentic AI 设计。根据 [AWorld 论文](https://arxiv.org/abs/2508.20404)，构建高性能 Agent 系统需要三个核心要素：
|
||
|
||
1. **算法（Algorithm）**：使 Agent 能够从环境交互中适应和改进的学习机制
|
||
2. **环境（Environment）**：提供丰富反馈和多样化挑战的复杂交互场景
|
||
3. **先验（Priors）**：当前大模型在推理、数学、视觉等领域的基础能力
|
||
|
||
AWorld Train 通过分布式架构解决了传统方法的核心瓶颈——**经验生成效率低下**。在 GAIA 基准测试中，我们将数据收集速度提升了 **14.6 倍**，使得大规模强化学习训练变得可行。
|
||
|
||
### ⚠️ 关于本教育性实验的重要说明
|
||
|
||
**GAIA（General AI Assistants Benchmark）** 是目前最具挑战性的 Agent 能力评测基准之一，也是 SOTA（State-of-the-Art）Agent 系统的竞技场。根据[论文](https://arxiv.org/abs/2508.20404)所示：
|
||
|
||
- **数据稀缺性**：GAIA validation set 仅包含 **165 个问题**，test set 约 **300 个问题**，远少于传统 RL 训练所需的数据量
|
||
- **计算资源需求**：论文中的 Qwen3-32B-AWorld 模型需要在 2 台 **8x A100 GPU 集群**上训练多天才能达到 32.23% 的性能，而这距离 SOTA 性能（80% 以上）还非常远
|
||
- **任务复杂度**：GAIA 问题涉及多模态理解、多步推理、工具链调用等，平均需要 10-20 轮交互才能完成
|
||
|
||
因此，本项目采用了教育友好的配置，使用 Qwen3-4B-Thinking-2507 作为基座模型，训练速度较快。
|
||
|
||
**本项目的目标是：**
|
||
- ✅ 演示完整的 "从实践中学习" 训练流程
|
||
- ✅ 理解 Agent-Environment 交互机制
|
||
- ✅ 实践 RL 算法（PPO/GRPO）在 Agent 训练中的应用
|
||
|
||
### 核心特性
|
||
|
||
- ⚡ **高效并发**：分布式任务执行，14.6x 数据收集加速
|
||
- 🔌 **框架无关**：支持 VeRL、OpenRLHF、AReaL、SWIFT 等主流 RL 框架
|
||
- 🛠️ **工具生态**：内置 26 个 MCP 服务器，提供 **126 个工具函数**，涵盖搜索、浏览器、代码执行、多模态处理等
|
||
- 📊 **长上下文**：支持 131K tokens 上下文，处理复杂多轮交互
|
||
- 🎯 **SOTA 性能**：Qwen3-32B-AWorld 在 GAIA 测试集达到 32.23% pass@1
|
||
|
||
---
|
||
|
||
## GAIA 环境工具生态
|
||
|
||
根据[论文](https://arxiv.org/abs/2508.20404)和 MCP Server 实现，AWorld 为 GAIA 任务提供了全面的工具支持，共计 **26 个 MCP 服务器**，**126 个工具函数**。以下是按类别的完整工具列表：
|
||
|
||
### 🌐 Web 交互工具（3 服务器，9 工具）
|
||
|
||
#### 1. Google Search Server (`googlesearch-server`)
|
||
- `search_google`: 使用 Google Custom Search API 进行网络搜索
|
||
- `get_search_capabilities`: 获取搜索服务能力信息
|
||
|
||
**典型应用**：查询实时信息、事实核查、多跳推理的起点
|
||
|
||
#### 2. Browser Use Server (`browser-server`)
|
||
- `browser_use`: 基于 LLM 的智能浏览器自动化（使用 browser-use 库）
|
||
- `get_browser_capabilities`: 获取浏览器自动化能力
|
||
|
||
**特性**：
|
||
- 自动处理机器人检测和验证码
|
||
- 支持表单填写、文件下载、内容提取
|
||
- 集成视觉理解和记忆功能
|
||
|
||
#### 3. Playwright Server (`ms-playwright`)
|
||
提供 **23 个精细化浏览器控制工具**：
|
||
- **导航**：`browser_navigate`, `browser_navigate_back`
|
||
- **交互**：`browser_click`, `browser_type`, `browser_hover`, `browser_drag`, `browser_select_option`
|
||
- **表单**：`browser_fill_form`, `browser_file_upload`
|
||
- **调试**：`browser_console_messages`, `browser_network_requests`, `browser_take_screenshot`
|
||
- **管理**：`browser_close`, `browser_resize`, `browser_tabs`, `browser_handle_dialog`
|
||
- **执行**：`browser_evaluate`, `browser_press_key`, `browser_wait_for`
|
||
- **快照**：`browser_snapshot`, `browser_install`
|
||
|
||
**对比**：`browser-server` 提供高级自动化，`ms-playwright` 提供细粒度控制
|
||
|
||
---
|
||
|
||
### 📄 文档处理工具（5 服务器，12 工具）
|
||
|
||
#### 4. Documents CSV Server (`documents-csv-server`)
|
||
- `extract_csv_content`: 提取和分析 CSV 文件内容（支持 Markdown/JSON 格式输出）
|
||
- `list_supported_formats`: 列出支持的 CSV 格式
|
||
|
||
#### 5. Documents DOCX Server (`documents-docx-server`)
|
||
- `extract_docx_content`: 提取 Word 文档内容（包括文本、表格、图片）
|
||
- `list_supported_formats`: 列出支持的 DOCX 格式
|
||
|
||
#### 6. Documents PPTX Server (`documents-pptx-server`)
|
||
- `extract_pptx_content`: 提取 PowerPoint 内容（幻灯片文本、注释、布局）
|
||
- `list_supported_formats`: 列出支持的 PPTX 格式
|
||
|
||
#### 7. Documents PDF Server (`documents-pdf-server`)
|
||
- `convert_document_to_markdown`: 将 PDF 转换为 Markdown（保留结构和格式）
|
||
|
||
#### 8. Documents TXT Server (`documents-txt-server`)
|
||
- `extract_text_content`: 提取纯文本文件内容
|
||
- `list_supported_formats`: 列出支持的文本编码
|
||
|
||
**GAIA 应用场景**：处理附件文件（GAIA 数据集 70% 的问题包含文档附件）
|
||
|
||
---
|
||
|
||
### 🎥 多媒体处理工具（3 服务器，12 工具）
|
||
|
||
#### 9. Media Audio Server (`media-audio-server`)
|
||
- `transcribe_audio`: 语音转文字（支持 Whisper API）
|
||
- `extract_audio_metadata`: 提取音频元数据（时长、比特率、采样率）
|
||
- `trim_audio`: 裁剪音频片段
|
||
- `list_supported_formats`: 列出支持的音频格式（MP3、WAV、M4A 等）
|
||
|
||
#### 10. Media Image Server (`media-image-server`)
|
||
- `extract_text_ocr`: OCR 文字识别（基于 Tesseract/Cloud Vision API）
|
||
- `analyze_image_ai`: AI 图像分析（场景识别、对象检测、描述生成）
|
||
- `get_image_metadata`: 提取图像元数据（尺寸、EXIF、拍摄时间）
|
||
|
||
#### 11. Media Video Server (`media-video-server`)
|
||
- `analyze_video`: 视频内容分析（场景分割、关键帧提取）
|
||
- `summarize_video`: 视频摘要生成
|
||
- `extract_keyframes`: 提取关键帧图像
|
||
|
||
#### 补充：独立多媒体工具
|
||
- **Audio Server** (`audio-server`): `mcp_transcribe_audio` - 高级语音转写
|
||
- **Image Server** (`image-server`): `mcp_image_recognition` - 图像识别和分类
|
||
|
||
**GAIA 应用**：约 40% 的 GAIA 问题涉及图片、音频或视频分析
|
||
|
||
---
|
||
|
||
### 💡 智能推理工具（3 服务器，6 工具）
|
||
|
||
#### 12. Intelligence Code Server (`intelligence-code-server`)
|
||
- `generate_python_code`: 生成和验证 Python 代码（用于数学计算、数据处理）
|
||
- `get_reasoning_capabilities`: 获取代码生成能力信息
|
||
|
||
#### 13. Intelligence Think Server (`intelligence-think-server`)
|
||
- `complex_problem_reasoning`: 复杂问题推理（数学证明、算法设计、逻辑谜题）
|
||
- `get_reasoning_capabilities`: 获取推理能力信息
|
||
|
||
**特点**：调用更强大的推理模型（如 GPT-4o、Claude 3.7 Sonnet）进行深度思考
|
||
|
||
#### 14. Intelligence Guard Server (`intelligence-guard-server`)
|
||
- `guarding_reasoning_process`: 推理过程保护和验证（防止幻觉、检查逻辑一致性）
|
||
- `get_guarding_capabilities`: 获取保护能力信息
|
||
|
||
**论文亮点**：这些"思考工具"使小模型能够调用大模型的推理能力，实现"站在巨人的肩膀上"
|
||
|
||
---
|
||
|
||
### 💻 代码执行工具（3 服务器，24 工具）
|
||
|
||
#### 15. Terminal Server (`terminal-server`)
|
||
- `execute_command`: 执行终端命令（Python、bash、系统命令）
|
||
- `get_command_history`: 获取命令执行历史
|
||
- `get_terminal_capabilities`: 获取终端能力信息
|
||
|
||
**安全特性**：命令白名单、超时控制、输出截断
|
||
|
||
#### 16. E2B Code Server (`e2b-code-server`)
|
||
- `e2b_upload_file`: 上传文件到沙箱
|
||
- `e2b_run_code`: 在隔离沙箱中执行代码（支持 Python、JavaScript、多种语言）
|
||
|
||
**优势**：完全隔离的执行环境，防止恶意代码影响主系统
|
||
|
||
#### 17. Terminal Controller (`terminal-controller`)
|
||
提供 **10 个高级终端管理工具**：
|
||
- `execute_command`, `get_command_history`, `get_current_directory`, `change_directory`
|
||
- `list_directory`, `write_file`, `read_file`
|
||
- `insert_file_content`, `delete_file_content`, `update_file_content`
|
||
|
||
**区别**：`terminal-server` 专注命令执行，`terminal-controller` 提供文件系统管理
|
||
|
||
---
|
||
|
||
### 📂 文件系统工具（1 服务器，14 工具）
|
||
|
||
#### 18. Filesystem Server (`filesystem-server`)
|
||
完整的文件操作能力：
|
||
- **读取**：`read_file`, `read_text_file`, `read_media_file`, `read_multiple_files`
|
||
- **写入**：`write_file`, `edit_file`
|
||
- **管理**：`create_directory`, `move_file`, `get_file_info`
|
||
- **搜索**：`search_files`, `list_directory`, `list_directory_with_sizes`, `directory_tree`
|
||
- **权限**：`list_allowed_directories` - 列出允许访问的目录
|
||
|
||
**GAIA 应用**：访问数据集附件（`/root/workspace/gaia_dataset/` 目录）
|
||
|
||
---
|
||
|
||
### 📊 Excel 处理工具（1 服务器，29 工具）
|
||
|
||
#### 19. Excel Server (`excel`)
|
||
提供企业级 Excel 操作能力：
|
||
|
||
**数据操作（9 个）**：
|
||
- `read_data_from_excel`, `write_data_to_excel`
|
||
- `insert_rows`, `insert_columns`, `delete_sheet_rows`, `delete_sheet_columns`
|
||
- `copy_range`, `delete_range`, `validate_excel_range`
|
||
|
||
**工作簿/表管理（7 个）**：
|
||
- `create_workbook`, `create_worksheet`, `copy_worksheet`
|
||
- `delete_worksheet`, `rename_worksheet`, `get_workbook_metadata`
|
||
|
||
**高级功能（13 个）**：
|
||
- `apply_formula`, `validate_formula_syntax`
|
||
- `format_range`, `create_chart`, `create_pivot_table`, `create_table`
|
||
- `merge_cells`, `unmerge_cells`, `get_merged_cells`
|
||
- `get_data_validation_info`
|
||
|
||
**GAIA 典型任务**：分析复杂的 Excel 数据表、计算统计指标、生成图表
|
||
|
||
---
|
||
|
||
### 🔍 知识检索工具（3 服务器，11 工具）
|
||
|
||
#### 20. Wikipedia Server (`wiki-server`)
|
||
- `search_wikipedia`: 搜索维基百科词条
|
||
- `get_article_content`: 获取完整文章内容
|
||
- `get_article_summary`: 获取文章摘要
|
||
- `get_article_categories`: 获取文章分类
|
||
- `get_article_links`: 获取文章链接
|
||
- `get_article_history`: 获取文章历史版本（用于时间敏感问题）
|
||
- `get_wikipedia_capabilities`: 获取 Wikipedia 服务能力
|
||
|
||
**特色功能**：支持多语言、历史版本查询（GAIA 中有"某年某月的人口数据"类问题）
|
||
|
||
#### 21. ArXiv Server (`parxiv-server`)
|
||
- `search_papers`: 搜索 arXiv 论文
|
||
- `get_paper_details`: 获取论文详细信息（摘要、作者、引用）
|
||
- `download_paper`: 下载论文 PDF
|
||
- `get_arxiv_capabilities`: 获取 arXiv 服务能力
|
||
- `get_categories`: 获取 arXiv 分类列表
|
||
|
||
#### 22. Wayback Machine Server (`wayback-server`)
|
||
- `list_archived_versions`: 列出网页的历史存档版本
|
||
- `get_archived_content`: 获取特定时间点的网页内容
|
||
- `get_wayback_capabilities`: 获取 Wayback Machine 能力
|
||
|
||
**GAIA 应用**：回答"2015 年某网站上的信息"这类历史查询问题
|
||
|
||
---
|
||
|
||
### 📥 其他实用工具（3 服务器，3 工具）
|
||
|
||
#### 23. Download Server (`download-server`)
|
||
- `download_file`: 下载网络文件到本地
|
||
- `get_download_capabilities`: 获取下载服务能力
|
||
|
||
#### 24. Read Web Server (`readweb-server`)
|
||
- 提供网页内容读取能力（具体工具由 MCP 配置定义）
|
||
|
||
#### 25. Google Search Alternative (`google-search`)
|
||
- `search`: 简化的搜索接口
|
||
- `read_webpage`: 读取网页内容
|
||
|
||
---
|
||
|
||
### 工具统计总结
|
||
|
||
| 类别 | 服务器数 | 工具数 | 关键能力 |
|
||
|------|---------|--------|---------|
|
||
| **Web 交互** | 3 | 32 | 搜索、智能浏览、精细控制 |
|
||
| **文档处理** | 5 | 12 | CSV、Word、PPT、PDF、TXT |
|
||
| **多媒体** | 5 | 14 | 音频转写、OCR、图像/视频分析 |
|
||
| **智能推理** | 3 | 6 | 代码生成、复杂推理、验证 |
|
||
| **代码执行** | 3 | 36 | 终端命令、沙箱执行、文件管理 |
|
||
| **文件系统** | 1 | 14 | 完整文件操作能力 |
|
||
| **Excel** | 1 | 29 | 企业级表格处理 |
|
||
| **知识检索** | 3 | 11 | Wikipedia、ArXiv、历史网页 |
|
||
| **其他** | 2 | 3 | 文件下载、网页读取 |
|
||
| **总计** | **26** | **126** | **涵盖 GAIA 所需的全部能力** |
|
||
|
||
### 工具调用示例（来自训练日志）
|
||
|
||
```python
|
||
# Google 搜索示例
|
||
Tool call: aworld-mcp__search_google
|
||
Tool args: {"query": "Wyoming population 2020", "num_results": 5}
|
||
Result: {"success": true, "results": [{"title": "Wyoming - Census Bureau", "snippet": "576,851..."}]}
|
||
|
||
# 文件系统示例
|
||
Tool call: aworld-mcp__list_directory
|
||
Tool args: {"path": "/root/workspace/gaia_dataset/2023/test"}
|
||
Result: ["[FILE] 021a5339-...-bd9b-9368b3efda7a.pdf", "[FILE] 03c577c9-...-f8f598de14c1.mp3", ...]
|
||
|
||
# CSV 处理示例（缺少 tabulate 依赖时会报错）
|
||
Tool call: aworld-mcp__extract_csv_content
|
||
Tool args: {"file_path": "/root/workspace/gaia_dataset/2023/test/52e8ce1c-...-67d1648779b9.csv"}
|
||
Error: "CSV extraction failed: Missing optional dependency 'tabulate'"
|
||
|
||
# Wikipedia 历史查询示例
|
||
Tool call: aworld-mcp__get_article_history
|
||
Tool args: {"title": "Cat", "date": "20191231", "language": "en"}
|
||
Result: {...historical Wikipedia content...}
|
||
```
|
||
|
||
---
|
||
|
||
## 核心架构
|
||
|
||
AWorld Train 采用四阶段训练流水线：
|
||
|
||
|
||
|
||
```
|
||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||
│ Environment │───▶│ Agent │───▶│ Adapter │───▶│ Training │
|
||
│ Setup │ │Construction │ │ Layer │ │ Framework │
|
||
└─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘
|
||
(MCP) (AWorld) (VeRL) (PPO/GRPO)
|
||
```
|
||
|
||
1. **环境配置**：部署 GAIA MCP Server，提供 20+ 工具能力
|
||
2. **Agent 构建**：实现自定义 AgentLoop，定义决策逻辑
|
||
3. **适配器集成**：统一接口，对接 RL 训练框架
|
||
4. **训练执行**：配置奖励函数和超参数，启动训练任务
|
||
|
||
---
|
||
|
||
## 快速开始
|
||
|
||
### 系统要求
|
||
|
||
| 组件 | 要求 |
|
||
|------|------|
|
||
| **操作系统** | Linux (推荐) / macOS / Windows |
|
||
| **硬件** | 最低 4 CPU 核心 + 8GB RAM<br>训练推荐 8x A100/H100 GPU |
|
||
| **软件** | Docker, NVIDIA Driver, CUDA 12.1+ |
|
||
|
||
### 安装训练框架
|
||
|
||
以 VeRL 为例，安装步骤如下：
|
||
|
||
```bash
|
||
# 可在任意目录运行；先设置本书仓库绝对路径。
|
||
export BOOK_ROOT=/absolute/path/to/ai-agent-book
|
||
|
||
# 1. 安装系统依赖（Ubuntu/Debian）
|
||
sudo apt-get update
|
||
sudo apt-get install -y build-essential git wget
|
||
|
||
# 2. 安装 CUDA Toolkit（匹配你的 GPU Driver）
|
||
# 参考：https://developer.nvidia.com/cuda-downloads
|
||
|
||
# 3. 安装 PyTorch（匹配 CUDA 版本）
|
||
pip install torch==2.4.0 torchvision==0.19.0 --index-url https://download.pytorch.org/whl/cu121
|
||
|
||
# 4. 将本书已核验的 AWorld fork 克隆到规范配套路径
|
||
git clone https://github.com/bojieli/AWorld.git "${BOOK_ROOT}/chapter8/AWorld"
|
||
git -C "${BOOK_ROOT}/chapter8/AWorld" checkout --detach a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec
|
||
|
||
# 5. 克隆并安装已核验的 VeRL 后端
|
||
git clone https://github.com/bojieli/verl.git "${BOOK_ROOT}/chapter8/verl"
|
||
git -C "${BOOK_ROOT}/chapter8/verl" checkout --detach 1593fc3a8cf894debdc3dece2a23ed739c282789
|
||
pip install -e "${BOOK_ROOT}/chapter8/verl"
|
||
```
|
||
|
||
**重要提示**：VeRL 的某些依赖需要在 CUDA 环境下编译，请确保先完成步骤 1-2。
|
||
|
||
---
|
||
|
||
### 配置 GAIA 环境
|
||
|
||
GAIA 环境通过 Docker 部署，提供 MCP（Model Context Protocol）服务。
|
||
|
||
#### 1. 下载数据集
|
||
|
||
```bash
|
||
# 从 Hugging Face 下载 GAIA 数据集
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/docker
|
||
mkdir -p gaia_dataset
|
||
|
||
# 使用 Hugging Face CLI 下载（需要先 pip install huggingface_hub）
|
||
huggingface-cli download gaia-benchmark/GAIA --repo-type dataset --local-dir gaia_dataset
|
||
```
|
||
|
||
#### 2. 配置环境变量
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/mcp_servers
|
||
cp .env_template .env
|
||
|
||
# 编辑 .env 文件，配置必要的 API 密钥
|
||
vim .env
|
||
```
|
||
|
||
`.env` 文件示例（部分字段）：
|
||
|
||
```bash
|
||
# OpenAI API（用于 intelligence-code-server 等）
|
||
OPENAI_API_KEY=your-openai-api-key
|
||
|
||
# Google Search API（用于 googlesearch-server）
|
||
GOOGLE_API_KEY=your-google-api-key
|
||
GOOGLE_CSE_ID=your-search-engine-id
|
||
|
||
# E2B API（用于代码执行沙箱）
|
||
E2B_API_KEY=your-e2b-api-key
|
||
```
|
||
|
||
#### 3. 启动 MCP Server
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env
|
||
bash run-local.sh
|
||
```
|
||
|
||
启动成功后，你将看到类似输出：
|
||
|
||
```
|
||
Starting services...
|
||
DISPLAY=:0
|
||
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
|
||
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
|
||
...
|
||
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
|
||
```
|
||
|
||
MCP Server 提供两个服务：
|
||
- **MCP 接口**：`http://localhost:8080/mcp`（Agent 工具调用）
|
||
- **VNC 界面**：`http://localhost:5901/vnc.html?autoconnect=true`（可视化调试）
|
||
|
||
#### 4. 验证环境
|
||
|
||
```bash
|
||
# 设置环境变量
|
||
export MCP_SERVER_URL=http://localhost:8080/mcp
|
||
|
||
# 测试连接（Python）
|
||
python3 << EOF
|
||
from train.adapter.verl.common import get_agent_tool_env_and_servers
|
||
|
||
config, servers = get_agent_tool_env_and_servers()
|
||
print(f"Available servers: {len(servers)}")
|
||
print(f"Sample servers: {servers[:5]}")
|
||
EOF
|
||
```
|
||
|
||
预期输出：
|
||
|
||
```
|
||
Available servers: 20
|
||
Sample servers: ['readweb-server', 'browser-server', 'documents-csv-server', ...]
|
||
```
|
||
|
||
---
|
||
|
||
## 构建自定义 Agent
|
||
|
||
### 实现 AgentLoop
|
||
|
||
创建自定义 Agent 的核心是继承 `AworldAgentLoop` 并实现 `build_agents()` 方法。以下是 GAIA Agent 的完整示例：
|
||
|
||
```python
|
||
# train/examples/train_gaia_with_aworld_verl/custom_agent_loop.py
|
||
|
||
from aworld.agents.llm_agent import Agent
|
||
from aworld.config import AgentConfig
|
||
from train.adapter.verl.aworld_agent_loop import AworldAgentLoop
|
||
from train.adapter.verl.common import get_agent_tool_env_and_servers
|
||
|
||
class GaiaAgentLoop(AworldAgentLoop):
|
||
"""GAIA 任务的自定义 Agent Loop"""
|
||
|
||
def build_agents(self):
|
||
# 获取 MCP 环境配置和可用服务列表
|
||
gaia_env_config, gaia_env_servers = get_agent_tool_env_and_servers()
|
||
|
||
# 构建 Agent 实例
|
||
return Agent(
|
||
conf=AgentConfig(
|
||
# LLM 服务地址由 VeRL 动态管理
|
||
llm_base_url=self.get_llm_server_address(),
|
||
llm_model_name=self.get_llm_server_model_name(),
|
||
llm_api_key="dummy", # VeRL 内部通信不需要真实 API Key
|
||
),
|
||
name="gaia_super_agent",
|
||
|
||
# 系统提示（定义 Agent 角色和能力）
|
||
system_prompt="""You are a helpful AI assistant specialized in solving complex tasks.
|
||
You have access to various tools including web search, code execution, and file analysis.
|
||
When given a task, break it down into steps and use available tools systematically.
|
||
Always provide your final answer in <answer>...</answer> tags.""",
|
||
|
||
# MCP 工具配置
|
||
mcp_config=gaia_env_config,
|
||
mcp_servers=gaia_env_servers,
|
||
)
|
||
```
|
||
|
||
### 配置 agent.yaml
|
||
|
||
在 `train/examples/train_gaia_with_aworld_verl/agent.yaml` 中注册你的 AgentLoop：
|
||
|
||
```yaml
|
||
- name: gaia_agent
|
||
_target_: train.examples.train_gaia_with_aworld_verl.custom_agent_loop.GaiaAgentLoop
|
||
```
|
||
|
||
### 高级场景
|
||
|
||
#### 多 Agent 系统
|
||
|
||
```python
|
||
from aworld.swarms.swarm import Swarm
|
||
|
||
class MultiAgentLoop(AworldAgentLoop):
|
||
def build_agents(self):
|
||
config, servers = get_agent_tool_env_and_servers()
|
||
|
||
# 创建专业化 Agent
|
||
researcher = Agent(
|
||
conf=AgentConfig(...),
|
||
name="researcher",
|
||
system_prompt="You are a research specialist...",
|
||
mcp_servers=["googlesearch-server", "wiki-server"]
|
||
)
|
||
|
||
coder = Agent(
|
||
conf=AgentConfig(...),
|
||
name="coder",
|
||
system_prompt="You are a coding expert...",
|
||
mcp_servers=["e2b-code-server", "terminal-server"]
|
||
)
|
||
|
||
# 构建 Swarm
|
||
return Swarm(
|
||
agents=[researcher, coder],
|
||
coordinator=researcher # 主协调 Agent
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
## 准备训练
|
||
|
||
### 1. 准备数据集
|
||
|
||
运行数据集生成脚本，将 GAIA 数据转换为训练格式：
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/train/examples/train_gaia_with_aworld_verl/gaia_datasets
|
||
|
||
python create_dataset.py \
|
||
--dataset_path ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/docker/gaia_dataset \
|
||
--output_dir ~/datasets \
|
||
--train_size 300 \
|
||
--test_size 100
|
||
```
|
||
|
||
这将生成：
|
||
- `~/datasets/train.parquet`：300 条训练样本
|
||
- `~/datasets/test.parquet`：100 条测试样本
|
||
|
||
数据格式（Parquet）：
|
||
|
||
| 字段 | 类型 | 说明 |
|
||
|------|------|------|
|
||
| `prompt` | List[Dict] | 格式化的聊天消息 `[{"role": "user", "content": "..."}]` |
|
||
| `data_source` | str | 数据来源标识 `"gaia"` |
|
||
| `ability` | str | 能力类别 `"agi"` |
|
||
| `reward_model` | Dict | 奖励配置 `{"style": "GAIA", "ground_truth": "..."}` |
|
||
| `extra_info` | Dict | 额外元数据（task_id, level 等） |
|
||
| `agent_name` | str | 目标 Agent 名称 |
|
||
|
||
### 2. 配置奖励函数
|
||
|
||
在 `train/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py` 中定义奖励逻辑：
|
||
|
||
```python
|
||
import re
|
||
from aworld.logs.util import logger
|
||
|
||
def gaia_reward_func(data_source, solution_str, ground_truth, extra_info=None):
|
||
"""
|
||
GAIA 任务的奖励函数
|
||
|
||
Args:
|
||
data_source: 数据来源标识
|
||
solution_str: Agent 生成的完整解答
|
||
ground_truth: 标准答案
|
||
extra_info: 额外信息（如 task_id, level）
|
||
|
||
Returns:
|
||
float: 奖励值（0.0 或 1.0）
|
||
"""
|
||
# 从 solution_str 中提取 <answer>...</answer> 标签内容
|
||
pattern = r'<answer>(.*?)</answer>'
|
||
match = re.search(pattern, solution_str, re.DOTALL | re.MULTILINE)
|
||
|
||
if not match:
|
||
logger.warning("No answer tag found in solution")
|
||
return 0.0
|
||
|
||
answer = match.group(1).strip()
|
||
logger.info(f"Extracted answer: {answer}, Ground truth: {ground_truth}")
|
||
|
||
# 使用 GAIA 标准评分器（支持数字、列表、字符串）
|
||
if question_scorer(answer, ground_truth):
|
||
return 1.0
|
||
else:
|
||
return 0.0
|
||
|
||
def question_scorer(model_answer: str, ground_truth: str) -> bool:
|
||
"""GAIA 标准评分逻辑（省略详细实现）"""
|
||
# 支持数字比较、列表比较、字符串归一化比较
|
||
# 详见完整代码
|
||
...
|
||
```
|
||
|
||
### 3. 配置训练脚本
|
||
|
||
编辑 `run.sh`，配置关键参数：
|
||
|
||
```bash
|
||
#!/usr/bin/env bash
|
||
set -xeuo pipefail
|
||
|
||
# ============ 集群拓扑 ============
|
||
export GPUS_PER_NODE=${GPUS_PER_NODE:-8}
|
||
export NNODES=${NNODES:-1}
|
||
|
||
# ============ 模型和数据 ============
|
||
model_path=${model_path:-Qwen/Qwen3-4B-Thinking-2507}
|
||
train_files=$DATA_ROOT/datasets/train.parquet
|
||
test_files=$DATA_ROOT/datasets/test.parquet
|
||
|
||
# ============ 自定义配置 ============
|
||
path_to_train="/root/AWorld/train"
|
||
agent_loop_config_path=${path_to_train}/examples/train_gaia_with_aworld_verl/agent.yaml
|
||
reward_fn_file_path=${path_to_train}/examples/train_gaia_with_aworld_verl/metrics/gaia_reward_function.py
|
||
reward_fn_name=gaia_reward_func
|
||
|
||
# ============ 训练超参数 ============
|
||
# PPO 算法配置
|
||
adv_estimator=grpo # 使用 Group Relative Policy Optimization
|
||
clip_ratio_low=0.2 # PPO clip 下界
|
||
clip_ratio_high=0.28 # PPO clip 上界
|
||
actor_lr=1e-6 # Actor 学习率
|
||
|
||
# 长上下文配置（AWorld 最新优化）
|
||
max_turns=32 # 最大交互轮数（从 8 提升到 32）
|
||
max_prompt_length=4096 # 提示最大长度（从 1024 提升到 4096）
|
||
max_response_length=4096 # 响应最大长度（从 2048 提升到 4096）
|
||
|
||
# 批次配置
|
||
train_batch_size=32 # 训练批次大小（从 1 提升到 32）
|
||
ppo_mini_batch_size=8 # PPO mini-batch 大小（4 个梯度更新）
|
||
n_resp_per_prompt=16 # 每个提示采样 16 个响应（从 1 提升）
|
||
n_resp_per_prompt_val=16 # 验证时采样数
|
||
|
||
# ============ MCP Server ============
|
||
export MCP_SERVER_URL=${MCP_SERVER_URL:-http://localhost:8080/mcp}
|
||
|
||
# ============ 性能优化 ============
|
||
export VLLM_USE_V1=1 # 使用 vLLM v1 引擎
|
||
export VLLM_ATTENTION_BACKEND=FLASH_ATTN # FlashAttention-2
|
||
infer_tp=1 # Tensor Parallel 大小
|
||
train_sp=1 # Sequence Parallel 大小
|
||
offload=true # 参数卸载到 CPU
|
||
|
||
# ============ VeRL 训练命令 ============
|
||
python3 -m verl.trainer.main_ppo \
|
||
algorithm.adv_estimator=$adv_estimator \
|
||
data.train_files="['$train_files']" \
|
||
data.val_files="['$test_files']" \
|
||
data.return_raw_chat=true \
|
||
data.train_batch_size=$train_batch_size \
|
||
data.max_prompt_length=$max_prompt_length \
|
||
data.max_response_length=$max_response_length \
|
||
actor_rollout_ref.model.path="$model_path" \
|
||
actor_rollout_ref.rollout.multi_turn.max_user_turns=$max_turns \
|
||
actor_rollout_ref.rollout.multi_turn.max_assistant_turns=$max_turns \
|
||
actor_rollout_ref.rollout.max_model_len=131072 \
|
||
actor_rollout_ref.rollout.max_num_batched_tokens=131072 \
|
||
actor_rollout_ref.rollout.gpu_memory_utilization=0.9 \
|
||
actor_rollout_ref.rollout.agent.agent_loop_config_path=$agent_loop_config_path \
|
||
custom_reward_function.path="${reward_fn_file_path}" \
|
||
custom_reward_function.name="${reward_fn_name}" \
|
||
trainer.logger=['console','wandb'] \
|
||
trainer.experiment_name=aworld_train_qwen3_4b \
|
||
trainer.save_freq=5 \
|
||
trainer.test_freq=5 \
|
||
+trainer.num_steps=300
|
||
```
|
||
|
||
---
|
||
|
||
## 启动训练
|
||
|
||
### 单机训练
|
||
|
||
```bash
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/train/examples/train_gaia_with_aworld_verl
|
||
|
||
# 启动训练（8卡 GPU）
|
||
export DATA_ROOT=~/datasets
|
||
export GPUS_PER_NODE=8
|
||
bash run.sh
|
||
```
|
||
|
||
### 多机训练（Slurm）
|
||
|
||
```bash
|
||
# 提交 Slurm 作业（2 节点，每节点 8 卡）
|
||
sbatch <<EOF
|
||
#!/bin/bash
|
||
#SBATCH --job-name=aworld-train
|
||
#SBATCH --nodes=2
|
||
#SBATCH --ntasks-per-node=1
|
||
#SBATCH --gpus-per-node=8
|
||
#SBATCH --time=48:00:00
|
||
|
||
export DATA_ROOT=/path/to/datasets
|
||
export NNODES=2
|
||
export GPUS_PER_NODE=8
|
||
|
||
srun bash run.sh
|
||
EOF
|
||
```
|
||
|
||
### 训练监控
|
||
|
||
训练过程支持多种日志后端：
|
||
|
||
```bash
|
||
# 1. 控制台输出
|
||
# 实时查看训练指标（loss, reward, KL divergence 等）
|
||
|
||
# 2. WandB 可视化（推荐）
|
||
# 访问 https://wandb.ai/<your-project>/aworld_train_qwen3_4b
|
||
|
||
# 3. TensorBoard
|
||
tensorboard --logdir ~/datasets/checkpoint/aworld_train_qwen3_4b
|
||
```
|
||
|
||
关键监控指标：
|
||
|
||
| 指标 | 说明 | 目标值 |
|
||
|------|------|--------|
|
||
| `reward/mean` | 平均奖励 | 逐步上升至 0.3+ |
|
||
| `reward/max` | 最大奖励 | 达到 1.0 |
|
||
| `actor/loss` | Actor 损失 | 稳定下降 |
|
||
| `rollout/response_length` | 响应长度 | 根据任务调整 |
|
||
| `rollout/num_turns` | 平均轮数 | 高效利用工具（5-15 轮） |
|
||
|
||
---
|
||
|
||
## 最新优化
|
||
|
||
基于最新代码修改（commit `a52d61d6`），我们进行了以下关键优化：
|
||
|
||
### 1. 长上下文支持（verl_provider.py）
|
||
|
||
```python
|
||
# 新增 max_model_len 参数动态配置
|
||
self.max_model_len = params.get("max_model_len", 24576)
|
||
```
|
||
|
||
**影响**：支持最长 131K tokens 的上下文窗口，处理复杂多轮对话和大规模工具调用历史。
|
||
|
||
### 2. 数据格式优化（create_dataset.py）
|
||
|
||
```python
|
||
# 提示格式化为聊天消息列表（适配 VeRL return_raw_chat 模式）
|
||
rl_dataset["prompt"].append([{"role": "user", "content": data["Question"]}])
|
||
```
|
||
|
||
**影响**：与 VeRL 的聊天模板系统无缝对接，避免格式转换开销。
|
||
|
||
### 3. 超参数调优（run.sh）
|
||
|
||
| 参数 | 旧值 | 新值 | 提升 |
|
||
|------|------|------|------|
|
||
| `max_turns` | 8 | 32 | 4x 交互深度 |
|
||
| `max_prompt_length` | 1024 | 4096 | 4x 输入容量 |
|
||
| `max_response_length` | 2048 | 4096 | 2x 输出容量 |
|
||
| `train_batch_size` | 1 | 32 | 32x 训练效率 |
|
||
| `n_resp_per_prompt` | 1 | 16 | 16x 样本多样性 |
|
||
|
||
**影响**：
|
||
- **更深层推理**：允许 Agent 进行更长时间的工具链调用和思考
|
||
- **更高效训练**：大批次训练加速收敛，多样本采样提升泛化能力
|
||
- **更稳定优化**：`ppo_mini_batch_size=8` 实现 4 次梯度更新，平衡训练稳定性和效率
|
||
|
||
### 4. 内存优化
|
||
|
||
```bash
|
||
# vLLM 配置
|
||
actor_rollout_ref.rollout.max_model_len=131072 # 模型上下文长度
|
||
actor_rollout_ref.rollout.max_num_batched_tokens=131072 # 批处理 token 数
|
||
actor_rollout_ref.rollout.gpu_memory_utilization=0.9 # GPU 内存利用率
|
||
```
|
||
|
||
**影响**：在 A100 80GB 上支持 32K+ tokens 的并发推理，充分利用 GPU 资源。
|
||
|
||
### 5. 新增 Qwen3-30B-A3B 训练脚本
|
||
|
||
```bash
|
||
# run_qwen3_30b_a3b.sh
|
||
infer_tp=4 # Tensor Parallel（推理）
|
||
train_sp=8 # Sequence Parallel（训练）
|
||
```
|
||
|
||
**影响**：支持更大规模模型训练，利用模型并行技术突破单卡限制。
|
||
|
||
---
|
||
|
||
## 故障排查
|
||
|
||
### Agent 训练过程输出示例
|
||
|
||
#### ✅ 正常推理流程
|
||
|
||
```
|
||
(AgentLoopWorker pid=448354) [agent] Content: Okay, let's see. So the user is asking about the population difference between the two states that have both Carl's Jr. and Hardee's fast food restaurants...
|
||
(AgentLoopWorker pid=448354) [agent] Tool call: aworld-mcp__search_google - ID: chatcmpl-tool-94b30baa
|
||
(AgentLoopWorker pid=448354) [agent] Tool args: {"query": "Wyoming population 2020", "num_results": 5}
|
||
(AgentLoopWorker pid=448354) [agent] Content: ["{\"success\": true, \"message\": {\"query\": \"Wyoming population 2020\", \"results\": [{\"title\": \"Wyoming - Census Bureau Profile\", \"snippet\": \"576,851. The Total Population for Wyoming is 576,851...\"...}
|
||
```
|
||
|
||
**说明**：Agent 正确调用工具并接收结果，推理链路完整。
|
||
|
||
#### ✅ 文件列表成功
|
||
|
||
```
|
||
(AgentLoopWorker pid=448358) [agent] Content: ["[FILE] 021a5339-744f-42b7-bd9b-9368b3efda7a.pdf\n[FILE] 03c577c9-4227-48a9-9b75-f8f598de14c1.mp3\n[FILE] 063800f6-8832-4856-972b-17b877612533.png\n..."]
|
||
(AgentLoopWorker pid=448358) [agent] Content: Okay, let's try to figure out how many horror titles are overdue based on the inventory file...
|
||
```
|
||
|
||
**说明**：文件系统工具正常工作，Agent 能够访问数据集文件。
|
||
|
||
---
|
||
|
||
### GAIA MCP Server 输出示例
|
||
|
||
#### ✅ 正常启动输出
|
||
|
||
```bash
|
||
$ docker logs gaia-mcp-server-gaia-mcp-server-1 -f
|
||
|
||
Starting services...
|
||
DISPLAY=:0
|
||
2025-10-06 05:20:42,368 - __main__ - INFO - Starting MCP Server Proxy...
|
||
2025-10-06 05:20:42,370 - mcp_server_proxy.mcp_server_proxy - INFO - Loaded MCP tool schema: mcp_tool_schema=
|
||
readweb-server:
|
||
browser-server:
|
||
- get_browser_capabilities
|
||
- browser_use
|
||
documents-csv-server:
|
||
- extract_csv_content
|
||
- list_supported_formats
|
||
googlesearch-server:
|
||
- search_google
|
||
- get_search_capabilities
|
||
...
|
||
2025-10-06 05:20:42,373 - mcp_server_proxy.mcp_server_proxy - INFO - Added MCP server executor: googlesearch-server
|
||
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
|
||
```
|
||
|
||
**检查点**：
|
||
- ✅ `Starting MCP Server Proxy` 出现
|
||
- ✅ 20+ 工具服务器被加载（`Added MCP server executor`）
|
||
- ✅ Uvicorn 在 8080 端口监听
|
||
|
||
#### ✅ 训练时正常请求
|
||
|
||
```
|
||
INFO: 208.64.254.164:36416 - "POST /mcp HTTP/1.1" 200 OK
|
||
2025-10-06 05:09:23,880 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
|
||
2025-10-06 05:09:27,006 - mcp.server.lowlevel.server - INFO - Processing request of type CallToolRequest
|
||
[10/06/25 05:09:27] INFO 🔍 Searching Google for: 'Speaker of the House that passed act...'
|
||
INFO ✅ Found 5 results in 0.42s
|
||
```
|
||
|
||
**说明**：Agent 正常调用 Google 搜索工具，请求响应快速（< 1s）。
|
||
|
||
---
|
||
|
||
### ❌ 常见错误及解决方案
|
||
|
||
#### 1. CSV 提取失败（缺少依赖）
|
||
|
||
```
|
||
ERROR CSV extraction failed: Missing optional dependency 'tabulate'.
|
||
Use pip or conda to install tabulate.
|
||
```
|
||
|
||
**原因**：pandas 的 `to_markdown()` 功能需要 `tabulate` 库。
|
||
|
||
**解决方案**：
|
||
|
||
```bash
|
||
# 进入 MCP Server Docker 容器
|
||
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash
|
||
|
||
# 安装缺失依赖
|
||
cd /app/mcp_servers/documents_server
|
||
source .venv/bin/activate
|
||
pip install tabulate
|
||
|
||
# 重启容器
|
||
exit
|
||
docker restart gaia-mcp-server-gaia-mcp-server-1
|
||
```
|
||
|
||
#### 2. OpenAI API 密钥错误
|
||
|
||
```
|
||
WARNING coding failed: Error code: 401 - {'error': {'message': 'Incorrect API key provided: [redacted-openrouter-key]', 'type': 'invalid_request_error', 'code': 'invalid_api_key'}}
|
||
```
|
||
|
||
**原因**：`intelligence-code-server` 等工具需要 OpenAI API 密钥生成代码。
|
||
|
||
**解决方案**：
|
||
|
||
```bash
|
||
# 方法 1：配置真实 OpenAI API Key
|
||
vim ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/mcp_servers/.env
|
||
# 添加：OPENAI_API_KEY=your-openai-api-key
|
||
|
||
# 方法 2：使用 OpenRouter 等兼容服务
|
||
# .env 中配置：
|
||
LLM_BASE_URL=https://openrouter.ai/api/v1
|
||
OPENAI_API_KEY=your-openrouter-api-key
|
||
|
||
# 重启服务
|
||
cd ${BOOK_ROOT}/chapter8/AWorld/env
|
||
bash run-local.sh
|
||
```
|
||
|
||
#### 3. Wikipedia API 429 限流
|
||
|
||
```
|
||
ERROR Wikipedia summary retrieval error:
|
||
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
|
||
```
|
||
|
||
**原因**：Wikipedia API 限流（429 Too Many Requests），但 Python 库没有正确处理。
|
||
|
||
**解决方案**：
|
||
|
||
```bash
|
||
# 方法 1：降低并发请求（调整训练参数）
|
||
train_batch_size=16 # 从 32 降低到 16
|
||
n_resp_per_prompt=8 # 从 16 降低到 8
|
||
|
||
# 方法 2：使用代理或切换 Wikipedia 镜像
|
||
# 在 wiki_server/.env 中配置：
|
||
WIKIPEDIA_BASE_URL=https://en.wikipedia.org/w/api.php
|
||
|
||
# 方法 3：添加请求重试逻辑（需修改代码）
|
||
# 在 wiki_server/src/wiki.py 中添加 exponential backoff
|
||
```
|
||
|
||
#### 4. 工具执行超时
|
||
|
||
```
|
||
2025-10-06 05:21:08,548 - mcp_server_proxy.mcp_server_executor - INFO - Starting tool server browser-server...
|
||
[10 秒后无响应]
|
||
```
|
||
|
||
**原因**：浏览器工具（Playwright）启动慢或资源不足。
|
||
|
||
**解决方案**：
|
||
|
||
```bash
|
||
# 增加超时配置
|
||
vim ${BOOK_ROOT}/chapter8/AWorld/env/gaia-mcp-server/mcp_servers/.env
|
||
# 添加：
|
||
TOOL_EXECUTION_TIMEOUT=120 # 从默认 60s 增加到 120s
|
||
|
||
# 预热浏览器环境
|
||
docker exec -it gaia-mcp-server-gaia-mcp-server-1 bash
|
||
cd /app/mcp_servers/browser_server
|
||
uv run python -c "from playwright.sync_api import sync_playwright; sync_playwright().start()"
|
||
```
|
||
|
||
#### 5. vLLM OOM（显存不足）
|
||
|
||
```
|
||
RuntimeError: CUDA out of memory. Tried to allocate 20.00 GiB (GPU 0; 79.35 GiB total capacity; 75.12 GiB already allocated; 2.31 GiB free; 78.90 GiB reserved in total by PyTorch)
|
||
```
|
||
|
||
**解决方案**：
|
||
|
||
```bash
|
||
# 方法 1：降低批次大小
|
||
train_batch_size=16 # 从 32 降低
|
||
ppo_mini_batch_size=4 # 从 8 降低
|
||
|
||
# 方法 2：降低 GPU 内存利用率
|
||
actor_rollout_ref.rollout.gpu_memory_utilization=0.75 # 从 0.9 降低
|
||
|
||
# 方法 3：启用 CPU offload
|
||
actor_rollout_ref.actor.fsdp_config.param_offload=true
|
||
actor_rollout_ref.actor.fsdp_config.optimizer_offload=true
|
||
|
||
# 方法 4：使用更大的 Tensor Parallel
|
||
infer_tp=4 # 从 1 增加到 4（需要 4 卡）
|
||
```
|
||
|
||
---
|
||
|
||
### 调试技巧
|
||
|
||
#### 1. 启用详细日志
|
||
|
||
```bash
|
||
# VeRL 训练日志
|
||
export RAY_LOGGING_LEVEL=DEBUG
|
||
export HYDRA_FULL_ERROR=1
|
||
|
||
# MCP Server 日志
|
||
docker logs -f gaia-mcp-server-gaia-mcp-server-1 --tail 100
|
||
|
||
# vLLM 推理日志
|
||
export VLLM_LOGGING_LEVEL=DEBUG
|
||
```
|
||
|
||
#### 2. 单步调试 Agent
|
||
|
||
```python
|
||
# test_agent_debug.py
|
||
from train.examples.train_gaia_with_aworld_verl.custom_agent_loop import GaiaAgentLoop
|
||
|
||
# 创建 AgentLoop（不启动训练）
|
||
loop = GaiaAgentLoop()
|
||
agent = loop.build_agents()
|
||
|
||
# 测试单个问题
|
||
response = agent.chat("What is the capital of France?")
|
||
print(response)
|
||
```
|
||
|
||
#### 3. 检查工具可用性
|
||
|
||
```bash
|
||
# 测试 MCP Server 健康状态
|
||
curl http://localhost:8080/health
|
||
|
||
# 列出所有工具
|
||
curl -X POST http://localhost:8080/mcp \
|
||
-H "Content-Type: application/json" \
|
||
-d '{"jsonrpc": "2.0", "method": "tools/list", "params": {}, "id": 1}'
|
||
```
|
||
|
||
---
|
||
|
||
## 性能基准
|
||
|
||
### GAIA 测试集结果（根据论文）
|
||
|
||
| 模型 | Pass@1 | 数据收集加速 | 训练时间 |
|
||
|------|--------|-------------|---------|
|
||
| GPT-4o (Baseline) | 27.91% | - | - |
|
||
| DeepSeek-V3 | 31.89% | - | - |
|
||
| **Qwen3-32B-AWorld** | **32.23%** | **14.6x** | 48h (8x A100) |
|
||
|
||
**关键发现**：
|
||
- 通过"从实践中学习"范式，32B 参数模型超越了 GPT-4o 和 DeepSeek-V3
|
||
- 分布式经验生成将数据收集时间从 7 天缩短至 12 小时
|
||
- 端到端训练（SFT + PPO）在 GAIA 验证集上提升 15+ 个百分点
|
||
|
||
### 硬件性能
|
||
|
||
| 配置 | Throughput | GPU 利用率 | 内存占用 |
|
||
|------|------------|-----------|---------|
|
||
| 1x A100 80GB (TP=1) | 120 tokens/s | 85% | 72GB |
|
||
| 4x A100 80GB (TP=4) | 450 tokens/s | 92% | 68GB/GPU |
|
||
| 8x A100 80GB (FSDP+TP) | 850 tokens/s | 95% | 70GB/GPU |
|
||
|
||
**优化建议**：
|
||
- **小模型（< 8B）**：单卡训练，`infer_tp=1`, `train_sp=1`
|
||
- **中等模型（8-30B）**：使用 TP=4 加速推理，`infer_tp=4`
|
||
- **大模型（> 30B）**：组合使用 TP 和 SP，`infer_tp=4, train_sp=8`
|
||
|
||
---
|
||
|
||
## 进阶主题
|
||
|
||
### 自定义奖励函数
|
||
|
||
除了 GAIA 的二元奖励，你还可以实现更复杂的奖励塑形：
|
||
|
||
```python
|
||
def dense_reward_func(data_source, solution_str, ground_truth, extra_info=None):
|
||
"""密集奖励函数（考虑中间步骤）"""
|
||
reward = 0.0
|
||
|
||
# 1. 工具使用奖励（鼓励探索）
|
||
num_tool_calls = solution_str.count("Tool call:")
|
||
reward += min(num_tool_calls * 0.1, 0.5) # 最多 0.5 分
|
||
|
||
# 2. 推理质量奖励（基于 CoT）
|
||
if "<think>" in solution_str and "</think>" in solution_str:
|
||
reward += 0.2 # 有思考过程
|
||
|
||
# 3. 最终答案奖励（主要分数）
|
||
if question_scorer(extract_answer(solution_str), ground_truth):
|
||
reward += 1.0
|
||
|
||
# 4. 效率惩罚（避免过度工具调用）
|
||
if num_tool_calls > 15:
|
||
reward -= 0.2
|
||
|
||
return reward
|
||
```
|
||
|
||
### 多任务训练
|
||
|
||
```python
|
||
# gaia_datasets/create_multitask_dataset.py
|
||
def create_multitask_dataset():
|
||
datasets = []
|
||
|
||
# 任务 1：GAIA
|
||
gaia_ds = load_gaia_dataset(...)
|
||
gaia_ds['task_type'] = 'gaia'
|
||
datasets.append(gaia_ds)
|
||
|
||
# 任务 2：Code Execution
|
||
code_ds = load_code_dataset(...)
|
||
code_ds['task_type'] = 'code'
|
||
datasets.append(code_ds)
|
||
|
||
# 任务 3：Web Navigation
|
||
web_ds = load_webarena_dataset(...)
|
||
web_ds['task_type'] = 'web'
|
||
datasets.append(web_ds)
|
||
|
||
# 混合采样
|
||
return pd.concat(datasets).sample(frac=1.0)
|
||
```
|
||
|
||
---
|
||
|
||
## 引用
|
||
|
||
如果你在研究中使用了 AWorld Train，请引用我们的论文：
|
||
|
||
```bibtex
|
||
@article{yu2025aworld,
|
||
title={AWorld: Orchestrating the Training Recipe for Agentic AI},
|
||
author={Yu, Chengyue and Lu, Siyuan and Zhuang, Chenyi and Wang, Dong and others},
|
||
journal={arXiv preprint arXiv:2508.20404},
|
||
year={2025}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 社区与支持
|
||
|
||
- **GitHub Issues**: [https://github.com/inclusionAI/AWorld/issues](https://github.com/inclusionAI/AWorld/issues)
|
||
- **Discord**: [https://discord.gg/aworld](https://discord.gg/aworld)
|
||
- **论文**: [https://arxiv.org/abs/2508.20404](https://arxiv.org/abs/2508.20404)
|
||
- **官方文档**: [https://inclusionai.github.io/AWorld/](https://inclusionai.github.io/AWorld/)
|
||
|
||
---
|
||
|
||
<div align="center">
|
||
|
||
**AWorld Train** — 让你的 Agent 从实践中学习
|
||
|
||
Made with ❤️ by [Inclusion AI](https://github.com/inclusionAI)
|
||
|
||
</div>
|
||
|
||
[license-image]: https://img.shields.io/badge/License-MIT-yellow.svg
|
||
[license-url]: https://opensource.org/licenses/MIT
|