liqiang/ai-agent-book
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
Details
dependency resolution / resolve (3.11) (push) Canceled after 0s
Details
dependency resolution / resolve (3.13) (push) Canceled after 0s
Details
deploy-pages / build (push) Canceled after 0s
Details
deploy-pages / deploy (push) Canceled after 0s
Details
i18n consistency check / check (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
Details
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
Details
web-search-agent tests / test (push) Canceled after 0s
Details
web-search-agent tests / agentbook (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
liqiang
2026-08-20 13:12:50 +00:00
commit b119135836
10275 changed files with 3284984 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
chapter4/DOCKER_DEPLOYMENT.md
+337
View File
@@ -0,0 +1,337 @@
# Docker Deployment Guide for MCP Servers
This guide explains how to deploy the three MCP servers (execution-tools, perception-tools, collaboration-tools) using Docker.
## Overview
All three MCP servers are containerized with Docker support, making them:
- **Portable**: Run anywhere Docker is available
- **Isolated**: Each server runs in its own environment
- **Reproducible**: Consistent behavior across different machines
- **Easy to deploy**: Simple setup with docker-compose
## Prerequisites
1. **Docker** (version 20.10 or later)
2. **Docker Compose** (version 2.0 or later)
3. **API Keys** for external services (OpenAI, Google, etc.)
## Quick Start
### 1. Set Up Environment Variables
Copy the example environment file and configure your API keys:
```bash
cd /Users/boj/ai-agent-book/projects/week4
cp .env.example .env
```
Edit `.env` and add your API keys:
```env
OPENAI_API_KEY=your-openai-api-key
GOOGLE_API_KEY=your-google-key
# ... other keys
```
### 2. Build and Run All Services
Use the provided script:
```bash
./build_and_run.sh
```
Or manually:
```bash
# Build all images
docker-compose build
# Start all services
docker-compose up -d
# View logs
docker-compose logs -f
```
### 3. Build Individual Services
To build/run a single service:
```bash
# Build execution-tools only
docker-compose build execution-tools
# Run execution-tools only
docker-compose up -d execution-tools
```
## Service Details
### Execution Tools
**Purpose**: Multi-language code execution with scientific computing support
**Languages Supported**:
- Python 3.11 (with NumPy, Pandas, Scikit-learn, etc.)
- JavaScript/Node.js 20.x
- TypeScript (with tsx/ts-node)
- Go 1.21
- Java 17 (OpenJDK)
- C++ (GCC)
- Rust
- PHP
- Bash
**Volume Mounts**:
- `execution-workspace:/workspace` - Code execution workspace
**Key Environment Variables**:
- `WORKSPACE_DIR`: Working directory for code execution
- `AUTO_VERIFY_CODE`: Automatically verify code before execution
- `AUTO_SUMMARIZE_COMPLEX_OUTPUT`: Summarize long outputs
### Perception Tools
**Purpose**: Document processing, web search, and data retrieval
**Features**:
- PDF/document processing
- Web search (Google, Arxiv)
- Data extraction and analysis
- OCR support (Tesseract)
**Volume Mounts**:
- `perception-data:/data` - Processed document storage
**Key Environment Variables**:
- `DATA_DIR`: Data storage directory
- `GOOGLE_API_KEY`: Google search API key
- `GOOGLE_CSE_ID`: Custom Search Engine ID
### Collaboration Tools
**Purpose**: Browser automation, Excel processing, HITL interactions
**Features**:
- Headless browser automation (Chromium)
- Excel file processing
- Human-in-the-loop interactions
- Chess game analysis
- Timer and notification tools
**Volume Mounts**:
- `collaboration-workspace:/workspace` - Working directory
**Key Environment Variables**:
- `WORKSPACE_DIR`: Working directory
- `DISPLAY`: X11 display (for headless browser)
## Local Development
For local development without Docker:
### Execution Tools
```bash
cd execution-tools
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env with your settings
python server.py
```
### Perception Tools
```bash
cd perception-tools
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp env.example .env
# Edit .env with your settings
python src/main.py
```
### Collaboration Tools
```bash
cd collaboration-tools
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp env.example .env
# Edit .env with your settings
python src/main.py
```
## Testing Multi-Language Code Execution
Once the execution-tools service is running, you can test different languages:
### Python Example
```python
code = """
import numpy as np
import pandas as pd
data = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
print(data.describe())
"""
# Execute via MCP: code_interpreter(code=code, language="python")
```
### JavaScript Example
```javascript
code = """
console.log('Hello from Node.js!');
const numbers = [1, 2, 3, 4, 5];
const sum = numbers.reduce((a, b) => a + b, 0);
console.log('Sum:', sum);
"""
# Execute via MCP: code_interpreter(code=code, language="javascript")
```
### Go Example
```go
code = """
package main
import "fmt"
func main() {
fmt.Println("Hello from Go!")
sum := 0
for i := 1; i <= 10; i++ {
sum += i
}
fmt.Printf("Sum: %d\\n", sum)
}
"""
# Execute via MCP: code_interpreter(code=code, language="go")
```
## Docker Commands Reference
```bash
# Build all services
docker-compose build
# Start all services
docker-compose up -d
# Stop all services
docker-compose down
# View logs
docker-compose logs -f [service-name]
# Restart a service
docker-compose restart [service-name]
# View running containers
docker-compose ps
# Execute command in container
docker-compose exec execution-tools bash
# Remove all containers and volumes
docker-compose down -v
# Rebuild a service
docker-compose up -d --build [service-name]
```
## Troubleshooting
### Issue: Service won't start
Check logs:
```bash
docker-compose logs [service-name]
```
### Issue: Permission denied
Ensure volumes have correct permissions:
```bash
docker-compose down -v
docker-compose up -d
```
### Issue: Out of memory
Increase Docker memory limit in Docker Desktop settings or add to docker-compose.yml:
```yaml
services:
execution-tools:
mem_limit: 4g
```
### Issue: Python packages missing
Rebuild the image:
```bash
docker-compose build --no-cache execution-tools
```
## Security Considerations
1. **Never commit .env files** with real API keys
2. **Use non-root users** in containers (already configured)
3. **Limit resource usage** with Docker resource constraints
4. **Keep images updated** regularly rebuild with latest security patches
5. **Use secrets management** for production deployments (Docker Swarm secrets, Kubernetes secrets)
## Production Deployment
For production deployments, consider:
1. **Orchestration**: Use Kubernetes or Docker Swarm
2. **Secrets Management**: Use external secret stores (Vault, AWS Secrets Manager)
3. **Monitoring**: Add Prometheus/Grafana for metrics
4. **Logging**: Centralized logging with ELK or Loki
5. **Resource Limits**: Set proper CPU/memory limits
6. **Health Checks**: Already configured in docker-compose.yml
7. **Auto-restart**: Already configured with `restart: unless-stopped`
## Architecture Diagram
```
┌─────────────────────────────────────────────────────────┐
│ MCP Client (Claude) │
└────────────┬────────────┬────────────┬──────────────────┘
│ │ │
│ stdio │ stdio │ stdio
│ │ │
┌─────────▼───────┐ ┌─▼──────────┐ ┌▼────────────────┐
│ execution-tools │ │ perception-│ │ collaboration- │
│ Container │ │ tools │ │ tools │
│ │ │ Container │ │ Container │
│ • Python 3.11 │ │ • Doc Proc │ │ • Browser │
│ • Node.js 20 │ │ • Search │ │ • Excel │
│ • Go 1.21 │ │ • OCR │ │ • HITL │
│ • Java 17 │ │ • APIs │ │ • Timers │
│ • C++/Rust/PHP │ │ │ │ │
└────────┬────────┘ └─┬──────────┘ └┬────────────────┘
│ │ │
▼ ▼ ▼
/workspace /data /workspace
(volume) (volume) (volume)
```
## Additional Resources
- [Docker Documentation](https://docs.docker.com/)
- [Docker Compose Reference](https://docs.docker.com/compose/)
- [MCP Protocol Specification](https://modelcontextprotocol.io/)
## Support
For issues or questions:
1. Check the logs: `docker-compose logs -f`
2. Review this documentation
3. Check the individual README files in each service directory
chapter4/EXPERIMENT_LEDGER.md
+96
View File
@@ -0,0 +1,96 @@
# Chapter 4 experiment ledger
This ledger separates execution coverage from the manuscript hypothesis and from external credential availability. `official_complete` is true only when every gate named by the manuscript has substantive real evidence. Mechanism tests and credential probes are retained, but never promoted as successful external executions.
| Experiment | Canonical run | Status | `official_complete` | Manifest SHA-256 |
| --- | --- | --- | --- | --- |
| 4-1 | `perception-tools/validation/experiment_4_1/real_mcp_dashscope_intl_20260730T070000Z` | blocked | false | `f93ee0ad9bd1121ed9e7c9d730bbaf85847d03e89c9024487cfdf9f62b8557ab` |
| 4-2 | `multimodal-agent/validation/runs/20260729T185433Z-4_2-e028c9db` | passed | true | `1a9cc7bfd48717e73a03ebbde7fd786c7da2811a15267715a3794c0f1220362e` |
| 4-3 | `execution-tools/validation/experiment_4_3/real_mcp_gui_20260802T093657Z` | blocked | false | `fde8976b91b149a61b7d468f4c825c1bdfdc9da3062cbfa66aaa1fd0f3d1966f` |
| 4-4 | `collaboration-tools/validation/experiment_4_4/real_mcp_human_20260803_v2` | blocked | false | `9fae8eadec1f9583ba03e21df5c8bc660cc8bec2ba328cf304bcaa0039bd97a3` |
| 4-5 | `agent-with-event-trigger/validation/experiment_4_5/credential_probe_20260730T064500Z` | blocked | false | `3f689dfee915503f61ca30e9b590e24c8950496ca90fbf365def83805e877d0a` |
| 4-6 | `async-agent/validation/experiment_4_6/real_subprocess_20260730T052500Z` | passed | true | `fff6b43a2e3a0b706fdd68bca289119f726d3f827f3f4d837e97321f7d48a825` |
| 4-7 | `active-tool-discovery/validation/experiment_4_7/qwen3_4b_exact_v2_20260730T130600Z` | passed | true | `ce9d6eda2237938e9ed7bf63a950d1b261526897c0597f2908b705e6d6430d0e` |
## Experiment 4-1 — perception MCP
Manuscript gates: a real MCP catalog covering search, multimodal understanding, filesystem operations, public data, and authorized private data.
- Passed: real MCP `tools/list`; web and local-knowledge search; HTTPS download and webpage reading; PDF/DOCX/PPTX extraction; OCR; local Whisper transcription; video parsing; DashScope international `qwen-vl-max` image and video analysis with response IDs, token usage, and latency; confined file read/search/list/copy/move/delete; three escape probes; Open-Meteo, Yahoo Finance, exchange-rate, Wikipedia, and arXiv calls.
- Blocked: Google Calendar and Notion. No usable OAuth token or Notion integration credential exists in the environment. The failed calls and credential-free preflight are retained.
- Failed provenance retained: the first DashScope attempt used the mainland endpoint with an international-region key and received 401; the corrected run uses `dashscope-intl.aliyuncs.com`.
## Experiment 4-2 — multimodal processing
Manuscript gates: run the same nontrivial image/PDF and questions through native multimodal, extract-to-text, and tool-on-demand paradigms, retaining real vision calls, tool-use traces, exact-answer quality, latency, usage, and an external judge for free-form output. The canonical run is retained under `multimodal-agent/validation/runs/20260729T185433Z-4_2-e028c9db/`.
## Experiment 4-3 — execution MCP
Manuscript gates: verified file write/edit, terminal timeout and dangerous-command review, sandboxed Python, long-output persistence, Excel operations, external system mutations, and browser/desktop/mobile execution.
- Passed: deterministic Python compiler and Node `--check` linter; structured invalid-code responses; workspace escape rejection; timeout; OpenRouter GPT-4.1-mini dangerous-command rejection with raw usage/latency receipts; Docker Python sandbox (`--network none`, read-only root, memory/CPU/PID limits); immutable full long-output retention; XLSX formulas rendered through LibreOffice and PyMuPDF; real HTTPS webhook; real headless Chromium navigation and screenshot; PR #605 created through the GitHub execution tool and then safely reused through query-before-mutation idempotency; headful Chromium on Xvfb driven through OS keyboard events with a hashed framebuffer; and a KVM-backed AndroidWorld API-33 emulator that opened Wi-Fi Settings, verified focus, captured pixels, and returned home through ADB input.
- Blocked: no Google Calendar or real email-provider credentials. Android, Computer Use, and GitHub are no longer blockers. The canonical run passes 13/15 gates while retaining `official_complete: false` for the two absent external mutations.
- Failed provenance retained: `real_mcp_gui_20260802T093348Z` established the GitHub/desktop/mobile gates but failed the spreadsheet gate because LibreOffice and the Chapter 4 PyMuPDF dependency were missing. The corrected canonical run installs/declares both and passes the spreadsheet gate; it reuses the already-open PR instead of creating a duplicate.
## Experiment 4-4 — collaboration MCP
Manuscript gates: sync/async sub-agent lifecycle, messages, cancellation/status, two context-passing strategies, HITL requests with timeout/default behavior, and real multi-channel notification.
- Passed: the canonical v2 run retains six unique Kimi K3 response/usage/latency receipts; real minimal and LLM-generated handoffs; privacy filtering; synchronous and asynchronous completion/status; follow-up messages; cancellation; a conservative timeout; and a live repository-user approval delivered to the same pending MCP request in 1,423.272 seconds within its four-hour response window. The independent validator checks the human/MCP IDs and decision, 55 tool receipts, all 61 manifest hashes, and credential absence.
- Blocked only on delivery: no real SMTP/SendGrid, Telegram, or Slack configuration exists. Credential-free preflights fail explicitly, so `official_complete` remains false even though the human-decision gate is now closed.
- Failed provenance retained: `real_mcp_human_20260803_v1` used a 30-minute live window; the response arrived just after timeout and exposed that an expired request could still be mutated. The failed run preserves the timeout and late-response receipts. The production HITL primitive now rejects late or duplicate responses to terminal requests, with focused regression tests. The earlier `real_mcp_kimi_20260730T063500Z` run also preserves the original too-short async polling failure.
## Experiment 4-5 — event-driven mailbox agent
Manuscript gates: three real inbound test-mailbox events processed FIFO: meeting/calendar conflict plus draft, complaint extraction plus high-priority notification, and marketing archive plus provider verification.
- The campaign fetched and hashed all eight official Unipile Email/Calendar schema documents and made credential-redacted live API probes.
- Blocked before mailbox mutation: the configured Unipile credential returns 401 with both documented `X-API-KEY` and diagnostic Bearer authentication. Therefore zero local/synthetic mail objects were substituted and no three-email success is claimed.
## Experiment 4-6 — interruptible asynchronous agent
All four exact manuscript scenarios passed with real OS subprocesses: a 3–5
second command remained non-blocking while the time question was answered;
queued instructions were appended once and produced a Japanese HTML artifact;
an interrupt terminated the real child process and the runtime recovered; and
the 3%/2%/1% parallel jobs triggered exactly one status query after the fast
job, preserved the >50% job, cancelled only the <=50% job, and produced a
hashed integrated report. The canonical summary is
`async-agent/validation/experiment_4_6/real_subprocess_20260730T052500Z/summary.json`.
## Experiment 4-7 — active tool discovery
The canonical campaign uses local Ollama `qwen3:4b`, 126 complete schemas
listed by the real perception MCP server, a 50,120-token schema catalog, a
local `all-MiniLM-L6-v2` index, five-schema user-history injection with a
cumulative status bar, and the three exact manuscript tasks in both arms. All
twelve formal gates are true. Both groups selected every required capability
and completed 3/3 tasks, so the manuscript's expected accuracy/completion
improvement was **not observed**: both arms scored 100%. Active discovery was
faster in this run (808.926 versus 2,590.820 seconds, 3.20×) and exposed much
less schema text (1,251 initial system tokens per treatment task plus 12,838
dynamic tokens across the group, versus 50,352 system tokens per control
task).
The successful aggregate must not be read as clean treatment behavior. On the
Apple task, Qwen first issued a vague discovery, malformed JSON, an irrelevant
Google search and a real but irrelevant `code_interpreter` call that wrote a
215-byte empty contributor chart; two premature finishes were rejected before
it discovered and executed `yfinance_quote` and `search_news`. The recovered
arXiv task retained two protocol parse errors and a redundant vague discovery.
Those trajectories remain in the canonical receipts.
Failed evidence is also preserved. The first exact campaign
`qwen3_4b_exact_20260730T061700Z` completed but had treatment at only 1/3 tasks
(manifest SHA-256
`e3b98be25fca51e3454e442f2e312ff84aad24c89c2d44a7c1e46628cdbebe09`).
The canonical v2 campaign's first terminal attempt hit real arXiv
429/503/disconnect failures; its final search succeeded only on turn 12, too
late to download. Its failed manifest SHA-256 is
`e18bc4465606087c195a2abafbd375048c2921233bae812ef3bc3f522eb9b86b`.
A bounded same-campaign resume archived that failed summary, manifest and task
receipt, reused the other five completed receipts, then made one fresh real
attempt. With the arXiv client page bounded to the requested three results,
the official endpoint succeeded on its first call and all three PDFs were
downloaded, signature-checked and hashed. No cached result or mock substituted
for either failed attempt.
chapter4/README.ar.md
+35
View File
@@ -0,0 +1,35 @@
# الفصل الرابع · الأدوات
> الأدوات هي أيدي الوكيل. يناقش تصنيف الأدوات ومبادئ التصميم العامة، وبروتوكول MCP وتحديات اختيار الأداة، وثلاثة أنواع من الأدوات (الإدراك، والتنفيذ، والتعاون)، والوكلاء غير المتزامنين المعتمدين على الأحداث.
← [العودة إلى الملف التمهيدي الرئيسي](../docs/ar/README.md) · 📖 [قراءة نص الفصل](../book-ar/chapter4.ar.md)
## كيفية قراءة التجارب
يستخدم النص هياكل آلية قصيرة لشرح تدفق التحكم؛ ويحتوي دليل التجارب على محولات SDK الكاملة والسجلات والاختبارات وأدلة القبول. لا حاجة لقراءة كل ملف سطرًا سطرًا.
- **Starter:** ابدأ بالهدف والأمر الأدنى وشروط القبول؛ وابدأ من [async-agent](async-agent/);
- **Builder:** تتبّع نقطة الدخول والحلقة الأساسية ومخطط الحالة/الرسائل والأدوات وأداة التحقق.
- **Maintainer:** ثم اقرأ الاختبارات وmanifest الأدلة ومعالجة الأعطال ومسارات التراجع ومحولات المزوّد.
في القراءة الأولى يمكنك تجاوز بيانات الاعتماد وطبقة العرض وتوافق المزوّد؛ عُد إليها عند إعادة إنتاج رقم.
## المشاريع المصاحبة
| التجربة | المشروع | النوع | الوصف |
| :--: | --- | :--: | --- |
| 4-1 | [أدوات الإدراك](perception-tools/) | ✅ | قم ببناء مجموعة شاملة من أدوات الإدراك، وتوفير إمكانيات البحث على الويب، والفهم متعدد الوسائط، وعمليات نظام الملفات، والوصول إلى مصادر البيانات العامة. تعتمد معظم الميزات على واجهات برمجة التطبيقات المجانية والمفتوحة (DuckDuckGo، وOpen-Meteo، وYahoo Finance، وOpenStreetMap، وما إلى ذلك) ولا تتطلب مفتاح API. |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [أدوات التنفيذ](execution-tools/) | ✅ | تنفيذ مجموعة من أدوات التنفيذ مع آليات السلامة، بما في ذلك عمليات الملفات، ومترجم الشفرة، والمحطة الافتراضية، وتكامل النظام الخارجي. منع العمليات الخطيرة من خلال آلية الموافقة الثانوية LLM، وتلخيص المخرجات المعقدة تلقائيًا، وإجراء التحقق من صحة بناء الجملة على الشفرة. |
| 4-4 | [أدوات التعاون](collaboration-tools/) | ✅ | توفير إمكانات تعاون شاملة، بما في ذلك أتمتة المتصفح (إطار استخدام المتصفح)، وHuman-in-the-Loop، والإشعارات متعددة القنوات (البريد الإلكتروني، وTelegram، وSlack، وDiscord)، وإدارة المؤقت. يدعم موافقة المسؤول على العمليات الحساسة وإرسال المهام المجدولة. |
| 4-5 | [الاكتشاف الاستباقي للأدوات](active-tool-discovery/) | ✅ | يقارن بين نهجين: إدراج مخططات أكثر من 120 أداة دفعة واحدة، والاكتشاف عند الطلب. ويحتفظ النهج الثاني بعدد قليل من الأدوات الأساسية وأداة `discover_tools` الوصفية في موجّه النظام، ثم يستخدم تشابه التضمينات لاسترجاع أكثر 3–5 أدوات متخصصة صلةً. ويخفض ذلك استهلاك الرموز ويحد من اختيار أداة عامة أو غير مناسبة من قائمة طويلة. |
| — | [اختيار الأداة النشطة](active-tool-selection/) | ✅ | تنفيذ آلية اختيار أداة ذكية تسمح للوكيل باختيار المجموعة الأكثر ملاءمة من الأدوات بشكل فعال بناءً على متطلبات المهمة، بدلاً من القبول السلبي لمجموعة أدوات محددة مسبقًا. |
> بالإضافة إلى ذلك، يوفر `chapter4/docker-compose.yml` و`chapter4/DOCKER_DEPLOYMENT.md` حلاً مرجعيًا لتخزين ونشر خوادم الأدوات MCP المذكورة أعلاه.
## أنواع المشاريع
| الأيقونة | النوع | المعنى |
| :--: | --- | --- |
| ✅ | **مستقل** | شفرة كاملة قابلة للتشغيل في هذا المستودع بعد إعداد مفتاح API |
| 📖 | **دليل إعادة الإنتاج** | وثائق تفصيلية تعتمد على مستودع خارجي يُجلب باستخدام `git clone` |
| 🚧 | **وثيقة التصميم** | وثيقة تصميم وخطة تنفيذ؛ أما الشفرة القابلة للتشغيل فما تزال قيد التطوير |
chapter4/README.en.md
+42
View File
@@ -0,0 +1,42 @@
# Chapter 4 · Tools
> Tools are the hands of an Agent. Discusses tool classification and general design principles, the MCP protocol and challenges of tool selection, three types of tools (perception, execution, collaboration), and event-driven asynchronous Agents.
← [Back to main README](../docs/en/README.md) · 📖 [Read chapter text](../book-en/chapter4.md)
## How to Read the Experiments
The prose uses short mechanism skeletons to explain control flow; the experiment directory contains complete SDK adapters, logs, tests, and acceptance evidence. You do not need to read every file line by line.
- **Starter:** Start with the goal, minimum command, and acceptance conditions; begin with [async-agent](async-agent/);
- **Builder:** Follow the entry point, core loop, state/message schema, tools, and verifier.
- **Maintainer:** Then read tests, evidence manifests, failure handling, rollback paths, and provider adapters.
On a first pass, skip credential loading, presentation code, and provider-compatibility layers; return when reproducing a number.
## Companion Projects
| Exp. | Project | Type | Description |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | Build a comprehensive set of perception tools, providing capabilities for web search, multimodal understanding, file system operations, and access to public data sources. Most features are based on free, open APIs (DuckDuckGo, Open-Meteo, Yahoo Finance, OpenStreetMap, etc.) and require no API key. |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | The canonical 20-call campaign passes 13/15 gates, including safe execution, a real GitHub PR, Xvfb desktop Computer Use, and KVM-backed Android actions; only authorized Calendar and email mutations remain blocked. |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | Provide comprehensive collaboration capabilities, including browser automation (browser-use framework), Human-in-the-Loop, multi-channel notifications (Email, Telegram, Slack, Discord), and timer management. Supports admin approval for sensitive operations and scheduled task dispatching. |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | Compares two paradigms: "injecting all 120+ tool schemas" vs. "active on-demand discovery." The latter keeps only a few basic tools and a `discover_tools` meta-tool in the system prompt, using embedding similarity to retrieve the 3-5 most relevant specialized tools from a tool library. This saves tokens and prevents the model from incorrectly selecting or misusing general tools from an overly long list. |
| — | [active-tool-selection](active-tool-selection/) | ✅ | Implement an intelligent tool selection mechanism that allows the Agent to actively choose the most suitable combination of tools based on task requirements, rather than passively accepting a predefined tool set. |
Runnable-project status is separate from manuscript acceptance. Experiments
4-1 through 4-5 have substantial real execution coverage but remain officially
incomplete because authorized private-data, Calendar/email mutation,
human-decision, notification, or real-mailbox gates are still blocked. The
Android and Computer Use gates for 4-3 now have substantive retained execution.
See the [experiment ledger](EXPERIMENT_LEDGER.md) for the exact boundary.
> Additionally, `chapter4/docker-compose.yml` and `chapter4/DOCKER_DEPLOYMENT.md` provide a reference solution for containerizing and deploying the aforementioned MCP tool servers.
## Project Types
| Icon | Type | Meaning |
| :--: | --- | --- |
| ✅ | **Standalone** | Full code in this repo, runs after configuring API Key |
| 📖 | **Reproduction Guide** | Detailed doc depending on **external repos** to `git clone` |
| 🚧 | **Design Doc** | Architecture/implementation plan only, runnable code still WIP |
chapter4/README.es.md
+36
View File
@@ -0,0 +1,36 @@
# Capítulo 4 · Herramientas
> Las herramientas son las manos del Agente: protocolo MCP, herramientas de percepción/ejecución/colaboración, Agentes asíncronos orientados a eventos
← [Volver al README principal](../docs/es/README.md) · 📖 [Leer texto del capítulo](../book-es/chapter4.es.md)
## Cómo leer los experimentos
El texto usa skeletons breves para explicar el flujo de control; el directorio de experimentos contiene adaptadores SDK completos, registros, pruebas y evidencias de aceptación. No hace falta leer cada archivo línea por línea.
- **Starter:** Empieza por el objetivo, el comando mínimo y la aceptación; comienza con [async-agent](async-agent/);
- **Builder:** Sigue el punto de entrada, el bucle central, el esquema de estado/mensajes, las herramientas y el verificador.
- **Maintainer:** Después revisa pruebas, manifiestos, fallos, rollback y adaptadores de proveedores.
En la primera pasada puedes omitir credenciales, presentación y compatibilidad de proveedores; vuelve al reproducir una cifra.
## Proyectos Complementarios
| Exp. | Proyecto | Tipo | Descripción |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | Herramientas MCP de percepción: búsqueda web, comprensión multimodal, sistema de archivos y fuentes abiertas |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | Herramientas MCP de ejecución: operaciones de archivos, intérprete de código, terminal virtual e integración externa |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | Herramientas MCP de colaboración: automatización de navegador, HITL, notificaciones y temporizadores |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | Comparación entre inyección completa de esquemas e inyección bajo demanda mediante meta-herramientas |
| — | [active-tool-selection](active-tool-selection/) | ✅ | Selección activa de la combinación de herramientas más adecuada según los requisitos de la tarea |
> Además, [`chapter4/docker-compose.yml`](docker-compose.yml) y [`chapter4/DOCKER_DEPLOYMENT.md`](DOCKER_DEPLOYMENT.md) proporcionan esquemas de despliegue en contenedores para los servidores MCP.
## Tipos de Proyectos
| Icono | Tipo | Significado |
| :--: | --- | --- |
| ✅ | **Autónomo** | Código completo en este repositorio, se ejecuta tras configurar la Clave API |
| 📖 | **Guía de Reproducción** | Documento detallado que depende de **repositorios externos** para realizar `git clone` |
| 🚧 | **Documento de Diseño** | Solo arquitectura/plan de implementación, el código ejecutable aún está en desarrollo |
chapter4/README.hu.md
+36
View File
@@ -0,0 +1,36 @@
# 4. fejezet · Eszközök
> Az eszközök az ágens kezei: eszközosztályozás és -tervezés, MCP-protokoll, érzékelési, végrehajtási és együttműködési eszközök, valamint eseményvezérelt aszinkron ágensek.
← [Vissza a magyar főoldalhoz](../docs/hu/README.md) · 📖 [A fejezet olvasása](../book-hu/chapter4.md)
## Hogyan olvassuk a kísérleteket?
A törzsszöveg rövid mechanizmus-skeletonokkal magyarázza a vezérlési folyamatot; a kísérleti könyvtárakban találhatók a teljes SDK-adapterek, naplók, tesztek és átvételi bizonyítékok. Nem kell minden fájlt sorról sorra elolvasni.
- **Starter:** Kezdje a céllal, a minimális paranccsal és az átvételi feltételekkel; induljon innen: [async-agent](async-agent/);
- **Builder:** Kövesse a belépési pontot, a fő ciklust, az állapot-/üzenetsémát, az eszközöket és az ellenőrzőt.
- **Maintainer:** Végül olvassa el a teszteket, a bizonyíték-manifeszteket, a hibakezelést, a visszaállítási útvonalakat és a provider-adaptereket.
Első olvasáskor átugorható a hitelesítő adatok betöltése, a megjelenítési réteg és a provider-kompatibilitás; a számok reprodukálásakor térjen vissza.
## Kapcsolódó projektek
| Kísérlet | Projekt | Típus | Leírás |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | Webes keresési, multimodális, fájlrendszer- és nyilvánosadat-eszközöket biztosít. |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | Fájlműveleteket, kódértelmezőt, virtuális terminált és biztonságos végrehajtási mechanizmusokat valósít meg. |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | Böngésző-automatizálást, emberi közreműködést, értesítéseket és időzítőket kínál. |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | Az összes eszközséma betöltését hasonlítja össze az igény szerinti aktív eszközfelderítéssel. |
| — | [active-tool-selection](active-tool-selection/) | ✅ | A feladat követelményei alapján kiválasztja a legmegfelelőbb eszközkombinációt. |
> A `chapter4/docker-compose.yml` és `chapter4/DOCKER_DEPLOYMENT.md` konténeres telepítési referenciát biztosít az MCP-szerverekhez.
## Projekttípusok
| Ikon | Típus | Jelentés |
| :--: | --- | --- |
| ✅ | **Önálló** | A teljes kód a repository-ban található, és az API-kulcsok beállítása után futtatható. |
| 📖 | **Reprodukciós útmutató** | Külső repository szükséges, amelyet külön kell `git clone` paranccsal letölteni. |
| 🚧 | **Folyamatban** | Az implementáció vagy az elfogadási bizonyíték még nem teljes. |
chapter4/README.id.md
+36
View File
@@ -0,0 +1,36 @@
# Bab 4 · Tool
> Tool adalah tangan Agent: klasifikasi dan desain tool, protokol MCP, tool persepsi/eksekusi/kolaborasi, serta Agent asinkron berbasis event.
← [Kembali ke README utama](../docs/id/README.md) · 📖 [Baca bab](../book-id/chapter4.md)
## Cara Membaca Eksperimen
Teks utama memakai skeleton mekanisme singkat untuk menjelaskan alur kontrol; direktori eksperimen berisi adapter SDK lengkap, log, pengujian, dan bukti penerimaan. Anda tidak perlu membaca setiap berkas baris demi baris.
- **Starter:** Mulai dari tujuan, perintah minimum, dan syarat penerimaan; awali dengan [async-agent](async-agent/);
- **Builder:** Telusuri titik masuk, loop inti, skema status/pesan, alat, dan verifier.
- **Maintainer:** Terakhir, baca pengujian, manifest bukti, penanganan kegagalan, rollback, dan adapter provider.
Pada pembacaan pertama, lewati kredensial, presentasi, dan kompatibilitas provider; kembali saat mereproduksi angka.
## Proyek Pendamping
| Eksperimen | Proyek | Jenis | Deskripsi |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | Menyediakan tool pencarian web, multimodal, sistem file, dan data publik. |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | Mengimplementasikan operasi file, interpreter kode, terminal virtual, dan pengamanan eksekusi. |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | Menyediakan browser automation, Human-in-the-Loop, notifikasi, dan timer. |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | Membandingkan injeksi seluruh schema tool dengan penemuan tool sesuai kebutuhan. |
| — | [active-tool-selection](active-tool-selection/) | ✅ | Memilih kombinasi tool yang paling sesuai berdasarkan kebutuhan tugas. |
> `chapter4/docker-compose.yml` dan `chapter4/DOCKER_DEPLOYMENT.md` menyediakan referensi deployment container untuk server MCP.
## Jenis Proyek
| Ikon | Jenis | Arti |
| :--: | --- | --- |
| ✅ | **Mandiri** | Kode lengkap tersedia di repositori dan dapat dijalankan setelah API Key dikonfigurasi. |
| 📖 | **Panduan Reproduksi** | Memerlukan repositori eksternal yang harus di-`git clone`. |
| 🚧 | **Dalam Proses** | Implementasi atau bukti penerimaan belum lengkap. |
chapter4/README.ja.md
+35
View File
@@ -0,0 +1,35 @@
# 第4章 · ツール
> ツールは Agent の手である。ツールの分類と一般的な設計原則、MCP プロトコルとツール選択の課題、3 種類のツール（知覚、実行、協調）、イベント駆動の非同期 Agent について論じる。
← [メイン README に戻る](../docs/ja/README.md) · 📖 [章の本文を読む](../book-ja/chapter4.ja.md)
## 実験の読み方
本文では短い mechanism skeleton で制御フローを説明し、実験ディレクトリには完全な SDK アダプター、ログ、テスト、受け入れ証拠を置きます。すべてのファイルを一行ずつ読む必要はありません。
- **Starter:** 目的・最小コマンド・受け入れ条件から始め、まず [async-agent](async-agent/);
- **Builder:** エントリポイント、中心ループ、状態／メッセージ schema、ツール、検証器を追います。
- **Maintainer:** 最後にテスト、証拠 manifest、失敗処理、rollback 経路、provider adapter を読みます。
初読では認証情報、表示層、provider 互換層を飛ばし、数値を再現するときに戻ってください。
## 付随プロジェクト
| 実験 | プロジェクト | 種類 | 説明 |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | Web 検索、マルチモーダル理解、ファイルシステム操作、公開データソースへのアクセス機能を提供する、包括的な知覚ツール群を構築する。ほとんどの機能は無料かつオープンな API（DuckDuckGo、Open-Meteo、Yahoo Finance、OpenStreetMap など）に基づいており、API キーを必要としない。 |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | ファイル操作、コードインタープリタ、仮想ターミナル、外部システム統合を含む、安全機構を備えた実行ツール群を実装する。二次的な LLM 承認機構によって危険な操作を防ぎ、複雑な出力を自動的に要約し、コードに対して構文検証を行う。 |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | ブラウザ自動化（browser-use フレームワーク）、Human-in-the-Loop、マルチチャネル通知（Email、Telegram、Slack、Discord）、タイマー管理を含む、包括的な協調能力を提供する。機密操作に対する管理者承認とスケジュールされたタスクのディスパッチをサポートする。 |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | 「120 以上のすべてのツールスキーマを注入する」方式と「能動的なオンデマンド発見」方式という 2 つのパラダイムを比較する。後者はシステムプロンプトにいくつかの基本ツールと `discover_tools` メタツールのみを残し、埋め込み類似度を用いてツールライブラリから最も関連性の高い 3〜5 個の専用ツールを取得する。これによりトークンを節約し、過度に長いリストからモデルが汎用ツールを誤って選択・誤用することを防ぐ。 |
| — | [active-tool-selection](active-tool-selection/) | ✅ | インテリジェントなツール選択機構を実装し、Agent が事前定義されたツールセットを受動的に受け入れるのではなく、タスク要件に基づいて最適なツールの組み合わせを能動的に選択できるようにする。 |
> さらに、`chapter4/docker-compose.yml` と `chapter4/DOCKER_DEPLOYMENT.md` は、前述の MCP ツールサーバーをコンテナ化してデプロイするための参考ソリューションを提供する。
## プロジェクトの種類
| アイコン | 種類 | 意味 |
| :--: | --- | --- |
| ✅ | **単独実行** | このリポジトリに完全なコードがあり、API キーを設定すれば実行できる |
| 📖 | **再現ガイド** | `git clone` が必要な**外部リポジトリ**に依存する詳細ドキュメント |
| 🚧 | **設計ドキュメント** | アーキテクチャ/実装計画のみで、実行可能なコードは未完成 |
chapter4/README.ko.md
+36
View File
@@ -0,0 +1,36 @@
# 제4장 · 도구
> 도구는 에이전트의 손입니다. 도구 분류와 일반 설계 원칙, MCP 프로토콜과 도구 선택의 어려움, 세 가지 도구 유형(인식, 실행, 협업), 이벤트 기반 비동기 에이전트를 다룹니다.
← [한국어 메인 README로 돌아가기](../docs/ko/README.md) · 📖 [제4장 본문 읽기](../book-ko/chapter4.ko.md)
## 실험 읽는 방법
본문은 짧은 메커니즘 skeleton으로 제어 흐름을 설명하고, 실험 디렉터리에는 완전한 SDK 어댑터·로그·테스트·검수 증거를 둡니다. 모든 파일을 줄 단위로 읽을 필요는 없습니다.
- **Starter:** 목표, 최소 명령, 검수 조건부터 시작하고 다음에서 출발하세요: [async-agent](async-agent/);
- **Builder:** 진입점, 핵심 루프, 상태/메시지 스키마, 도구와 verifier를 따라갑니다.
- **Maintainer:** 마지막으로 테스트, 증거 manifest, 실패 처리, rollback 경로와 provider adapter를 읽습니다.
첫 읽기에서는 credential, UI, provider 호환 계층을 건너뛰고 수치를 재현할 때 돌아오세요.
## 연계 프로젝트
| 실험 | 프로젝트 | 유형 | 설명 |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | 웹 검색, 멀티모달 이해, 파일 시스템 작업, 공개 데이터 소스 접근 기능을 아우르는 인식 도구 모음을 구축합니다. 대부분 무료 공개 API(DuckDuckGo, Open-Meteo, Yahoo Finance, OpenStreetMap 등)를 사용하므로 API 키가 필요하지 않습니다. |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | 파일 작업, 코드 인터프리터, 가상 터미널, 외부 시스템 연동을 포함한 안전장치 내장 실행 도구 모음을 구현합니다. 보조 LLM 승인으로 위험한 작업을 막고, 복잡한 출력을 자동으로 요약하며, 코드 문법을 검증합니다. |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | 브라우저 자동화(browser-use 프레임워크), Human-in-the-Loop, 다채널 알림(이메일, Telegram, Slack, Discord), 타이머 관리를 포함한 종합 협업 기능을 제공합니다. 민감한 작업에 대한 관리자 승인과 예약 작업 실행을 지원합니다. |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | ‘120개가 넘는 도구 스키마를 모두 주입’하는 방식과 ‘필요할 때 능동적으로 발견’하는 방식을 비교합니다. 후자는 시스템 프롬프트에 몇 가지 기본 도구와 `discover_tools` 메타 도구만 유지하고, 임베딩 유사도로 도구 라이브러리에서 가장 관련 있는 전문 도구 3~5개를 검색합니다. 토큰을 절약하고, 지나치게 긴 목록 때문에 모델이 일반 도구를 잘못 선택하거나 오용하는 문제를 막습니다. |
| — | [active-tool-selection](active-tool-selection/) | ✅ | 미리 정한 도구 집합을 수동적으로 받아들이는 대신, 에이전트가 작업 요구에 따라 가장 적절한 도구 조합을 능동적으로 선택하는 지능형 도구 선택 메커니즘을 구현합니다. |
> 또한 [`chapter4/docker-compose.yml`](docker-compose.yml)과 [`chapter4/DOCKER_DEPLOYMENT.md`](DOCKER_DEPLOYMENT.md)에는 앞서 소개한 MCP 도구 서버를 컨테이너화하고 배포하는 참고 솔루션이 있습니다.
## 프로젝트 유형
| 아이콘 | 유형 | 의미 |
| :--: | --- | --- |
| ✅ | **독립 실행** | 전체 코드가 이 저장소에 있으며, API 키를 설정하면 실행할 수 있습니다. |
| 📖 | **재현 가이드** | **외부 저장소**를 `git clone`해야 하는 상세 안내 문서입니다. |
| 🚧 | **설계 문서** | 아키텍처와 구현 계획만 있으며, 실행 가능한 코드는 아직 작성 중입니다. |
chapter4/README.md
+40
View File
@@ -0,0 +1,40 @@
# 第 4 章 · 工具
> 工具是 Agent 的双手：MCP 协议，感知/执行/协作三类工具，以及工具规模化后的主动发现
← [返回主目录](../README.md) · 📖 [读本章正文](../book/chapter4.md)
## 如何阅读实验
正文 skeleton 只保留工具安全门、事件循环和主动发现的控制关系；实现和真实门禁在以下项目：
- **Starter**：从 [execution-tools](execution-tools/) 的 `python cli.py demo` 离线调用开始，先找 schema 校验、风险分类和结果验证；
- **Builder**：阅读 [async-agent](async-agent/) 的 AgentRuntime._dispatcher、_handle_interrupt 与并行工具任务，再看 [active-tool-discovery](active-tool-discovery/) 的检索/追加 schema 路径；
- **Maintainer**：检查权限策略、沙盒清理、取消确认、原始 provider 回执和 EXPERIMENT_LEDGER.md。
首次可跳过 MCP transport、Web UI 和 provider 适配器；先运行再按上述入口读核心循环。
## 配套项目
| 编号 | 项目 | 类型 | 一句话说明 |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | 感知工具 MCP：网络搜索、多模态理解、文件系统、公共数据源（DuckDuckGo/Open-Meteo/Yahoo/OpenStreetMap），大多无需 API Key |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | 对比原生多模态、提取为文本、工具化分析三种策略在保真度、成本和灵活性上的权衡 |
| 4-3 | [execution-tools](execution-tools/) | ✅ | 执行工具 MCP：20 次正式调用已通过 13/15 门禁，含 GitHub PR、Xvfb 桌面 Computer Use 与 KVM Android 实机操作；仅真实日历/邮件授权仍阻塞 |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | 协作工具 MCP：浏览器自动化、HITL、Email/Telegram/Slack/Discord 通知、定时器，支持管理员审批 |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | Qwen3-4B 真实对照中两组均 3/3 完成、准确率均 100%（未证明准确率提升）；主动发现的 schema 暴露和实测用时显著更低，但轨迹仍含无关调用与过早结束 |
| — | [active-tool-selection](active-tool-selection/) | ✅ | 让 Agent 根据任务需求主动选择最合适的工具组合，而非被动接受预定义工具集 |
> 此外，[`chapter4/docker-compose.yml`](docker-compose.yml) 与 [`chapter4/DOCKER_DEPLOYMENT.md`](DOCKER_DEPLOYMENT.md) 提供了将上述 MCP 工具服务器容器化部署的参考方案。
## 正式实验验收
真实运行、原始收据、哈希、逐项门禁与外部凭据阻塞项统一记录在 [EXPERIMENT_LEDGER.md](EXPERIMENT_LEDGER.md)。代码“可独立运行”不等于本机当前凭据已满足论文实验：4-1 至 4-5 的可执行核心均已通过真实运行，但授权私有数据、外部通知或真实邮箱门禁仍按证据诚实标为 blocked；不会用 mock 结果代替。
## 项目类型说明
| 图标 | 类型 | 含义 |
| :--: | --- | --- |
| ✅ | **可独立运行** | 本仓库自带完整代码，配置好 API Key 即可运行 |
| 📖 | **复现指南** | 依赖需自行 `git clone` 的**外部仓库**（训练框架、评测基准等） |
| 🚧 | **设计文档** | 仅包含架构与实现方案，可运行代码仍在完善中 |
chapter4/README.ru.md
+35
View File
@@ -0,0 +1,35 @@
# Глава 4 · Инструменты
> Инструменты — это руки агента. Обсуждаются классификация инструментов и общие принципы их проектирования, протокол MCP и трудности выбора инструментов, три типа инструментов (восприятие, исполнение, сотрудничество) и событийно-управляемые асинхронные агенты.
← [К оглавлению](../docs/ru/README.md) · 📖 [Читать главу](../book-ru/chapter4.md)
## Как читать эксперименты
В основном тексте короткие скелеты механизмов объясняют поток управления; в каталогах экспериментов находятся полные адаптеры SDK, журналы, тесты и приёмочные доказательства. Читать каждый файл построчно не требуется.
- **Starter:** Начните с цели, минимальной команды и условий приёмки; начните с [async-agent](async-agent/);
- **Builder:** Проследите точку входа, основной цикл, схему состояния/сообщений, инструменты и проверяющий модуль.
- **Maintainer:** Затем изучите тесты, манифесты доказательств, обработку сбоев, откат и адаптеры провайдеров.
При первом чтении можно пропустить ключи, слой представления и совместимость провайдеров; вернитесь при воспроизведении чисел.
## Сопутствующие проекты
| Эксп. | Проект | Тип | Описание |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | Строит полноценный набор инструментов восприятия: веб-поиск, мультимодальное понимание, операции с файловой системой и доступ к публичным источникам данных. Большинство функций основано на бесплатных открытых API (DuckDuckGo, Open-Meteo, Yahoo Finance, OpenStreetMap и др.) и не требует API-ключа. |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | Реализует набор инструментов исполнения с механизмами безопасности: операции с файлами, интерпретатор кода, виртуальный терминал и интеграция с внешними системами. Предотвращает опасные операции через вторичный механизм подтверждения LLM, автоматически суммирует сложные выводы и проверяет синтаксис кода. |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | Даёт полный набор возможностей сотрудничества: автоматизация браузера (фреймворк browser-use), Human-in-the-Loop, многоканальные уведомления (Email, Telegram, Slack, Discord) и управление таймерами. Поддерживает одобрение чувствительных операций администратором и запуск задач по расписанию. |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | Сравнивает две парадигмы: «внедрить схемы всех 120+ инструментов» и «активное обнаружение по требованию». Во второй в системном промпте остаются лишь несколько базовых инструментов и мета-инструмент `discover_tools`, а по сходству эмбеддингов из библиотеки извлекаются 3–5 наиболее релевантных специализированных инструментов. Это экономит токены и не даёт модели ошибочно выбрать или неправильно применить инструмент из чрезмерно длинного списка. |
| — | [active-tool-selection](active-tool-selection/) | ✅ | Реализует интеллектуальный выбор инструментов, позволяя агенту активно подбирать наиболее подходящую их комбинацию под задачу, а не пассивно принимать предопределённый набор. |
> Дополнительно `chapter4/docker-compose.yml` и `chapter4/DOCKER_DEPLOYMENT.md` дают эталонное решение для контейнеризации и развёртывания упомянутых MCP-серверов инструментов.
## Типы проектов
| Значок | Тип | Значение |
| :--: | --- | --- |
| ✅ | **Автономный** | Полный код в этом репозитории, запускается после настройки API-ключа |
| 📖 | **Гайд по воспроизведению** | Подробный документ, зависящий от **внешних репозиториев** через `git clone` |
| 🚧 | **Проектный документ** | Только архитектура/план реализации, рабочий код ещё в разработке |
chapter4/README.ta.md
+34
View File
@@ -0,0 +1,34 @@
# அத்தியாயம் 4 · கருவிகள்
> கருவிகள் ஏஜெண்டின் கைகளாகும். கருவி வகைப்பாடு மற்றும் பொதுவான வடிவமைப்புக் கொள்கைகள், MCP நெறிமுறை மற்றும் கருவித் தேர்வின் சவால்கள், உணர்தல்/செயலாக்கம்/ஒத்துழைப்பு என்ற மூன்று வகைக் கருவிகள், மற்றும் நிகழ்வு-இயக்கப்படும் ஒத்திசைவற்ற ஏஜென்ட் ஆகியவற்றை விவரிக்கிறது.
← [முக்கிய README க்குத் திரும்பு](../docs/ta/README.md) · 📖 [அத்தியாய உரையைப் படி](../book-ta/chapter4.ta.md)
## சோதனைகளை எப்படிப் படிப்பது
முதன்மை உரை குறுகிய mechanism skeleton-களால் control flow-ஐ விளக்குகிறது; முழு SDK adapters, logs, tests, acceptance evidence ஆகியவை experiment கோப்பகத்தில் உள்ளன. ஒவ்வொரு கோப்பையும் வரி வரியாகப் படிக்க வேண்டியதில்லை.
- **Starter:** இலக்கு, குறைந்தபட்ச கட்டளை, ஏற்றுக்கொள்ளும் நிபந்தனைகளில் தொடங்குங்கள்; முதலில் [async-agent](async-agent/);
- **Builder:** நுழைவுப் புள்ளி, மையச் சுழற்சி, state/message schema, கருவிகள், verifier ஆகியவற்றைப் பின்தொடருங்கள்.
- **Maintainer:** பின்னர் tests, evidence manifest, தோல்வி கையாளல், rollback பாதை, provider adapter ஆகியவற்றைப் படியுங்கள்.
முதல் வாசிப்பில் credentials, UI, provider-compatibility அடுக்குகளைத் தவிர்க்கலாம்; முடிவுகளை மீண்டும் உருவாக்கும்போது திரும்பிப் பாருங்கள்.
## துணை திட்டங்கள்
| சோதனை | Project | Type | Description |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | வலைத் தேடல், பல்முறை (multimodal) புரிதல், கோப்பு முறைமைச் செயல்பாடுகள் மற்றும் பொது தரவு மூலங்களை அணுகும் திறன் ஆகியவற்றை வழங்கும் விரிவான உணர்தல் கருவிகள் தொகுப்பை உருவாக்குகிறது. பெரும்பாலான அம்சங்கள் இலவச, திறந்த API-கள் (DuckDuckGo, Open-Meteo, Yahoo Finance, OpenStreetMap போன்றவை) அடிப்படையிலானவை; API விசை இல்லாமலேயே பயன்படுத்தலாம். |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | கோப்புச் செயல்பாடுகள், குறியீடு விளக்கி, மெய்நிகர் முனையம் மற்றும் வெளிப்புற அமைப்பு ஒருங்கிணைப்பு ஆகியவற்றை உள்ளடக்கிய, பாதுகாப்பு வழிமுறைகள் கொண்ட செயலாக்கக் கருவிகள் தொகுப்பைச் செயல்படுத்துகிறது. LLM இரண்டாம்-நிலை ஒப்புதல் வழிமுறையின் மூலம் ஆபத்தான செயல்பாடுகளைத் தடுத்து, சிக்கலான வெளியீடுகளை தானாகச் சுருக்கி, குறியீட்டின் தொடரியலைச் சரிபார்க்கிறது. |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | உலாவித் தன்னியக்கம் (browser-use கட்டமைப்பு), மனித-கணினி ஒத்துழைப்பு (Human-in-the-Loop), பல-சேனல் அறிவிப்புகள் (Email, Telegram, Slack, Discord) மற்றும் டைமர் நிர்வாகம் ஆகியவற்றை உள்ளடக்கிய விரிவான ஒத்துழைப்புத் திறன்களை வழங்குகிறது. உணர்திறன் செயல்பாடுகளுக்கான நிர்வாகி ஒப்புதல் மற்றும் திட்டமிடப்பட்ட பணிகளின் அட்டவணைப்படுத்தலை ஆதரிக்கிறது. |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | "120+ கருவி schema-களை முழுமையாக உட்செலுத்துதல்" மற்றும் "தேவைப்படும்போது முன்னெச்சரிக்கையுடன் கண்டுபிடித்தல்" என்ற இரண்டு முன்னுதாரணங்களை ஒப்பிடுகிறது: பிந்தையது system-இல் சில அடிப்படைக் கருவிகள் + ஒரு `discover_tools` மெட்டா-கருவியை மட்டுமே வைத்திருந்து, உட்பொதிவு ஒற்றுமையைப் பயன்படுத்தி கருவி நூலகத்திலிருந்து மிகவும் தொடர்புடைய 3-5 சிறப்புக் கருவிகளை மீட்டெடுக்கிறது—இது டோக்கனை மிச்சப்படுத்துவதுடன், மிக நீண்ட கருவிப் பட்டியலில் மாதிரி பொதுநோக்குக் கருவிகளை தவறாகத் தேர்ந்தெடுப்பது/தவறாகப் பயன்படுத்துவதையும் தவிர்க்கிறது. |
| — | [active-tool-selection](active-tool-selection/) | ✅ | நுண்ணறிவுக் கருவி தேர்வு வழிமுறையைச் செயல்படுத்துகிறது — முன்னரே வரையறுக்கப்பட்ட கருவித் தொகுப்பை ஏஜென்ட் செயலற்ற முறையில் ஏற்றுக்கொள்வதற்குப் பதிலாக, பணித் தேவைகளுக்கு ஏற்ப மிகவும் பொருத்தமான கருவிக் கலவையை தானாக முன்வந்து தேர்ந்தெடுக்க முடிகிறது. |
## திட்ட வகைகள்
| சின்னம் | வகை | பொருள் |
| :--: | --- | --- |
| ✅ | **தனித்து இயங்கும்** | முழு குறியீடு இந்த களஞ்சியத்தில், API Key உள்ளமைத்தவுடன் இயங்கும் |
| 📖 | **மறு உருவாக்க வழிகாட்டி** | **வெளிப்புற களஞ்சியங்களை** `git clone` செய்ய வேண்டிய விரிவான ஆவணம் |
| 🚧 | **வடிவமைப்பு ஆவணம்** | கட்டமைப்பு/செயலாக்கத் திட்டம் மட்டும், இயங்கும் குறியீடு இன்னும் WIP |
chapter4/README.tr.md
+36
View File
@@ -0,0 +1,36 @@
# Bölüm 4 · Araçlar
> Araçlar bir Agent'ın elleridir. Araç sınıflandırması ve genel tasarım ilkelerini, MCP protokolünü ve araç seçimi zorluklarını, üç tür aracı (algı, yürütme, işbirliği) ve olay güdümlü asenkron Agent'ları ele alır.
← [Ana README'ye dön](../README.tr.md) · 📖 [Bölüm metnini oku](../book-tr/chapter4.tr.md)
## Deneyler nasıl okunur
Metin, kontrol akışını açıklamak için kısa mekanizma skeleton'ları kullanır; deney dizininde tam SDK adaptörleri, günlükler, testler ve kabul kanıtı bulunur. Her dosyayı satır satır okumanız gerekmez.
- **Starter:** Hedef, en kısa komut ve kabul koşullarıyla başlayın; önce [async-agent](async-agent/);
- **Builder:** Giriş noktasını, ana döngüyü, durum/mesaj şemasını, araçları ve doğrulayıcıyı izleyin.
- **Maintainer:** Son olarak testleri, kanıt manifestlerini, hata işlemeyi, rollback yollarını ve sağlayıcı adaptörlerini okuyun.
İlk okumada kimlik bilgisi yükleme, sunum katmanı ve sağlayıcı uyumluluğunu atlayıp sayıları yeniden üretirken dönün.
## Eşlik Eden Projeler
| Proje | Tür | Açıklama |
| --- | :--: | --- |
| [perception-tools](perception-tools/) | ✅ | Web araması, çok modlu anlama, dosya sistemi işlemleri ve kamuya açık veri kaynaklarına erişim yetenekleri sunan kapsamlı bir algı aracı seti inşa eder. Çoğu özellik ücretsiz, açık API'lere dayanır (DuckDuckGo, Open-Meteo, Yahoo Finance, OpenStreetMap vb.) ve API anahtarı gerektirmez. |
| [multimodal-agent](multimodal-agent/) | ✅ | Çok modlu işleme: yerel çok modlu, metne çıkarım ve araç tabanlı analizi karşılaştırır. |
| [execution-tools](execution-tools/) | ✅ | Dosya işlemleri, bir kod yorumlayıcı, sanal terminal ve harici sistem entegrasyonu dahil, güvenlik mekanizmalarına sahip bir yürütme aracı seti uygular. İkincil bir LLM onay mekanizmasıyla tehlikeli işlemleri önler, karmaşık çıktıları otomatik özetler ve kod üzerinde sözdizimi doğrulaması yapar. |
| [collaboration-tools](collaboration-tools/) | ✅ | Tarayıcı otomasyonu (browser-use çerçevesi), İnsan-Döngüde (Human-in-the-Loop), çok kanallı bildirimler (E-posta, Telegram, Slack, Discord) ve zamanlayıcı yönetimi dahil kapsamlı işbirliği yetenekleri sunar. Hassas işlemler için yönetici onayını ve zamanlanmış görev dağıtımını destekler. |
| [active-tool-selection](active-tool-selection/) | ✅ | Agent'ın önceden tanımlanmış bir araç kümesini pasif olarak kabul etmek yerine, görev gereksinimlerine göre en uygun araç kombinasyonunu aktif olarak seçmesini sağlayan akıllı bir araç seçim mekanizması uygular. |
| [active-tool-discovery](active-tool-discovery/) | ✅ | İki paradigmayı karşılaştırır: "120+ araç şemasının tümünü enjekte etmek" ile "aktif, istendiğinde keşif." İkincisi, sistem isteminde yalnızca birkaç temel araç ve bir `discover_tools` meta-aracını tutar; bir araç kütüphanesinden en ilgili 3-5 uzman aracı getirmek için gömme benzerliğini kullanır. Bu, token tasarrufu sağlar ve modelin aşırı uzun bir listeden yanlış araç seçmesini veya yanlış kullanmasını önler. |
> Ayrıca, `chapter4/docker-compose.yml` ve `chapter4/DOCKER_DEPLOYMENT.md`, yukarıdaki MCP araç sunucularını konteynerleştirme ve dağıtma için bir referans çözüm sunar.
## Proje Türleri
| İkon | Tür | Anlamı |
| :--: | --- | --- |
| ✅ | **Bağımsız** | Bu depoda tam kod, API Key yapılandırıldıktan sonra çalışır |
| 📖 | **Yeniden Üretim Rehberi** | `git clone` ile **harici depolara** bağımlı ayrıntılı belge |
| 🚧 | **Tasarım Belgesi** | Yalnızca mimari/uygulama planı, çalıştırılabilir kod henüz hazır değil |
chapter4/README.vi.md
+34
View File
@@ -0,0 +1,34 @@
# Chương 4 · Công cụ
> công cụ là đôi tay của Agent. Trình bày phân loại công cụ và nguyên tắc thiết kế tổng quát, giao thức MCP và thách thức chọn công cụ, ba loại công cụ cảm nhận/thực thi/cộng tác, cũng như Agent bất đồng bộ hướng sự kiện.
← [Về README chính](../docs/vi/README.md) · 📖 [Đọc nội dung chương](../book-vi/chapter4.vi.md)
## Cách đọc các thí nghiệm
Phần văn bản dùng skeleton cơ chế ngắn để giải thích luồng điều khiển; thư mục thí nghiệm chứa adapter SDK đầy đủ, log, kiểm thử và bằng chứng nghiệm thu. Không cần đọc từng tệp theo từng dòng.
- **Starter:** Bắt đầu từ mục tiêu, lệnh tối thiểu và điều kiện nghiệm thu; hãy bắt đầu với [async-agent](async-agent/);
- **Builder:** Lần theo điểm vào, vòng lặp lõi, schema trạng thái/tin nhắn, công cụ và verifier.
- **Maintainer:** Sau đó đọc test, manifest bằng chứng, xử lý lỗi, đường rollback và adapter nhà cung cấp.
Lần đầu có thể bỏ qua credential, lớp trình bày và tương thích provider; quay lại khi cần tái tạo số liệu.
## Dự án đi kèm
| Thí nghiệm | Project | Type | Description |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | Xây dựng bộ công cụ cảm nhận toàn diện, cung cấp khả năng tìm kiếm web, hiểu đa phương thức, thao tác hệ thống tệp và truy cập nguồn dữ liệu công cộng. Phần lớn chức năng dựa trên API mở miễn phí (DuckDuckGo, Open-Meteo, Yahoo Finance, OpenStreetMap, v.v.) và không cần API key. |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | Triển khai bộ công cụ thực thi có cơ chế an toàn, bao gồm thao tác file, code interpreter, terminal ảo và tích hợp hệ thống bên ngoài. Dùng cơ chế phê duyệt lần hai bằng LLM để ngăn thao tác nguy hiểm, tự động tóm tắt đầu ra phức tạp và kiểm tra cú pháp mã. |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | Cung cấp năng lực cộng tác toàn diện, gồm tự động hóa trình duyệt (framework browser-use), phối hợp người-máy (Human-in-the-Loop), thông báo đa kênh (Email, Telegram, Slack, Discord) và quản lý bộ hẹn giờ. Hỗ trợ phê duyệt quản trị viên cho thao tác nhạy cảm và lập lịch tác vụ định kỳ. |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | So sánh hai mô thức “nhồi toàn bộ hơn 120 tool schema” và “chủ động phát hiện theo nhu cầu”: mô thức sau chỉ giữ một số ít công cụ nền tảng + một meta-tool `discover_tools` trong system, dùng độ tương tự embedding để truy xuất 3–5 công cụ chuyên dụng liên quan nhất từ thư viện công cụ, vừa tiết kiệm token vừa tránh mô hình chọn sai/lạm dụng công cụ chung khi danh sách công cụ quá dài. |
| — | [active-tool-selection](active-tool-selection/) | ✅ | Triển khai cơ chế chọn công cụ thông minh, giúp Agent chủ động chọn tổ hợp công cụ phù hợp nhất theo nhu cầu nhiệm vụ, thay vì thụ động tiếp nhận bộ công cụ định nghĩa sẵn. |
## Phân loại dự án
| Biểu tượng | Loại | Ý nghĩa |
| :--: | --- | --- |
| ✅ | **Chạy độc lập** | Có mã đầy đủ trong kho, chạy được sau khi cấu hình API Key |
| 📖 | **Hướng dẫn tái hiện** | Tài liệu chi tiết, cần `git clone` **kho ngoài** |
| 🚧 | **Tài liệu thiết kế** | Chỉ có kiến trúc/phương án, mã chạy được đang hoàn thiện |
chapter4/README.zh-TW.md
+36
View File
@@ -0,0 +1,36 @@
# 第 4 章 · 工具
> 工具是 Agent 的雙手：MCP 協議、感知/執行/協作三類工具、事件驅動非同步 Agent、主動工具發現
← [返回主目錄](../docs/zh-TW/README.md) · 📖 [讀本章正文](../book/chapter4.md)
## 如何閱讀實驗
正文用短小的機制 skeleton 說明控制流；實驗目錄放完整的 SDK 適配、日誌、測試與驗收證據，不需要逐行讀完每個檔案。
- **Starter:** 先讀目標、最小指令與驗收條件；可從 [async-agent](async-agent/);
- **Builder:** 沿著入口、核心迴圈、狀態／訊息 schema、工具與驗證器閱讀。
- **Maintainer:** 最後再看測試、證據 manifest、失敗處理、回滾路徑與 provider adapter。
第一次閱讀可先跳過憑證載入、展示層和 provider 相容層；要重現數字時再回來查看。
## 配套專案
| 編號 | 專案 | 型別 | 一句話說明 |
| :--: | --- | :--: | --- |
| 4-1 | [perception-tools](perception-tools/) | ✅ | 感知工具 MCP：網路搜尋、多模態理解、檔案系統、公共資料來源（DuckDuckGo/Open-Meteo/Yahoo/OpenStreetMap），大多無需 API Key |
| 4-2 | [multimodal-agent](multimodal-agent/) | ✅ | Multimodal processing: compare native multimodal, extract-to-text, and tool-based analysis. |
| 4-3 | [execution-tools](execution-tools/) | ✅ | 執行工具 MCP：檔案操作、程式碼直譯器、虛擬終端機、外部系統整合，LLM 二次審批防誤操作 |
| 4-4 | [collaboration-tools](collaboration-tools/) | ✅ | 協作工具 MCP：瀏覽器自動化、HITL、Email/Telegram/Slack/Discord 通知、計時器，支援管理員審批 |
| 4-5 | [active-tool-discovery](active-tool-discovery/) | ✅ | 對比「全量注入 120+ 工具 schema」與「少量基礎工具 + discover_tools 元工具按需檢索」，省 token 防錯選 |
| — | [active-tool-selection](active-tool-selection/) | ✅ | 讓 Agent 根據任務需求主動選擇最合適的工具組合，而非被動接受預定義工具集 |
> 此外，[`chapter4/docker-compose.yml`](docker-compose.yml) 與 [`chapter4/DOCKER_DEPLOYMENT.md`](DOCKER_DEPLOYMENT.md) 提供了將上述 MCP 工具伺服器容器化部署的參考方案。
## 專案型別說明
| 圖示 | 型別 | 含義 |
| :--: | --- | --- |
| ✅ | **可獨立執行** | 本倉庫自帶完整程式碼，設定好 API Key 即可執行 |
| 📖 | **復現指南** | 依賴需自行 `git clone` 的**外部倉庫**（訓練框架、評測基準等） |
| 🚧 | **設計文件** | 僅包含架構與實現方案，可執行程式碼仍在完善中 |
chapter4/active-tool-discovery/.gitignore
+3
View File
@@ -0,0 +1,3 @@
.cache/
.env
__pycache__/
chapter4/active-tool-discovery/README.md
+500
View File
@@ -0,0 +1,500 @@
# Active Tool Discovery / 主动工具发现
> Companion code for *AI Agents in Depth*, Chapter 4 — **Experiment 4-7 ★★★: Proactive Tool Discovery**. Compares full injection, retrieval prefilter, and active discovery on a 126-tool library.
> 配套《深入理解 AI Agent》第 4 章 **实验 4-7 ★★★：主动工具发现**。在 126 个跨领域工具上对比全量注入、检索预筛选与主动发现。
← [Chapter 4 index / 返回第 4 章目录](../README.md)
## Code map
- **Run first:** `python demo.py --offline` (mechanism smoke without credentials); use `run_exact_experiment.py` only for the formal real-tool campaign.
- **Start here:** `discovery.py::ToolIndex` builds/searches the index; `agent.py::run_active_discovery` shows on-demand schema injection.
- **Core behavior:** compare `run_full_injection`, `run_retrieval_prefilter` and `run_active_discovery`; the formal runner’s task loop is in `run_exact_experiment.py::run_agent_task`.
- **State / protocol:** available-tool names, appended schemas, JSON action records and per-task receipts.
- **Verifier:** `test_exact_experiment.py`, `run_exact_experiment.py::derive_acceptance` and the manifest/receipt checks.
- **Experiment variable:** catalog size, retrieval `top_k`, strategy and task mix; compare latency, tokens and exact completion.
- **Skip on first pass:** embedding backends, MCP transport and failure-attempt archival.
## Canonical manuscript campaign / 正式实验活动
`run_exact_experiment.py` replaces the mechanism-only demo for formal
Experiment 4-7 acceptance. It obtains all 126 complete schemas from the real
perception MCP server, proves the full-catalog control exceeds 50K measured
tokens, uses local Ollama `qwen3:4b` for both arms, retrieves five candidates
with `all-MiniLM-L6-v2`, and executes every selected tool through MCP against
real public APIs/local processes. No mock tool result can satisfy a formal
gate. Runs are resumable and store the gzipped catalog, model receipts, MCP
receipts, artifacts, hashes, paired metrics, and an honest hypothesis result.
```bash
python run_exact_experiment.py --campaign-id my-qwen3-4b-run
# after an interruption:
python run_exact_experiment.py --campaign-id my-qwen3-4b-run --resume
```
`demo.py` and `offline_backend.py` remain useful teaching/CI paths, but their
lightweight tool outputs and scripted model do not count as formal evidence.
The completed canonical evidence is
[`validation/experiment_4_7/qwen3_4b_exact_v2_20260730T130600Z/summary.json`](validation/experiment_4_7/qwen3_4b_exact_v2_20260730T130600Z/summary.json),
with manifest SHA-256
`88d622db4981207a9980c30abea4eb8dc2621161ded80be0cb2bb8582833153c`.
All twelve gates passed. Control and treatment both selected every required
capability and completed all three tasks (100% versus 100%), so the predicted
accuracy/completion improvement was not observed. Treatment elapsed time was
808.926 seconds versus 2,590.820 seconds for control (3.20× faster). Its
initial system prompt was 1,251 tokens per task and it dynamically injected
12,838 schema tokens across all tasks; control used 50,352 system-prompt tokens
per task.
The pass does not hide weak-model detours. In the treatment Apple trajectory,
Qwen made an irrelevant search and code call (leaving a 215-byte empty SVG)
and attempted to finish twice before the completion gate forced discovery of
the stock and news specialists. The arXiv trajectory also retained malformed
actions and a redundant discovery. The first v2 terminal attempt additionally
retains real 429/503/disconnect receipts under `failed_attempts/`; a single
bounded resume archived it and retried only that incomplete task. Completed
receipts are never replayed, and a third real attempt is refused.
`run_exact_experiment.py` 是实验 4-7 的正式运行器：从真实感知 MCP 读取 126 个完整
schema，验证控制组超过 50K token，两组都使用本地 Ollama `qwen3:4b`，实验组用
`all-MiniLM-L6-v2` 每次检索五个候选，并通过 MCP 调用真实公共 API 或本地进程执行所选
工具。mock 结果不能通过正式门禁；中断后可用 `--resume` 续跑。`demo.py` 与离线后端仅作
教学/CI 机制自检，不是正式验收证据。
正式证据为
[`validation/experiment_4_7/qwen3_4b_exact_v2_20260730T130600Z/summary.json`](validation/experiment_4_7/qwen3_4b_exact_v2_20260730T130600Z/summary.json)，
manifest SHA-256 为
`88d622db4981207a9980c30abea4eb8dc2621161ded80be0cb2bb8582833153c`。
12 项门禁全部通过；对照组与实验组均完成 3/3 任务，准确率均为 100%，因此正文
预期的准确率/完成率提升并未出现。实验组用时 808.926 秒，对照组为
2,590.820 秒（快 3.20×）；实验组每任务初始 system prompt 为 1,251 token，
三任务合计动态注入 12,838 token，对照组每任务则为 50,352 token。
成功轨迹仍保留了 Apple 任务的无关搜索/代码调用、215 字节空 SVG、两次过早结束，
以及 arXiv 任务的格式错误和冗余发现；不把通过解读为“工具选择过程干净”。
---
## English
### Purpose
When an Agent has hundreds of tools, a common approach is to inject every tool JSON schema into the system prompt. That creates two problems:
1. **Token waste**: Full schemas for 126 tools are about **11.6k tokens**, re-billed on every reasoning step.
2. **Instruction-following degradation**: On slightly vague tasks, the model “casts a wide net” and calls generic fallbacks (`web_search` / `google_search` / `universal_search`) together with specialized tools—or even replaces specialized tools with generic search (e.g. looking up a stock price via generic `web_search`).
**Active discovery** keeps only a few base tools plus a `discover_tools(need)` meta-tool in the system prompt. When the model hits a capability gap, it describes the need in natural language; the system retrieves the 3–5 most relevant specialized tools via embedding similarity, appends their schemas as a **user message** (protecting the system-prefix KV cache), and updates the status bar of available tools.
### Mechanisms
```
tools_library.py 126 cross-domain teaching tools (finance/web/arxiv/github/geo/weather/media/...; 17 domains)
demo.py uses lightweight outputs; the formal runner instead uses perception MCP schemas/execution
Intentionally mixes 8 generic/near-synonym tools (web_search, etc.) with inflated descriptions
select_tools(size): subset by --tool-set-size to show full injection cost growing with catalog size
discovery.py Pluggable embedding backend + tool vector index; OpenAIEmbedder uses text-embedding-3-small
and caches to .cache/; search(need) = embed need, cosine similarity vs tool vectors, return top-k
agent.py Three ReAct strategies (text protocol: model outputs one JSON tool call per step)
- run_full_injection: all 126 schemas in system prompt
- run_retrieval_prefilter: one-shot top-n retrieve by initial query (book’s “retrieval prefilter”)
- run_active_discovery: base tools + discover_tools; retrieve-on-demand during execution
offline_backend.py Offline backend: LocalEmbedder (local bag-of-words hash) + MockChatClient (scripted mock)
so --offline runs end-to-end without any API key (token/latency real; accuracy = heuristic routing)
demo.py Same tasks under selected strategies; prints token / latency / call traces / exact match; summary table
```
**Why “text inject + text parse” instead of native OpenAI function calling?**
Native function-calling is heavily optimized for tool choice and rarely errs even with hundreds of tools, so it cannot demonstrate long-context instruction-following degradation. Putting schemas in the prompt as plain text and letting the model emit JSON tool calls is the control condition—and matches the book’s “inject schemas into the system prompt (tens of thousands of tokens)” setup.
**Why does embedding retrieval reduce wrong picks?** Generic tools like `web_search` claim to “do everything,” so their semantics are diluted; specialized tools (e.g. `search_news`) have focused descriptions. For a focused `need` (“recent Tesla news”), specialized tools score higher and rank first; generics often never enter top-k and are never loaded—retrieval acts as a precision filter.
**Why isn’t retrieval prefilter enough?** Prefilter (`run_retrieval_prefilter`) matches only the **initial query** once and injects top-n tools. On multi-step cross-domain tasks (e.g. stock price + news), the initial vector often favors the first domain; the second sub-task’s specialized tool may miss top-n. Active discovery defers discovery until each real `need` appears and retrieves separately (offline self-check shows prefilter missing the second tool on half of multi-step tasks—see table below).
### How to run
```bash
# From the repository root: use the shared Chapter 4 environment
uv sync --locked --python 3.12 --extra ch4
# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat
# pip fallback when uv is not installed:
# python -m pip install -e ".[ch4]"
cd chapter4/active-tool-discovery
# Single-project compatibility path, still supported during migration:
# python -m pip install -r requirements.txt
# Path A: offline mechanism self-check (no keys; token/latency real; accuracy = heuristic routing only)
python demo.py --offline
# Path B: real model (needed for small-model instruction-following degradation)
cp env.example .env # set OPENAI_API_KEY (chat + embeddings both use OpenAI)
# Fallback: if OPENAI_API_KEY is unset but OPENROUTER_API_KEY is set, chat routes via OpenRouter
# (model mapped to openai/gpt-5.6-luna, etc.); tool retrieval falls back to local hash embeddings
# (OpenRouter has no embeddings API).
python demo.py # all 8 tasks × three strategies
python demo.py --strategies full,discovery # compare only two strategies
python demo.py --tasks finance+news,crypto+news # selected tasks (comma-separated)
python demo.py --tasks 'opinion(诱导)' # quote task ids that contain parentheses
python demo.py --tool-set-size 20 # smaller catalog: full-injection disadvantage shrinks
python demo.py --query '查英伟达股价再搜点相关新闻' --offline # one-off natural-language task
python demo.py --offline --output results/offline.json # export structured results
```
Default model `gpt-5.6-luna`; override with `--model` or env: `python demo.py --model gpt-5.6-luna`.
First run builds tool embeddings and caches under `.cache/`. Full flags: `python demo.py --help`
(`--query / --tasks / --strategies / --tool-set-size / --top-k / --prefilter-n / --model / --embed-model / --max-steps / --offline / --output`).
### Adaptation / extension
- **Swap chat model**: `MODEL=gpt-4.1-mini python demo.py`; swap embeddings with `EMBED_MODEL=text-embedding-3-large` (cache rebuilds automatically when the embed signature changes).
- **Swap provider / gateway**: chat and embeddings both use the OpenAI SDK; `OpenAI()` reads `OPENAI_BASE_URL`, so any **OpenAI-compatible** gateway works via `OPENAI_BASE_URL=https://your-gateway/v1` (endpoint must offer both chat and embeddings).
- **Swap tasks / inputs**: edit `TASKS` in `tools_library.py` (each has `prompt` and scoring capability slots), or use `--tasks` / `--query`; grow/shrink the catalog in `ALL_TOOLS` in the same file.
- **Offline self-check**: `--offline` uses `offline_backend.py` (local hash embeddings + scripted mock). Good for CI / offline / pipeline smoke tests. It reproduces token/latency structure and “prefilter misses second tool,” not real-model long-context choice behavior (see real gpt-5.6-luna results below).
### Offline mechanism self-check (`python demo.py --offline`)
One real `--offline` run (8 tasks × three strategies). **Token/latency are real tiktoken/wall-clock**; **accuracy only reflects scripted heuristic routing**, not a real model—the mock is a “strong router” and never degrades, so full injection also scores perfectly.
| Strategy | Exact match | Task complete | Avg inject tokens | Total inject tokens | Avg latency (s) |
|---|---|---|---|---|---|
| Full injection | 8/8 | 8/8 | 11630 | 93040 | 0.008 |
| Retrieval prefilter | 4/8 | 4/8 | 1030 | 8236 | 0.006 |
| Active discovery | 8/8 | 8/8 | 974 | 7796 | 0.010 |
Two **real, reproducible structural** takeaways:
1. **Tokens diverge as catalog size grows**: full injection is fixed at 11,630 tokens/task; prefilter and discovery inject ~1,000 (**~11.9×** smaller). With `--tool-set-size 20` the gap shrinks to ~1.8×—confirming “more tools → full injection hurts more.”
2. **Prefilter structurally misses tools on multi-step cross-domain tasks**: one-shot top-10 misses the second specialized tool on 4/8 tasks (e.g. `academic(诱导)` top-10 has no `arxiv_search`); active discovery retrieves per emerging `need` and hits 8/8.
### Conclusions (one real run, gpt-5.6-luna, 2026-07)
> One real LLM run (`python demo.py --model gpt-5.6-luna`, 8 tasks × three strategies, OpenAI chat + `text-embedding-3-small`). gpt-5.6-luna is a reasoning model that only supports default `temperature=1` (no `temperature=0`; code falls back on that error), so this is a **single non-deterministic** run. Scoring: ✅ exact match (all capability slots, no generic fallback misuse); ⚠️ completed but also picked a generic tool; ❌ failed (missed specialized tool, abandoned, or 0 tool calls).
| Task | Full | Prefilter | Discovery | Full tokens | Discovery tokens |
|---|---|---|---|---|---|
| finance+news | ✅ | ❌ | ✅ | 11630 | 883 |
| arxiv+download | ✅ | ❌ | ✅ | 11630 | 927 |
| github+viz | ❌ | ❌ | ❌ | 11630 | 295 |
| weather+calendar | ❌ | ✅ | ✅ | 11630 | 1055 |
| forex+weather | ✅ | ✅ | ❌ | 11630 | 295 |
| crypto+news | ❌ | ⚠️ | ❌ | 11630 | 295 |
| opinion(诱导) | ⚠️ | ❌ | ✅ | 11630 | 688 |
| academic(诱导) | ⚠️ | ⚠️ | ❌ | 11630 | 295 |
| **Exact match** | **3/8** | **2/8** | **4/8** | | |
| **Task complete** | **5/8** | **4/8** | **4/8** | | |
| **Total inject tokens** | | | | **93040** | **4733** |
(Prefilter avg 971 tokens/task, total 7768; mean latency ~11.5 / 9.6 / 10.7 s for the three strategies—measured this run.)
1. **Token savings remain robust**: full injection fixed at **11,630 tokens/task**; discovery **295–1,055**, total 93,040 → 4,733 (**~19.7×**). Part of the larger ratio is gpt-5.6-luna abandoning some tasks without `discover_tools` (only 3 base tools = 295 tokens). The structural gain still holds: full injection re-bills tens of thousands; on-demand injects thousands or fewer.
2. **Book core phenomenon on two “inducement” tasks**: with vague wording, full injection grabs generic fallbacks—
- `opinion(诱导)`: full called `search_news, search_news, web_search, search_tweets` (⚠️ included generic **`web_search`**); discovery retrieved `search_news / get_news_by_source / ...` (**no** `web_search`) and only used specialized news tools (✅).
- `academic(诱导)`: full called 8 tools including **`google_search / universal_search / ask_knowledge_base`** (⚠️); prefilter also misused `google_search / universal_search`.
3. **Another real behavior this run**: conservative reasoning models sometimes **`finish` with 0 tool calls** (“cannot access real-time data”), lowering absolute accuracy for all strategies. Main failure mode was abandon/skip steps—not only wrong tool choice—and it appears under both full injection and discovery.
4. **Boundaries**:
- Control setup is “schemas as plain text + model emits JSON tool calls”; mock tools return placeholders, so conservative models may refuse—major source of low accuracy here.
- With `temperature=1` only, per-task outcomes vary across runs; structural conclusions (token savings; full injection misusing generics on inducement tasks) stay directionally stable.
- For cleaner, reproducible mechanism checks (token/latency + prefilter missing second tool), use the `--offline` table above.
> **One line:** On gpt-5.6-luna, active discovery’s steadiest win is still tokens (~19.7× this run); on vague inducement tasks, embedding retrieval keeps inflated generics (`web_search` / `google_search` / `universal_search`) out of the candidate set. Strong reasoners’ conservative “give up” behavior also lowers absolute accuracy for every strategy—read the table above as-is.
### Model ↔ scaffolding trade-off (weak gpt-4o-mini vs strong gpt-5.6-luna)
> Does stronger models make this scaffolding useless? Comparing the gpt-5.6-luna (strong) run with a gpt-4o-mini (weak) run on the same 8 tasks × three strategies (`python demo.py --model gpt-4o-mini`, 2026-07, same scoring). Scaffolding has two values: one **fades** as models strengthen; one is **model-independent**.
**Weak model gpt-4o-mini real summary:**
| Strategy | Exact match | Task complete | Total inject tokens | Avg latency (s) |
|---|---|---|---|---|
| Full injection | 5/8 | **8/8** | 93040 | 8.38 |
| Retrieval prefilter | 7/8 | 7/8 | 7768 | 4.90 |
| Active discovery | **8/8** | **8/8** | 7266 | 7.65 |
(token 93040 → 7266, **~12.8×**.)
#### Value 1: avoid misusing generic tools — **fades** with stronger models
- **Weak gpt-4o-mini:** under full injection never abandons (8/8 complete) but wide-nets generics on 3 tasks → **5/8 exact**. Discovery blocks inflated generics: **8/8 exact, 0 generic misuse, still 8/8 complete** (+3 exact tasks, zero completion loss).
- **Strong gpt-5.6-luna:** generic misuse only on 2 inducement tasks; discovery cleans those but exact only **3/8 → 4/8 (+1)** and completion **falls 5/8 → 4/8**. Main failure is abandon, not wrong pick—retrieval cannot fix “no tool call at all.” The weakness scaffolding targets is thin on strong models, so that value fades.
#### Value 2: inject-token savings — **persists** regardless of model strength
Full injection is always **11,630 tokens/task** (all 126 schemas in system). On-demand injects a few hundred to ~1k:
- Weak gpt-4o-mini: 93,040 → 7,266, **~12.8×**
- Strong gpt-5.6-luna: 93,040 → 4,733, **~19.7×** (larger partly because abandon skips `discover_tools`)
Token savings hold on both models and grow with catalog size—the hard reason to keep scaffolding in the strong-model era.
#### One-line summary
> **Stronger models weaken “help it not pick the wrong tool”** (gpt-4o-mini full 5/8→discovery 8/8 exact, zero completion loss; gpt-5.6-luna only 3/8→4/8 and completion drops, because losses are “abandon” not “wrong pick”). **Token savings stay** (~12.8× / ~19.7×). On strong models, the main case for active discovery shifts from “fix instruction-following degradation” to “control context cost.”
### Files
- `tools_library.py` — 126 tool defs + `select_tools` + mock execution + 8 eval tasks / scoring
- `discovery.py` — pluggable embedders (`OpenAIEmbedder`) + vector index / similarity search
- `agent.py` — three strategies (full / prefilter / discovery) ReAct loops + token stats
- `offline_backend.py` — `LocalEmbedder` + `MockChatClient` for `--offline`
- `demo.py` — multi-strategy CLI demo
- `requirements.txt` / `env.example`
---
## 中文
### 目的
当一个 Agent 拥有上百个工具时，常见做法是把全部工具的 JSON schema 一次性塞进 system prompt。
这会带来两个问题：
1. **token 浪费**：126 个工具的完整 schema 约 **1.16 万 token**，每一步推理都要重复计费。
2. **指令遵循退化**：措辞稍泛的任务下，模型会"广撒网"地把通用兜底工具（`web_search` /
`google_search` / `universal_search`）和专用工具一起调用，甚至用通用搜索替代专用工具
—— 即书中所说的"查股价却选了通用 web_search"。
**主动发现**只在 system 里保留少量基础工具 + 一个 `discover_tools(need)` 元工具。模型遇到能力缺口时，
用自然语言描述需求，系统用嵌入相似度从工具库检索 3-5 个最相关的专用工具，把它们的 schema 作为
**user message** 追加进对话（保护 system 前缀的 KV Cache），并更新状态栏可用工具列表。
### 机制
```
tools_library.py 126 个跨领域工具（finance/web/arxiv/github/geo/weather/media/... 共 17 个领域）
每个工具有真实 name/description/parameters；执行为轻量 mock（重点是"选对工具"）
其中故意混入 8 个"通用/近义"工具（web_search 等），它们的描述夸大自己无所不能
select_tools(size)：按 --tool-set-size 截取子集，演示"工具集越大全量注入越吃亏"
discovery.py 可插拔嵌入后端 + 工具向量索引；OpenAIEmbedder 用 text-embedding-3-small 生成向量
并缓存到 .cache/；search(need) = 把 need 向量化后与工具向量做余弦相似度返回 top-k
agent.py 三种策略的 ReAct 循环（文本协议：模型每步输出一个 JSON 工具调用）
- run_full_injection：126 个工具 schema 全部写进 system prompt
- run_retrieval_prefilter：按初始查询一次性检索 top-n 工具注入（书中"检索式预筛选"）
- run_active_discovery：基础工具 + discover_tools，执行中按需检索加载
offline_backend.py 离线后端：LocalEmbedder（本地哈希词袋嵌入）+ MockChatClient（脚本化 mock 模型），
让 --offline 无需任何 API key 即可跑通全流程（token/延迟真实，准确率仅反映启发式路由）
demo.py 对同一组任务分别跑所选策略，打印 token / 延迟 / 调用轨迹 / 是否精确选对，并汇总对比表
```
**为什么用"文本注入 + 文本解析"而不是 OpenAI 原生 function calling？**
原生 function-calling 接口对工具选择做了很强的约束优化，即使上百个工具也极少选错，无法体现书中
所述的"超长上下文指令遵循退化"。把 schema 当作纯文本塞进 prompt、让模型自己以 JSON 输出工具调用，
才是控制组的真实机制，也才能观察到退化。这也正是书中"把 schema 注入 system prompt（几万 token）"的写法。
**为什么嵌入检索能避免错选？** 通用工具 `web_search` 的描述"什么都能做"，语义被稀释；而专用工具
（如 `search_news`）描述聚焦。对一个聚焦的 `need`（"获取特斯拉最近的新闻"），聚焦的专用工具余弦相似度
更高、排在前面，通用工具往往进不了 top-k，于是根本不会被加载 —— 检索层天然起到了"精度过滤"作用。
**检索预筛选为什么不够？** 检索预筛选（`run_retrieval_prefilter`）只按**初始查询**做一次语义匹配、
一次性注入 top-n 工具。对"查股价 + 搜新闻"这类多步跨领域任务，初始查询的向量往往偏向第一个领域，
第二个子任务需要的专用工具可能挤不进 top-n，模型执行到一半才发现"想调用的工具根本没在清单里"——
这正是书中指出的一次性匹配的内在局限。主动发现把"发现"延后到执行中、按每个真实浮现的 `need` 分别检索，
从而补齐这一缺口（离线自检里可直接观察到：检索预筛选在半数多步任务上漏掉了第二个工具，见下表）。
### 运行
```bash
# 在仓库根目录使用统一的第 4 章环境
uv sync --locked --python 3.12 --extra ch4
# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat
# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch4]"
cd chapter4/active-tool-discovery
# 迁移期间仍支持单项目兼容路径：
# python -m pip install -r requirements.txt
# 方式 A：离线机制自检（无需任何 key；token/延迟真实，准确率仅反映启发式路由）
python demo.py --offline
# 方式 B：真实模型（体现小模型"指令遵循退化"需要真实 LLM）
cp env.example .env # 填入 OPENAI_API_KEY（chat 与 embeddings 都用 OpenAI）
# 兜底：若无 OPENAI_API_KEY 但设置了 OPENROUTER_API_KEY，chat 会自动改走 OpenRouter
#（模型映射到 openai/gpt-5.6-luna 等），工具检索退用本地哈希嵌入（OpenRouter 无 embeddings 接口）。
python demo.py # 全部 8 任务 × 三种策略
python demo.py --strategies full,discovery # 只跑其中两种策略对比
python demo.py --tasks finance+news,crypto+news # 只跑指定任务（逗号分隔）
python demo.py --tasks 'opinion(诱导)' # 含括号的任务 id 记得加引号
python demo.py --tool-set-size 20 # 缩小工具集，看全量注入的劣势如何随规模放大
python demo.py --query '查英伟达股价再搜点相关新闻' --offline # 临时单条自然语言任务
python demo.py --offline --output results/offline.json # 导出结构化结果
```
默认模型 `gpt-5.6-luna`，可用 `--model` 或 env 覆盖：`python demo.py --model gpt-5.6-luna`。
首次运行会为工具生成嵌入向量并缓存到 `.cache/`，之后复用。`python demo.py --help` 查看全部参数
（`--query / --tasks / --strategies / --tool-set-size / --top-k / --prefilter-n / --model / --embed-model / --max-steps / --offline / --output`）。
### 如何适配 / 扩展
- **换模型**：`MODEL=gpt-4.1-mini python demo.py`（chat 模型）；`EMBED_MODEL=text-embedding-3-large` 换嵌入模型
（换嵌入模型会因签名变化自动重建 `.cache/` 索引）。
- **换供应商 / 网关**：chat 与 embeddings 都走 OpenAI SDK，`OpenAI()` 会自动读取环境变量 `OPENAI_BASE_URL`，
因此指向任意 **OpenAI 兼容**的网关/代理只需 `OPENAI_BASE_URL=https://your-gateway/v1`（该端点需同时提供 chat 与 embeddings）。
- **换任务 / 输入**：编辑 `tools_library.py` 里的 `TASKS`（每条含 `prompt` 与判分用的能力槽位），或用
`--tasks` 只跑其中几条，或用 `--query` 传一句临时需求；想扩充工具库同样在 `tools_library.py` 的 `ALL_TOOLS` 中增删。
- **离线自检**：`--offline` 用 `offline_backend.py` 的本地哈希嵌入 + 脚本化 mock 模型，无需任何 key，
适合 CI、无网环境或快速验证流水线；它复现的是 token/延迟结构与"检索预筛选一次性漏工具"的机制，
不复现真实模型在长上下文工具墙下的选择行为（后者见下方 gpt-5.6-luna 真实结果）。
### 离线机制自检（本地嵌入 + mock 模型，`python demo.py --offline`）
下表为一次真实的 `--offline` 运行（8 任务 × 三策略）。**token/延迟是 tiktoken/wall-clock 真实测量**；
**准确率仅反映脚本化启发式路由，不代表真实模型能力**——mock 模型是"强路由器"，不会退化，所以全量注入也拿满分。
| 策略 | 精确选对 | 任务完成 | 平均注入 token | 总注入 token | 平均延迟(s) |
|---|---|---|---|---|---|
| 全量注入 | 8/8 | 8/8 | 11630 | 93040 | 0.008 |
| 检索预筛选 | 4/8 | 4/8 | 1030 | 8236 | 0.006 |
| 主动发现 | 8/8 | 8/8 | 974 | 7796 | 0.010 |
离线自检要传达的**两个真实、可复现的结构性结论**：
1. **token 随工具集规模放大而分化**：全量注入固定 11,630 token/任务；检索预筛选与主动发现按需只注入
约 1,000 token（**~11.9× 精简**）。用 `--tool-set-size 20` 缩小工具集，差距收敛到 ~1.8×——印证"工具越多、
全量注入越吃亏"。
2. **检索预筛选在多步跨领域任务上结构性漏工具**：一次性 top-10 检索在 8 个任务里有 4 个漏掉了第二个子任务
所需的专用工具（如 `academic(诱导)` 的 top-10 里根本没有 `arxiv_search`），模型执行到一半调不到工具 →
子任务失败；主动发现按每个真实浮现的 `need` 分别检索，8/8 补齐。
### 结论（基于一次真实运行，gpt-5.6-luna，2026-07）
> 说明：下表是一次真实 LLM 运行（`python demo.py --model gpt-5.6-luna`，8 任务 × 三策略，OpenAI 直连
> chat + `text-embedding-3-small` 检索）。gpt-5.6-luna 是推理型模型，仅支持默认 `temperature=1`
> （不支持 `temperature=0`，代码遇到该报错会自动回退到默认温度），故本次为**单次、非确定性**运行；
> token/延迟为真实测量，逐任务的选择结果可能随采样波动。判定：✅=精确选对（覆盖全部能力槽位且未错选
> 通用兜底工具）；⚠️=完成但顺手错选了通用工具；❌=出错（漏用专用工具或中途放弃、0 次工具调用）。
| 任务 | 全量注入 | 检索预筛选 | 主动发现 | 全量 token | 发现 token |
|---|---|---|---|---|---|
| finance+news | ✅ | ❌ | ✅ | 11630 | 883 |
| arxiv+download | ✅ | ❌ | ✅ | 11630 | 927 |
| github+viz | ❌ | ❌ | ❌ | 11630 | 295 |
| weather+calendar | ❌ | ✅ | ✅ | 11630 | 1055 |
| forex+weather | ✅ | ✅ | ❌ | 11630 | 295 |
| crypto+news | ❌ | ⚠️ | ❌ | 11630 | 295 |
| opinion(诱导) | ⚠️ | ❌ | ✅ | 11630 | 688 |
| academic(诱导) | ⚠️ | ⚠️ | ❌ | 11630 | 295 |
| **精确选对** | **3/8** | **2/8** | **4/8** | | |
| **任务完成** | **5/8** | **4/8** | **4/8** | | |
| **总注入 token** | | | | **93040** | **4733** |
（检索预筛选平均 971 token/任务、总 7768；三策略平均延迟约 11.5 / 9.6 / 10.7 s，均为本次真实测量。）
1. **token 节省依旧稳健（且更悬殊）**：全量注入每任务固定注入 **11,630 token**；主动发现按需加载后仅
**295~1,055 token**，合计 93,040 → 4,733（**~19.7×**）。需诚实说明：本次比值偏大，部分是因为
gpt-5.6-luna 在若干任务上直接放弃、根本没触发 `discover_tools`（此时只注入 3 个基础工具 = 295 token）。
即便如此，"全量注入固定重复计费上万 token、按需发现只注入千级 token"这一结构性收益不受影响。
2. **书中核心现象在两个"诱导任务"上如实复现**：措辞偏泛时，全量注入会顺手抓通用兜底工具——
- `opinion(诱导)`（"特斯拉最近的新闻舆论风向"）：全量注入调用了 `search_news, search_news,
web_search, search_tweets`，把通用的 **`web_search`** 也用上（⚠️ 错选）；主动发现检索到
`search_news / get_news_by_source / ...`（**没有** `web_search`），只调用专用新闻工具，**干净选对（✅）**。
- `academic(诱导)`（"量子计算最新科研进展"）：全量注入一口气调用了 8 个工具，其中
**`google_search / universal_search / ask_knowledge_base`** 三个都是通用兜底（⚠️）；检索预筛选也错选了
`google_search / universal_search`。这正是书中"上百工具的工具墙 + 措辞含糊 → 广撒网抓通用工具"的写照。
3. **本次运行暴露的另一类真实行为（与早期 gpt-4o-mini 运行不同，须如实记录）**：gpt-5.6-luna 是偏保守的
推理型模型，在多个任务上**没有调用（mock）工具就提前 `finish`**，理由多为"无法访问实时数据/工具"
（如 `github+viz`、`weather+calendar` 的全量注入，以及 `forex+weather`、`crypto+news`、`academic` 的
主动发现，均出现 0 次工具调用）。这压低了三种策略的绝对准确率，也意味着本次**得不出**"清晰任务下模型
面对工具墙一律选对"的结论——恰恰相反，放弃/漏步成了主要失分点，且这类失分在全量注入与主动发现上都存在。
4. **如实说明的边界**：
- 本实验用"schema 当纯文本注入 + 模型自行输出 JSON 工具调用"的控制组机制来观察长上下文选择行为；
mock 工具返回的是占位数据，保守的推理模型有时会识破并拒绝作答，这是本次准确率偏低的一大来源。
- 因 gpt-5.6-luna 仅支持默认 `temperature=1`，逐任务结果具随机性；重复运行时哪些任务"放弃"、哪些
"错选通用工具"会有波动，但两条结构性结论（token 节省、诱导任务下全量注入误用通用工具）方向稳定。
- 想要更干净、可复现的机制自检（token/延迟结构 + 检索预筛选一次性漏工具），见上方 `--offline` 表。
> 一句话：**在 gpt-5.6-luna 上，主动工具发现最稳的收益仍是 token（本次 ~19.7×）；在措辞含糊、通用工具
> 易被误用的"诱导任务"上，嵌入检索确实把 `web_search / google_search / universal_search` 等夸大其词的
> 通用工具挡在候选之外。但这一版真实运行也提醒：强推理模型保守的"放弃"行为会同时拉低各策略的绝对准确率，
> 单次结果需按上表如实解读。**
### 模型 ↔ 脚手架此消彼长（弱模型 gpt-4o-mini vs 强模型 gpt-5.6-luna，均为真实运行）
> 这一节回答一个直接的问题：**模型变强，这套"主动工具发现"脚手架是不是就没用了？**
> 我们把上面的 gpt-5.6-luna（强）结果，与同样 8 任务 × 三策略、OpenAI 直连 chat +
> `text-embedding-3-small` 检索的 **gpt-4o-mini（弱）** 真实运行放在一起对照
> （`python demo.py --model gpt-4o-mini`，2026-07，判定口径同上）。结论是：脚手架有两种价值，
> 一种随模型变强而**淡出**，另一种与模型强弱**无关、始终存在**。
**弱模型 gpt-4o-mini 真实汇总：**
| 策略 | 精确选对 | 任务完成 | 总注入 token | 平均延迟(s) |
|---|---|---|---|---|
| 全量注入 | 5/8 | **8/8** | 93040 | 8.38 |
| 检索预筛选 | 7/8 | 7/8 | 7768 | 4.90 |
| 主动发现 | **8/8** | **8/8** | 7266 | 7.65 |
（token 93040 → 7266，**~12.8×** 精简。）
#### 价值一：避免"错选通用工具"—— 随模型变强而**淡出**（fading）
- **弱模型 gpt-4o-mini：脚手架价值巨大且干净。** 全量注入下，gpt-4o-mini 从不放弃（任务完成 8/8），
但在 3 个任务上"广撒网"抓了通用兜底工具——`crypto+news` 用了 `web_search`，两个诱导任务
`opinion` / `academic` 各自把 `web_search / google_search / universal_search` 一并调用——
于是全量注入只有 **5/8 精确**。主动发现让嵌入检索把这些夸大其词的通用工具**挡在候选之外**，
gpt-4o-mini 根本无从误用：**8/8 精确、0 次通用工具误用、且任务完成不降（仍 8/8）**。
即"全量 5/8 → 发现 8/8 精确，+3 个任务，零完成损失"——这正是书中"上百工具的工具墙 +
措辞含糊 → 广撒网抓通用工具"的弱模型病症，脚手架把它一次性治好。
- **强模型 gpt-5.6-luna：同一价值明显缩水。** 它在全量注入下的"通用工具误用"只剩 2 个任务
（`opinion` / `academic`），比 gpt-4o-mini 的 3 个更少；主动发现把这两处也擦干净，但精确率只从
**3/8 提到 4/8（+1）**，而且**任务完成反而从 5/8 降到 4/8**。原因在于强推理模型的主要失分点
**不是"选错工具"，而是"直接放弃"**：多个任务它 0 次工具调用就 `finish`（理由多为"无法访问实时数据"），
这类失分检索层无法修复，工具可见得更少时甚至略微加剧。**换言之，脚手架专治的"错选通用工具"这一弱点，
在强模型上本就稀薄，收益随之淡出。**
#### 价值二：节省注入 token —— 与模型强弱**无关、始终存在**（persisting）
全量注入无论模型强弱都固定为 **11,630 token/任务**（把 126 个工具 schema 全塞进 system），
这是纯结构性开销。按需发现只注入几百到一千余 token：
- 弱模型 gpt-4o-mini：93,040 → 7,266，**~12.8×**；
- 强模型 gpt-5.6-luna：93,040 → 4,733，**~19.7×**（比值更大，部分是因为它常放弃、根本没触发 `discover_tools`，
只注入 3 个基础工具）。
两个模型上 token 节省都稳稳成立，且随工具集变大而放大——**这份收益不因模型变强而消失**，
是脚手架在"强模型时代"仍然值得保留的硬理由。
#### 一句话小结
> **模型越强，脚手架"帮它别选错工具"的价值越淡（gpt-4o-mini 全量 5/8→发现 8/8 精确、零完成损失；
> gpt-5.6-luna 仅 3/8→4/8 且完成还降了，因为它的失分是"放弃"而非"错选"）；但"省 token"的价值
> 与模型强弱无关、始终存在（弱模型 ~12.8×、强模型 ~19.7×，全量注入恒为 11,630 token/任务）。
> 所以在强模型上，主动工具发现的主要理由从"纠正指令遵循退化"转向"控制上下文成本"。**
### 文件
- `tools_library.py` — 126 个工具定义 + `select_tools` 子集截取 + mock 执行 + 8 个评测任务与判分标准
- `discovery.py` — 可插拔嵌入后端（`OpenAIEmbedder`）+ 工具向量索引与相似度检索（`discover_tools`/预筛选的后端）
- `agent.py` — 三种策略（全量注入 / 检索预筛选 / 主动发现）的 ReAct 循环与 token 统计
- `offline_backend.py` — 离线后端：`LocalEmbedder` + `MockChatClient`，支撑 `--offline` 无 key 自检
- `demo.py` — 一键多策略对比演示（含 CLI：`--query/--tasks/--strategies/--tool-set-size/--offline/--output` 等）
- `requirements.txt` / `env.example`
---
## Notes / 说明
- Commands, paths, env vars, and measured tables are identical on both language sides.
- 两侧命令、路径、环境变量与实测表格保持一致。
- Offline path needs no API key; real-model tables are single-run, non-deterministic (temperature=1).
- 离线路径无需 API Key；真实模型表为单次非确定性运行（temperature=1）。
chapter4/active-tool-discovery/agent.py
+281
View File
@@ -0,0 +1,281 @@
"""
三种工具发现策略的 Agent 循环（文本/ReAct 协议）。
为什么用"文本注入 + 文本解析工具调用"而不是 OpenAI 原生 function calling？
—— 本实验要复现的正是书中所述：把 120+ 工具 schema **一次性注入 system prompt（几万 token）**，
模型在超长上下文下"指令遵循退化"。OpenAI 原生 function-calling 接口对工具选择做了很强的
约束/优化，即使上百个工具也很少选错，无法体现该退化；而把 schema 当作纯文本塞进 prompt、
让模型自己以 JSON 形式输出工具调用，才是书中控制组的真实机制，也才能观察到退化。
协议：模型每一步只输出一个 JSON：
{"thought": "...", "tool": "工具名", "arguments": {...}}
任务完成时输出：
{"thought": "...", "tool": "finish", "arguments": {"answer": "..."}}
1) run_full_injection —— 对照组（全量注入）
system prompt 里以文本列出全部 126 个工具。injected_tokens = 该工具清单文本的 token 数。
2) run_retrieval_prefilter —— 对照组之二（检索预筛选）
按用户初始查询做**一次性**语义检索，只把 top-n 个候选工具注入 system prompt。
token 已大幅下降，但一次性匹配无法预见执行中才浮现的跨领域需求（书中所述局限）。
3) run_active_discovery —— 实验组（主动发现）
system prompt 只列出少量基础工具 + discover_tools 元工具。
模型调用 discover_tools(need) 时，用嵌入相似度返回 3-5 个候选工具，其文本清单作为
**user message** 追加进对话（保护 system 前缀 KV Cache），并更新状态栏可用工具列表。
injected_tokens = 基础工具 + discover_tools + 实际发现加载的工具清单的 token 数。
"""
import json
import re
from typing import Dict, List
import tiktoken
from discovery import ToolIndex # noqa: F401 (类型提示用)
from tools_library import (ALL_TOOLS, BASE_TOOL_NAMES, TOOL_IMPLS,
TOOLS_BY_NAME)
try:
_ENC = tiktoken.get_encoding("o200k_base") # gpt-4o 系列编码
except Exception:
_ENC = tiktoken.get_encoding("cl100k_base")
# ---------------------------------------------------------------------------
# 工具清单文本渲染 & token 统计
# ---------------------------------------------------------------------------
def render_tool(tool: Dict) -> str:
"""把单个工具渲染成完整 JSON schema 文本（与真实注入到 prompt 的形式一致）。"""
return json.dumps(tool["function"], ensure_ascii=False, indent=2)
def render_tools(tools: List[Dict]) -> str:
return "\n".join(render_tool(t) for t in tools)
def count_tokens(text: str) -> int:
return len(_ENC.encode(text)) if text else 0
# discover_tools 元工具（也用文本形式呈现给模型）
DISCOVER_TOOL = {
"type": "function",
"function": {
"name": "discover_tools",
"description": ("发现新工具：当缺少合适的专用工具时调用它，用一句自然语言描述你需要的"
"『能力』(need)，系统会用语义检索返回最匹配的若干专用工具及其定义，之后即可调用它们。"),
"parameters": {"type": "object",
"properties": {"need": {"type": "string"}}, "required": ["need"]},
},
}
FINISH_TOOL_DESC = "- finish(answer: string): 所有子任务都完成后调用，给出最终回答。"
_PROTOCOL = (
"你每一步都必须、且只能输出一个 JSON 对象，不要输出任何多余文字，格式为：\n"
'{"thought": "简要思考", "tool": "工具名", "arguments": {参数键值}}\n'
"系统会执行该工具并把结果返回给你，然后你再输出下一步。\n"
"当且仅当任务的所有子任务都已用合适的工具完成后，输出："
'{"thought": "...", "tool": "finish", "arguments": {"answer": "最终回答"}}\n'
"注意：请为每个子任务选择最匹配的『专用工具』，而不是笼统的通用搜索工具。"
)
def _extract_json(text: str):
"""从模型回复里抽取第一个 JSON 对象。"""
text = text.strip()
text = re.sub(r"^```(?:json)?|```$", "", text, flags=re.MULTILINE).strip()
# 找到第一个 { 到匹配的 }
start = text.find("{")
if start == -1:
return None
depth = 0
for i in range(start, len(text)):
if text[i] == "{":
depth += 1
elif text[i] == "}":
depth -= 1
if depth == 0:
try:
return json.loads(text[start:i + 1])
except json.JSONDecodeError:
return None
return None
def _run_loop(client, model, system_prompt, task_prompt, available_names,
on_discover=None, max_steps=10):
"""
文本 ReAct 循环。
available_names: set，当前允许调用的工具名（不含 discover_tools/finish）。
—— 主动发现模式下会随 discover_tools 动态增长。
返回 (called_tools, trace, finished)。
"""
messages = [{"role": "system", "content": system_prompt},
{"role": "user", "content": task_prompt}]
called: List[str] = []
trace: List[str] = []
finished = False
for _ in range(max_steps):
try:
resp = client.chat.completions.create(
model=model, messages=messages, temperature=0)
except Exception as e:
# 部分推理型模型（如 gpt-5.x）只支持默认 temperature=1，此时退回默认值重试。
if "temperature" in str(e):
resp = client.chat.completions.create(
model=model, messages=messages)
else:
raise
content = resp.choices[0].message.content or ""
messages.append({"role": "assistant", "content": content})
action = _extract_json(content)
if action is None or "tool" not in action:
trace.append(f"[格式错误] 模型未输出合法 JSON: {content[:80]!r}")
messages.append({"role": "user",
"content": "你的回复不是合法的 JSON，请只输出规定格式的 JSON 对象。"})
continue
name = action.get("tool")
args = action.get("arguments") or {}
if name == "finish":
trace.append(f"[finish] {str(args.get('answer',''))[:100]}")
finished = True
break
if name == "discover_tools" and on_discover is not None:
need = args.get("need", "")
result_text, new_names = on_discover(need)
called.append(name)
trace.append(f"[discover_tools] need='{need}' -> {new_names}")
available_names.update(new_names)
messages.append({"role": "user", "content": result_text})
continue
# 普通工具调用
if name not in available_names:
# 该工具当前不可用（主动发现里还没发现 / 预筛选没选中 / 或纯属幻觉）——
# 不计入 called（未真正执行），判分因此能体现该子任务失败。
trace.append(f"[不可用] {name}")
hint = ("该工具当前不可用。"
+ ("请先用 discover_tools 发现所需能力的工具。" if on_discover else
"请从工具清单中选择一个存在的工具。"))
messages.append({"role": "user", "content": hint})
continue
called.append(name)
impl = TOOL_IMPLS.get(name)
result = impl(args) if impl else json.dumps({"error": f"unknown tool {name}"})
trace.append(f"[call] {name}({json.dumps(args, ensure_ascii=False)})")
messages.append({"role": "user", "content": f"工具 {name} 返回：{result}"})
return called, trace, finished
# ---------------------------------------------------------------------------
# 对照组：全量注入
# ---------------------------------------------------------------------------
def run_full_injection(client, model, task_prompt: str, tools: List[Dict] = None,
max_steps: int = 10) -> Dict:
tools = tools if tools is not None else ALL_TOOLS
tools_text = render_tools(tools) + "\n" + FINISH_TOOL_DESC
injected = count_tokens(tools_text)
system = (
f"你是一个智能助手。下面是你可以使用的全部工具清单（共 {len(tools)} 个），"
"请根据任务选择最合适的工具来完成。若任务包含多个子任务，请确保每个子任务都被处理。\n\n"
"【工具清单】\n" + tools_text + "\n\n" + _PROTOCOL
)
available = {t["function"]["name"] for t in tools}
called, trace, finished = _run_loop(client, model, system, task_prompt, available,
max_steps=max_steps)
return {"mode": "full_injection", "injected_tokens": injected,
"num_tools_exposed": len(tools), "called": called,
"trace": trace, "finished": finished}
# ---------------------------------------------------------------------------
# 对照组之二：检索预筛选（书中"检索式预筛选"）
# —— 按用户初始查询做**一次性**语义检索，只把 top-n 个候选工具注入 system prompt。
# 它介于"全量注入"与"主动发现"之间：token 已大幅下降，但只匹配一次，无法预见
# 任务执行中才浮现的跨领域需求（书中所述的内在局限）——若第二个子任务所需的
# 专用工具没被这一次检索选中，模型就无从调用它，导致该子任务失败。
# ---------------------------------------------------------------------------
def run_retrieval_prefilter(client, model, task_prompt: str, index, top_n: int = 10,
tools: List[Dict] = None, max_steps: int = 10) -> Dict:
tools = tools if tools is not None else ALL_TOOLS
tbn = {t["function"]["name"]: t for t in tools}
hits = index.search(task_prompt, top_k=top_n)
picked = [name for name, _ in hits if name in tbn]
picked_tools = [tbn[n] for n in picked]
tools_text = render_tools(picked_tools) + "\n" + FINISH_TOOL_DESC
injected = count_tokens(tools_text)
system = (
f"你是一个智能助手。系统已根据你的任务预先检索出下列可能相关的工具（共 {len(picked_tools)} 个），"
"请从中选择合适的工具完成任务。若某个子任务在清单中找不到合适的工具，请如实说明。\n\n"
"【工具清单】\n" + tools_text + "\n\n" + _PROTOCOL
)
available = set(picked)
called, trace, finished = _run_loop(client, model, system, task_prompt, available,
max_steps=max_steps)
return {"mode": "retrieval_prefilter", "injected_tokens": injected,
"num_tools_exposed": len(picked_tools), "prefiltered": picked,
"called": called, "trace": trace, "finished": finished}
# ---------------------------------------------------------------------------
# 实验组：主动发现
# ---------------------------------------------------------------------------
def run_active_discovery(client, model, task_prompt: str, index, top_k=4,
tools: List[Dict] = None, max_steps: int = 10) -> Dict:
tools = tools if tools is not None else ALL_TOOLS
tbn = {t["function"]["name"]: t for t in tools}
base_tools = [tbn[n] for n in BASE_TOOL_NAMES]
base_text = (render_tools(base_tools) + "\n"
+ render_tool(DISCOVER_TOOL) + "\n" + FINISH_TOOL_DESC)
discovered_names = set() # 本轮实际发现加载的专用工具
discovered_texts: List[str] = [] # 对应的文本清单（用于统计按需注入 token）
available = set(BASE_TOOL_NAMES)
def on_discover(need: str):
hits = index.search(need, top_k=top_k)
names, lines = [], []
for name, score in hits:
if name in BASE_TOOL_NAMES:
continue
names.append(name)
lines.append(render_tool(tbn[name]) + f" (相似度 {score:.3f})")
if name not in discovered_names:
discovered_names.add(name)
discovered_texts.append(render_tool(tbn[name]))
status = f"\n\n【状态栏｜当前可用工具】{sorted(available | set(names))}"
body = ("discover_tools 匹配到以下专用工具，已加载，可直接调用：\n"
+ "\n".join(lines) + status)
return body, names
system = (
"你是一个智能助手。你当前只掌握少量基础工具（见下）。"
"当任务需要你没有的能力时，先调用 discover_tools，用自然语言描述你需要的能力，"
"系统会返回并加载匹配的专用工具，然后你再调用它们。"
"若任务包含多个子任务（如既要查询又要下载），请针对每一项能力分别调用 discover_tools，"
"并在结束前确认每个子任务都已用合适的工具完成。\n\n"
"【基础工具】\n" + base_text + "\n\n" + _PROTOCOL
)
called, trace, finished = _run_loop(client, model, system, task_prompt,
available, on_discover=on_discover,
max_steps=max_steps)
injected = count_tokens(base_text) + count_tokens("\n".join(discovered_texts))
return {"mode": "active_discovery", "injected_tokens": injected,
"num_tools_exposed": len(BASE_TOOL_NAMES) + 1 + len(discovered_names),
"discovered": sorted(discovered_names),
"called": called, "trace": trace, "finished": finished}
chapter4/active-tool-discovery/demo.py
+264
View File
@@ -0,0 +1,264 @@
"""
实验 8-4 演示：主动工具发现 vs 检索预筛选 vs 全量注入
对同一组跨领域任务，在 126 个工具的工具库上分别用三种"工具发现"策略运行，
并在一次运行里输出可对比的表格（准确率 / 注入 token / 延迟）：
- full_injection 全量注入：126 个工具 schema 一次性进上下文（对照组，书中控制组）。
- retrieval_prefilter 检索预筛选：按初始查询做**一次性**语义检索，只注入 top-n 候选工具。
- active_discovery 主动发现：少量基础工具 + discover_tools 元工具，执行中按需检索加载。
核心论点（第 8 章）：当工具规模达到上百个时，"把所有工具塞进上下文"在 token 上昂贵、
且对小模型的指令遵循是灾难；主动发现按需加载，token 大幅下降、选择更精准。
用法（详见 --help）：
python demo.py # 默认：全部任务 × 三种策略（需 OPENAI_API_KEY）
python demo.py --offline # 离线自检：本地嵌入 + mock 模型，无需任何 key
python demo.py --tasks finance+news,crypto+news
python demo.py --strategies full,discovery --tool-set-size 30
python demo.py --query "查一下英伟达股价再搜点相关新闻" --offline
python demo.py --offline --output results/offline.json
"""
import argparse
import json
import os
import sys
import time
from tools_library import TASKS, grade, select_tools, ALL_TOOLS
def _to_openrouter_model(model: str) -> str:
"""把常见模型名映射到 OpenRouter 命名空间（用于无 OPENAI_API_KEY 的兜底路径）。"""
if not model:
return "openai/gpt-5.6-luna"
if "/" in model:
return model
if model.startswith("gpt-"):
return "openai/" + model
if model.startswith("claude-"):
return "anthropic/claude-opus-4.8"
return "openai/gpt-5.6-luna"
# 策略注册表：key -> (中文名, 需要 index 吗)
STRATEGIES = {
"full": ("全量注入", False),
"prefilter": ("检索预筛选", True),
"discovery": ("主动发现", True),
}
STRATEGY_ORDER = ["full", "prefilter", "discovery"]
def build_parser() -> argparse.ArgumentParser:
ap = argparse.ArgumentParser(
prog="demo.py",
formatter_class=argparse.RawDescriptionHelpFormatter,
description="实验 8-4：主动工具发现 vs 检索预筛选 vs 全量注入。\n"
"在 126 个工具的工具库上，对多任务一次性输出『准确率 / 注入 token / 延迟』对比表，\n"
"验证第 8 章论点：上百工具场景下，主动按需发现优于把全部工具塞进上下文。",
epilog="示例：\n"
" python demo.py --offline # 无需 key 的离线机制自检\n"
" python demo.py --strategies full,discovery --tasks finance+news\n"
" python demo.py --query '查英伟达股价并搜相关新闻' --offline\n")
ap.add_argument("--query", metavar="TEXT",
help="临时单任务：直接给一句自然语言需求，跳过内置任务集（判分槽位按关键词自动推断）。")
ap.add_argument("--tasks", metavar="IDS",
help="逗号分隔的内置任务 id（见 tools_library.TASKS），缺省跑全部 8 个任务。"
"含括号的 id 记得加引号，如 'opinion(诱导)'。")
ap.add_argument("--strategies", metavar="LIST", default="full,prefilter,discovery",
help="逗号分隔的策略，取值 full/prefilter/discovery，缺省三者全跑并对比。")
ap.add_argument("--tool-set-size", type=int, default=None, metavar="N",
help="把工具库截取为 N 个工具（始终保留基础/通用/任务相关工具）。"
"缺省用全部 126 个——用小 N 可对比『工具集越大，全量注入越吃亏』。")
ap.add_argument("--top-k", type=int, default=4, metavar="K",
help="主动发现中 discover_tools 每次返回的候选工具数（默认 4）。")
ap.add_argument("--prefilter-n", type=int, default=10, metavar="N",
help="检索预筛选一次性注入的候选工具数（默认 10）。")
ap.add_argument("--model", default=os.getenv("MODEL", "gpt-5.6-luna"), metavar="NAME",
help="对话模型名（默认取环境变量 MODEL 或 gpt-5.6-luna）；离线模式下忽略。")
ap.add_argument("--embed-model", default=os.getenv("EMBED_MODEL", "text-embedding-3-small"),
metavar="NAME", help="嵌入模型名（默认 text-embedding-3-small）；离线模式下忽略。")
ap.add_argument("--max-steps", type=int, default=10, metavar="N",
help="单个任务的 ReAct 最大步数（默认 10）。")
ap.add_argument("--offline", action="store_true",
help="离线机制自检：用本地哈希嵌入 + 脚本化 mock 模型，无需任何 API key。"
"token/延迟为真实测量，准确率仅反映启发式路由、不代表真实模型能力。")
ap.add_argument("--output", metavar="PATH",
help="把逐任务、逐策略的结构化结果写入该 JSON 文件。")
return ap
def _fmt_grade(g):
tag = "✅ 精确选对" if g["precise"] else ("⚠️ 完成但错选" if g["correct"] else "❌ 出错")
detail = f"{g['filled_slots']}/{g['total_slots']} 能力槽位命中"
extra = ""
if g["missed_slots"]:
extra += f"｜漏用: {[s[0] for s in g['missed_slots']]}"
if g["used_generic_substitute"]:
extra += f"｜错选通用工具: {g['used_generic_substitute']}"
return f"{tag}（{detail}{extra}）"
def _make_task_from_query(query: str):
"""把临时 --query 包装成带判分槽位的任务（槽位按关键词推断）。"""
from offline_backend import match_intents
slots = [[tool] for tool, _ in match_intents(query)]
return {"id": "adhoc", "prompt": query, "required_slots": slots}
def run_strategy(key, client, model, prompt, index, tools, args):
"""执行一种策略并返回 (result_dict, latency_s)。"""
from agent import (run_active_discovery, run_full_injection,
run_retrieval_prefilter)
t0 = time.perf_counter()
if key == "full":
res = run_full_injection(client, model, prompt, tools=tools, max_steps=args.max_steps)
elif key == "prefilter":
res = run_retrieval_prefilter(client, model, prompt, index,
top_n=args.prefilter_n, tools=tools, max_steps=args.max_steps)
else:
res = run_active_discovery(client, model, prompt, index,
top_k=args.top_k, tools=tools, max_steps=args.max_steps)
return res, time.perf_counter() - t0
def main():
args = build_parser().parse_args()
strategies = [s.strip() for s in args.strategies.split(",") if s.strip()]
bad = [s for s in strategies if s not in STRATEGIES]
if bad:
print(f"未知策略: {bad}，可选: {list(STRATEGIES)}")
sys.exit(2)
strategies.sort(key=STRATEGY_ORDER.index)
# ---- 任务集 ----
if args.query:
tasks = [_make_task_from_query(args.query)]
else:
tasks = TASKS
if args.tasks:
want = set(args.tasks.split(","))
tasks = [t for t in TASKS if t["id"] in want]
if not tasks:
print(f"没有匹配的任务 id：{args.tasks}")
sys.exit(2)
tools = select_tools(args.tool_set_size, tasks)
need_index = any(STRATEGIES[s][1] for s in strategies)
# ---- 后端（在线 OpenAI / 离线 mock）----
if args.offline:
from offline_backend import LocalEmbedder, MockChatClient
from discovery import ToolIndex
client = MockChatClient()
model = "mock-offline"
embedder = LocalEmbedder()
print("=" * 92)
print("离线机制自检模式：本地哈希嵌入 + 脚本化 mock 模型（无需 API key）。")
print(" · token / 延迟为真实测量；准确率仅反映启发式路由，不代表真实模型能力。")
print(" · 观察点：三种策略的 token 差距，以及『检索预筛选一次性匹配』的结构性漏工具。")
print("=" * 92)
else:
try:
from dotenv import load_dotenv
from openai import OpenAI
except ImportError:
print("缺少 openai / python-dotenv，请先 pip install -r requirements.txt，"
"或改用 --offline 离线自检。")
sys.exit(1)
load_dotenv()
from discovery import OpenAIEmbedder, ToolIndex
if os.getenv("OPENAI_API_KEY"):
# 直连 OpenAI：chat + embeddings 都走 OpenAI
client = OpenAI()
model = args.model
embedder = OpenAIEmbedder(client, model=args.embed_model)
elif os.getenv("OPENROUTER_API_KEY"):
# 统一兜底：OpenRouter 只代理 chat completions，没有 embeddings 接口，
# 因此对话走 OpenRouter（真实模型），工具检索改用本地哈希嵌入。
from offline_backend import LocalEmbedder
client = OpenAI(api_key=os.getenv("OPENROUTER_API_KEY"),
base_url="https://openrouter.ai/api/v1")
model = _to_openrouter_model(args.model)
embedder = LocalEmbedder()
print("未检测到 OPENAI_API_KEY，改走 OpenRouter 兜底：")
print(f" · 对话模型: {model}（真实调用）")
print(" · 工具检索: 本地哈希嵌入（OpenRouter 无 embeddings 接口）。")
else:
print("请设置 OPENAI_API_KEY 或 OPENROUTER_API_KEY（见 env.example），"
"或改用 --offline 离线自检。")
sys.exit(1)
index = ToolIndex(embedder, tools=tools) if need_index else None
print(f"模型: {model} | 嵌入: {embedder.name} | 工具库: {len(tools)} 个 "
f"| 任务数: {len(tasks)} | 策略: {[STRATEGIES[s][0] for s in strategies]}\n")
# ---- 逐任务运行 ----
records = [] # 每条: {task, strategy, result, grade, latency}
for task in tasks:
print("=" * 92)
print(f"任务 [{task['id']}]: {task['prompt']}")
print("-" * 92)
for key in strategies:
res, latency = run_strategy(key, client, model, task["prompt"], index, tools, args)
g = grade(task, res["called"])
records.append({"task": task["id"], "strategy": key, "result": res,
"grade": g, "latency_s": round(latency, 3)})
cname = STRATEGIES[key][0]
print(f"[{cname}] 注入 {res['injected_tokens']:>6} tokens "
f"（暴露 {res['num_tools_exposed']} 个工具） 延迟 {latency:5.2f}s")
if key == "prefilter":
print(f" 预筛选命中: {res['prefiltered']}")
if key == "discovery":
for line in res["trace"]:
if line.startswith("[discover_tools]"):
print(f" {line}")
print(f" 发现并加载: {res['discovered']}")
print(f" 调用轨迹: {res['called']}")
print(f" 判定: {_fmt_grade(g)}")
print()
_print_summary(tasks, strategies, records)
if args.output:
os.makedirs(os.path.dirname(os.path.abspath(args.output)), exist_ok=True)
payload = {"model": model, "embedder": embedder.name, "tool_set_size": len(tools),
"offline": args.offline, "strategies": strategies,
"records": records}
json.dump(payload, open(args.output, "w", encoding="utf-8"),
ensure_ascii=False, indent=2)
print(f"\n结构化结果已写入: {args.output}")
def _print_summary(tasks, strategies, records):
n = len(tasks)
print("=" * 92)
print("汇总对比（『精确选对』= 覆盖全部能力槽位 且 未错选通用兜底工具）")
print("=" * 92)
header = f"{'策略':<14}{'精确选对':>10}{'任务完成':>10}{'平均注入token':>16}{'总注入token':>14}{'平均延迟(s)':>12}"
print(header)
print("-" * 92)
for key in strategies:
rs = [r for r in records if r["strategy"] == key]
precise = sum(int(r["grade"]["precise"]) for r in rs)
correct = sum(int(r["grade"]["correct"]) for r in rs)
tok = [r["result"]["injected_tokens"] for r in rs]
lat = [r["latency_s"] for r in rs]
avg_tok = sum(tok) / len(tok) if tok else 0
avg_lat = sum(lat) / len(lat) if lat else 0
print(f"{STRATEGIES[key][0]:<12}{f'{precise}/{n}':>10}{f'{correct}/{n}':>10}"
f"{avg_tok:>16.0f}{sum(tok):>14}{avg_lat:>12.3f}")
print("-" * 92)
if "full" in strategies and "discovery" in strategies:
ft = sum(r["result"]["injected_tokens"] for r in records if r["strategy"] == "full")
at = sum(r["result"]["injected_tokens"] for r in records if r["strategy"] == "discovery")
if at:
print(f"注入 token：全量注入 {ft} vs 主动发现 {at}，平均每任务精简约 {ft/at:.1f} 倍。")
if __name__ == "__main__":
main()
chapter4/active-tool-discovery/discovery.py
+102
View File
@@ -0,0 +1,102 @@
"""
主动工具发现的核心：用嵌入向量相似度，从 126 个工具里
按自然语言"能力需求"检索出最相关的 3-5 个候选工具。
- 工具向量：对每个工具用 "name: description" 生成 embedding，并缓存到本地
.cache/tool_embeddings_<embedder>.json，避免每次运行都重新计算。
- discover_tools(need)：把 need 向量化，与工具向量做余弦相似度，返回 top-k。
嵌入后端是可插拔的（见 `Embedder` 协议）：
- OpenAIEmbedder：调用 OpenAI embeddings API（默认，联网，效果最好）。
- 离线模式（--offline）使用 offline_backend.LocalEmbedder（本地哈希词袋，无需 API），
用于在没有 key 时验证整条流水线与量化 token/延迟。
"""
import hashlib
import json
import os
import re
from typing import Dict, List, Tuple
from tools_library import ALL_TOOLS
EMBED_MODEL = os.getenv("EMBED_MODEL", "text-embedding-3-small")
_CACHE_DIR = os.path.join(os.path.dirname(__file__), ".cache")
def _tool_text(tool: Dict) -> str:
f = tool["function"]
return f"{f['name']}: {f['description']}"
def _cosine(a: List[float], b: List[float]) -> float:
dot = sum(x * y for x, y in zip(a, b))
na = sum(x * x for x in a) ** 0.5
nb = sum(y * y for y in b) ** 0.5
return dot / (na * nb + 1e-9)
class OpenAIEmbedder:
"""基于 OpenAI embeddings API 的嵌入后端。"""
def __init__(self, client, model: str = None):
self.client = client
self.model = model or EMBED_MODEL
self.name = self.model
def embed(self, texts: List[str]) -> List[List[float]]:
resp = self.client.embeddings.create(model=self.model, input=texts)
return [d.embedding for d in resp.data]
class ToolIndex:
"""工具向量索引 + 相似度检索。
embedder: 具备 `.embed(texts) -> List[vec]` 与 `.name` 的对象；
为向后兼容，也可直接传入 OpenAI client（会自动包装为 OpenAIEmbedder）。
tools: 参与索引的工具子集，缺省为全部 ALL_TOOLS（配合 --tool-set-size 使用）。
"""
def __init__(self, embedder, tools: List[Dict] = None):
self.embedder = embedder if hasattr(embedder, "embed") else OpenAIEmbedder(embedder)
tools = tools if tools is not None else ALL_TOOLS
self.names = [t["function"]["name"] for t in tools]
self.texts = [_tool_text(t) for t in tools]
self.vectors = self._load_or_build()
def _cache_file(self) -> str:
safe = re.sub(r"[^A-Za-z0-9_.-]", "_", self.embedder.name)
return os.path.join(_CACHE_DIR, f"tool_embeddings_{safe}.json")
def _signature(self) -> str:
h = hashlib.sha256()
h.update(self.embedder.name.encode())
for t in self.texts:
h.update(t.encode())
return h.hexdigest()[:16]
def _load_or_build(self) -> Dict[str, List[float]]:
sig = self._signature()
cache_file = self._cache_file()
if os.path.exists(cache_file):
try:
cached = json.load(open(cache_file, encoding="utf-8"))
if cached.get("signature") == sig:
return cached["vectors"]
except Exception:
pass
# 缓存缺失或失效 -> 调用嵌入后端批量生成
print(f"[discovery] 正在用 {self.embedder.name} 为 {len(self.texts)} 个工具生成嵌入向量 ...")
embeddings = self.embedder.embed(self.texts)
vectors = {name: vec for name, vec in zip(self.names, embeddings)}
os.makedirs(_CACHE_DIR, exist_ok=True)
json.dump({"signature": sig, "vectors": vectors},
open(cache_file, "w", encoding="utf-8"))
return vectors
def search(self, need: str, top_k: int = 4) -> List[Tuple[str, float]]:
"""返回与 need 最相关的 top_k 个 (工具名, 相似度)。"""
q = self.embedder.embed([need])[0]
scored = [(name, _cosine(q, self.vectors[name])) for name in self.names]
scored.sort(key=lambda x: x[1], reverse=True)
return scored[:top_k]
chapter4/active-tool-discovery/env.example
+15
View File
@@ -0,0 +1,15 @@
# 复制为 .env 后填入你的 key
# 本实验默认用 OpenAI（chat + embeddings 均用 OpenAI）
# 若暂时没有 key，可先跑离线机制自检：python demo.py --offline（本地嵌入 + mock 模型，无需下面的 key）
OPENAI_API_KEY=your-openai-api-key
# 统一兜底：若没有 OPENAI_API_KEY，但设置了 OPENROUTER_API_KEY，则对话（chat）自动
# 改走 OpenRouter（模型映射到 openai/gpt-5.6-luna 等）；因 OpenRouter 无 embeddings
# 接口，工具检索会退用本地哈希嵌入。
# OPENROUTER_API_KEY=your-openrouter-api-key
# 可选：对话模型，默认 gpt-5.6-luna
# MODEL=gpt-5.6-luna
# 可选：嵌入模型，默认 text-embedding-3-small（仅 OpenAI 直连路径使用）
# EMBED_MODEL=text-embedding-3-small
chapter4/active-tool-discovery/experiment_protocol.json
+46
View File
@@ -0,0 +1,46 @@
{
"experiment": "4-7",
"title": "Active tool discovery with Qwen3-4B and the perception MCP server",
"authority": "book/chapter4.md:624",
"model": "qwen3:4b",
"model_runtime": "ollama",
"minimum_mcp_tools": 120,
"minimum_control_schema_tokens": 50000,
"control": {
"system_tools": "all complete schemas returned by perception MCP tools/list",
"required_behavior": "plan and execute using the full catalog"
},
"treatment": {
"system_tools": ["web_search", "code_interpreter", "discover_tools"],
"discovery_top_k": 5,
"schema_injection_role": "user",
"status_bar_updates": true,
"base_tool_boundary": "Generic web search and local code cannot substitute for authoritative domain-specific retrieval; discover each missing specialist at the moment the gap arises."
},
"tasks": [
{
"id": "apple_stock_news",
"prompt": "Query Apple's latest stock price and search related current news to explain the movement.",
"required_capabilities": ["specialized_stock_quote", "current_web_news"]
},
{
"id": "transformer_arxiv_download",
"prompt": "Find the latest transformer papers on arXiv and download the top three PDFs.",
"required_capabilities": ["arxiv_search", "file_download"],
"required_downloads": 3
},
{
"id": "github_contributors_visualization",
"prompt": "Analyze contributor statistics for openai/openai-python and generate a visualization report.",
"required_capabilities": ["github_contributors", "code_interpreter"],
"required_visualizations": 1
}
],
"acceptance": {
"catalog_from_mcp": true,
"no_mock_tool_results": true,
"all_receipts_machine_readable": true,
"raw_schema_catalog_gzipped_and_hashed": true,
"compare_accuracy_and_completion_honestly": true
}
}
chapter4/active-tool-discovery/offline_backend.py
+195
View File
@@ -0,0 +1,195 @@
"""
离线后端：让整条流水线在**没有 OpenAI key** 时也能跑通，用于验证机制、
量化 token/延迟，并让读者零成本复现"三种策略"的对比结构。
包含两部分：
1) LocalEmbedder —— 本地哈希词袋嵌入（中文字 unigram/bigram + 英文词），
无需联网即可支撑 discover_tools / 检索预筛选的语义相似度。
2) MockChatClient —— 一个确定性的"脚本化"模型，接口与 OpenAI 客户端一致
（client.chat.completions.create(...).choices[0].message.content）。
它按关键词把任务拆成若干子任务，遵循 ReAct 文本协议逐步调用工具。
重要边界说明：
- MockChatClient 是一个**强启发式路由器**，不代表真实小模型的能力，因此它**不会**复现
书中"超长上下文下指令遵循退化、错选通用工具"的现象——那需要真实的小参数量模型。
- 离线模式下真实可复现的是：① 各策略注入的 token 量（tiktoken 真实计算）；
② 检索预筛选"一次性匹配"的结构性局限（若第二个子任务的专用工具没被初始检索选中，
模型就调用不到它 → 该子任务失败）；③ 主动发现按需加载后仍能补齐工具、完成任务。
- 要观察真实模型在长上下文工具墙下的选择行为，请配置真实模型（见 README 中 gpt-5.6-luna 的真实结果表）。
"""
import hashlib
import json
import re
from types import SimpleNamespace
from typing import Dict, List, Tuple
from tools_library import TOOLS_BY_NAME
_DIM = 512
# ---------------------------------------------------------------------------
# 1) 本地嵌入后端
# ---------------------------------------------------------------------------
def _tokens(text: str) -> List[str]:
"""把中英文混合文本切成词袋 token：英文按词（并拆下划线），中文按字 unigram + bigram。"""
text = text.lower()
toks: List[str] = []
for w in re.findall(r"[a-z0-9]+", text):
toks.append(w)
han = re.findall(r"[一-鿿]", text)
toks += han
toks += [han[i] + han[i + 1] for i in range(len(han) - 1)]
return toks
class LocalEmbedder:
"""哈希词袋嵌入：确定性、无需联网。相似度由中英文关键词重叠驱动。"""
name = "local-hash-%d" % _DIM
def embed(self, texts: List[str]) -> List[List[float]]:
out = []
for t in texts:
vec = [0.0] * _DIM
for tok in _tokens(t):
h = int(hashlib.md5(tok.encode()).hexdigest(), 16)
vec[h % _DIM] += 1.0
norm = sum(x * x for x in vec) ** 0.5 or 1.0
out.append([x / norm for x in vec])
return out
# ---------------------------------------------------------------------------
# 2) 脚本化 mock 模型
# ---------------------------------------------------------------------------
# 意图规则：把任务关键词映射到"应当使用的专用工具"及一句能力需求描述。
# 顺序有意义（如"预报"类天气须排在通用"天气"之前）。
INTENT_RULES: List[Tuple[str, str, str]] = [
(r"股价|股票", "get_stock_price", "查询某股票的实时价格与涨跌幅"),
(r"以太坊|比特币|加密|\beth\b|\bbtc\b", "get_crypto_price", "查询加密货币的实时价格"),
(r"日元|汇率|美元.*换|换.*(日元|美元|欧元)|兑换", "get_forex_rate", "查询两种法定货币的外汇汇率"),
(r"论文|arxiv|文献|量子计算|科研进展|研究进展", "arxiv_search", "在学术论文库检索最新论文"),
(r"下载", "download_file", "从 URL 下载文件保存到本地"),
(r"贡献", "github_list_contributors", "获取 GitHub 仓库的贡献者提交统计"),
(r"图表|可视化|画个|画图|画一", "render_chart", "根据数据渲染可视化图表"),
(r"预报|未来|周日|这周|明天|后天|下周", "get_weather_forecast", "查询某城市未来若干天的天气预报"),
(r"天气", "get_current_weather", "查询某城市的实时天气"),
(r"日历|日程|活动|记一个|记录一个", "create_calendar_event", "在日历上创建一个事件"),
(r"新闻|舆论|消息|报道|风向", "search_news", "按关键词检索相关的最新新闻"),
]
def match_intents(prompt: str) -> List[Tuple[str, str]]:
"""返回任务涉及的 (专用工具名, 能力需求描述) 列表（去重、保序）。"""
needed: List[Tuple[str, str]] = []
seen = set()
for pat, tool, phrase in INTENT_RULES:
if re.search(pat, prompt, re.IGNORECASE) and tool not in seen:
needed.append((tool, phrase))
seen.add(tool)
# 天气去重：若命中"预报"则不再单独要求"实时天气"。
if "get_weather_forecast" in seen and "get_current_weather" in seen:
needed = [(t, p) for t, p in needed if t != "get_current_weather"]
return needed
_ARG_HINTS = {
"symbol": "AAPL", "location": "北京", "query": "查询", "url": "https://example.com/f.pdf",
"path": "/tmp/paper.pdf", "owner": "pytorch", "repo": "pytorch", "base": "USD",
"quote": "JPY", "title": "户外徒步", "start": "2026-07-19T09:00", "end": "2026-07-19T12:00",
"days": 3, "data": "[]", "chart_type": "bar", "code": "print('ok')", "max_results": 3,
}
def _fill_args(tool_name: str) -> Dict:
tool = TOOLS_BY_NAME.get(tool_name)
if not tool:
return {}
props = tool["function"]["parameters"]["properties"]
args = {}
for key, spec in props.items():
if key in _ARG_HINTS:
args[key] = _ARG_HINTS[key]
elif spec.get("type") == "integer":
args[key] = 1
else:
args[key] = "auto"
return args
def _extract_json(text: str):
text = text.strip()
start = text.find("{")
if start == -1:
return None
depth = 0
for i in range(start, len(text)):
if text[i] == "{":
depth += 1
elif text[i] == "}":
depth -= 1
if depth == 0:
try:
return json.loads(text[start:i + 1])
except json.JSONDecodeError:
return None
return None
def _json(thought: str, tool: str, arguments: Dict) -> str:
return json.dumps({"thought": thought, "tool": tool, "arguments": arguments},
ensure_ascii=False)
class MockChatClient:
"""确定性脚本模型；接口与 OpenAI 客户端子集兼容。"""
def __init__(self):
self.chat = SimpleNamespace(completions=SimpleNamespace(create=self._create))
def _create(self, model=None, messages=None, temperature=0, **kw):
content = self._respond(messages or [])
msg = SimpleNamespace(content=content)
return SimpleNamespace(choices=[SimpleNamespace(message=msg)])
def _respond(self, messages: List[Dict]) -> str:
system = messages[0]["content"] if messages and messages[0]["role"] == "system" else ""
task_prompt = next((m["content"] for m in messages if m["role"] == "user"), "")
full_text = "\n".join(m.get("content", "") for m in messages)
has_discover = "discover_tools" in system
# 当前"可用工具" = 出现在对话文本中的工具名（system 注入 / discover 追加）。
available = set(re.findall(r'"name":\s*"([a-zA-Z_][a-zA-Z0-9_]*)"', full_text))
available.discard("discover_tools")
prior = []
for m in messages:
if m["role"] == "assistant":
a = _extract_json(m.get("content", ""))
if a and "tool" in a:
prior.append(a)
called_ok = {a["tool"] for a in prior if a["tool"] in available}
discover_needs = [((a.get("arguments") or {}).get("need", ""))
for a in prior if a.get("tool") == "discover_tools"]
attempted = [a["tool"] for a in prior
if a["tool"] not in available and a["tool"] not in ("discover_tools", "finish")]
for tool, phrase in match_intents(task_prompt):
if tool in called_ok:
continue
if tool in available:
return _json(f"调用专用工具 {tool}", tool, _fill_args(tool))
# 目标工具当前不可用
if has_discover:
if discover_needs.count(phrase) >= 1:
continue # 已发现过仍未命中 -> 放弃该子任务
return _json(f"我需要一个能『{phrase}』的工具，先发现它", "discover_tools",
{"need": phrase})
else:
if attempted.count(tool) >= 1:
continue # 清单里没有该工具，尝试过一次即放弃
return _json(f"任务需要 {tool}，尝试调用", tool, _fill_args(tool))
return _json("所有子任务已处理", "finish", {"answer": "已完成可完成的子任务。"})
chapter4/active-tool-discovery/requirements.txt
+3
View File
@@ -0,0 +1,3 @@
openai>=1.30.0
tiktoken>=0.5.0
python-dotenv>=1.0.0
chapter4/active-tool-discovery/run_exact_experiment.py
+1214
View File
File diff suppressed because it is too large Load Diff
chapter4/active-tool-discovery/test_exact_experiment.py
+190
View File
@@ -0,0 +1,190 @@
"""Contract tests for the exact Experiment 4-7 runner (no model/API calls)."""
from __future__ import annotations
import asyncio
import importlib.util
import json
import sys
from copy import deepcopy
from pathlib import Path
import pytest
HERE = Path(__file__).resolve().parent
RUNNER_PATH = HERE / "run_exact_experiment.py"
SPEC = importlib.util.spec_from_file_location("experiment_4_7_runner", RUNNER_PATH)
runner = importlib.util.module_from_spec(SPEC)
assert SPEC.loader is not None
sys.modules[SPEC.name] = runner
SPEC.loader.exec_module(runner)
def test_protocol_is_exact_book_contract():
protocol = json.loads((HERE / "experiment_protocol.json").read_text(encoding="utf-8"))
assert protocol["model"] == "qwen3:4b"
assert protocol["minimum_mcp_tools"] >= 120
assert protocol["minimum_control_schema_tokens"] >= 50000
assert protocol["treatment"]["system_tools"] == [
"web_search", "code_interpreter", "discover_tools"
]
assert "cannot substitute" in protocol["treatment"]["base_tool_boundary"]
assert "structured market quote" in runner.TREATMENT_GUIDANCE
assert "call discover_tools separately" in runner.TREATMENT_GUIDANCE
assert len(protocol["tasks"]) == 3
def test_plan_grading_requires_both_cross_domain_slots():
task = runner.TASKS[0]
incomplete = runner.grade_plan(task, [{"tool": "web_search"}])
complete = runner.grade_plan(task, [
{"tool": "yfinance_quote"}, {"tool": "web_search"}
])
assert incomplete["accuracy"] == 0.5
assert not incomplete["all_required_capabilities_selected"]
assert complete["accuracy"] == 1.0
assert complete["all_required_capabilities_selected"]
def test_visualization_code_writes_real_svg(tmp_path):
output = tmp_path / "contributors.svg"
code = runner.visualization_code([
{"login": "alice", "contributions": 7},
{"login": "bob", "contributions": 3},
], output)
namespace = {}
exec(compile(code, "<test>", "exec"), namespace)
assert output.read_text(encoding="utf-8").startswith("<svg")
assert output.stat().st_size > 100
def _real_receipt(tool: str, backend: str = "live.example") -> dict:
return {
"tool": tool,
"success": True,
"transport": "mcp-stdio",
"mcp_result_is_error": False,
"backend_provenance": {"backend": backend, "origin": "live-api"},
"simulation_markers": [],
"substantive_observation": True,
"payload": {"success": True, "data": {"observed": True}},
}
def test_real_execution_gate_rejects_missing_required_receipt():
record = {"execution": {"receipts": [_real_receipt("yfinance_quote")]}}
assert not runner._required_receipts_real(record, runner.TASKS[0])
def test_real_execution_gate_rejects_failed_receipt():
receipts = [_real_receipt("yfinance_quote"), _real_receipt("web_search")]
receipts[1]["success"] = False
record = {"execution": {"receipts": receipts}}
assert not runner._required_receipts_real(record, runner.TASKS[0])
def test_real_execution_gate_rejects_tampered_mock_provenance():
receipts = [_real_receipt("yfinance_quote"), _real_receipt("web_search")]
tampered = deepcopy(receipts)
tampered[0]["backend_provenance"] = {"backend": "mock-server", "origin": "mock"}
tampered[0]["simulation_markers"] = ["mock"]
record = {"execution": {"receipts": tampered}}
assert not runner._required_receipts_real(record, runner.TASKS[0])
def test_acceptance_status_fails_closed_without_campaign_receipts():
protocol = json.loads((HERE / "experiment_protocol.json").read_text(encoding="utf-8"))
result = runner.derive_acceptance([], [], {}, protocol, {}, {})
assert result["status"] == "failed"
assert not result["gates"]["real_mcp_execution_only"]
assert not any(result["gates"].values())
def test_run_group_resume_reuses_only_compatible_receipt(tmp_path):
task = runner.TASKS[0]
task_dir = tmp_path / "control" / task["id"]
task_dir.mkdir(parents=True)
expected = {
"strategy": "control",
"task": task["id"],
"model": runner.MODEL,
"execution": {"task_complete": True},
}
(task_dir / "receipt.json").write_text(json.dumps(expected), encoding="utf-8")
original_tasks = runner.TASKS
runner.TASKS = [task]
try:
records = asyncio.run(
runner.run_group(None, [], None, "control", tmp_path, resume=True)
)
finally:
runner.TASKS = original_tasks
assert records == [expected]
def test_run_group_resume_archives_and_retries_one_incomplete_attempt(tmp_path):
task = runner.TASKS[0]
task_dir = tmp_path / "treatment" / task["id"]
task_dir.mkdir(parents=True)
failed = {
"strategy": "treatment",
"task": task["id"],
"model": runner.MODEL,
"execution": {"task_complete": False},
}
(task_dir / "receipt.json").write_text(json.dumps(failed), encoding="utf-8")
(task_dir / "partial.svg").write_text("<svg/>", encoding="utf-8")
recovered = {
"strategy": "treatment",
"task": task["id"],
"model": runner.MODEL,
"execution": {"task_complete": True},
}
async def fake_run_agent_task(*_args, **_kwargs):
assert not (task_dir / "receipt.json").exists()
assert not (task_dir / "partial.svg").exists()
return recovered
original_tasks = runner.TASKS
original_run_agent_task = runner.run_agent_task
runner.TASKS = [task]
runner.run_agent_task = fake_run_agent_task
try:
records = asyncio.run(
runner.run_group(None, [], None, "treatment", tmp_path, resume=True)
)
finally:
runner.TASKS = original_tasks
runner.run_agent_task = original_run_agent_task
archive = task_dir / "failed_attempts" / "attempt-1"
assert records == [recovered]
assert json.loads((archive / "receipt.json").read_text(encoding="utf-8")) == failed
assert (archive / "partial.svg").read_text(encoding="utf-8") == "<svg/>"
assert json.loads((task_dir / "receipt.json").read_text(encoding="utf-8")) == recovered
def test_run_group_resume_refuses_third_real_attempt(tmp_path):
task = runner.TASKS[0]
task_dir = tmp_path / "treatment" / task["id"]
(task_dir / "failed_attempts" / "attempt-1").mkdir(parents=True)
failed = {
"strategy": "treatment",
"task": task["id"],
"model": runner.MODEL,
"execution": {"task_complete": False},
}
(task_dir / "receipt.json").write_text(json.dumps(failed), encoding="utf-8")
original_tasks = runner.TASKS
runner.TASKS = [task]
try:
with pytest.raises(RuntimeError, match="maximum two real attempts exhausted"):
asyncio.run(
runner.run_group(None, [], None, "treatment", tmp_path, resume=True)
)
finally:
runner.TASKS = original_tasks
chapter4/active-tool-discovery/tools_library.py
+475
View File
@@ -0,0 +1,475 @@
"""
实验 8-4 工具库：120+ 个跨领域工具定义。
设计要点：
1) 每个工具都有真实可读的 name / description / parameters（OpenAI function schema）。
2) 领域覆盖 finance / news / web / arxiv / file / github / code / geo / weather /
media / language / email / db / ecommerce / social / crypto / util 等。
3) 故意混入大量"通用/近义"工具（web_search / universal_search / quick_answer ...），
它们在全量注入时会与"专用工具"竞争，诱导模型错选（如查股价用 web_search）。
4) 工具执行只做轻量 mock —— 本实验关心的是"能否选对工具"，不是工具真实结果。
对外导出：
- ALL_TOOLS : List[dict] OpenAI tools 数组（126 个）
- TOOLS_BY_NAME : Dict[str, dict]
- TOOL_IMPLS : Dict[str, callable] mock 执行函数
- BASE_TOOL_NAMES : 主动发现模式下 system 里保留的少量基础工具
- GENERIC_TOOL_NAMES : 通用/兜底工具集合（用于统计"是否用通用工具替代了专用工具"）
- select_tools : 按 --tool-set-size 截取工具子集（演示工具集规模的影响）
- TASKS : List[dict] 评测任务及其判分标准
"""
from typing import Dict, List
def _tool(name: str, description: str, params: Dict) -> Dict:
"""构造一个 OpenAI function-calling tool schema。"""
return {
"type": "function",
"function": {
"name": name,
"description": description,
"parameters": {
"type": "object",
"properties": params,
"required": list(params.keys()),
},
},
}
def _s(desc: str) -> Dict:
return {"type": "string", "description": desc}
def _i(desc: str) -> Dict:
return {"type": "integer", "description": desc}
# ---------------------------------------------------------------------------
# 工具定义（按领域分组）
# ---------------------------------------------------------------------------
_DEFS: List[Dict] = []
# --- finance（金融专用，10）---
_DEFS += [
_tool("get_stock_price", "获取指定股票代码的实时最新股价、涨跌幅与成交量（专业金融数据源）。",
{"symbol": _s("股票代码，如 AAPL、TSLA")}),
_tool("get_stock_history", "获取某支股票的历史 K 线行情数据。",
{"symbol": _s("股票代码"), "range": _s("时间范围，如 1mo/1y")}),
_tool("get_company_financials", "获取上市公司的财报数据（营收、利润、资产负债表）。",
{"symbol": _s("股票代码")}),
_tool("get_forex_rate", "获取两种法定货币之间的实时外汇汇率。",
{"base": _s("基准货币，如 USD"), "quote": _s("报价货币，如 JPY")}),
_tool("get_crypto_price", "获取指定加密货币的实时价格（USD 计价）。",
{"symbol": _s("加密货币代码，如 BTC、ETH")}),
_tool("get_market_index", "获取股票市场指数的实时点位，如标普500、纳斯达克。",
{"index": _s("指数代码，如 SPX、IXIC")}),
_tool("get_earnings_calendar", "查询某公司的财报发布日历。", {"symbol": _s("股票代码")}),
_tool("get_analyst_ratings", "获取分析师对某股票的评级与目标价。", {"symbol": _s("股票代码")}),
_tool("get_dividend_history", "获取某股票的历史分红派息记录。", {"symbol": _s("股票代码")}),
_tool("convert_currency", "按最新汇率把一笔金额从一种货币换算为另一种货币。",
{"amount": {"type": "number", "description": "金额"},
"from_currency": _s("源货币"), "to_currency": _s("目标货币")}),
]
# --- news（新闻专用，4）---
_DEFS += [
_tool("search_news", "按关键词检索最新新闻文章，返回标题、来源、时间与摘要（新闻聚合源）。",
{"query": _s("检索关键词"), "lang": _s("语言，如 zh/en")}),
_tool("get_top_headlines", "获取某分类/国家的头条新闻。",
{"category": _s("分类，如 business/tech"), "country": _s("国家代码，如 us/cn")}),
_tool("get_news_by_source", "获取指定新闻媒体源的最新报道。", {"source": _s("媒体源，如 reuters")}),
_tool("summarize_article", "抓取并总结一篇新闻文章的核心内容。", {"url": _s("文章 URL")}),
]
# --- web / generic（通用检索，诱导错选，8）---
_DEFS += [
_tool("web_search", "通用联网搜索，可查询几乎任何实时信息并给出答案，"
"包括股票价格、汇率、天气、新闻、百科、代码、地理等各类问题——一个工具满足大部分查询需求。",
{"query": _s("搜索关键词")}),
_tool("universal_search", "万能搜索助手，可回答任何主题的问题，"
"覆盖金融、科技、生活、学术等所有领域的信息查询。", {"query": _s("查询")}),
_tool("quick_answer", "对任意问题给出快速简短的答案，适用于价格、天气、新闻、常识等各种即时提问。",
{"question": _s("问题")}),
_tool("google_search", "使用 Google 搜索网页，可查询任意主题的最新信息。", {"query": _s("查询")}),
_tool("bing_search", "使用 Bing 搜索网页，可查询任意主题的最新信息。", {"query": _s("查询")}),
_tool("fetch_url", "抓取给定 URL 的网页原始内容。", {"url": _s("网页 URL")}),
_tool("scrape_webpage", "抓取网页并按 CSS 选择器提取结构化内容。",
{"url": _s("网页 URL"), "selector": _s("CSS 选择器")}),
_tool("ask_knowledge_base", "向通用知识库提问，返回百科式答案。", {"question": _s("问题")}),
]
# --- arxiv / academic（学术专用，5）---
_DEFS += [
_tool("arxiv_search", "在 arXiv 论文库中检索论文，按相关度/时间返回论文标题、作者、摘要与 PDF 链接。",
{"query": _s("检索关键词"), "max_results": _i("返回论文数量")}),
_tool("arxiv_get_paper", "根据 arXiv ID 获取单篇论文的详细信息。", {"arxiv_id": _s("arXiv 编号")}),
_tool("semantic_scholar_search", "在 Semantic Scholar 检索学术论文。", {"query": _s("关键词")}),
_tool("get_citations", "获取某篇论文的引用列表。", {"paper_id": _s("论文 ID")}),
_tool("search_pubmed", "在 PubMed 检索生物医学文献。", {"query": _s("关键词")}),
]
# --- file / download（文件下载，10）---
_DEFS += [
_tool("download_file", "从给定 URL 下载文件（PDF/图片/压缩包等）并保存到本地。",
{"url": _s("文件 URL"), "path": _s("本地保存路径")}),
_tool("upload_file", "把本地文件上传到远端存储。", {"path": _s("本地文件路径")}),
_tool("read_file", "读取本地文本文件内容。", {"path": _s("文件路径")}),
_tool("write_file", "把内容写入本地文件。", {"path": _s("文件路径"), "content": _s("写入内容")}),
_tool("list_directory", "列出目录下的文件。", {"path": _s("目录路径")}),
_tool("delete_file", "删除本地文件。", {"path": _s("文件路径")}),
_tool("convert_document", "转换文档格式，如 docx→pdf。",
{"path": _s("文件路径"), "target_format": _s("目标格式")}),
_tool("extract_text_from_pdf", "从 PDF 文件中抽取文本。", {"path": _s("PDF 路径")}),
_tool("compress_files", "把多个文件压缩为一个压缩包。", {"paths": _s("逗号分隔的文件路径")}),
_tool("unzip_archive", "解压压缩包。", {"path": _s("压缩包路径")}),
]
# --- github / dev（代码托管专用，8）---
_DEFS += [
_tool("github_get_repo", "获取 GitHub 仓库的基本信息（stars、语言、描述等）。",
{"owner": _s("仓库所有者"), "repo": _s("仓库名")}),
_tool("github_list_contributors", "列出 GitHub 仓库的贡献者及各自的提交数（专用 GitHub API）。",
{"owner": _s("仓库所有者"), "repo": _s("仓库名")}),
_tool("github_list_issues", "列出 GitHub 仓库的 issues。",
{"owner": _s("仓库所有者"), "repo": _s("仓库名")}),
_tool("github_get_commits", "获取 GitHub 仓库的提交历史。",
{"owner": _s("仓库所有者"), "repo": _s("仓库名")}),
_tool("github_search_code", "在 GitHub 上按关键词搜索代码。", {"query": _s("搜索关键词")}),
_tool("github_get_pull_requests", "列出 GitHub 仓库的 PR。",
{"owner": _s("仓库所有者"), "repo": _s("仓库名")}),
_tool("github_get_user", "获取 GitHub 用户资料。", {"username": _s("用户名")}),
_tool("gitlab_get_project", "获取 GitLab 项目信息。", {"project_id": _s("项目 ID")}),
]
# --- code / analysis（代码执行与可视化，6）---
_DEFS += [
_tool("code_interpreter", "在沙箱中执行 Python 代码，可做数据分析、统计并绘制/生成可视化图表。",
{"code": _s("要执行的 Python 代码")}),
_tool("render_chart", "根据给定数据直接渲染柱状图/折线图/饼图等可视化图表。",
{"data": _s("JSON 数据"), "chart_type": _s("图表类型，如 bar/line/pie")}),
_tool("run_shell_command", "在服务器上执行 shell 命令。", {"command": _s("命令")}),
_tool("lint_code", "对代码做静态检查。", {"code": _s("代码"), "language": _s("语言")}),
_tool("format_code", "格式化代码。", {"code": _s("代码"), "language": _s("语言")}),
_tool("execute_sql", "执行 SQL 查询。", {"query": _s("SQL 语句")}),
]
# --- geo / maps（地理，6）---
_DEFS += [
_tool("geocode_address", "把地址转换为经纬度坐标。", {"address": _s("地址")}),
_tool("reverse_geocode", "把经纬度转换为地址。", {"lat": _s("纬度"), "lon": _s("经度")}),
_tool("get_directions", "获取两地之间的导航路线。", {"origin": _s("起点"), "destination": _s("终点")}),
_tool("get_distance", "计算两地之间的距离。", {"origin": _s("起点"), "destination": _s("终点")}),
_tool("search_places", "在指定位置附近搜索地点/商户。",
{"query": _s("关键词"), "location": _s("位置")}),
_tool("get_timezone", "根据坐标获取时区。", {"lat": _s("纬度"), "lon": _s("经度")}),
]
# --- weather（天气专用，3）---
_DEFS += [
_tool("get_current_weather", "获取指定城市的实时天气（气温、湿度、天气状况）。", {"location": _s("城市名")}),
_tool("get_weather_forecast", "获取指定城市未来若干天的天气预报（专业气象数据源）。",
{"location": _s("城市名"), "days": _i("预报天数")}),
_tool("get_air_quality", "获取指定城市的空气质量指数 AQI。", {"location": _s("城市名")}),
]
# --- media（多媒体，6）---
_DEFS += [
_tool("generate_image", "根据文字提示生成图片。", {"prompt": _s("图片描述")}),
_tool("caption_image", "为图片生成文字描述。", {"url": _s("图片 URL")}),
_tool("transcribe_audio", "把音频转写为文字。", {"url": _s("音频 URL")}),
_tool("text_to_speech", "把文字合成为语音。", {"text": _s("文本")}),
_tool("video_summarize", "总结一段视频的内容。", {"url": _s("视频 URL")}),
_tool("ocr_image", "识别图片中的文字。", {"url": _s("图片 URL")}),
]
# --- language / NLP（文本处理，8）---
_DEFS += [
_tool("translate_text", "把文本翻译为目标语言。", {"text": _s("文本"), "target_lang": _s("目标语言")}),
_tool("detect_language", "检测文本语言。", {"text": _s("文本")}),
_tool("summarize_text", "对一段文本做摘要。", {"text": _s("文本")}),
_tool("paraphrase_text", "改写/润色一段文本。", {"text": _s("文本")}),
_tool("correct_grammar", "纠正文本语法错误。", {"text": _s("文本")}),
_tool("sentiment_analysis", "分析文本情感倾向。", {"text": _s("文本")}),
_tool("extract_keywords", "从文本中抽取关键词。", {"text": _s("文本")}),
_tool("named_entity_recognition", "识别文本中的命名实体。", {"text": _s("文本")}),
]
# --- email / comm / calendar（通讯与日程，7）---
_DEFS += [
_tool("send_email", "发送一封电子邮件。",
{"to": _s("收件人"), "subject": _s("主题"), "body": _s("正文")}),
_tool("read_inbox", "读取邮箱中的邮件。", {"folder": _s("文件夹，如 inbox")}),
_tool("create_calendar_event", "在用户日历上创建一个日程/事件（专用日历服务）。",
{"title": _s("事件标题"), "start": _s("开始时间"), "end": _s("结束时间")}),
_tool("list_calendar_events", "列出某日期的日历事件。", {"date": _s("日期 YYYY-MM-DD")}),
_tool("send_slack_message", "向 Slack 频道发送消息。", {"channel": _s("频道"), "text": _s("内容")}),
_tool("send_sms", "发送短信。", {"number": _s("手机号"), "text": _s("内容")}),
_tool("make_phone_call", "拨打电话并播报脚本。", {"number": _s("电话"), "script": _s("话术")}),
]
# --- database / storage（存储，7）---
_DEFS += [
_tool("query_database", "在业务数据库上执行只读查询。", {"sql": _s("SQL 查询")}),
_tool("insert_record", "向数据表插入记录。", {"table": _s("表名"), "data": _s("JSON 数据")}),
_tool("get_record", "按主键读取一条记录。", {"table": _s("表名"), "id": _s("主键")}),
_tool("redis_get", "读取 Redis 键值。", {"key": _s("键")}),
_tool("redis_set", "写入 Redis 键值。", {"key": _s("键"), "value": _s("值")}),
_tool("s3_upload", "上传文件到 S3。", {"bucket": _s("桶"), "key": _s("对象键"), "path": _s("本地路径")}),
_tool("s3_download", "从 S3 下载文件。", {"bucket": _s("桶"), "key": _s("对象键")}),
]
# --- ecommerce / travel（电商与出行，8）---
_DEFS += [
_tool("search_products", "在电商平台搜索商品。", {"query": _s("关键词")}),
_tool("get_product_details", "获取商品详情。", {"product_id": _s("商品 ID")}),
_tool("add_to_cart", "把商品加入购物车。", {"product_id": _s("商品 ID"), "qty": _i("数量")}),
_tool("track_shipment", "查询快递物流。", {"tracking_no": _s("运单号")}),
_tool("search_flights", "搜索航班。",
{"origin": _s("出发地"), "destination": _s("目的地"), "date": _s("日期")}),
_tool("search_hotels", "搜索酒店。",
{"location": _s("城市"), "checkin": _s("入住日"), "checkout": _s("离店日")}),
_tool("book_restaurant", "预订餐厅。",
{"name": _s("餐厅名"), "time": _s("时间"), "party": _i("人数")}),
_tool("get_product_reviews", "获取商品评价。", {"product_id": _s("商品 ID")}),
]
# --- social（社交，5）---
_DEFS += [
_tool("post_tweet", "发布一条推文。", {"text": _s("内容")}),
_tool("search_tweets", "搜索推文。", {"query": _s("关键词")}),
_tool("get_user_profile", "获取社交平台用户资料。",
{"platform": _s("平台"), "username": _s("用户名")}),
_tool("get_trending_topics", "获取热门话题。", {"region": _s("地区")}),
_tool("get_reddit_posts", "获取某 subreddit 的帖子。", {"subreddit": _s("版块")}),
]
# --- crypto / blockchain（区块链，3）---
_DEFS += [
_tool("get_wallet_balance", "查询链上钱包余额。", {"address": _s("钱包地址")}),
_tool("get_gas_price", "查询链上 gas 价格。", {"chain": _s("链名，如 ethereum")}),
_tool("get_nft_metadata", "获取 NFT 元数据。", {"contract": _s("合约地址"), "token_id": _s("token ID")}),
]
# --- misc util（杂项工具，10）---
_DEFS += [
_tool("calculator", "做数学表达式计算。", {"expression": _s("数学表达式")}),
_tool("get_current_time", "获取指定时区的当前时间。", {"timezone": _s("时区，如 Asia/Shanghai")}),
_tool("generate_uuid", "生成一个 UUID。", {"version": _i("UUID 版本")}),
_tool("get_random_number", "生成一个区间内的随机数。", {"min": _i("最小值"), "max": _i("最大值")}),
_tool("url_shortener", "生成短链接。", {"url": _s("原始 URL")}),
_tool("qr_code_generator", "生成二维码。", {"data": _s("二维码内容")}),
_tool("password_generator", "生成随机密码。", {"length": _i("密码长度")}),
_tool("get_ip_info", "查询 IP 归属地信息。", {"ip": _s("IP 地址")}),
_tool("dns_lookup", "查询域名 DNS 记录。", {"domain": _s("域名")}),
_tool("ping_host", "测试主机连通性。", {"host": _s("主机名")}),
]
# --- 更多领域工具（补足 120+，12）---
_DEFS += [
_tool("get_commodity_price", "获取大宗商品（黄金/原油等）实时价格。", {"commodity": _s("商品名，如 gold/oil")}),
_tool("get_bond_yield", "获取国债收益率。", {"country": _s("国家"), "maturity": _s("期限，如 10y")}),
_tool("get_flight_status", "查询航班实时状态。", {"flight_no": _s("航班号")}),
_tool("get_traffic_info", "查询某路段的实时路况。", {"road": _s("路段/城市")}),
_tool("book_taxi", "叫一辆出租车/网约车。", {"pickup": _s("上车地点"), "dropoff": _s("目的地")}),
_tool("get_horoscope", "获取星座运势。", {"sign": _s("星座")}),
_tool("get_recipe", "根据食材/菜名获取菜谱。", {"dish": _s("菜名")}),
_tool("get_definition", "查询词语释义。", {"word": _s("词语")}),
_tool("currency_list", "列出支持的货币代码。", {"region": _s("地区")}),
_tool("get_holidays", "查询某国某年的法定节假日。", {"country": _s("国家"), "year": _i("年份")}),
_tool("unit_convert", "单位换算（长度/重量/温度等）。",
{"value": {"type": "number", "description": "数值"}, "from_unit": _s("源单位"), "to_unit": _s("目标单位")}),
_tool("get_wikipedia_summary", "获取维基百科词条摘要。", {"title": _s("词条标题")}),
]
# ---------------------------------------------------------------------------
# 导出结构
# ---------------------------------------------------------------------------
ALL_TOOLS: List[Dict] = _DEFS
TOOLS_BY_NAME: Dict[str, Dict] = {t["function"]["name"]: t for t in ALL_TOOLS}
assert len(ALL_TOOLS) == len(TOOLS_BY_NAME), "工具名有重复！"
# 主动发现模式下 system 保留的少量基础工具（不含任何专用领域工具）。
BASE_TOOL_NAMES = ["calculator", "get_current_time"]
# 通用/兜底工具：若在需要专用工具的任务中调用了这些，视为"用通用工具替代了专用工具"。
GENERIC_TOOL_NAMES = {
"web_search", "universal_search", "quick_answer", "google_search",
"bing_search", "fetch_url", "scrape_webpage", "ask_knowledge_base",
}
def select_tools(size: int = None, tasks: "List[Dict]" = None) -> List[Dict]:
"""按 --tool-set-size 截取一个工具子集，用于演示"工具集规模"对各策略的影响。
子集**始终**包含：基础工具、全部通用/兜底工具（诱导项）、以及所选任务判分槽位涉及的
专用工具；其余名额按 ALL_TOOLS 原顺序补足，直到达到 size 个。
size 为空或 >= 全库规模时返回全部工具（默认行为）。
"""
if size is None or size >= len(ALL_TOOLS):
return ALL_TOOLS
keep = set(BASE_TOOL_NAMES) | set(GENERIC_TOOL_NAMES)
for task in (tasks if tasks is not None else TASKS):
for slot in task["required_slots"]:
keep.update(slot)
size = max(size, len(keep))
required = [t for t in ALL_TOOLS if t["function"]["name"] in keep]
others = [t for t in ALL_TOOLS if t["function"]["name"] not in keep]
return required + others[: size - len(required)]
# ---------------------------------------------------------------------------
# mock 执行
# ---------------------------------------------------------------------------
def _mock_result(name: str, args: Dict) -> str:
"""为常用工具返回像样的 mock 结果，其余返回通用占位结果。"""
import json
canned = {
"get_stock_price": {"symbol": args.get("symbol"), "price": 227.52,
"change_pct": -1.83, "currency": "USD", "source": "NASDAQ"},
"get_crypto_price": {"symbol": args.get("symbol"), "price": 3125.4, "currency": "USD"},
"get_forex_rate": {"base": args.get("base"), "quote": args.get("quote"), "rate": 156.7},
"convert_currency": {"amount": args.get("amount"), "from": args.get("from_currency"),
"to": args.get("to_currency"), "result": 15670.0, "rate": 156.7},
"search_news": {"results": [
{"title": "Apple shares slip on iPhone demand concerns", "source": "Reuters"},
{"title": "Analysts weigh in on AAPL pullback", "source": "Bloomberg"}]},
"arxiv_search": {"results": [
{"id": "2406.00001", "title": "Efficient Transformers Revisited",
"pdf": "https://arxiv.org/pdf/2406.00001"},
{"id": "2406.00002", "title": "Sparse Attention Transformers",
"pdf": "https://arxiv.org/pdf/2406.00002"},
{"id": "2406.00003", "title": "Transformer Scaling Laws 2024",
"pdf": "https://arxiv.org/pdf/2406.00003"}]},
"download_file": {"saved": args.get("path"), "bytes": 482113, "status": "ok"},
"github_list_contributors": {"contributors": [
{"login": "alice", "commits": 1240}, {"login": "bob", "commits": 830},
{"login": "carol", "commits": 617}]},
"code_interpreter": {"stdout": "chart saved to /tmp/contrib.png", "status": "ok"},
"render_chart": {"chart": "/tmp/contrib.png", "status": "ok"},
"get_weather_forecast": {"location": args.get("location"),
"forecast": [{"day": "Sun", "cond": "Sunny", "high": 31}]},
"get_current_weather": {"location": args.get("location"), "cond": "Clear", "temp": 28},
"create_calendar_event": {"event": args.get("title"), "status": "created"},
}
if name in canned:
return json.dumps(canned[name], ensure_ascii=False)
return json.dumps({"tool": name, "args": args, "status": "ok",
"result": f"<{name} 的 mock 结果>"}, ensure_ascii=False)
# 所有工具共用一个 mock 分发器
TOOL_IMPLS: Dict[str, callable] = {
name: (lambda args, n=name: _mock_result(n, args)) for name in TOOLS_BY_NAME
}
# ---------------------------------------------------------------------------
# 评测任务及判分标准
# ---------------------------------------------------------------------------
# required_slots: List[List[str]]
# 每个内层 list 是"一个能力槽位"的可接受工具集合（任一命中即算填上该槽位）。
# 一个任务判为"选对"，当且仅当所有槽位都被填上。
# 这些任务都需要跨领域协作，且都存在"通用工具易被误选"的陷阱。
TASKS: List[Dict] = [
{
"id": "finance+news",
"prompt": "苹果公司最近股价怎么样？帮我看看有没有相关新闻能解释一下原因。",
"required_slots": [
["get_stock_price"],
["search_news", "get_top_headlines", "get_news_by_source"],
],
},
{
"id": "arxiv+download",
"prompt": "我想看看 transformer 领域最新的研究论文，帮我找几篇最新的，并把排在前三的下载下来。",
"required_slots": [
["arxiv_search"],
["download_file"],
],
},
{
"id": "github+viz",
"prompt": "帮我看看 pytorch/pytorch 这个仓库都有谁贡献最多，最好能画个各人提交量的图表。",
"required_slots": [
["github_list_contributors"],
["code_interpreter", "render_chart"],
],
},
{
"id": "weather+calendar",
"prompt": "这周日北京天气怎么样？要是晴天的话，帮我在日历里记一个'户外徒步'的活动。",
"required_slots": [
["get_weather_forecast"],
["create_calendar_event"],
],
},
{
"id": "forex+weather",
"prompt": "100 美元现在能换多少日元？顺便告诉我东京现在的天气怎么样。",
"required_slots": [
["get_forex_rate", "convert_currency"],
["get_current_weather"],
],
},
{
"id": "crypto+news",
"prompt": "以太坊现在多少钱一个？另外有什么最新的相关消息吗？",
"required_slots": [
["get_crypto_price"],
["search_news", "get_top_headlines", "get_news_by_source"],
],
},
# 下面两个是"通用工具诱导"任务：措辞偏泛，容易让模型误用 web_search 等通用兜底工具，
# 而其实存在更合适的专用工具。用来体现"全量注入错选通用工具、主动发现选对专用工具"。
{
"id": "opinion(诱导)",
"prompt": "帮我了解一下特斯拉这家公司最近的新闻舆论风向。",
"required_slots": [
["search_news", "get_news_by_source", "get_top_headlines"],
],
},
{
"id": "academic(诱导)",
"prompt": "帮我了解一下最近'量子计算'方面有什么新的科研进展。",
"required_slots": [
["arxiv_search", "semantic_scholar_search", "search_pubmed"],
],
},
]
def grade(task: Dict, called_tools: List[str]) -> Dict:
"""根据实际调用的工具给某个任务打分。"""
called = set(called_tools)
filled = []
missed = []
for slot in task["required_slots"]:
if any(t in called for t in slot):
filled.append(slot)
else:
missed.append(slot)
used_generic = sorted(called & GENERIC_TOOL_NAMES)
correct = len(missed) == 0
return {
"correct": correct, # 是否覆盖了全部能力槽位
# 精确选对 = 覆盖全部能力槽位 且 没有误用通用兜底工具（web_search 等）
"precise": correct and not used_generic,
"filled_slots": len(filled),
"total_slots": len(task["required_slots"]),
"missed_slots": missed,
"used_generic_substitute": used_generic,
}
if __name__ == "__main__":
print(f"工具总数: {len(ALL_TOOLS)}")
print(f"基础工具: {BASE_TOOL_NAMES}")
print(f"任务数: {len(TASKS)}")
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/catalog.schemas.json.gz
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/catalog_receipt.json
+24
View File
@@ -0,0 +1,24 @@
{
"transport": "mcp-stdio",
"server": "/Users/boj/book/ai-agent-book/chapter4/perception-tools/src/main.py",
"tools_list_received": true,
"server_name": "perception-tools",
"server_version": "1.26.0",
"tool_count": 126,
"unique_tool_count": 126,
"schema_tokens_o200k": 50120,
"schema_bytes": 206819,
"schema_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627",
"required_tools_present": {
"web_search": true,
"code_interpreter": true,
"yfinance_quote": true,
"search_news": true,
"arxiv_search": true,
"arxiv_download": true,
"github_list_contributors": true
},
"catalog_gzip_bytes": 21244,
"catalog_gzip_sha256": "822bdadf46573ca850447f1ead6b526c01f46f73d35c90000895f891584d24bd",
"catalog_gzip_content_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627"
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/control/apple_stock_news/receipt.json
+1778
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/control/github_contributors_visualization/contributors.svg
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="500"><rect width="100%" height="100%" fill="white"/><text x="450" y="32" text-anchor="middle" font-size="22">openai/openai-python contributors</text><rect x="70" y="70" width="68" height="350" fill="#4f46e5"/><text x="104.0" y="65" text-anchor="middle" font-size="11">700</text><text x="104.0" y="440" text-anchor="end" transform="rotate(-35 104.0 440)" font-size="10">stainless-app[bot]</text><rect x="146" y="282" width="68" height="138" fill="#4f46e5"/><text x="180.0" y="277" text-anchor="middle" font-size="11">276</text><text x="180.0" y="440" text-anchor="end" transform="rotate(-35 180.0 440)" font-size="10">stainless-bot</text><rect x="222" y="372" width="68" height="48" fill="#4f46e5"/><text x="256.0" y="367" text-anchor="middle" font-size="11">96</text><text x="256.0" y="440" text-anchor="end" transform="rotate(-35 256.0 440)" font-size="10">RobertCraigie</text><rect x="298" y="401" width="68" height="19" fill="#4f46e5"/><text x="332.0" y="396" text-anchor="middle" font-size="11">39</text><text x="332.0" y="440" text-anchor="end" transform="rotate(-35 332.0 440)" font-size="10">apcha-oai</text><rect x="374" y="406" width="68" height="14" fill="#4f46e5"/><text x="408.0" y="401" text-anchor="middle" font-size="11">29</text><text x="408.0" y="440" text-anchor="end" transform="rotate(-35 408.0 440)" font-size="10">hallacy</text><rect x="450" y="413" width="68" height="7" fill="#4f46e5"/><text x="484.0" y="408" text-anchor="middle" font-size="11">14</text><text x="484.0" y="440" text-anchor="end" transform="rotate(-35 484.0 440)" font-size="10">rachellim</text><rect x="526" y="414" width="68" height="6" fill="#4f46e5"/><text x="560.0" y="409" text-anchor="middle" font-size="11">13</text><text x="560.0" y="440" text-anchor="end" transform="rotate(-35 560.0 440)" font-size="10">logankilpatrick</text><rect x="602" y="416" width="68" height="4" fill="#4f46e5"/><text x="636.0" y="411" text-anchor="middle" font-size="11">9</text><text x="636.0" y="440" text-anchor="end" transform="rotate(-35 636.0 440)" font-size="10">dtmeadows</text><rect x="678" y="416" width="68" height="4" fill="#4f46e5"/><text x="712.0" y="411" text-anchor="middle" font-size="11">9</text><text x="712.0" y="440" text-anchor="end" transform="rotate(-35 712.0 440)" font-size="10">kristapratico</text><rect x="754" y="417" width="68" height="3" fill="#4f46e5"/><text x="788.0" y="412" text-anchor="middle" font-size="11">7</text><text x="788.0" y="440" text-anchor="end" transform="rotate(-35 788.0 440)" font-size="10">ddeville</text></svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/control/github_contributors_visualization/receipt.json
+2825
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/control/transformer_arxiv_download/papers/2607.27184v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/control/transformer_arxiv_download/papers/2607.27185v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/control/transformer_arxiv_download/papers/2607.27188v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/control/transformer_arxiv_download/receipt.json
+1653
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/embedding_receipt.json
+12
View File
@@ -0,0 +1,12 @@
{
"model": "sentence-transformers/all-MiniLM-L6-v2",
"backend": "local-transformers-mean-pooling",
"device": "cpu",
"local_files_only": true,
"catalog_text_count": 126,
"texts_sha256": "3da0897f88acbd0d32a3a77578bd135a91d631a8b85775ff12232eb5aafe3b9d",
"cache_path": "/Users/boj/book/ai-agent-book/chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json",
"cache_sha256": "23051857081c61e9dfc184d868fdaecd0bc816360133eeb951cbb6593fdda9c9",
"vector_count": 126,
"vector_dimensions": 384
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json
+48645
View File
File diff suppressed because it is too large Load Diff
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/manifest.json
+105
View File
@@ -0,0 +1,105 @@
{
"generated_at": "2026-07-30T05:05:11.922419+00:00",
"files": [
{
"path": "catalog.schemas.json.gz",
"bytes": 21244,
"sha256": "822bdadf46573ca850447f1ead6b526c01f46f73d35c90000895f891584d24bd"
},
{
"path": "catalog_receipt.json",
"bytes": 864,
"sha256": "9f4072990692bb6a0f5c4a6e23f18f2a4bf9938478cf60b2ffd73fe5ee766a7a"
},
{
"path": "control/apple_stock_news/receipt.json",
"bytes": 73358,
"sha256": "ecd66de27d40f03f8aaafe02fe1826fff1917efac68078031d0cb3117edfbceb"
},
{
"path": "control/github_contributors_visualization/contributors.svg",
"bytes": 2602,
"sha256": "d14524e843fa88f3ad322848fc7fa8cad7d5153c79f7d74c7be5e24ba3880b0a"
},
{
"path": "control/github_contributors_visualization/receipt.json",
"bytes": 124615,
"sha256": "400e65e13aad11428f6ef43ee115001a820440df25f6ce24e66c0328f7a48a0f"
},
{
"path": "control/transformer_arxiv_download/papers/2607.27184v1.pdf",
"bytes": 1095498,
"sha256": "73f5c9a80cbfd807ed5bc14c963b7c9fd18a9de6eb30c8845b0505a79abd9536"
},
{
"path": "control/transformer_arxiv_download/papers/2607.27185v1.pdf",
"bytes": 797471,
"sha256": "4e4ff0947d28d37b7cfcfd743b50f1538945b6acfa598e74b084659436780754"
},
{
"path": "control/transformer_arxiv_download/papers/2607.27188v1.pdf",
"bytes": 1176383,
"sha256": "ecbe0ac9ae6768860d7f9f10a6986f86d4e98f9c73fd398e44894a8ac29b42e8"
},
{
"path": "control/transformer_arxiv_download/receipt.json",
"bytes": 67851,
"sha256": "a9cc68669d65e3836f25ad436c0448355bac27d7ac30784de3f7fe918f4c3347"
},
{
"path": "embedding_receipt.json",
"bytes": 601,
"sha256": "a1f160a9903b88259ba789587a9c2c6f3fc3408c80ec54c97b99fe269116bd40"
},
{
"path": "index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json",
"bytes": 1354409,
"sha256": "23051857081c61e9dfc184d868fdaecd0bc816360133eeb951cbb6593fdda9c9"
},
{
"path": "protocol.json",
"bytes": 1738,
"sha256": "a9b40108c154e5b492aad47443e9d290e7d509a6ee5d75e5b77e365fdda8d8f9"
},
{
"path": "summary.json",
"bytes": 3008,
"sha256": "94b993ff8fd5f96fc048c95575c44f6e04393e97dd3e910d7f679815d9cd1958"
},
{
"path": "treatment/apple_stock_news/receipt.json",
"bytes": 33672,
"sha256": "2ac858befb44d83cbe494ebf80b06c0dfb6da2f29d9c78c2c4e218e0b716c05b"
},
{
"path": "treatment/github_contributors_visualization/contributors.svg",
"bytes": 215,
"sha256": "01d4f904edcaefb2563db27c7a3879eed9ed538df1dbd85fc86ce184bd84466f"
},
{
"path": "treatment/github_contributors_visualization/receipt.json",
"bytes": 51082,
"sha256": "96d0bc297187151c87efa0a4da928af908debc971abfa970f06bdd9575cea1d3"
},
{
"path": "treatment/transformer_arxiv_download/papers/2607.27184v1.pdf",
"bytes": 1095498,
"sha256": "73f5c9a80cbfd807ed5bc14c963b7c9fd18a9de6eb30c8845b0505a79abd9536"
},
{
"path": "treatment/transformer_arxiv_download/papers/2607.27185v1.pdf",
"bytes": 797471,
"sha256": "4e4ff0947d28d37b7cfcfd743b50f1538945b6acfa598e74b084659436780754"
},
{
"path": "treatment/transformer_arxiv_download/papers/2607.27188v1.pdf",
"bytes": 1176383,
"sha256": "ecbe0ac9ae6768860d7f9f10a6986f86d4e98f9c73fd398e44894a8ac29b42e8"
},
{
"path": "treatment/transformer_arxiv_download/receipt.json",
"bytes": 141065,
"sha256": "b052456553112648946f63b75e3fdf9bdf25ef6fe446f48d3d297ff91ebb09b3"
}
]
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/protocol.json
+58
View File
@@ -0,0 +1,58 @@
{
"experiment": "4-5",
"title": "Active tool discovery with Qwen3-4B and the perception MCP server",
"authority": "book/chapter4.md:624",
"model": "qwen3:4b",
"model_runtime": "ollama",
"minimum_mcp_tools": 120,
"minimum_control_schema_tokens": 50000,
"control": {
"system_tools": "all complete schemas returned by perception MCP tools/list",
"required_behavior": "plan and execute using the full catalog"
},
"treatment": {
"system_tools": [
"web_search",
"code_interpreter",
"discover_tools"
],
"discovery_top_k": 5,
"schema_injection_role": "user",
"status_bar_updates": true
},
"tasks": [
{
"id": "apple_stock_news",
"prompt": "Query Apple's latest stock price and search related current news to explain the movement.",
"required_capabilities": [
"specialized_stock_quote",
"current_web_news"
]
},
{
"id": "transformer_arxiv_download",
"prompt": "Find the latest transformer papers on arXiv and download the top three PDFs.",
"required_capabilities": [
"arxiv_search",
"file_download"
],
"required_downloads": 3
},
{
"id": "github_contributors_visualization",
"prompt": "Analyze contributor statistics for openai/openai-python and generate a visualization report.",
"required_capabilities": [
"github_contributors",
"code_interpreter"
],
"required_visualizations": 1
}
],
"acceptance": {
"catalog_from_mcp": true,
"no_mock_tool_results": true,
"all_receipts_machine_readable": true,
"raw_schema_catalog_gzipped_and_hashed": true,
"compare_accuracy_and_completion_honestly": true
}
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/summary.json
+81
View File
@@ -0,0 +1,81 @@
{
"experiment": "4-5",
"campaign_id": "qwen3_4b_exact_20260730T061700Z",
"generated_at": "2026-07-30T05:05:11.594208+00:00",
"model": "qwen3:4b",
"catalog": {
"transport": "mcp-stdio",
"server": "/Users/boj/book/ai-agent-book/chapter4/perception-tools/src/main.py",
"tools_list_received": true,
"server_name": "perception-tools",
"server_version": "1.26.0",
"tool_count": 126,
"unique_tool_count": 126,
"schema_tokens_o200k": 50120,
"schema_bytes": 206819,
"schema_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627",
"required_tools_present": {
"web_search": true,
"code_interpreter": true,
"yfinance_quote": true,
"search_news": true,
"arxiv_search": true,
"arxiv_download": true,
"github_list_contributors": true
},
"catalog_gzip_bytes": 21244,
"catalog_gzip_sha256": "822bdadf46573ca850447f1ead6b526c01f46f73d35c90000895f891584d24bd",
"catalog_gzip_content_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627"
},
"embedding": {
"model": "sentence-transformers/all-MiniLM-L6-v2",
"backend": "local-transformers-mean-pooling",
"device": "cpu",
"local_files_only": true,
"catalog_text_count": 126,
"texts_sha256": "3da0897f88acbd0d32a3a77578bd135a91d631a8b85775ff12232eb5aafe3b9d",
"cache_path": "/Users/boj/book/ai-agent-book/chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json",
"cache_sha256": "23051857081c61e9dfc184d868fdaecd0bc816360133eeb951cbb6593fdda9c9",
"vector_count": 126,
"vector_dimensions": 384
},
"comparison": {
"control": {
"tasks": 3,
"mean_tool_selection_accuracy": 1.0,
"tasks_with_all_required_capabilities": 3,
"tasks_completed": 3,
"elapsed_seconds": 2382.981
},
"treatment": {
"tasks": 3,
"mean_tool_selection_accuracy": 0.6666666666666666,
"tasks_with_all_required_capabilities": 1,
"tasks_completed": 1,
"elapsed_seconds": 813.557
}
},
"dynamic_schema_injection_tokens": {
"apple_stock_news": 0,
"transformer_arxiv_download": 1852,
"github_contributors_visualization": 0
},
"acceptance": {
"status": "failed",
"gates": {
"exact_model_with_qwen_response_receipts": true,
"catalog_from_mcp_and_hash_matches": true,
"tool_count_at_least_120": true,
"control_over_50k_complete_schema_tokens": true,
"three_tasks_each_group": true,
"real_mcp_execution_only": false,
"all_tasks_completed_with_required_artifacts": false,
"treatment_discovery_history_and_status_verified": false,
"identical_tasks_model_runtime_and_catalog": true,
"local_embedding_index_receipted": true,
"dynamic_schema_injection_tokens_recorded": false,
"comparison_metrics_present_for_both_arms": true
}
},
"status": "failed"
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/treatment/apple_stock_news/receipt.json
+304
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/treatment/github_contributors_visualization/contributors.svg
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="500"><rect width="100%" height="100%" fill="white"/><text x="450" y="32" text-anchor="middle" font-size="22">openai/openai-python contributors</text></svg>
Side by Side

After

Width:  |  Height:  |  Size: 215 B

chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/treatment/github_contributors_visualization/receipt.json
+439
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/treatment/transformer_arxiv_download/papers/2607.27184v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/treatment/transformer_arxiv_download/papers/2607.27185v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/treatment/transformer_arxiv_download/papers/2607.27188v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_20260730T061700Z/treatment/transformer_arxiv_download/receipt.json
+1738
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/catalog.schemas.json.gz
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/catalog_receipt.json
+24
View File
@@ -0,0 +1,24 @@
{
"transport": "mcp-stdio",
"server": "/Users/boj/book/ai-agent-book/chapter4/perception-tools/src/main.py",
"tools_list_received": true,
"server_name": "perception-tools",
"server_version": "1.26.0",
"tool_count": 126,
"unique_tool_count": 126,
"schema_tokens_o200k": 50120,
"schema_bytes": 206819,
"schema_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627",
"required_tools_present": {
"web_search": true,
"code_interpreter": true,
"yfinance_quote": true,
"search_news": true,
"arxiv_search": true,
"arxiv_download": true,
"github_list_contributors": true
},
"catalog_gzip_bytes": 21244,
"catalog_gzip_sha256": "dd911ebfaf881037289bcf3c64cfe48fb2b4639438800860d50b9ec79615b903",
"catalog_gzip_content_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627"
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/control/apple_stock_news/receipt.json
+1778
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/control/github_contributors_visualization/contributors.svg
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="500"><rect width="100%" height="100%" fill="white"/><text x="450" y="32" text-anchor="middle" font-size="22">openai/openai-python contributors</text><rect x="70" y="70" width="68" height="350" fill="#4f46e5"/><text x="104.0" y="65" text-anchor="middle" font-size="11">700</text><text x="104.0" y="440" text-anchor="end" transform="rotate(-35 104.0 440)" font-size="10">stainless-app[bot]</text><rect x="146" y="282" width="68" height="138" fill="#4f46e5"/><text x="180.0" y="277" text-anchor="middle" font-size="11">276</text><text x="180.0" y="440" text-anchor="end" transform="rotate(-35 180.0 440)" font-size="10">stainless-bot</text><rect x="222" y="372" width="68" height="48" fill="#4f46e5"/><text x="256.0" y="367" text-anchor="middle" font-size="11">96</text><text x="256.0" y="440" text-anchor="end" transform="rotate(-35 256.0 440)" font-size="10">RobertCraigie</text><rect x="298" y="401" width="68" height="19" fill="#4f46e5"/><text x="332.0" y="396" text-anchor="middle" font-size="11">39</text><text x="332.0" y="440" text-anchor="end" transform="rotate(-35 332.0 440)" font-size="10">apcha-oai</text><rect x="374" y="406" width="68" height="14" fill="#4f46e5"/><text x="408.0" y="401" text-anchor="middle" font-size="11">29</text><text x="408.0" y="440" text-anchor="end" transform="rotate(-35 408.0 440)" font-size="10">hallacy</text><rect x="450" y="413" width="68" height="7" fill="#4f46e5"/><text x="484.0" y="408" text-anchor="middle" font-size="11">14</text><text x="484.0" y="440" text-anchor="end" transform="rotate(-35 484.0 440)" font-size="10">rachellim</text><rect x="526" y="414" width="68" height="6" fill="#4f46e5"/><text x="560.0" y="409" text-anchor="middle" font-size="11">13</text><text x="560.0" y="440" text-anchor="end" transform="rotate(-35 560.0 440)" font-size="10">logankilpatrick</text><rect x="602" y="416" width="68" height="4" fill="#4f46e5"/><text x="636.0" y="411" text-anchor="middle" font-size="11">9</text><text x="636.0" y="440" text-anchor="end" transform="rotate(-35 636.0 440)" font-size="10">dtmeadows</text><rect x="678" y="416" width="68" height="4" fill="#4f46e5"/><text x="712.0" y="411" text-anchor="middle" font-size="11">9</text><text x="712.0" y="440" text-anchor="end" transform="rotate(-35 712.0 440)" font-size="10">kristapratico</text><rect x="754" y="417" width="68" height="3" fill="#4f46e5"/><text x="788.0" y="412" text-anchor="middle" font-size="11">7</text><text x="788.0" y="440" text-anchor="end" transform="rotate(-35 788.0 440)" font-size="10">ddeville</text></svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/control/github_contributors_visualization/receipt.json
+2825
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/control/transformer_arxiv_download/papers/2607.27184v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/control/transformer_arxiv_download/papers/2607.27185v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/control/transformer_arxiv_download/papers/2607.27188v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/control/transformer_arxiv_download/receipt.json
+1653
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/embedding_receipt.json
+12
View File
@@ -0,0 +1,12 @@
{
"model": "sentence-transformers/all-MiniLM-L6-v2",
"backend": "local-transformers-mean-pooling",
"device": "cpu",
"local_files_only": true,
"catalog_text_count": 126,
"texts_sha256": "3da0897f88acbd0d32a3a77578bd135a91d631a8b85775ff12232eb5aafe3b9d",
"cache_path": "/Users/boj/book/ai-agent-book/chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json",
"cache_sha256": "23051857081c61e9dfc184d868fdaecd0bc816360133eeb951cbb6593fdda9c9",
"vector_count": 126,
"vector_dimensions": 384
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/failed_campaign_attempts/attempt-1/manifest.failed.json
+10
View File
@@ -0,0 +1,10 @@
{
"generated_at": "2026-07-30T06:17:57.036863+00:00",
"files": [
{
"path": "summary.failed.json",
"bytes": 3017,
"sha256": "cc817f77c7c268ced5dd5a19fc28afe2b8ca5f9466c339a9a29600812ba84c07"
}
]
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/failed_campaign_attempts/attempt-1/summary.failed.json
+81
View File
@@ -0,0 +1,81 @@
{
"experiment": "4-5",
"campaign_id": "qwen3_4b_exact_v2_20260730T130600Z",
"generated_at": "2026-07-30T06:17:56.800966+00:00",
"model": "qwen3:4b",
"catalog": {
"transport": "mcp-stdio",
"server": "/Users/boj/book/ai-agent-book/chapter4/perception-tools/src/main.py",
"tools_list_received": true,
"server_name": "perception-tools",
"server_version": "1.26.0",
"tool_count": 126,
"unique_tool_count": 126,
"schema_tokens_o200k": 50120,
"schema_bytes": 206819,
"schema_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627",
"required_tools_present": {
"web_search": true,
"code_interpreter": true,
"yfinance_quote": true,
"search_news": true,
"arxiv_search": true,
"arxiv_download": true,
"github_list_contributors": true
},
"catalog_gzip_bytes": 21244,
"catalog_gzip_sha256": "dd911ebfaf881037289bcf3c64cfe48fb2b4639438800860d50b9ec79615b903",
"catalog_gzip_content_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627"
},
"embedding": {
"model": "sentence-transformers/all-MiniLM-L6-v2",
"backend": "local-transformers-mean-pooling",
"device": "cpu",
"local_files_only": true,
"catalog_text_count": 126,
"texts_sha256": "3da0897f88acbd0d32a3a77578bd135a91d631a8b85775ff12232eb5aafe3b9d",
"cache_path": "/Users/boj/book/ai-agent-book/chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json",
"cache_sha256": "23051857081c61e9dfc184d868fdaecd0bc816360133eeb951cbb6593fdda9c9",
"vector_count": 126,
"vector_dimensions": 384
},
"comparison": {
"control": {
"tasks": 3,
"mean_tool_selection_accuracy": 1.0,
"tasks_with_all_required_capabilities": 3,
"tasks_completed": 3,
"elapsed_seconds": 2590.82
},
"treatment": {
"tasks": 3,
"mean_tool_selection_accuracy": 0.8333333333333334,
"tasks_with_all_required_capabilities": 2,
"tasks_completed": 2,
"elapsed_seconds": 1513.24
}
},
"dynamic_schema_injection_tokens": {
"apple_stock_news": 6201,
"transformer_arxiv_download": 4005,
"github_contributors_visualization": 2632
},
"acceptance": {
"status": "failed",
"gates": {
"exact_model_with_qwen_response_receipts": true,
"catalog_from_mcp_and_hash_matches": true,
"tool_count_at_least_120": true,
"control_over_50k_complete_schema_tokens": true,
"three_tasks_each_group": true,
"real_mcp_execution_only": false,
"all_tasks_completed_with_required_artifacts": false,
"treatment_discovery_history_and_status_verified": true,
"identical_tasks_model_runtime_and_catalog": true,
"local_embedding_index_receipted": true,
"dynamic_schema_injection_tokens_recorded": true,
"comparison_metrics_present_for_both_arms": true
}
},
"status": "failed"
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json
+48645
View File
File diff suppressed because it is too large Load Diff
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/manifest.json
+125
View File
@@ -0,0 +1,125 @@
{
"generated_at": "2026-07-30T06:23:19.609663+00:00",
"files": [
{
"path": "catalog.schemas.json.gz",
"bytes": 21244,
"sha256": "dd911ebfaf881037289bcf3c64cfe48fb2b4639438800860d50b9ec79615b903"
},
{
"path": "catalog_receipt.json",
"bytes": 864,
"sha256": "caa34a27d7483e4aec5de0be0af7124dab2cd3b546de1566709febc4c16f962c"
},
{
"path": "control/apple_stock_news/receipt.json",
"bytes": 71818,
"sha256": "79cd1fb10cefdb57e153353e79213fb8d72dbc84a877c53d042a7f0d50c94977"
},
{
"path": "control/github_contributors_visualization/contributors.svg",
"bytes": 2602,
"sha256": "d14524e843fa88f3ad322848fc7fa8cad7d5153c79f7d74c7be5e24ba3880b0a"
},
{
"path": "control/github_contributors_visualization/receipt.json",
"bytes": 124341,
"sha256": "9eac4f261867cdfdfebbcee18c846f3b496e30e21550e49cd8685ceb144c2391"
},
{
"path": "control/transformer_arxiv_download/papers/2607.27184v1.pdf",
"bytes": 1095498,
"sha256": "73f5c9a80cbfd807ed5bc14c963b7c9fd18a9de6eb30c8845b0505a79abd9536"
},
{
"path": "control/transformer_arxiv_download/papers/2607.27185v1.pdf",
"bytes": 797471,
"sha256": "4e4ff0947d28d37b7cfcfd743b50f1538945b6acfa598e74b084659436780754"
},
{
"path": "control/transformer_arxiv_download/papers/2607.27188v1.pdf",
"bytes": 1176383,
"sha256": "ecbe0ac9ae6768860d7f9f10a6986f86d4e98f9c73fd398e44894a8ac29b42e8"
},
{
"path": "control/transformer_arxiv_download/receipt.json",
"bytes": 67884,
"sha256": "22dc028746590c1086f8649c8f918543c2c3ff59f381c7374929eb0af64e3cbc"
},
{
"path": "embedding_receipt.json",
"bytes": 604,
"sha256": "88157c12bfd7df963bbc6ce411375930f744060a255b8cf39dacb80132a051e1"
},
{
"path": "failed_campaign_attempts/attempt-1/manifest.failed.json",
"bytes": 228,
"sha256": "eea533322acb44f6eb7bd97ad671ffc3ad1f9fd6a3dceaa634f363060ec1c980"
},
{
"path": "failed_campaign_attempts/attempt-1/summary.failed.json",
"bytes": 3017,
"sha256": "cc817f77c7c268ced5dd5a19fc28afe2b8ca5f9466c339a9a29600812ba84c07"
},
{
"path": "index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json",
"bytes": 1354409,
"sha256": "23051857081c61e9dfc184d868fdaecd0bc816360133eeb951cbb6593fdda9c9"
},
{
"path": "protocol.json",
"bytes": 1927,
"sha256": "4b63335edec11b5ef501972dfaf38aad49d35c6e67aaa625770ba3279c701f42"
},
{
"path": "summary.json",
"bytes": 3000,
"sha256": "c493ab24ca11bd2ef3bb425f46b778cee7bfb1a1c5e07beac03547a2dcdc7924"
},
{
"path": "treatment/apple_stock_news/contributors.svg",
"bytes": 215,
"sha256": "01d4f904edcaefb2563db27c7a3879eed9ed538df1dbd85fc86ce184bd84466f"
},
{
"path": "treatment/apple_stock_news/receipt.json",
"bytes": 87893,
"sha256": "8533019d4b7e2f5707f98e5444639205784b04c7528ecd68cbf35c086568e612"
},
{
"path": "treatment/github_contributors_visualization/contributors.svg",
"bytes": 2602,
"sha256": "d14524e843fa88f3ad322848fc7fa8cad7d5153c79f7d74c7be5e24ba3880b0a"
},
{
"path": "treatment/github_contributors_visualization/receipt.json",
"bytes": 51397,
"sha256": "b95c3d568bceec2de7c91638e9f74f87ea1e83b2a20e6ca26efba8ac6b027883"
},
{
"path": "treatment/transformer_arxiv_download/failed_attempts/attempt-1/receipt.json",
"bytes": 155879,
"sha256": "e65227aa379495c7a4208d2df8bc9d45da5e88257aec6854bbee41886fa32595"
},
{
"path": "treatment/transformer_arxiv_download/papers/2607.27184v1.pdf",
"bytes": 1095498,
"sha256": "73f5c9a80cbfd807ed5bc14c963b7c9fd18a9de6eb30c8845b0505a79abd9536"
},
{
"path": "treatment/transformer_arxiv_download/papers/2607.27185v1.pdf",
"bytes": 797471,
"sha256": "4e4ff0947d28d37b7cfcfd743b50f1538945b6acfa598e74b084659436780754"
},
{
"path": "treatment/transformer_arxiv_download/papers/2607.27188v1.pdf",
"bytes": 1176383,
"sha256": "ecbe0ac9ae6768860d7f9f10a6986f86d4e98f9c73fd398e44894a8ac29b42e8"
},
{
"path": "treatment/transformer_arxiv_download/receipt.json",
"bytes": 66186,
"sha256": "6800b522dc05040ae46c864e05a673df4d17872051a041c46b18a821aba83ced"
}
]
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/protocol.json
+59
View File
@@ -0,0 +1,59 @@
{
"experiment": "4-5",
"title": "Active tool discovery with Qwen3-4B and the perception MCP server",
"authority": "book/chapter4.md:624",
"model": "qwen3:4b",
"model_runtime": "ollama",
"minimum_mcp_tools": 120,
"minimum_control_schema_tokens": 50000,
"control": {
"system_tools": "all complete schemas returned by perception MCP tools/list",
"required_behavior": "plan and execute using the full catalog"
},
"treatment": {
"system_tools": [
"web_search",
"code_interpreter",
"discover_tools"
],
"discovery_top_k": 5,
"schema_injection_role": "user",
"status_bar_updates": true,
"base_tool_boundary": "Generic web search and local code cannot substitute for authoritative domain-specific retrieval; discover each missing specialist at the moment the gap arises."
},
"tasks": [
{
"id": "apple_stock_news",
"prompt": "Query Apple's latest stock price and search related current news to explain the movement.",
"required_capabilities": [
"specialized_stock_quote",
"current_web_news"
]
},
{
"id": "transformer_arxiv_download",
"prompt": "Find the latest transformer papers on arXiv and download the top three PDFs.",
"required_capabilities": [
"arxiv_search",
"file_download"
],
"required_downloads": 3
},
{
"id": "github_contributors_visualization",
"prompt": "Analyze contributor statistics for openai/openai-python and generate a visualization report.",
"required_capabilities": [
"github_contributors",
"code_interpreter"
],
"required_visualizations": 1
}
],
"acceptance": {
"catalog_from_mcp": true,
"no_mock_tool_results": true,
"all_receipts_machine_readable": true,
"raw_schema_catalog_gzipped_and_hashed": true,
"compare_accuracy_and_completion_honestly": true
}
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/summary.json
+81
View File
@@ -0,0 +1,81 @@
{
"experiment": "4-5",
"campaign_id": "qwen3_4b_exact_v2_20260730T130600Z",
"generated_at": "2026-07-30T06:23:19.390598+00:00",
"model": "qwen3:4b",
"catalog": {
"transport": "mcp-stdio",
"server": "/Users/boj/book/ai-agent-book/chapter4/perception-tools/src/main.py",
"tools_list_received": true,
"server_name": "perception-tools",
"server_version": "1.26.0",
"tool_count": 126,
"unique_tool_count": 126,
"schema_tokens_o200k": 50120,
"schema_bytes": 206819,
"schema_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627",
"required_tools_present": {
"web_search": true,
"code_interpreter": true,
"yfinance_quote": true,
"search_news": true,
"arxiv_search": true,
"arxiv_download": true,
"github_list_contributors": true
},
"catalog_gzip_bytes": 21244,
"catalog_gzip_sha256": "dd911ebfaf881037289bcf3c64cfe48fb2b4639438800860d50b9ec79615b903",
"catalog_gzip_content_sha256": "77b57f69c57243f3295e6b2c3cf493420fe1bdbdd816df7d6374a1640221d627"
},
"embedding": {
"model": "sentence-transformers/all-MiniLM-L6-v2",
"backend": "local-transformers-mean-pooling",
"device": "cpu",
"local_files_only": true,
"catalog_text_count": 126,
"texts_sha256": "3da0897f88acbd0d32a3a77578bd135a91d631a8b85775ff12232eb5aafe3b9d",
"cache_path": "/Users/boj/book/ai-agent-book/chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/index/embeddings-all-MiniLM-L6-v2-3da0897f88acbd0d32a3.json",
"cache_sha256": "23051857081c61e9dfc184d868fdaecd0bc816360133eeb951cbb6593fdda9c9",
"vector_count": 126,
"vector_dimensions": 384
},
"comparison": {
"control": {
"tasks": 3,
"mean_tool_selection_accuracy": 1.0,
"tasks_with_all_required_capabilities": 3,
"tasks_completed": 3,
"elapsed_seconds": 2590.82
},
"treatment": {
"tasks": 3,
"mean_tool_selection_accuracy": 1.0,
"tasks_with_all_required_capabilities": 3,
"tasks_completed": 3,
"elapsed_seconds": 808.926
}
},
"dynamic_schema_injection_tokens": {
"apple_stock_news": 6201,
"transformer_arxiv_download": 4005,
"github_contributors_visualization": 2632
},
"acceptance": {
"status": "passed",
"gates": {
"exact_model_with_qwen_response_receipts": true,
"catalog_from_mcp_and_hash_matches": true,
"tool_count_at_least_120": true,
"control_over_50k_complete_schema_tokens": true,
"three_tasks_each_group": true,
"real_mcp_execution_only": true,
"all_tasks_completed_with_required_artifacts": true,
"treatment_discovery_history_and_status_verified": true,
"identical_tasks_model_runtime_and_catalog": true,
"local_embedding_index_receipted": true,
"dynamic_schema_injection_tokens_recorded": true,
"comparison_metrics_present_for_both_arms": true
}
},
"status": "passed"
}
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/apple_stock_news/contributors.svg
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="500"><rect width="100%" height="100%" fill="white"/><text x="450" y="32" text-anchor="middle" font-size="22">openai/openai-python contributors</text></svg>
Side by Side

After

Width:  |  Height:  |  Size: 215 B

chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/apple_stock_news/receipt.json
+1190
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/github_contributors_visualization/contributors.svg
+1
View File
@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" width="900" height="500"><rect width="100%" height="100%" fill="white"/><text x="450" y="32" text-anchor="middle" font-size="22">openai/openai-python contributors</text><rect x="70" y="70" width="68" height="350" fill="#4f46e5"/><text x="104.0" y="65" text-anchor="middle" font-size="11">700</text><text x="104.0" y="440" text-anchor="end" transform="rotate(-35 104.0 440)" font-size="10">stainless-app[bot]</text><rect x="146" y="282" width="68" height="138" fill="#4f46e5"/><text x="180.0" y="277" text-anchor="middle" font-size="11">276</text><text x="180.0" y="440" text-anchor="end" transform="rotate(-35 180.0 440)" font-size="10">stainless-bot</text><rect x="222" y="372" width="68" height="48" fill="#4f46e5"/><text x="256.0" y="367" text-anchor="middle" font-size="11">96</text><text x="256.0" y="440" text-anchor="end" transform="rotate(-35 256.0 440)" font-size="10">RobertCraigie</text><rect x="298" y="401" width="68" height="19" fill="#4f46e5"/><text x="332.0" y="396" text-anchor="middle" font-size="11">39</text><text x="332.0" y="440" text-anchor="end" transform="rotate(-35 332.0 440)" font-size="10">apcha-oai</text><rect x="374" y="406" width="68" height="14" fill="#4f46e5"/><text x="408.0" y="401" text-anchor="middle" font-size="11">29</text><text x="408.0" y="440" text-anchor="end" transform="rotate(-35 408.0 440)" font-size="10">hallacy</text><rect x="450" y="413" width="68" height="7" fill="#4f46e5"/><text x="484.0" y="408" text-anchor="middle" font-size="11">14</text><text x="484.0" y="440" text-anchor="end" transform="rotate(-35 484.0 440)" font-size="10">rachellim</text><rect x="526" y="414" width="68" height="6" fill="#4f46e5"/><text x="560.0" y="409" text-anchor="middle" font-size="11">13</text><text x="560.0" y="440" text-anchor="end" transform="rotate(-35 560.0 440)" font-size="10">logankilpatrick</text><rect x="602" y="416" width="68" height="4" fill="#4f46e5"/><text x="636.0" y="411" text-anchor="middle" font-size="11">9</text><text x="636.0" y="440" text-anchor="end" transform="rotate(-35 636.0 440)" font-size="10">dtmeadows</text><rect x="678" y="416" width="68" height="4" fill="#4f46e5"/><text x="712.0" y="411" text-anchor="middle" font-size="11">9</text><text x="712.0" y="440" text-anchor="end" transform="rotate(-35 712.0 440)" font-size="10">kristapratico</text><rect x="754" y="417" width="68" height="3" fill="#4f46e5"/><text x="788.0" y="412" text-anchor="middle" font-size="11">7</text><text x="788.0" y="440" text-anchor="end" transform="rotate(-35 788.0 440)" font-size="10">ddeville</text></svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.5 KiB

chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/github_contributors_visualization/receipt.json
+703
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/transformer_arxiv_download/failed_attempts/attempt-1/receipt.json
+1619
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/transformer_arxiv_download/papers/2607.27184v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/transformer_arxiv_download/papers/2607.27185v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/transformer_arxiv_download/papers/2607.27188v1.pdf
BIN
View File
Binary file not shown.
chapter4/active-tool-discovery/validation/experiment_4_5/qwen3_4b_exact_v2_20260730T130600Z/treatment/transformer_arxiv_download/receipt.json
+909
View File
File diff suppressed because one or more lines are too long
chapter4/active-tool-selection/ARCHITECTURE.md
+596
View File
@@ -0,0 +1,596 @@
# Architecture Deep Dive
This document provides a detailed explanation of the active tool selection system architecture, inspired by MCP-Zero.
## Table of Contents
1. [System Overview](#system-overview)
2. [Core Components](#core-components)
3. [Active Discovery Flow](#active-discovery-flow)
4. [Semantic Routing Algorithm](#semantic-routing-algorithm)
5. [Comparison: Active vs Passive](#comparison-active-vs-passive)
6. [Performance Optimization](#performance-optimization)
7. [Design Decisions](#design-decisions)
## System Overview
The active tool selection system consists of four major components working together:
```
┌─────────────────────────────────────────────────────────┐
│ User Task │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Active Tool Agent │
│ • Task analysis │
│ • Capability gap identification │
│ • Structured tool request generation │
│ • Tool usage and task execution │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Hierarchical Semantic Router │
│ Stage 1: Server-level routing (platform matching) │
│ Stage 2: Tool-level routing (operation matching) │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Tool Knowledge Base │
│ 8 Servers × 40+ Tools │
│ Organized by domain/platform │
└─────────────────────────────────────────────────────────┘
```
## Core Components
### 1. Active Tool Agent (`agent.py`)
The agent is responsible for:
#### Task Analysis
```python
def execute_task(self, task: str):
# 1. Initialize with empty toolset
self.available_tools = []
# 2. Analyze task to identify capability needs
# 3. Generate structured tool requests
# 4. Iteratively discover and load tools
# 5. Execute task with discovered tools
```
#### Tool Request Generation
Agent generates structured requests in this format:
```xml
<tool_request>
server: [platform/domain description]
tool: [operation description]
</tool_request>
```
**Example:**
```xml
<tool_request>
server: GitHub for repository operations
tool: search repositories by keywords and filters
</tool_request>
```
#### Iterative Discovery
The agent can make multiple tool requests as understanding evolves:
```python
# Iteration 1: Basic need identified
Request: "GitHub repository access"
→ Load: github_search_repos, github_list_issues
# Iteration 2: Additional need identified
Request: "File system operations for local storage"
→ Load: fs_read_file, fs_write_file
# Iteration 3: Analysis need identified
Request: "Data visualization and statistics"
→ Load: analytics_summarize, analytics_visualize
```
### 2. Semantic Router (`semantic_router.py`)
Implements two-stage hierarchical routing:
#### Stage 1: Server-Level Routing
Matches tool requests to relevant servers (platforms):
```python
def _route_to_servers(self, request: str, top_k: int):
# 1. Vectorize request using TF-IDF
request_vector = self.server_vectorizer.transform([request])
# 2. Calculate cosine similarity with all servers
similarities = cosine_similarity(request_vector, self.server_embeddings)
# 3. Return top-K servers by similarity
top_indices = np.argsort(similarities)[::-1][:top_k]
return [(self.servers[idx], similarities[idx]) for idx in top_indices]
```
**Why This Works:**
- Reduces search space from all tools to tools in relevant servers
- Platform/domain matching is coarse-grained and reliable
- Example: "GitHub" request → GitHub server (not filesystem server)
#### Stage 2: Tool-Level Routing
Matches requests to specific tools within selected servers:
```python
def _route_to_tools(self, server: ServerDefinition, request: str, top_k: int):
# 1. Get server-specific vectorizer and embeddings
vectorizer = self.tool_vectorizers[server.name]
tool_embeddings = server._tool_embeddings
# 2. Vectorize request
request_vector = vectorizer.transform([request])
# 3. Calculate similarity with tools in this server
similarities = cosine_similarity(request_vector, tool_embeddings)
# 4. Return top-K tools
top_indices = np.argsort(similarities)[::-1][:top_k]
return [(server.tools[idx], similarities[idx]) for idx in top_indices]
```
**Why This Works:**
- Fine-grained matching within relevant domain
- Tool descriptions are more specific than server descriptions
- Example: "search repositories" → github_search_repos (not github_create_issue)
#### Score Combination
Final tool scores combine both stages:
```python
combined_score = 0.3 * server_score + 0.7 * tool_score
```
**Rationale:**
- Server score (30%): Ensures tool is from relevant domain
- Tool score (70%): Prioritizes operation-level match
- Weighted combination prevents cross-domain false positives
### 3. Tool Knowledge Base (`tool_knowledge_base.py`)
Organized hierarchically:
```
Knowledge Base
├── GitHub Server
│ ├── github_search_repos
│ ├── github_create_pr
│ ├── github_list_issues
│ ├── github_get_file
│ └── github_create_issue
├── Filesystem Server
│ ├── fs_read_file
│ ├── fs_write_file
│ ├── fs_list_directory
│ ├── fs_delete_file
│ └── fs_search_files
├── Database Server
│ ├── db_query
│ ├── db_insert
│ ├── db_update
│ ├── db_delete
│ └── db_schema
└── ... (5 more servers)
```
**Design Principles:**
1. **Hierarchical Organization**: Tools grouped by platform/domain
2. **Rich Descriptions**: Both servers and tools have semantic descriptions
3. **Standard Schema**: OpenAI function calling format
4. **Extensible**: Easy to add new servers/tools
### 4. Configuration (`config.py`)
Centralized configuration for all components:
```python
# LLM Settings
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_BASE_URL = os.getenv("OPENAI_BASE_URL")
OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-5.6-luna")
# Routing Thresholds
SIMILARITY_THRESHOLD = 0.3 # Minimum similarity for match
TOP_K_SERVERS = 3 # Servers to search
TOP_K_TOOLS = 5 # Tools per server
# Agent Limits
MAX_TOOL_REQUESTS = 5 # Max discovery iterations
```
## Active Discovery Flow
Detailed flow of active tool discovery:
```
┌─────────────────────────────────────────────────────────┐
│ Step 1: Task Submission │
│ User: "Search for Python ML repos on GitHub" │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Step 2: Task Analysis (Agent) │
│ • Identifies need for repository search capability │
│ • Current tools: None │
│ • Decision: Request GitHub tools │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Step 3: Tool Request Generation │
│ <tool_request> │
│ server: GitHub for repository operations │
│ tool: search repositories by keywords │
│ </tool_request> │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Step 4: Semantic Routing │
│ Stage 1: Server routing │
│ • github: 0.89 ✓ │
│ • filesystem: 0.12 │
│ • web: 0.24 │
│ │
│ Stage 2: Tool routing (GitHub server) │
│ • github_search_repos: 0.94 ✓ │
│ • github_list_issues: 0.45 │
│ • github_get_file: 0.31 │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Step 5: Tool Loading │
│ Loaded: [github_search_repos] │
│ Available tools count: 1 │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Step 6: Task Execution │
│ Agent uses github_search_repos to complete task │
└──────────────────────┬──────────────────────────────────┘
↓
┌─────────────────────────────────────────────────────────┐
│ Step 7: Response │
│ Results returned to user │
│ Metrics: 1 tool loaded, ~2000 tokens used │
└─────────────────────────────────────────────────────────┘
```
### Multi-Iteration Example
Complex task requiring multiple tool discovery iterations:
```
Task: "Clone repo, analyze code, visualize metrics, email report"
Iteration 1:
Analysis: Need GitHub access
Request: GitHub repository operations
Loaded: github tools (2 tools)
Iteration 2:
Analysis: Need file system for code storage
Request: Filesystem operations
Loaded: filesystem tools (3 tools total)
Iteration 3:
Analysis: Need analytics for code analysis
Request: Data analytics and visualization
Loaded: analytics tools (5 tools total)
Iteration 4:
Analysis: Need communication for email
Request: Email communication
Loaded: communication tools (6 tools total)
Execution: Use all 6 tools to complete task
```
## Semantic Routing Algorithm
### TF-IDF Vectorization
Tools and requests are converted to vectors using TF-IDF:
```python
# Build vocabulary from all tool descriptions
vectorizer = TfidfVectorizer(stop_words='english')
# Server descriptions
server_docs = [f"{s.name} {s.description}" for s in servers]
server_matrix = vectorizer.fit_transform(server_docs)
# Tool descriptions (per server)
tool_docs = [f"{t.name} {t.description}" for t in tools]
tool_matrix = vectorizer.fit_transform(tool_docs)
```
**What is TF-IDF?**
- **TF (Term Frequency)**: How often a word appears in a document
- **IDF (Inverse Document Frequency)**: How rare a word is across documents
- **TF-IDF**: Words that are frequent in a document but rare overall get high scores
**Example:**
```
Server: "GitHub repository management and version control"
Tool: "search repositories by keywords"
Request: "find GitHub repositories"
TF-IDF vectors capture semantic overlap:
- "repository" appears in all three → medium weight
- "GitHub" appears in server and request → strong match
- "search" appears in tool and request → strong match
```
### Cosine Similarity
Measures similarity between vectors:
```python
similarity = cosine_similarity(request_vector, tool_vector)
# Returns value between 0 (orthogonal) and 1 (identical)
```
**Geometric Interpretation:**
```
If vectors point in same direction → similar (score near 1)
If vectors are perpendicular → dissimilar (score near 0)
```
**Example Scores:**
```
Request: "search for repositories"
• github_search_repos: 0.92 (strong match)
• github_create_pr: 0.31 (weak match)
• fs_read_file: 0.08 (no match)
```
### Threshold Filtering
Tools below similarity threshold are filtered out:
```python
SIMILARITY_THRESHOLD = 0.3
relevant_tools = [
tool for tool, score in tool_scores
if score >= SIMILARITY_THRESHOLD
]
```
**Why 0.3?**
- Balance between precision and recall
- Captures semantic overlap without false positives
- Empirically determined from testing
## Comparison: Active vs Passive
### Passive Tool Injection (Traditional)
```python
class PassiveToolAgent:
def __init__(self):
# Load ALL tools at initialization
self.all_tools = load_all_40_plus_tools()
def execute_task(self, task):
# Inject all tool schemas into prompt
response = llm.complete(
messages=[{"role": "user", "content": task}],
tools=self.all_tools # 40+ tool schemas
)
```
**Problems:**
1. **Massive Context**: 30k-50k tokens just for tool schemas
2. **Poor Scalability**: Adding 10 tools increases every request by 5k tokens
3. **Lost Autonomy**: Agent selects from pre-defined set
4. **Cognitive Overload**: LLM must process irrelevant tools
### Active Tool Discovery (MCP-Zero Approach)
```python
class ActiveToolAgent:
def __init__(self):
# Start with empty toolset
self.available_tools = []
def execute_task(self, task):
# Iteratively discover tools as needed
while not task_complete:
# Agent identifies capability gaps
if need_more_tools:
request = agent.generate_tool_request()
new_tools = router.discover_tools(request)
self.available_tools.extend(new_tools)
else:
# Use available tools
execute_with_tools(self.available_tools)
```
**Benefits:**
1. **Minimal Context**: 2k-5k tokens (only needed tools)
2. **Efficient Scaling**: Adding 100 tools doesn't affect simple tasks
3. **Preserved Autonomy**: Agent controls capability acquisition
4. **Focused Processing**: LLM sees only relevant tools
### Performance Comparison Table
| Metric | Passive | Active | Improvement |
|--------|---------|--------|-------------|
| **Initial Tools** | 40 | 0 | N/A |
| **Tools for Simple Task** | 40 | 2-3 | 92-95% reduction |
| **Tokens (Simple Task)** | 45,000 | 2,500 | 94% reduction |
| **Tokens (Complex Task)** | 50,000 | 8,000 | 84% reduction |
| **Scalability** | O(n) | O(k) | k << n |
| **Agent Autonomy** | Low | High | Qualitative |
where:
- n = total tools in ecosystem
- k = tools needed for specific task
## Performance Optimization
### 1. Embedding Precomputation
Tool embeddings are computed once at initialization:
```python
def __init__(self, servers):
# Precompute all embeddings
self._build_server_index()
self._build_tool_indices()
# Query time: just cosine similarity
# No re-vectorization needed
```
**Benefit**: O(1) query time instead of O(n) vectorization
### 2. Hierarchical Search
Two-stage routing reduces complexity:
```python
# Without hierarchy: Search all 40 tools
# Complexity: O(40) similarity comparisons
# With hierarchy: Search 8 servers, then top-3 servers
# Stage 1: O(8) server comparisons
# Stage 2: O(5) tool comparisons per server = O(15)
# Total: O(8 + 15) = O(23)
# Savings: 40 - 23 = 17 comparisons (42% reduction)
```
**Scales Better**:
- 100 tools, 10 servers: 100 vs 35 comparisons (65% reduction)
- 1000 tools, 20 servers: 1000 vs 120 comparisons (88% reduction)
### 3. Caching Potential
Future optimization: Cache routing results:
```python
# Cache structure
routing_cache = {
"search GitHub repos": ["github_search_repos", ...],
"read local file": ["fs_read_file", ...]
}
# Cache hit: O(1) lookup
# Cache miss: Fall back to semantic routing
```
## Design Decisions
### Why TF-IDF Instead of Neural Embeddings?
**Chosen**: TF-IDF with cosine similarity
**Alternatives Considered**:
- Sentence-BERT embeddings
- OpenAI embeddings (text-embedding-ada-002)
**Rationale**:
1. **Educational Clarity**: TF-IDF is easier to understand and debug
2. **No API Calls**: Works offline without additional costs
3. **Sufficient Performance**: Tool descriptions are technical and keyword-rich
4. **Fast**: No model inference required
**When Neural Embeddings Better**:
- Natural language queries (less technical)
- Semantic nuances important
- Large corpus with synonyms
### Why Two-Stage Routing?
**Alternatives Considered**:
- Flat search over all tools
- Clustering-based search
- Retrieval-augmented generation (RAG)
**Rationale**:
1. **Matches Mental Model**: Users think "GitHub" → "search repos"
2. **Reduces False Positives**: "search" alone might match wrong domain
3. **Improves Precision**: Server context narrows tool search
4. **Scalable**: Logarithmic complexity vs linear
### Why Structured Requests?
**Format**:
```xml
<tool_request>
server: [domain]
tool: [operation]
</tool_request>
```
**Alternatives Considered**:
- Free-form natural language
- JSON format
- Function calling
**Rationale**:
1. **Explicit Structure**: Server + tool decomposition matches routing stages
2. **Easy Parsing**: Simple string matching
3. **LLM-Friendly**: Clear format reduces ambiguity
4. **Semantic Alignment**: Request format matches knowledge base organization
### Why Simulated Tool Execution?
**Decision**: Tools return simulated results instead of real execution
**Rationale**:
1. **Educational Focus**: Demonstrates discovery, not execution
2. **Safety**: No real API calls or file operations
3. **Portability**: Works without external dependencies
4. **Simplicity**: Focus on architecture, not integration
**Future Enhancement**: Connect to real APIs for production use
### Why 3 Servers and 5 Tools?
**Configuration**:
```python
TOP_K_SERVERS = 3
TOP_K_TOOLS = 5
```
**Rationale**:
1. **Balance**: Captures relevant tools without overwhelming context
2. **Empirical**: Tested on various tasks, 3×5=15 tools usually sufficient
3. **Context Window**: 15 tool schemas ≈ 3k-5k tokens (manageable)
4. **Fallback**: Can request more tools if initial set insufficient
**Tuning Guidelines**:
- Simple tasks: Decrease to 2×3 = 6 tools
- Complex tasks: Increase to 5×7 = 35 tools
- Large ecosystems: Keep ratio, not absolute numbers
## Conclusion
The active tool selection architecture demonstrates that:
1. **Hierarchical routing** reduces search complexity while maintaining precision
2. **Active discovery** preserves agent autonomy and scales efficiently
3. **Iterative extension** allows toolchains to evolve with task understanding
4. **Semantic matching** (even with simple TF-IDF) works well for tool discovery
This architecture represents a fundamental shift from passive tool injection to active capability acquisition, enabling agents to operate effectively in ecosystems with hundreds or thousands of available tools.
chapter4/active-tool-selection/README.md
+1104
View File
File diff suppressed because it is too large Load Diff
chapter4/active-tool-selection/agent.py
+541
View File
@@ -0,0 +1,541 @@
"""
Active Tool Discovery Agent.
Implements an LLM agent that actively requests tools on-demand rather than
having all tool schemas injected into the prompt. Inspired by MCP-Zero.
"""
from typing import List, Dict, Any, Optional
from openai import OpenAI
from tool_knowledge_base import ToolDefinition, ServerDefinition, create_tool_knowledge_base
from semantic_router import SemanticRouter, StructuredRequestParser
import config
class ActiveToolAgent:
"""
Agent that actively discovers and requests tools as needed.
Key principles:
1. Maintains minimal context by not injecting all tools upfront
2. Actively requests specific tools when capability gaps are identified
3. Iteratively builds toolchain as task understanding evolves
"""
def __init__(self, servers: Optional[List[ServerDefinition]] = None,
model: Optional[str] = None):
self.client = OpenAI(
api_key=config.OPENAI_API_KEY,
base_url=config.OPENAI_BASE_URL
)
self.model = model or config.OPENAI_MODEL
# Initialize tool knowledge base (callers may inject a padded/custom catalog)
self.servers = servers if servers is not None else create_tool_knowledge_base()
self.router = SemanticRouter(self.servers)
# Agent state
self.conversation_history = []
self.available_tools: List[ToolDefinition] = [] # Currently loaded tools
self.tool_request_count = 0
# Metrics
self.metrics = {
'tokens_used': 0,
'tool_requests': 0,
'tools_loaded': 0,
'api_calls': 0,
'tools_called': [] # Names of tools the model actually invoked
}
def execute_task(self, task: str) -> Dict[str, Any]:
"""
Execute a task with active tool discovery.
The agent will:
1. Analyze the task
2. Identify capability gaps
3. Request specific tools
4. Execute with discovered tools
Returns execution results with metrics.
"""
self.conversation_history = []
self.available_tools = []
self.tool_request_count = 0
# Initial system message explaining active tool discovery
system_message = self._create_system_message()
self.conversation_history.append({
"role": "system",
"content": system_message
})
# Add user task
self.conversation_history.append({
"role": "user",
"content": task
})
# Iterative tool discovery and execution
max_iterations = config.MAX_TOOL_REQUESTS
for iteration in range(max_iterations):
# Get agent response
response = self._call_llm()
self.metrics['api_calls'] += 1
# Check if agent is requesting tools
tool_request = StructuredRequestParser.parse_request(response)
if tool_request:
# Agent is requesting tools - discover and provide them
self._handle_tool_request(tool_request, response)
self.tool_request_count += 1
self.metrics['tool_requests'] += 1
else:
# Agent has what it needs and is responding
self.conversation_history.append({
"role": "assistant",
"content": response
})
break
return {
'response': response,
'metrics': self.metrics,
'tools_loaded': [t.name for t in self.available_tools],
'conversation': self.conversation_history
}
def _create_system_message(self) -> str:
"""Create system message explaining active tool discovery."""
return """You are an autonomous AI agent with active tool discovery capabilities.
Instead of having all possible tools available upfront, you can actively request tools as you need them. This allows you to:
1. Maintain a minimal context footprint
2. Focus on relevant capabilities for the current task
3. Iteratively build your toolchain as your understanding evolves
When you identify a capability gap, request tools using this format:
<tool_request>
server: [describe the platform/domain you need, e.g., "GitHub for repository operations" or "filesystem for local file access"]
tool: [describe the specific operation you need, e.g., "search repositories" or "read file contents"]
</tool_request>
After requesting tools, they will be provided to you. You can then use them to accomplish the task.
Process:
1. Analyze the task and identify what capabilities you need
2. Request specific tools if you don't have them yet
3. Once you have the necessary tools, use them to complete the task
4. Respond with your findings or results
Current available tools: None (request tools as needed)"""
def _call_llm(self) -> str:
"""Call LLM with current context and available tools."""
kwargs = {
"model": self.model,
"messages": self.conversation_history,
"temperature": config.AGENT_TEMPERATURE
}
# Add tools if available
if self.available_tools:
kwargs["tools"] = [tool.to_schema() for tool in self.available_tools]
kwargs["tool_choice"] = "auto"
response = self.client.chat.completions.create(**kwargs)
# Track token usage
# response.usage is Optional in the OpenAI SDK: the attribute always
# exists, but is None when the provider omits token accounting.
if getattr(response, 'usage', None):
self.metrics['tokens_used'] += response.usage.total_tokens
# Extract response content
message = response.choices[0].message
# Handle tool calls if present
if message.tool_calls:
return self._handle_tool_calls(message)
return message.content or ""
def _handle_tool_request(self, tool_request: Dict[str, str], full_response: str):
"""
Handle tool request from agent.
Args:
tool_request: Parsed tool request with 'server' and 'tool' fields
full_response: Full response text from agent
"""
# Combine server and tool descriptions for routing
query = f"{tool_request['server']} {tool_request['tool']}"
# Use semantic router to find relevant tools
discovered_tools = self.router.route_request(query)
if not discovered_tools:
# No tools found
feedback = f"""No tools found matching your request. Please refine your request or proceed without additional tools.
Your request was:
- Server: {tool_request['server']}
- Tool: {tool_request['tool']}"""
else:
# Add discovered tools to available tools
new_tools = []
for tool in discovered_tools:
if tool not in self.available_tools:
self.available_tools.append(tool)
new_tools.append(tool)
self.metrics['tools_loaded'] += 1
tool_list = "\n".join([f"- {t.name}: {t.description}" for t in new_tools])
feedback = f"""Tools discovered and loaded ({len(new_tools)} new tools):
{tool_list}
You can now use these tools to complete the task. Please proceed."""
# Add agent's request and system's response to history
self.conversation_history.append({
"role": "assistant",
"content": full_response
})
self.conversation_history.append({
"role": "user",
"content": feedback
})
def _handle_tool_calls(self, message) -> str:
"""Handle actual tool execution (simulated for demo)."""
# For this educational demo, we simulate tool execution
tool_results = []
for tool_call in message.tool_calls:
func_name = tool_call.function.name
self.metrics['tools_called'].append(func_name)
# Simulate tool execution
result = f"[Simulated] Tool '{func_name}' executed successfully with result: Success"
tool_results.append({
"tool_call_id": tool_call.id,
"output": result
})
# Add tool call message to history
self.conversation_history.append({
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": tc.id,
"type": "function",
"function": {
"name": tc.function.name,
"arguments": tc.function.arguments
}
}
for tc in message.tool_calls
]
})
# Add tool results to history
for result in tool_results:
self.conversation_history.append({
"role": "tool",
"tool_call_id": result["tool_call_id"],
"content": result["output"]
})
# Get final response after tool execution
return self._call_llm()
def reset(self):
"""Reset agent state."""
self.conversation_history = []
self.available_tools = []
self.tool_request_count = 0
self.metrics = {
'tokens_used': 0,
'tool_requests': 0,
'tools_loaded': 0,
'api_calls': 0,
'tools_called': []
}
class RetrievalToolAgent:
"""
One-shot retrieval agent (semantic tool retrieval / "工具检索").
This is the RAG-style middle ground between passive injection and active
discovery: before the very first LLM call, it retrieves the top-k tools most
semantically relevant to the task and injects *only* those. There is no extra
discovery round-trip — tool selection is delegated to the retriever, turning the
"which of hundreds of tools" problem into a knowledge-retrieval problem.
This directly embodies the mechanism the chapter attributes to Anthropic's
on-demand tool retrieval experiment: fewer, more relevant tool schemas in
context both cut token cost and reduce the model's selection errors.
"""
def __init__(self, servers: Optional[List[ServerDefinition]] = None,
model: Optional[str] = None, top_k: Optional[int] = None):
self.client = OpenAI(
api_key=config.OPENAI_API_KEY,
base_url=config.OPENAI_BASE_URL
)
self.model = model or config.OPENAI_MODEL
self.top_k = top_k if top_k is not None else config.TOP_K_TOOLS
self.servers = servers if servers is not None else create_tool_knowledge_base()
self.router = SemanticRouter(self.servers)
self.conversation_history = []
self.available_tools: List[ToolDefinition] = []
self.metrics = {
'tokens_used': 0,
'tools_loaded': 0,
'api_calls': 0,
'tools_called': []
}
def execute_task(self, task: str) -> Dict[str, Any]:
"""Retrieve top-k relevant tools for the task, then execute in one shot."""
self.conversation_history = []
# Retrieval step (no LLM call): pick the top-k most relevant tools.
self.available_tools = self.router.retrieve(task, self.top_k)
self.metrics['tools_loaded'] = len(self.available_tools)
tool_list = "\n".join(
f"- {t.name}: {t.description}" for t in self.available_tools
)
system_message = f"""You are an AI agent. A retrieval system has pre-selected the \
{len(self.available_tools)} tools below as most relevant to the user's task.
{tool_list}
Analyze the task and call the appropriate tool(s) to complete it."""
self.conversation_history.append({"role": "system", "content": system_message})
self.conversation_history.append({"role": "user", "content": task})
response = self._call_llm()
self.metrics['api_calls'] += 1
return {
'response': response,
'metrics': self.metrics,
'tools_loaded': [t.name for t in self.available_tools],
'conversation': self.conversation_history
}
def _call_llm(self) -> str:
"""Call LLM with only the retrieved tools injected."""
kwargs = {
"model": self.model,
"messages": self.conversation_history,
"temperature": config.AGENT_TEMPERATURE
}
if self.available_tools:
kwargs["tools"] = [tool.to_schema() for tool in self.available_tools]
kwargs["tool_choice"] = "auto"
response = self.client.chat.completions.create(**kwargs)
# response.usage is Optional in the OpenAI SDK: the attribute always
# exists, but is None when the provider omits token accounting.
if getattr(response, 'usage', None):
self.metrics['tokens_used'] += response.usage.total_tokens
message = response.choices[0].message
if message.tool_calls:
return self._handle_tool_calls(message)
return message.content or ""
def _handle_tool_calls(self, message) -> str:
"""Handle tool execution (simulated)."""
tool_results = []
for tool_call in message.tool_calls:
func_name = tool_call.function.name
self.metrics['tools_called'].append(func_name)
result = f"[Simulated] Tool '{func_name}' executed successfully"
tool_results.append({"tool_call_id": tool_call.id, "output": result})
self.conversation_history.append({
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": tc.id,
"type": "function",
"function": {
"name": tc.function.name,
"arguments": tc.function.arguments
}
}
for tc in message.tool_calls
]
})
for result in tool_results:
self.conversation_history.append({
"role": "tool",
"tool_call_id": result["tool_call_id"],
"content": result["output"]
})
return self._call_llm()
def reset(self):
"""Reset agent state."""
self.conversation_history = []
self.available_tools = []
self.metrics = {
'tokens_used': 0,
'tools_loaded': 0,
'api_calls': 0,
'tools_called': []
}
class PassiveToolAgent:
"""
Traditional agent with all tools injected upfront (for comparison).
This approach:
1. Injects all tool schemas into the initial prompt
2. Massive context overhead
3. Reduces agent to passive tool selector
"""
def __init__(self, servers: Optional[List[ServerDefinition]] = None,
model: Optional[str] = None):
self.client = OpenAI(
api_key=config.OPENAI_API_KEY,
base_url=config.OPENAI_BASE_URL
)
self.model = model or config.OPENAI_MODEL
# Load ALL tools upfront
self.servers = servers if servers is not None else create_tool_knowledge_base()
self.all_tools = []
for server in self.servers:
self.all_tools.extend(server.tools)
self.conversation_history = []
self.metrics = {
'tokens_used': 0,
'tools_loaded': len(self.all_tools),
'api_calls': 0,
'tools_called': []
}
def execute_task(self, task: str) -> Dict[str, Any]:
"""Execute task with all tools pre-loaded."""
self.conversation_history = []
# System message
system_message = f"""You are an AI agent with access to {len(self.all_tools)} tools across multiple domains.
All available tools have been pre-loaded. Analyze the task and use the appropriate tools to complete it."""
self.conversation_history.append({
"role": "system",
"content": system_message
})
self.conversation_history.append({
"role": "user",
"content": task
})
# Call LLM with ALL tools
response = self._call_llm()
self.metrics['api_calls'] += 1
return {
'response': response,
'metrics': self.metrics,
'tools_loaded': [t.name for t in self.all_tools],
'conversation': self.conversation_history
}
def _call_llm(self) -> str:
"""Call LLM with ALL tools injected."""
kwargs = {
"model": self.model,
"messages": self.conversation_history,
"temperature": config.AGENT_TEMPERATURE,
"tools": [tool.to_schema() for tool in self.all_tools],
"tool_choice": "auto"
}
response = self.client.chat.completions.create(**kwargs)
# Track token usage
# response.usage is Optional in the OpenAI SDK: the attribute always
# exists, but is None when the provider omits token accounting.
if getattr(response, 'usage', None):
self.metrics['tokens_used'] += response.usage.total_tokens
message = response.choices[0].message
# Handle tool calls (simulated)
if message.tool_calls:
return self._handle_tool_calls(message)
return message.content or ""
def _handle_tool_calls(self, message) -> str:
"""Handle tool execution (simulated)."""
tool_results = []
for tool_call in message.tool_calls:
func_name = tool_call.function.name
self.metrics['tools_called'].append(func_name)
result = f"[Simulated] Tool '{func_name}' executed successfully"
tool_results.append({
"tool_call_id": tool_call.id,
"output": result
})
# Add to history
self.conversation_history.append({
"role": "assistant",
"content": None,
"tool_calls": [
{
"id": tc.id,
"type": "function",
"function": {
"name": tc.function.name,
"arguments": tc.function.arguments
}
}
for tc in message.tool_calls
]
})
for result in tool_results:
self.conversation_history.append({
"role": "tool",
"tool_call_id": result["tool_call_id"],
"content": result["output"]
})
return self._call_llm()
def reset(self):
"""Reset agent state."""
self.conversation_history = []
self.metrics = {
'tokens_used': 0,
'tools_loaded': len(self.all_tools),
'api_calls': 0,
'tools_called': []
}
chapter4/active-tool-selection/benchmark.py
+203
View File
@@ -0,0 +1,203 @@
"""
Tool-selection benchmark and offline evaluation.
Provides a small labeled benchmark (task -> ground-truth tool) and utilities to
quantify, *without any API calls*, the core claim of the chapter: when a tool
ecosystem grows to hundreds of tools, retrieving the few relevant tools on demand
keeps the right tool reachable while slashing the token cost of dumping every tool
schema into context.
Two things are measured here deterministically:
1. Retrieval recall@k — is the ground-truth tool among the tools a strategy
places in the model's context?
2. Context schema tokens — how many tokens the injected tool schemas cost.
End-to-end accuracy/latency (whether the model actually *calls* the right tool)
requires an API key and lives in demo_comparison.py.
"""
from typing import List, Dict
from tool_knowledge_base import (
ToolDefinition,
ServerDefinition,
create_tool_knowledge_base,
get_all_tools,
calculate_total_tokens,
)
from semantic_router import SemanticRouter
# Labeled benchmark: each task has one (or a few acceptable) ground-truth tool(s).
# Queries are in English to match the English tool descriptions used by the
# TF-IDF router (see tool_knowledge_base.py).
BENCHMARK_TASKS: List[Dict] = [
{
"name": "GitHub repo search",
"task": "Search GitHub for popular Python machine learning repositories with more than 10000 stars",
"gold_tools": ["github_search_repos"],
},
{
"name": "Read config file",
"task": "Read the contents of the local configuration file at /etc/app/config.json",
"gold_tools": ["fs_read_file"],
},
{
"name": "List directory",
"task": "List all files and subdirectories under the /var/log directory",
"gold_tools": ["fs_list_directory"],
},
{
"name": "Summary statistics",
"task": "Calculate the mean, median and standard deviation of last quarter's sales figures",
"gold_tools": ["analytics_summarize"],
},
{
"name": "Send email",
"task": "Send the quarterly performance summary email to the team members",
"gold_tools": ["comm_send_email"],
},
{
"name": "Deploy to production",
"task": "Deploy version 2.3.0 of the application to the production environment",
"gold_tools": ["devops_deploy"],
},
{
"name": "SQL query",
"task": "Run a SQL query on the database to count the number of active users per region",
"gold_tools": ["db_query"],
},
{
"name": "Upload to cloud",
"task": "Upload the local report file to the cloud storage bucket",
"gold_tools": ["cloud_upload_storage"],
},
{
"name": "Scrape prices",
"task": "Scrape the prices of all products listed on the given web page",
"gold_tools": ["web_scrape"],
},
{
"name": "Monitor service",
"task": "Get the current CPU and memory monitoring metrics for the staging service",
"gold_tools": ["devops_monitor"],
},
]
def make_distractor_servers(num_tools: int, start_index: int = 1,
tools_per_server: int = 5) -> List[ServerDefinition]:
"""
Generate synthetic *distractor* servers/tools to inflate the catalog size.
These are deliberately generic "internal service" operations. They add real
schema tokens and act as retrieval noise, so we can study how each strategy
scales as the ecosystem grows to hundreds of tools — without hand-writing
hundreds of realistic tools. They are clearly named ``svcN_opM`` so nobody
mistakes them for the real catalog.
"""
servers: List[ServerDefinition] = []
created = 0
server_idx = start_index
while created < num_tools:
n = min(tools_per_server, num_tools - created)
tools = []
for j in range(1, n + 1):
op = created + j
tools.append(ToolDefinition(
name=f"svc{server_idx}_op{j}",
description=(
f"Auxiliary internal-service operation {op} for background "
f"housekeeping on internal resource group {server_idx}"
),
parameters={
"type": "object",
"properties": {
"resource_id": {"type": "string", "description": "Internal resource identifier"},
"options": {"type": "object", "description": "Operation options"},
},
"required": ["resource_id"],
},
server=f"internal_service_{server_idx}",
))
servers.append(ServerDefinition(
name=f"internal_service_{server_idx}",
description=f"Internal auxiliary service {server_idx} for background housekeeping operations",
tools=tools,
))
created += n
server_idx += 1
return servers
def build_catalog(num_tools: int = 0) -> List[ServerDefinition]:
"""
Build the tool catalog, optionally padded with distractor tools.
Args:
num_tools: Target total number of tools. 0 (default) keeps the real
catalog untouched. Values below the real catalog size are ignored
(we never drop real tools); larger values pad with distractors.
"""
servers = create_tool_knowledge_base()
real_count = len(get_all_tools(servers))
if num_tools and num_tools > real_count:
servers = servers + make_distractor_servers(num_tools - real_count)
return servers
def evaluate_offline(servers: List[ServerDefinition], top_k: int,
tasks: List[Dict] = None) -> Dict:
"""
Deterministically compare tool-selection strategies (no API calls).
Returns a dict with per-strategy aggregate metrics and per-task retrieval
details. Two strategies are directly comparable offline:
* ``all-tools`` — inject every tool schema. Recall is 1.0 by construction
(the gold tool is always present) but token cost grows with the catalog.
* ``retrieval`` — inject only the top-k retrieved tools. Recall is measured;
token cost stays roughly flat as the catalog grows.
(The ``active`` MCP-Zero strategy needs the model in the loop, so it is only
evaluated in the online benchmark.)
"""
tasks = tasks or BENCHMARK_TASKS
router = SemanticRouter(servers)
all_tools = get_all_tools(servers)
all_tools_tokens = calculate_total_tokens(all_tools)
per_task = []
retrieval_hits = 0
retrieval_tokens_sum = 0
for t in tasks:
retrieved = router.retrieve(t["task"], top_k)
retrieved_names = [tool.name for tool in retrieved]
hit = any(g in retrieved_names for g in t["gold_tools"])
retrieval_hits += int(hit)
retrieval_tokens_sum += calculate_total_tokens(retrieved)
per_task.append({
"name": t["name"],
"gold_tools": t["gold_tools"],
"retrieved": retrieved_names,
"hit": hit,
})
n = len(tasks)
return {
"num_tools": len(all_tools),
"top_k": top_k,
"per_task": per_task,
"strategies": {
"all-tools": {
"tools_in_context": len(all_tools),
"avg_schema_tokens": all_tools_tokens,
"recall": 1.0,
},
"retrieval": {
"tools_in_context": top_k,
"avg_schema_tokens": retrieval_tokens_sum / n,
"recall": retrieval_hits / n,
},
},
}
chapter4/active-tool-selection/config.py
+56
View File
@@ -0,0 +1,56 @@
"""Configuration for Active Tool Selection Agent."""
import os
from dotenv import load_dotenv
load_dotenv()
# LLM Configuration
LLM_PROVIDER = os.getenv("LLM_PROVIDER", "openai").lower()
LLM_PROVIDER = {"qwen": "dashscope", "bailian": "dashscope"}.get(LLM_PROVIDER, LLM_PROVIDER)
if LLM_PROVIDER == "dashscope":
OPENAI_API_KEY = os.getenv("DASHSCOPE_API_KEY")
OPENAI_BASE_URL = os.getenv(
"DASHSCOPE_BASE_URL", "https://dashscope.aliyuncs.com/compatible-mode/v1"
)
OPENAI_MODEL = os.getenv("DASHSCOPE_MODEL", "qwen3.7-plus")
else:
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
OPENAI_BASE_URL = os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1")
OPENAI_MODEL = os.getenv("OPENAI_MODEL", "gpt-5.6-luna")
def _map_model_for_openrouter(model: str) -> str:
"""Map a plain model id onto OpenRouter's `provider/model` form.
Ids that already contain "/" pass through unchanged; gpt-*/o1-*/o3-*/o4-*
become openai/…; claude-* becomes anthropic/claude-opus-4.8.
"""
if "/" in model:
return model
m = model.lower()
if m.startswith(("gpt-", "o1-", "o3-", "o4-")):
return f"openai/{model}"
if m.startswith("claude-"):
return "anthropic/claude-opus-4.8"
return model
# Universal fallback + gpt-5.x preference: route through OpenRouter when no direct
# OPENAI_API_KEY is configured, OR when the model is a gpt-5.x id (incl. gpt-5.6*)
# which needs OpenAI org-verification on the direct API. Explicit OPENAI_BASE_URL /
# OPENAI_MODEL overrides are kept.
_OR_KEY = os.getenv("OPENROUTER_API_KEY")
if _OR_KEY and (not OPENAI_API_KEY or OPENAI_MODEL.lower().startswith("gpt-5")):
OPENAI_API_KEY = _OR_KEY
if not os.getenv("OPENAI_BASE_URL"):
OPENAI_BASE_URL = "https://openrouter.ai/api/v1"
OPENAI_MODEL = _map_model_for_openrouter(OPENAI_MODEL)
# Agent Configuration
AGENT_TEMPERATURE = 0.7
MAX_TOOL_REQUESTS = 5 # Maximum number of tool discovery iterations
# Semantic Routing Configuration
SIMILARITY_THRESHOLD = 0.15 # Minimum similarity score for tool matching
TOP_K_SERVERS = 3 # Number of top servers to search
TOP_K_TOOLS = 5 # Number of top tools to return per server
chapter4/active-tool-selection/demo_comparison.py
+535
View File
@@ -0,0 +1,535 @@
"""
Comparison Demo: Active vs Passive Tool Selection.
Demonstrates the efficiency gains of active tool discovery compared to
traditional passive tool injection approach.
"""
import argparse
import json
import time
from tabulate import tabulate
from agent import ActiveToolAgent, PassiveToolAgent, RetrievalToolAgent
from tool_knowledge_base import create_tool_knowledge_base, calculate_total_tokens
import benchmark
import config
def print_section(title: str):
"""Print a formatted section header."""
print("\n" + "=" * 80)
print(f" {title}")
print("=" * 80 + "\n")
def run_comparison_demo():
"""Run side-by-side comparison of active vs passive approaches."""
print_section("Active Tool Discovery vs Passive Tool Injection Comparison")
# Show knowledge base statistics
servers = create_tool_knowledge_base()
all_tools = []
for server in servers:
all_tools.extend(server.tools)
total_tokens = calculate_total_tokens(all_tools)
print("📊 Tool Knowledge Base Statistics:")
print(f" • Total Servers: {len(servers)}")
print(f" • Total Tools: {len(all_tools)}")
print(f" • Estimated tokens for all tool schemas: ~{total_tokens:,}")
print()
# Test tasks
test_tasks = [
{
"name": "GitHub Repository Search",
"task": "Find popular Python machine learning repositories on GitHub with more than 10k stars"
},
{
"name": "File System Operation",
"task": "Read the configuration file at /etc/app/config.json and list all API keys"
},
{
"name": "Data Analytics",
"task": "Calculate summary statistics (mean, median, std) for the sales data in the last quarter"
},
{
"name": "Multi-Domain Task",
"task": "Clone the repository, analyze the code files, and generate a visualization of code complexity metrics"
}
]
results = []
for test in test_tasks:
print(f"\n🔍 Testing: {test['name']}")
print(f" Task: {test['task']}")
print()
# Test with Active Agent
print(" [Active Agent] Executing...")
active_agent = ActiveToolAgent()
active_result = active_agent.execute_task(test['task'])
# Test with Passive Agent
print(" [Passive Agent] Executing...")
passive_agent = PassiveToolAgent()
passive_result = passive_agent.execute_task(test['task'])
# Calculate efficiency metrics
token_reduction = (1 - active_result['metrics']['tokens_used'] /
passive_result['metrics']['tokens_used']) * 100
tools_loaded_active = active_result['metrics']['tools_loaded']
tools_loaded_passive = passive_result['metrics']['tools_loaded']
results.append({
'Task': test['name'],
'Active Tokens': f"{active_result['metrics']['tokens_used']:,}",
'Passive Tokens': f"{passive_result['metrics']['tokens_used']:,}",
'Token Reduction': f"{token_reduction:.1f}%",
'Active Tools': tools_loaded_active,
'Passive Tools': tools_loaded_passive,
'Tool Reduction': f"{(1 - tools_loaded_active/tools_loaded_passive)*100:.1f}%"
})
print(f" ✓ Active: {active_result['metrics']['tokens_used']:,} tokens, {tools_loaded_active} tools")
print(f" ✓ Passive: {passive_result['metrics']['tokens_used']:,} tokens, {tools_loaded_passive} tools")
print(f" 💡 Reduction: {token_reduction:.1f}% tokens saved")
# Display results table
print_section("Comparison Results")
print(tabulate(results, headers='keys', tablefmt='grid'))
# Calculate averages
avg_token_reduction = sum(
float(r['Token Reduction'].rstrip('%')) for r in results
) / len(results)
print(f"\n📈 Summary:")
print(f" • Average token reduction: {avg_token_reduction:.1f}%")
print(f" • Active approach: Loads only {results[0]['Active Tools']} tools on average")
print(f" • Passive approach: Loads all {results[0]['Passive Tools']} tools upfront")
print()
print("💡 Key Insights:")
print(" • Active tool discovery maintains minimal context footprint")
print(" • Significant token savings (80-98% in typical scenarios)")
print(" • Agent autonomy preserved - discovers tools as needed")
print(" • Scales efficiently as tool ecosystem grows")
def demo_active_discovery_process():
"""Demonstrate the active discovery process in detail."""
print_section("Active Tool Discovery Process Demonstration")
task = "Search for Python repositories on GitHub and analyze their README files"
print(f"📝 Task: {task}\n")
agent = ActiveToolAgent()
result = agent.execute_task(task)
print("🔄 Discovery Process:")
print(f" • Tool requests made: {result['metrics']['tool_requests']}")
print(f" • Tools loaded: {result['metrics']['tools_loaded']}")
print(f" • API calls: {result['metrics']['api_calls']}")
print(f" • Total tokens: {result['metrics']['tokens_used']:,}")
print()
print("🛠️ Tools Discovered:")
for i, tool in enumerate(result['tools_loaded'], 1):
print(f" {i}. {tool}")
print()
print("💬 Conversation Flow:")
for i, msg in enumerate(result['conversation'], 1):
role = msg['role'].upper()
content = msg.get('content', '[Tool Call]')
if content and len(content) > 100:
content = content[:100] + "..."
print(f" {i}. [{role}] {content}")
print()
print("✅ Final Response:")
print(f" {result['response']}")
def demo_semantic_routing():
"""Demonstrate hierarchical semantic routing."""
print_section("Hierarchical Semantic Routing Demonstration")
from semantic_router import SemanticRouter
servers = create_tool_knowledge_base()
router = SemanticRouter(servers)
test_queries = [
"I need to search for repositories on GitHub",
"Read a file from the local filesystem",
"Query the database for user information",
"Send an email notification to the team",
"Deploy the application to production environment"
]
print("🎯 Testing semantic routing for various requests:\n")
for query in test_queries:
print(f"📌 Request: '{query}'")
details = router.get_routing_details(query, top_k_servers=2, top_k_tools=3)
print(" Stage 1 - Server Routing:")
for server in details['stage1_servers']:
print(f" • {server['name']}: {server['score']:.3f}")
print(" Stage 2 - Tool Routing:")
for tool in details['final_tools']:
print(f" • {tool['name']} ({tool['server']}): {tool['score']:.3f}")
print()
def demo_iterative_capability_extension():
"""Demonstrate iterative capability extension."""
print_section("Iterative Capability Extension Demonstration")
print("🎯 Complex Multi-Step Task:")
task = """Perform a comprehensive analysis:
1. Search GitHub for Python data science repositories
2. Download the top repository
3. Analyze the code structure
4. Generate visualization of dependencies
5. Send summary report via email"""
print(f"{task}\n")
agent = ActiveToolAgent()
result = agent.execute_task(task)
print("📊 Capability Extension Timeline:")
print(f" • Initial tools: 0")
print(f" • Tools after request 1: GitHub tools")
print(f" • Tools after request 2: Filesystem + GitHub")
print(f" • Tools after request 3: Analytics + Filesystem + GitHub")
print(f" • Tools after request 4: Communication + Analytics + Filesystem + GitHub")
print()
print(f" Total tool requests: {result['metrics']['tool_requests']}")
print(f" Final toolchain size: {result['metrics']['tools_loaded']} tools")
print()
print("💡 The agent iteratively built a cross-domain toolchain as task understanding evolved!")
def run_offline_benchmark(servers, top_k: int, scaling: bool = True) -> dict:
"""
Deterministic (no-API) strategy comparison: retrieval recall vs token cost.
This is the heart of the experiment and runs without any API key. It shows
that as the tool catalog grows, injecting all tools makes context token cost
explode, while on-demand retrieval keeps cost roughly flat and still surfaces
the right tool (recall).
"""
print_section("Offline Strategy Comparison (deterministic, no API)")
result = benchmark.evaluate_offline(servers, top_k)
strat = result['strategies']
print(f"Benchmark tasks: {len(benchmark.BENCHMARK_TASKS)} "
f"Catalog size: {result['num_tools']} tools Retrieval top-k: {top_k}\n")
rows = [
{
'Strategy': 'all-tools (dump everything)',
'Tools in context': strat['all-tools']['tools_in_context'],
'Schema tokens': f"{strat['all-tools']['avg_schema_tokens']:,}",
'Recall (gold reachable)': f"{strat['all-tools']['recall']*100:.0f}%",
},
{
'Strategy': f'retrieval (top-{top_k})',
'Tools in context': strat['retrieval']['tools_in_context'],
'Schema tokens': f"{strat['retrieval']['avg_schema_tokens']:,.0f}",
'Recall (gold reachable)': f"{strat['retrieval']['recall']*100:.0f}%",
},
]
print(tabulate(rows, headers='keys', tablefmt='grid'))
token_saving = (1 - strat['retrieval']['avg_schema_tokens'] /
strat['all-tools']['avg_schema_tokens']) * 100
print(f"\n=> Retrieval keeps {strat['retrieval']['recall']*100:.0f}% recall while cutting "
f"tool-schema tokens by {token_saving:.1f}% "
f"({strat['all-tools']['avg_schema_tokens']:,} -> "
f"{strat['retrieval']['avg_schema_tokens']:,.0f}).")
# Per-task retrieval detail (which tools were surfaced, and whether the gold hit)
print("\nPer-task retrieval (top-k tools surfaced for each task):")
detail_rows = [
{
'Task': p['name'],
'Gold tool': ', '.join(p['gold_tools']),
'Hit': '✓' if p['hit'] else '✗',
'Retrieved (top-k)': ', '.join(p['retrieved']),
}
for p in result['per_task']
]
print(tabulate(detail_rows, headers='keys', tablefmt='github'))
scaling_result = None
if scaling:
print_section("Scaling: token cost as the catalog grows")
sizes = [size for size in [50, 100, 200, 400] if size >= result['num_tools']]
if not sizes or sizes[0] != result['num_tools']:
sizes = [result['num_tools']] + sizes
scaling_rows = []
scaling_result = []
for size in sizes:
padded = benchmark.build_catalog(size)
r = benchmark.evaluate_offline(padded, top_k)
s = r['strategies']
scaling_rows.append({
'Catalog tools': r['num_tools'],
'all-tools tokens': f"{s['all-tools']['avg_schema_tokens']:,}",
f'retrieval(top-{top_k}) tokens': f"{s['retrieval']['avg_schema_tokens']:,.0f}",
'retrieval recall': f"{s['retrieval']['recall']*100:.0f}%",
})
scaling_result.append({
'num_tools': r['num_tools'],
'all_tools_tokens': s['all-tools']['avg_schema_tokens'],
'retrieval_tokens': s['retrieval']['avg_schema_tokens'],
'retrieval_recall': s['retrieval']['recall'],
})
print(tabulate(scaling_rows, headers='keys', tablefmt='grid'))
print("\n=> all-tools token cost grows with the catalog; retrieval stays roughly flat.")
return {'benchmark': result, 'scaling': scaling_result}
STRATEGY_AGENTS = {
'all': ('all-tools', PassiveToolAgent),
'retrieval': ('retrieval', RetrievalToolAgent),
'active': ('active (MCP-Zero)', ActiveToolAgent),
}
def _build_agent(strategy: str, servers, top_k: int, model: str):
"""Instantiate the agent for a strategy (needs a valid API key)."""
if strategy == 'retrieval':
return RetrievalToolAgent(servers=servers, model=model, top_k=top_k)
_, cls = STRATEGY_AGENTS[strategy]
return cls(servers=servers, model=model)
def run_online_benchmark(servers, strategies, top_k: int, model: str,
tasks=None) -> dict:
"""
End-to-end benchmark (requires API key): does the model actually CALL the
ground-truth tool, at what token cost and latency, under each strategy?
"""
tasks = tasks or benchmark.BENCHMARK_TASKS
print_section("Online End-to-End Benchmark (requires API)")
print(f"Model: {model} Tasks: {len(tasks)} "
f"Catalog: {sum(len(s.tools) for s in servers)} tools "
f"Retrieval top-k: {top_k}\n")
agents = {st: _build_agent(st, servers, top_k, model) for st in strategies}
rows = []
raw = {}
for st in strategies:
label = STRATEGY_AGENTS[st][0]
agent = agents[st]
hits = 0
tokens_sum = 0
latency_sum = 0.0
tools_ctx_sum = 0
per_task = []
print(f"[{label}] running {len(tasks)} tasks...")
for t in tasks:
agent.reset()
start = time.time()
res = agent.execute_task(t['task'])
elapsed = time.time() - start
called = res['metrics'].get('tools_called', [])
hit = any(g in called for g in t['gold_tools'])
hits += int(hit)
tokens_sum += res['metrics']['tokens_used']
latency_sum += elapsed
tools_ctx_sum += res['metrics']['tools_loaded']
per_task.append({
'task': t['name'],
'gold': t['gold_tools'],
'called': called,
'hit': hit,
'tokens': res['metrics']['tokens_used'],
'latency': round(elapsed, 2),
})
n = len(tasks)
rows.append({
'Strategy': label,
'Accuracy (calls gold)': f"{hits/n*100:.0f}%",
'Avg tools in ctx': f"{tools_ctx_sum/n:.1f}",
'Avg tokens': f"{tokens_sum/n:,.0f}",
'Avg latency (s)': f"{latency_sum/n:.2f}",
})
raw[st] = {'accuracy': hits / n, 'per_task': per_task}
print()
print(tabulate(rows, headers='keys', tablefmt='grid'))
return raw
def run_single_query(servers, strategies, query: str, top_k: int, model: str) -> dict:
"""Run a single ad-hoc query through the chosen strategies (requires API)."""
print_section("Single Query")
print(f"Query: {query}\n")
raw = {}
for st in strategies:
label = STRATEGY_AGENTS[st][0]
agent = _build_agent(st, servers, top_k, model)
res = agent.execute_task(query)
print(f"[{label}]")
print(f" tools in context : {res['metrics']['tools_loaded']}")
print(f" tools loaded : {', '.join(res['tools_loaded']) or '(none)'}")
print(f" tools called : {', '.join(res['metrics'].get('tools_called', [])) or '(none)'}")
print(f" tokens used : {res['metrics']['tokens_used']:,}")
print()
raw[st] = {
'tools_loaded': res['tools_loaded'],
'tools_called': res['metrics'].get('tools_called', []),
'tokens_used': res['metrics']['tokens_used'],
}
return raw
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="demo_comparison.py",
formatter_class=argparse.RawDescriptionHelpFormatter,
description=(
"主动工具选择实验：对比\"把全部工具塞进上下文\"与\"按需检索工具\"两类策略。\n"
"在工具数量增长到上百个时，动态检索（retrieval）在保持召回率的同时大幅降低\n"
"上下文 token 成本，并减少模型的选择错误。\n\n"
"策略说明：\n"
" all-tools 一次性注入全部工具（传统被动式基线）\n"
" retrieval 按任务语义检索 top-k 个工具后再注入（工具检索 / RAG 式）\n"
" active MCP-Zero 式主动发现：模型迭代地请求所需工具\n\n"
"离线表格（召回率 / token 成本 / 随工具规模的扩展性）无需 API Key 即可运行；\n"
"端到端准确率与延迟对比需要配置 API Key。"
),
epilog=(
"示例：\n"
" python demo_comparison.py --offline # 仅离线对比，无需 API\n"
" python demo_comparison.py --offline --num-tools 200 # 扩展到 200 个工具再对比\n"
" python demo_comparison.py --strategy compare # 三种策略端到端对比（需 API）\n"
" python demo_comparison.py --query \"部署到生产环境\" --strategy retrieval\n"
" python demo_comparison.py --output results.json # 保存结果为 JSON"
),
)
parser.add_argument(
"--strategy", choices=["all", "retrieval", "active", "compare"],
default="compare",
help="端到端评测使用的策略；compare 表示三种策略全部对比（默认：compare）",
)
parser.add_argument(
"--query", type=str, default=None,
help="只对单条查询运行选定策略（需要 API），而不是跑整个基准集",
)
parser.add_argument(
"--num-tools", type=int, default=0, metavar="N",
help="将工具目录扩充到 N 个（用合成干扰工具补齐，用于观察扩展性）；0 表示保持真实目录（默认：0）",
)
parser.add_argument(
"--top-k", type=int, default=config.TOP_K_TOOLS, metavar="K",
help=f"retrieval 策略检索的工具数量（默认：{config.TOP_K_TOOLS}）",
)
parser.add_argument(
"--model", type=str, default=config.OPENAI_MODEL,
help=f"覆盖使用的 LLM 模型（默认：{config.OPENAI_MODEL}）",
)
parser.add_argument(
"--output", type=str, default=None, metavar="PATH",
help="将结果写入 JSON 文件",
)
parser.add_argument(
"--offline", action="store_true",
help="仅运行离线确定性对比（召回率/token 成本），不进行任何 API 调用",
)
parser.add_argument(
"--legacy-demos", action="store_true",
help="额外运行原有的叙事式演示（语义路由、迭代发现等，需要 API）",
)
return parser
def _has_api_key() -> bool:
return bool(config.OPENAI_API_KEY)
def main(argv=None):
parser = build_parser()
args = parser.parse_args(argv)
print("""
╔════════════════════════════════════════════════════════════════════════════╗
║ Active Tool Selection — Strategy Comparison ║
║ Inspired by MCP-Zero (arXiv:2506.01056) ║
╚════════════════════════════════════════════════════════════════════════════╝
""")
servers = benchmark.build_catalog(args.num_tools)
strategies = ["all", "retrieval", "active"] if args.strategy == "compare" else [args.strategy]
results = {
'config': {
'num_tools': sum(len(s.tools) for s in servers),
'top_k': args.top_k,
'model': args.model,
'strategies': strategies,
}
}
# 1) Offline deterministic comparison — always runs, no API needed.
if not args.query:
results['offline'] = run_offline_benchmark(servers, args.top_k)
# 2) Online end-to-end comparison — needs an API key.
if args.offline:
print("\n[offline mode] 跳过所有需要 API 的评测。")
elif not _has_api_key():
print("\n[提示] 未检测到 OPENAI_API_KEY，跳过端到端评测（准确率/延迟）。")
print(" 配置 .env 后可运行端到端对比；或使用 --offline 显式仅跑离线部分。")
else:
if args.query:
results['single_query'] = run_single_query(
servers, strategies, args.query, args.top_k, args.model)
else:
results['online'] = run_online_benchmark(
servers, strategies, args.top_k, args.model)
if args.legacy_demos:
run_comparison_demo()
demo_active_discovery_process()
demo_semantic_routing()
demo_iterative_capability_extension()
if args.output:
with open(args.output, "w", encoding="utf-8") as f:
json.dump(results, f, ensure_ascii=False, indent=2)
print(f"\n结果已写入 {args.output}")
print_section("Takeaway")
print(
"当工具数量增长到上百个时，把全部工具塞进上下文既浪费 token 又干扰决策；\n"
"按需检索把\"工具选择\"问题转化为\"知识检索\"问题——在保持召回率的同时\n"
"把工具描述的 token 成本压到很低，也减少了模型的选择错误。\n\n"
"参考：MCP-Zero 论文 (https://arxiv.org/pdf/2506.01056)"
)
if __name__ == "__main__":
main()
chapter4/active-tool-selection/env.example
+23
View File
@@ -0,0 +1,23 @@
# Provider: openai (default) or dashscope/qwen/bailian
LLM_PROVIDER=openai
# OpenAI API Configuration
OPENAI_API_KEY=your_api_key_here
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-5.6-luna
# Alibaba Cloud Model Studio / Bailian (Qwen)
# DASHSCOPE_API_KEY=your_dashscope_api_key_here
# DASHSCOPE_MODEL=qwen3.7-plus
# DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
# Or use compatible APIs (Kimi, DeepSeek, etc.)
# OPENAI_BASE_URL=https://api.moonshot.cn/v1
# OPENAI_MODEL=kimi-k3
# Universal OpenRouter fallback:
# If OPENAI_API_KEY is not set but OPENROUTER_API_KEY is, config.py routes
# through OpenRouter automatically (base_url=https://openrouter.ai/api/v1) and
# maps the model id to provider/model form (gpt-* -> openai/…,
# claude-* -> anthropic/claude-opus-4.8, ids with "/" pass through).
# OPENROUTER_API_KEY=your-openrouter-api-key
chapter4/active-tool-selection/examples.py
+212
View File
@@ -0,0 +1,212 @@
"""
Example use cases demonstrating active tool selection.
"""
from agent import ActiveToolAgent
from semantic_router import SemanticRouter
from tool_knowledge_base import create_tool_knowledge_base
def example_github_workflow():
"""Example: GitHub development workflow."""
print("\n" + "=" * 70)
print("Example 1: GitHub Development Workflow")
print("=" * 70 + "\n")
agent = ActiveToolAgent()
task = """I need to:
1. Search for Python testing frameworks on GitHub
2. Find issues labeled 'good-first-issue' in the top repository
3. Create a new branch and make changes
4. Create a pull request"""
print(f"Task:\n{task}\n")
result = agent.execute_task(task)
print(f"\n✅ Tools discovered: {len(result['tools_loaded'])}")
print(f" {', '.join(result['tools_loaded'])}")
print(f"\n📊 Metrics:")
print(f" • Tokens used: {result['metrics']['tokens_used']:,}")
print(f" • Tool requests: {result['metrics']['tool_requests']}")
print(f" • API calls: {result['metrics']['api_calls']}")
def example_data_pipeline():
"""Example: Data processing pipeline."""
print("\n" + "=" * 70)
print("Example 2: Data Processing Pipeline")
print("=" * 70 + "\n")
agent = ActiveToolAgent()
task = """Build a data pipeline:
1. Query the database for last month's sales data
2. Calculate summary statistics
3. Create visualizations (bar charts and trend lines)
4. Upload results to cloud storage
5. Send notification email to stakeholders"""
print(f"Task:\n{task}\n")
result = agent.execute_task(task)
print(f"\n✅ Cross-domain toolchain built:")
for i, tool in enumerate(result['tools_loaded'], 1):
print(f" {i}. {tool}")
print(f"\n📊 Efficiency:")
print(f" • Only {len(result['tools_loaded'])} tools loaded (out of 35 available)")
print(f" • Token savings: ~90% compared to loading all tools")
def example_devops_automation():
"""Example: DevOps automation task."""
print("\n" + "=" * 70)
print("Example 3: DevOps Automation")
print("=" * 70 + "\n")
agent = ActiveToolAgent()
task = """Automate deployment process:
1. Check monitoring metrics for the staging environment
2. If metrics are healthy, trigger production deployment pipeline
3. Monitor deployment progress and logs
4. If any errors occur, automatically rollback
5. Send deployment status notification"""
print(f"Task:\n{task}\n")
result = agent.execute_task(task)
print(f"\n✅ DevOps toolchain assembled:")
print(f" Tools: {', '.join(result['tools_loaded'])}")
print(f"\n💡 Active discovery enabled iterative refinement:")
print(f" • Started with monitoring tools")
print(f" • Added deployment tools when needed")
print(f" • Included notification tools at the end")
def example_semantic_search():
"""Example: Demonstrate semantic search capabilities."""
print("\n" + "=" * 70)
print("Example 4: Semantic Tool Search")
print("=" * 70 + "\n")
servers = create_tool_knowledge_base()
router = SemanticRouter(servers)
queries = [
"I need to version control my code",
"Store and retrieve structured data",
"Make HTTP requests to APIs",
"Analyze datasets and create graphs",
"Configure cloud infrastructure"
]
print("Testing semantic understanding of tool requests:\n")
for query in queries:
print(f"🔍 Query: '{query}'")
tools = router.route_request(query, top_k_servers=1, top_k_tools=3)
if tools:
print(f" ✓ Found: {', '.join([t.name for t in tools])}")
else:
print(f" ✗ No matching tools found")
print()
def example_multi_turn_discovery():
"""Example: Multi-turn conversation with progressive tool discovery."""
print("\n" + "=" * 70)
print("Example 5: Multi-Turn Progressive Discovery")
print("=" * 70 + "\n")
print("Scenario: Agent progressively discovers tools across multiple turns\n")
agent = ActiveToolAgent()
# Turn 1: Initial request
print("👤 User: Search for machine learning repositories")
result1 = agent.execute_task("Search for machine learning repositories")
print(f"🤖 Agent loaded: {', '.join(result1['tools_loaded'][:2])}")
print()
# Turn 2: Additional requirements emerge
print("👤 User: Now download the README files and analyze them")
result2 = agent.execute_task("Download README files and analyze them")
print(f"🤖 Agent additionally loaded: filesystem and analytics tools")
print()
# Turn 3: Visualization needed
print("👤 User: Create a visualization comparing repository sizes")
result3 = agent.execute_task("Create a visualization comparing repository sizes")
print(f"🤖 Agent additionally loaded: visualization tools")
print()
print("💡 Tools were discovered on-demand as the conversation evolved!")
print(" This demonstrates the iterative capability extension principle.")
def example_efficiency_comparison():
"""Example: Show efficiency comparison with metrics."""
print("\n" + "=" * 70)
print("Example 6: Efficiency Comparison")
print("=" * 70 + "\n")
from agent import PassiveToolAgent
task = "List files in the current directory"
print(f"Task: {task}\n")
# Active approach
print("🔄 Active Tool Discovery:")
active_agent = ActiveToolAgent()
active_result = active_agent.execute_task(task)
print(f" • Tools loaded: {active_result['metrics']['tools_loaded']}")
print(f" • Tokens used: {active_result['metrics']['tokens_used']:,}")
print()
# Passive approach
print("📚 Passive Tool Injection:")
passive_agent = PassiveToolAgent()
passive_result = passive_agent.execute_task(task)
print(f" • Tools loaded: {passive_result['metrics']['tools_loaded']}")
print(f" • Tokens used: {passive_result['metrics']['tokens_used']:,}")
print()
# Comparison
reduction = (1 - active_result['metrics']['tokens_used'] /
passive_result['metrics']['tokens_used']) * 100
print(f"📊 Efficiency Gain:")
print(f" • Token reduction: {reduction:.1f}%")
print(f" • Tool reduction: {active_result['metrics']['tools_loaded']} vs {passive_result['metrics']['tools_loaded']}")
print()
print("💡 For simple tasks requiring 1-2 tools, active discovery achieves")
print(" massive efficiency gains while maintaining full capability!")
if __name__ == "__main__":
print("""
╔════════════════════════════════════════════════════════════════════════════╗
║ ║
║ Active Tool Selection Examples ║
║ ║
╚════════════════════════════════════════════════════════════════════════════╝
""")
# Run all examples
example_github_workflow()
example_data_pipeline()
example_devops_automation()
example_semantic_search()
example_multi_turn_discovery()
example_efficiency_comparison()
print("\n" + "=" * 70)
print("All examples completed!")
print("=" * 70 + "\n")
chapter4/active-tool-selection/quickstart.py
+133
View File
@@ -0,0 +1,133 @@
"""
Quick Start for Active Tool Selection.
Run this script to see a basic demonstration of active tool discovery.
"""
from agent import ActiveToolAgent, PassiveToolAgent
from tool_knowledge_base import create_tool_knowledge_base, calculate_total_tokens
def main():
print("""
╔════════════════════════════════════════════════════════════════════════════╗
║ ║
║ Active Tool Selection - Quick Start ║
║ Inspired by MCP-Zero (arXiv:2506.01056) ║
║ ║
╚════════════════════════════════════════════════════════════════════════════╝
This demonstration shows how active tool discovery enables agents to:
• Maintain minimal context footprint
• Actively request tools as needed
• Scale efficiently with ecosystem growth
""")
# Show knowledge base info
print("📚 Tool Knowledge Base:")
servers = create_tool_knowledge_base()
total_tools = sum(len(server.tools) for server in servers)
total_tokens = calculate_total_tokens([tool for server in servers for tool in server.tools])
print(f" • Servers: {len(servers)}")
print(f" • Total tools: {total_tools}")
print(f" • Token cost if all injected: ~{total_tokens:,} tokens")
print()
# Example task
task = "Search for Python web frameworks on GitHub with more than 5000 stars"
print(f"🎯 Example Task:\n {task}\n")
# Test with active agent
print("=" * 80)
print("1️⃣ ACTIVE TOOL DISCOVERY")
print("=" * 80)
print("\n⏳ Agent is analyzing task and discovering needed tools...\n")
active_agent = ActiveToolAgent()
active_result = active_agent.execute_task(task)
print(f"✅ Task completed with active discovery:\n")
print(f" 📊 Metrics:")
print(f" • Tools loaded: {active_result['metrics']['tools_loaded']} (out of {total_tools})")
print(f" • Tokens used: {active_result['metrics']['tokens_used']:,}")
print(f" • Tool requests: {active_result['metrics']['tool_requests']}")
print(f" • API calls: {active_result['metrics']['api_calls']}")
print()
print(f" 🛠️ Tools discovered:")
for tool in active_result['tools_loaded']:
print(f" • {tool}")
print()
# Test with passive agent
print("=" * 80)
print("2️⃣ PASSIVE TOOL INJECTION (Traditional Approach)")
print("=" * 80)
print(f"\n⏳ Agent has all {total_tools} tools pre-loaded...\n")
passive_agent = PassiveToolAgent()
passive_result = passive_agent.execute_task(task)
print(f"✅ Task completed with passive injection:\n")
print(f" 📊 Metrics:")
print(f" • Tools loaded: {passive_result['metrics']['tools_loaded']} (all tools)")
print(f" • Tokens used: {passive_result['metrics']['tokens_used']:,}")
print(f" • API calls: {passive_result['metrics']['api_calls']}")
print()
# Comparison
print("=" * 80)
print("3️⃣ COMPARISON")
print("=" * 80)
print()
token_reduction = (1 - active_result['metrics']['tokens_used'] /
passive_result['metrics']['tokens_used']) * 100
tool_reduction = (1 - active_result['metrics']['tools_loaded'] /
passive_result['metrics']['tools_loaded']) * 100
print(f"📊 Efficiency Gains:\n")
print(f" Token Usage:")
print(f" • Active: {active_result['metrics']['tokens_used']:,} tokens")
print(f" • Passive: {passive_result['metrics']['tokens_used']:,} tokens")
print(f" • Reduction: {token_reduction:.1f}% 🎉")
print()
print(f" Tools Loaded:")
print(f" • Active: {active_result['metrics']['tools_loaded']} tools")
print(f" • Passive: {passive_result['metrics']['tools_loaded']} tools")
print(f" • Reduction: {tool_reduction:.1f}% 🎯")
print()
print("=" * 80)
print("💡 KEY INSIGHTS")
print("=" * 80)
print("""
1. Active Discovery maintains agent autonomy
→ Agent decides what tools it needs, when it needs them
2. Massive efficiency gains
→ 80-98% token reduction for typical tasks
3. Scales with ecosystem growth
→ Adding 100 more tools doesn't bloat every request
4. Iterative capability extension
→ Toolchain evolves as task understanding deepens
5. Semantic routing enables precision
→ Tools matched by meaning, not just keywords
""")
print("🎓 Next Steps:")
print(" • Run 'python demo_comparison.py' for comprehensive comparison")
print(" • Run 'python examples.py' for more use cases")
print(" • See README.md for architecture details")
print()
print("📄 Reference: MCP-Zero paper - https://arxiv.org/pdf/2506.01056")
print()
if __name__ == "__main__":
main()
chapter4/active-tool-selection/requirements.txt
+6
View File
@@ -0,0 +1,6 @@
openai>=1.0.0
python-dotenv>=1.0.0
numpy>=1.24.0
scikit-learn>=1.3.0
requests>=2.31.0
tabulate>=0.9.0
chapter4/active-tool-selection/semantic_router.py
+292
View File
@@ -0,0 +1,292 @@
"""
Hierarchical Semantic Routing for Tool Discovery.
Implements a two-stage algorithm for matching tool requests to relevant tools:
1. Server-level routing: Filter candidate servers by domain/platform
2. Tool-level routing: Rank tools within selected servers by semantic similarity
This approach reduces search complexity while maintaining precision, inspired by MCP-Zero.
"""
from typing import List, Dict, Tuple
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from tool_knowledge_base import ServerDefinition, ToolDefinition
import config
class SemanticRouter:
"""Hierarchical semantic routing for tool discovery."""
def __init__(self, servers: List[ServerDefinition]):
self.servers = servers
self.server_vectorizer = TfidfVectorizer(stop_words='english')
self.tool_vectorizers: Dict[str, TfidfVectorizer] = {}
# Precompute server embeddings
self._build_server_index()
# Precompute tool embeddings for each server
self._build_tool_indices()
def _build_server_index(self):
"""Build TF-IDF index for servers."""
if not self.servers:
self.server_embeddings = None
return
server_descriptions = [f"{s.name} {s.description}" for s in self.servers]
try:
self.server_embeddings = self.server_vectorizer.fit_transform(server_descriptions)
except ValueError:
self.server_embeddings = None
def _build_tool_indices(self):
"""Build TF-IDF indices for tools within each server."""
for server in self.servers:
if not server.tools:
continue
tool_descriptions = [
f"{tool.name} {tool.description}"
for tool in server.tools
]
vectorizer = TfidfVectorizer(stop_words='english')
try:
embeddings = vectorizer.fit_transform(tool_descriptions)
except ValueError:
embeddings = None
self.tool_vectorizers[server.name] = vectorizer
# Store embeddings on server for later use
server._tool_embeddings = embeddings
def route_request(self, tool_request: str, top_k_servers: int = None,
top_k_tools: int = None) -> List[ToolDefinition]:
"""
Route a tool request to relevant tools using hierarchical semantic matching.
Args:
tool_request: Natural language description of needed tool
top_k_servers: Number of top servers to search (default from config)
top_k_tools: Number of tools to return per server (default from config)
Returns:
List of relevant tools ranked by relevance
"""
if top_k_servers is None:
top_k_servers = config.TOP_K_SERVERS
if top_k_tools is None:
top_k_tools = config.TOP_K_TOOLS
# Stage 1: Server-level routing
relevant_servers = self._route_to_servers(tool_request, top_k_servers)
# Stage 2: Tool-level routing within selected servers
relevant_tools = []
for server, server_score in relevant_servers:
tools_with_scores = self._route_to_tools(server, tool_request, top_k_tools)
# Combine server and tool scores
for tool, tool_score in tools_with_scores:
combined_score = 0.3 * server_score + 0.7 * tool_score
relevant_tools.append((tool, combined_score))
# Sort by combined score and filter by threshold
relevant_tools.sort(key=lambda x: x[1], reverse=True)
relevant_tools = [
(tool, score) for tool, score in relevant_tools
if score >= config.SIMILARITY_THRESHOLD
]
# Return top tools
return [tool for tool, _ in relevant_tools[:top_k_tools * top_k_servers]]
def retrieve(self, query: str, top_k: int) -> List[ToolDefinition]:
"""
Flat top-k tool retrieval across ALL servers (single-shot RAG-style routing).
Unlike ``route_request`` (which first narrows to a few candidate servers),
this scores every tool in every server and returns the global top-k. It is the
most direct embodiment of "turn tool selection into knowledge retrieval": given
the task description, fetch only the handful of tools most likely to be relevant.
Args:
query: Natural language task/request description
top_k: Number of tools to return
Returns:
Up to ``top_k`` tools ranked by combined (server + tool) similarity.
"""
# Score against every server so no candidate tool is filtered out prematurely.
relevant_servers = self._route_to_servers(query, len(self.servers))
scored_tools = []
for server, server_score in relevant_servers:
for tool, tool_score in self._route_to_tools(server, query, len(server.tools)):
combined_score = 0.3 * server_score + 0.7 * tool_score
scored_tools.append((tool, combined_score))
scored_tools.sort(key=lambda x: x[1], reverse=True)
return [tool for tool, _ in scored_tools[:top_k]]
def _route_to_servers(self, request: str, top_k: int) -> List[Tuple[ServerDefinition, float]]:
"""
Stage 1: Route request to top-k relevant servers.
Args:
request: Tool request description
top_k: Number of top servers to return
Returns:
List of (server, similarity_score) tuples
"""
if not self.servers:
return []
if self.server_embeddings is None:
return [(server, 0.0) for server in self.servers[:top_k]]
# Vectorize the request
request_vector = self.server_vectorizer.transform([request])
# Calculate similarities with all servers
similarities = cosine_similarity(request_vector, self.server_embeddings)[0]
# Get top-k servers
top_indices = np.argsort(similarities)[::-1][:top_k]
return [(self.servers[idx], similarities[idx]) for idx in top_indices]
def _route_to_tools(self, server: ServerDefinition, request: str,
top_k: int) -> List[Tuple[ToolDefinition, float]]:
"""
Stage 2: Route request to top-k relevant tools within a server.
Args:
server: Server to search within
request: Tool request description
top_k: Number of top tools to return
Returns:
List of (tool, similarity_score) tuples
"""
if server.name not in self.tool_vectorizers or getattr(server, "_tool_embeddings", None) is None:
return []
vectorizer = self.tool_vectorizers[server.name]
tool_embeddings = server._tool_embeddings
if tool_embeddings is None:
return []
# Vectorize the request
request_vector = vectorizer.transform([request])
if request_vector.getnnz() == 0:
return []
# Calculate similarities with all tools in this server
similarities = cosine_similarity(request_vector, tool_embeddings)[0]
# Get top-k tools
top_indices = np.argsort(similarities)[::-1][:top_k]
return [(server.tools[idx], similarities[idx]) for idx in top_indices]
def get_routing_details(self, tool_request: str, top_k_servers: int = None,
top_k_tools: int = None) -> Dict:
"""
Get detailed routing information for debugging/visualization.
Returns a dictionary with:
- request: Original request
- stage1_servers: List of servers with scores
- stage2_tools: List of tools with scores per server
- final_tools: Final ranked list of tools
"""
if top_k_servers is None:
top_k_servers = config.TOP_K_SERVERS
if top_k_tools is None:
top_k_tools = config.TOP_K_TOOLS
# Stage 1: Server routing
relevant_servers = self._route_to_servers(tool_request, top_k_servers)
# Stage 2: Tool routing
stage2_results = {}
all_tools = []
for server, server_score in relevant_servers:
tools_with_scores = self._route_to_tools(server, tool_request, top_k_tools)
stage2_results[server.name] = {
'server_score': server_score,
'tools': [(tool.name, tool_score) for tool, tool_score in tools_with_scores]
}
# Calculate combined scores
for tool, tool_score in tools_with_scores:
combined_score = 0.3 * server_score + 0.7 * tool_score
all_tools.append((tool, combined_score, server.name))
# Sort and filter
all_tools.sort(key=lambda x: x[1], reverse=True)
final_tools = [
{'name': tool.name, 'server': server, 'score': score}
for tool, score, server in all_tools[:top_k_tools * top_k_servers]
if score >= config.SIMILARITY_THRESHOLD
]
return {
'request': tool_request,
'stage1_servers': [
{'name': s.name, 'score': score}
for s, score in relevant_servers
],
'stage2_tools': stage2_results,
'final_tools': final_tools
}
class StructuredRequestParser:
"""
Parse structured tool requests from LLM.
MCP-Zero uses structured requests in format:
<tool_request>
server: [platform/domain description]
tool: [operation description]
</tool_request>
"""
@staticmethod
def parse_request(text: str) -> Dict[str, str]:
"""
Parse structured tool request from text.
Returns dict with 'server' and 'tool' fields, or None if not found.
"""
if '<tool_request>' not in text:
return None
start = text.find('<tool_request>')
end = text.find('</tool_request>', start + len('<tool_request>'))
if end == -1:
return None
request_text = text[start + len('<tool_request>'):end].strip()
result = {}
for line in request_text.split('\n'):
line = line.strip()
if line.startswith('server:'):
result['server'] = line[7:].strip()
elif line.startswith('tool:'):
result['tool'] = line[5:].strip()
return result if 'server' in result and 'tool' in result else None
@staticmethod
def format_request(server_desc: str, tool_desc: str) -> str:
"""Format a structured tool request."""
return f"""<tool_request>
server: {server_desc}
tool: {tool_desc}
</tool_request>"""
chapter4/active-tool-selection/tests/conftest.py
+9
View File
@@ -0,0 +1,9 @@
"""Test import bootstrap for the active-tool-selection experiment."""
from pathlib import Path
import sys
EXPERIMENT_ROOT = Path(__file__).resolve().parents[1]
if str(EXPERIMENT_ROOT) not in sys.path:
sys.path.insert(0, str(EXPERIMENT_ROOT))
chapter4/active-tool-selection/tests/test_basic.py
+219
View File
@@ -0,0 +1,219 @@
"""
Basic tests to verify the active tool selection system works correctly.
Run without API key to test core functionality.
"""
from tool_knowledge_base import create_tool_knowledge_base, calculate_total_tokens, get_all_tools
from semantic_router import SemanticRouter, StructuredRequestParser
def test_knowledge_base():
"""Test that knowledge base loads correctly."""
print("Testing Knowledge Base...")
servers = create_tool_knowledge_base()
all_tools = get_all_tools(servers)
assert len(servers) == 8, f"Expected 8 servers, got {len(servers)}"
assert len(all_tools) > 30, f"Expected 30+ tools, got {len(all_tools)}"
# Check each server has tools
for server in servers:
assert len(server.tools) > 0, f"Server {server.name} has no tools"
assert server.description, f"Server {server.name} missing description"
# Check tool schemas
for tool in all_tools:
schema = tool.to_schema()
assert 'type' in schema, f"Tool {tool.name} missing type"
assert 'function' in schema, f"Tool {tool.name} missing function"
total_tokens = calculate_total_tokens(all_tools)
print(f" ✓ Loaded {len(servers)} servers with {len(all_tools)} tools")
print(f" ✓ Estimated tokens: {total_tokens:,}")
print()
def test_semantic_router():
"""Test semantic routing functionality."""
print("Testing Semantic Router...")
servers = create_tool_knowledge_base()
router = SemanticRouter(servers)
# Test server routing with realistic queries
test_queries = [
("search for GitHub repositories", ["github"]),
("read a file from filesystem", ["filesystem"]),
("query database for users", ["database"]),
("send an email notification", ["communication"]),
("deploy to production environment", ["devops"])
]
for query, expected_servers in test_queries:
tools = router.route_request(query, top_k_servers=3, top_k_tools=3)
assert len(tools) > 0, f"No tools found for query: {query}"
# Check that tools are from expected servers
tool_servers = {tool.server for tool in tools}
assert any(exp in tool_servers for exp in expected_servers), \
f"Expected servers {expected_servers}, got {tool_servers} for query: {query}"
print(f" ✓ Semantic routing working correctly")
print(f" ✓ All test queries matched appropriate servers")
print()
def test_structured_request_parser():
"""Test structured request parsing."""
print("Testing Structured Request Parser...")
# Valid request
valid_request = """
Some text before
<tool_request>
server: GitHub for repository operations
tool: search repositories by keywords
</tool_request>
Some text after
"""
parsed = StructuredRequestParser.parse_request(valid_request)
assert parsed is not None, "Failed to parse valid request"
assert 'server' in parsed, "Missing server in parsed request"
assert 'tool' in parsed, "Missing tool in parsed request"
assert "GitHub" in parsed['server'], "Server description incorrect"
assert "search" in parsed['tool'], "Tool description incorrect"
# Invalid request (missing tags)
invalid_request = "Just some text without proper tags"
parsed_invalid = StructuredRequestParser.parse_request(invalid_request)
assert parsed_invalid is None, "Should return None for invalid request"
# Test formatting
formatted = StructuredRequestParser.format_request(
"GitHub operations",
"search repositories"
)
assert "<tool_request>" in formatted, "Missing opening tag"
assert "</tool_request>" in formatted, "Missing closing tag"
assert "server:" in formatted, "Missing server field"
assert "tool:" in formatted, "Missing tool field"
print(f" ✓ Request parsing working correctly")
print(f" ✓ Request formatting working correctly")
print()
def test_routing_details():
"""Test detailed routing information."""
print("Testing Routing Details...")
servers = create_tool_knowledge_base()
router = SemanticRouter(servers)
query = "I need to search for Python repositories on GitHub"
details = router.get_routing_details(query, top_k_servers=3, top_k_tools=3)
assert 'request' in details, "Missing request in details"
assert 'stage1_servers' in details, "Missing stage1 in details"
assert 'stage2_tools' in details, "Missing stage2 in details"
assert 'final_tools' in details, "Missing final_tools in details"
assert len(details['stage1_servers']) > 0, "No servers in stage1"
assert len(details['final_tools']) > 0, "No final tools"
# Check structure
for server in details['stage1_servers']:
assert 'name' in server, "Server missing name"
assert 'score' in server, "Server missing score"
assert 0 <= server['score'] <= 1, "Server score out of range"
for tool in details['final_tools']:
assert 'name' in tool, "Tool missing name"
assert 'server' in tool, "Tool missing server"
assert 'score' in tool, "Tool missing score"
print(f" ✓ Routing details structure correct")
print(f" ✓ Stage 1: {len(details['stage1_servers'])} servers")
print(f" ✓ Stage 2: {len(details['final_tools'])} final tools")
print()
def test_tool_schemas():
"""Test that tool schemas are properly formatted for OpenAI."""
print("Testing Tool Schemas...")
servers = create_tool_knowledge_base()
all_tools = get_all_tools(servers)
for tool in all_tools:
schema = tool.to_schema()
# Check OpenAI function calling format
assert schema['type'] == 'function', f"Tool {tool.name} has wrong type"
assert 'function' in schema, f"Tool {tool.name} missing function"
func = schema['function']
assert 'name' in func, f"Tool {tool.name} missing name"
assert 'description' in func, f"Tool {tool.name} missing description"
assert 'parameters' in func, f"Tool {tool.name} missing parameters"
params = func['parameters']
assert params['type'] == 'object', f"Tool {tool.name} parameters not object type"
assert 'properties' in params, f"Tool {tool.name} missing properties"
print(f" ✓ All {len(all_tools)} tool schemas properly formatted")
print(f" ✓ Compatible with OpenAI function calling")
print()
def run_all_tests():
"""Run all basic tests."""
print("""
╔════════════════════════════════════════════════════════════════════════════╗
║ ║
║ Active Tool Selection - Basic Tests ║
║ ║
╚════════════════════════════════════════════════════════════════════════════╝
""")
try:
test_knowledge_base()
test_semantic_router()
test_structured_request_parser()
test_routing_details()
test_tool_schemas()
print("=" * 80)
print("✅ ALL TESTS PASSED")
print("=" * 80)
print()
print("The active tool selection system is working correctly!")
print()
print("Next steps:")
print(" 1. Configure your API key in .env")
print(" 2. Run 'python quickstart.py' for a demonstration")
print(" 3. Run 'python demo_comparison.py' for comprehensive comparison")
print()
except AssertionError as e:
print()
print("=" * 80)
print("❌ TEST FAILED")
print("=" * 80)
print(f"Error: {e}")
print()
raise
except Exception as e:
print()
print("=" * 80)
print("❌ UNEXPECTED ERROR")
print("=" * 80)
print(f"Error: {e}")
print()
raise
if __name__ == "__main__":
run_all_tests()
chapter4/active-tool-selection/tests/test_parse_request_closing_tag.py
+19
View File
@@ -0,0 +1,19 @@
from semantic_router import StructuredRequestParser
def test_parse_request_preceding_closing_tag_mention():
text = (
"Note: Do not format as </tool_request> without an opening tag.\n\n"
"<tool_request>\n"
"server: GitHub for repository operations\n"
"tool: search repositories by keywords\n"
"</tool_request>\n"
)
parsed = StructuredRequestParser.parse_request(text)
assert parsed is not None, "Failed to parse tool request when </tool_request> is mentioned in preceding text"
assert parsed["server"] == "GitHub for repository operations"
assert parsed["tool"] == "search repositories by keywords"
if __name__ == "__main__":
test_parse_request_preceding_closing_tag_mention()
chapter4/active-tool-selection/tests/test_usage_none.py
+52
View File
@@ -0,0 +1,52 @@
"""Regression test: agent must tolerate providers that return usage=None.
The OpenAI SDK response object always HAS a `usage` attribute (pydantic
field), but it deserializes as None when the provider omits token accounting.
The old `hasattr(response, 'usage')` guard was therefore ineffective and
`response.usage.total_tokens` raised AttributeError, crashing execute_task.
"""
import os
import sys
from types import SimpleNamespace
os.environ.setdefault("OPENAI_API_KEY", "test-key") # OpenAI() requires a key at construction
sys.path.insert(0, os.path.dirname(__file__))
from agent import ActiveToolAgent, RetrievalToolAgent, PassiveToolAgent
from tool_knowledge_base import ToolDefinition, ServerDefinition
AGENT_CLASSES = [ActiveToolAgent, RetrievalToolAgent, PassiveToolAgent]
def _catalog():
tool = ToolDefinition(
name="demo_tool",
description="demo tool",
parameters={"type": "object", "properties": {}},
server="demo",
)
return [ServerDefinition(name="demo", description="demo server", tools=[tool])]
def _client_with_usage(usage):
"""Fake OpenAI client; response mimics the SDK object (usage attr always present)."""
message = SimpleNamespace(content="final answer", tool_calls=None)
response = SimpleNamespace(choices=[SimpleNamespace(message=message)], usage=usage)
completions = SimpleNamespace(create=lambda **kwargs: response)
return SimpleNamespace(chat=SimpleNamespace(completions=completions))
def test_usage_none_does_not_crash():
for cls in AGENT_CLASSES:
agent = cls(servers=_catalog())
agent.client = _client_with_usage(None)
result = agent.execute_task("do something trivial")
assert result["metrics"]["tokens_used"] == 0, cls.__name__
def test_usage_still_accumulated_when_present():
for cls in AGENT_CLASSES:
agent = cls(servers=_catalog())
agent.client = _client_with_usage(SimpleNamespace(total_tokens=42))
result = agent.execute_task("do something trivial")
assert result["metrics"]["tokens_used"] == 42, cls.__name__
chapter4/active-tool-selection/tool_knowledge_base.py
+624
View File
@@ -0,0 +1,624 @@
"""
Tool Knowledge Base - Simulates MCP servers with various tools.
This module defines a comprehensive knowledge base of tools organized by domains (servers),
similar to the MCP (Model Context Protocol) ecosystem. Each server represents a platform
or service domain with specific tools.
"""
from typing import List, Dict, Any
class ToolDefinition:
"""Represents a single tool with its metadata."""
def __init__(self, name: str, description: str, parameters: Dict[str, Any], server: str):
self.name = name
self.description = description
self.parameters = parameters
self.server = server
def to_schema(self) -> Dict[str, Any]:
"""Convert to OpenAI function schema format."""
return {
"type": "function",
"function": {
"name": self.name,
"description": self.description,
"parameters": self.parameters
}
}
def __repr__(self):
return f"Tool(name={self.name}, server={self.server})"
class ServerDefinition:
"""Represents a server (domain) containing multiple tools."""
def __init__(self, name: str, description: str, tools: List[ToolDefinition]):
self.name = name
self.description = description
self.tools = tools
def __repr__(self):
return f"Server(name={self.name}, tools={len(self.tools)})"
# Define comprehensive tool knowledge base
def create_tool_knowledge_base() -> List[ServerDefinition]:
"""
Create a comprehensive tool knowledge base organized by servers.
This simulates the MCP ecosystem with multiple domains:
- GitHub: Repository management and code operations
- Filesystem: File system operations
- Database: Data storage and retrieval
- Web: HTTP requests and web scraping
- Analytics: Data analysis and visualization
- Communication: Email and messaging
- DevOps: Deployment and monitoring
- Cloud: Cloud service operations
"""
servers = []
# GitHub Server
github_tools = [
ToolDefinition(
name="github_search_repos",
description="Search for GitHub repositories using keywords, filters, and sorting options",
parameters={
"type": "object",
"properties": {
"query": {"type": "string", "description": "Search query (e.g., 'language:python stars:>1000')"},
"sort": {"type": "string", "enum": ["stars", "forks", "updated"], "description": "Sort by field"},
"per_page": {"type": "integer", "description": "Results per page (max 100)"}
},
"required": ["query"]
},
server="github"
),
ToolDefinition(
name="github_create_pr",
description="Create a pull request in a GitHub repository",
parameters={
"type": "object",
"properties": {
"repo": {"type": "string", "description": "Repository name (owner/repo)"},
"title": {"type": "string", "description": "PR title"},
"body": {"type": "string", "description": "PR description"},
"head": {"type": "string", "description": "Branch to merge from"},
"base": {"type": "string", "description": "Branch to merge into"}
},
"required": ["repo", "title", "head", "base"]
},
server="github"
),
ToolDefinition(
name="github_list_issues",
description="List issues in a GitHub repository with filtering options",
parameters={
"type": "object",
"properties": {
"repo": {"type": "string", "description": "Repository name (owner/repo)"},
"state": {"type": "string", "enum": ["open", "closed", "all"], "description": "Issue state"},
"labels": {"type": "array", "items": {"type": "string"}, "description": "Filter by labels"}
},
"required": ["repo"]
},
server="github"
),
ToolDefinition(
name="github_get_file",
description="Get contents of a file from a GitHub repository",
parameters={
"type": "object",
"properties": {
"repo": {"type": "string", "description": "Repository name (owner/repo)"},
"path": {"type": "string", "description": "File path in repository"},
"branch": {"type": "string", "description": "Branch name (default: main)"}
},
"required": ["repo", "path"]
},
server="github"
),
ToolDefinition(
name="github_create_issue",
description="Create a new issue in a GitHub repository",
parameters={
"type": "object",
"properties": {
"repo": {"type": "string", "description": "Repository name (owner/repo)"},
"title": {"type": "string", "description": "Issue title"},
"body": {"type": "string", "description": "Issue description"},
"labels": {"type": "array", "items": {"type": "string"}, "description": "Issue labels"}
},
"required": ["repo", "title"]
},
server="github"
)
]
servers.append(ServerDefinition("github", "GitHub repository management and version control operations", github_tools))
# Filesystem Server
filesystem_tools = [
ToolDefinition(
name="fs_read_file",
description="Read the contents of a file from the local filesystem",
parameters={
"type": "object",
"properties": {
"path": {"type": "string", "description": "File path to read"},
"encoding": {"type": "string", "description": "File encoding (default: utf-8)"}
},
"required": ["path"]
},
server="filesystem"
),
ToolDefinition(
name="fs_write_file",
description="Write content to a file in the local filesystem",
parameters={
"type": "object",
"properties": {
"path": {"type": "string", "description": "File path to write"},
"content": {"type": "string", "description": "Content to write"},
"mode": {"type": "string", "enum": ["w", "a"], "description": "Write mode (w=overwrite, a=append)"}
},
"required": ["path", "content"]
},
server="filesystem"
),
ToolDefinition(
name="fs_list_directory",
description="List files and directories in a given path",
parameters={
"type": "object",
"properties": {
"path": {"type": "string", "description": "Directory path to list"},
"recursive": {"type": "boolean", "description": "List recursively"},
"pattern": {"type": "string", "description": "File pattern filter (e.g., '*.py')"}
},
"required": ["path"]
},
server="filesystem"
),
ToolDefinition(
name="fs_delete_file",
description="Delete a file or directory from the filesystem",
parameters={
"type": "object",
"properties": {
"path": {"type": "string", "description": "Path to delete"},
"recursive": {"type": "boolean", "description": "Delete directories recursively"}
},
"required": ["path"]
},
server="filesystem"
),
ToolDefinition(
name="fs_search_files",
description="Search for files containing specific text or matching patterns",
parameters={
"type": "object",
"properties": {
"path": {"type": "string", "description": "Directory to search in"},
"query": {"type": "string", "description": "Text to search for"},
"file_pattern": {"type": "string", "description": "File pattern (e.g., '*.py')"}
},
"required": ["path", "query"]
},
server="filesystem"
)
]
servers.append(ServerDefinition("filesystem", "Local filesystem operations for reading, writing, and managing files", filesystem_tools))
# Database Server
database_tools = [
ToolDefinition(
name="db_query",
description="Execute a SQL query on the database and return results",
parameters={
"type": "object",
"properties": {
"sql": {"type": "string", "description": "SQL query to execute"},
"database": {"type": "string", "description": "Database name"},
"timeout": {"type": "integer", "description": "Query timeout in seconds"}
},
"required": ["sql"]
},
server="database"
),
ToolDefinition(
name="db_insert",
description="Insert data into a database table",
parameters={
"type": "object",
"properties": {
"table": {"type": "string", "description": "Table name"},
"data": {"type": "object", "description": "Data to insert as key-value pairs"},
"database": {"type": "string", "description": "Database name"}
},
"required": ["table", "data"]
},
server="database"
),
ToolDefinition(
name="db_update",
description="Update records in a database table",
parameters={
"type": "object",
"properties": {
"table": {"type": "string", "description": "Table name"},
"data": {"type": "object", "description": "Data to update"},
"where": {"type": "string", "description": "WHERE clause condition"},
"database": {"type": "string", "description": "Database name"}
},
"required": ["table", "data", "where"]
},
server="database"
),
ToolDefinition(
name="db_delete",
description="Delete records from a database table",
parameters={
"type": "object",
"properties": {
"table": {"type": "string", "description": "Table name"},
"where": {"type": "string", "description": "WHERE clause condition"},
"database": {"type": "string", "description": "Database name"}
},
"required": ["table", "where"]
},
server="database"
),
ToolDefinition(
name="db_schema",
description="Get schema information for database tables",
parameters={
"type": "object",
"properties": {
"table": {"type": "string", "description": "Table name (optional, returns all if omitted)"},
"database": {"type": "string", "description": "Database name"}
},
"required": []
},
server="database"
)
]
servers.append(ServerDefinition("database", "Database operations for querying and manipulating structured data", database_tools))
# Web Server
web_tools = [
ToolDefinition(
name="web_get",
description="Make HTTP GET request to a URL and return the response",
parameters={
"type": "object",
"properties": {
"url": {"type": "string", "description": "URL to request"},
"headers": {"type": "object", "description": "HTTP headers"},
"params": {"type": "object", "description": "Query parameters"}
},
"required": ["url"]
},
server="web"
),
ToolDefinition(
name="web_post",
description="Make HTTP POST request to a URL with data",
parameters={
"type": "object",
"properties": {
"url": {"type": "string", "description": "URL to post to"},
"data": {"type": "object", "description": "Data to send"},
"headers": {"type": "object", "description": "HTTP headers"}
},
"required": ["url", "data"]
},
server="web"
),
ToolDefinition(
name="web_scrape",
description="Scrape and extract data from a web page using CSS selectors",
parameters={
"type": "object",
"properties": {
"url": {"type": "string", "description": "URL to scrape"},
"selector": {"type": "string", "description": "CSS selector for elements to extract"},
"attributes": {"type": "array", "items": {"type": "string"}, "description": "Attributes to extract"}
},
"required": ["url", "selector"]
},
server="web"
),
ToolDefinition(
name="web_download",
description="Download a file from a URL to local filesystem",
parameters={
"type": "object",
"properties": {
"url": {"type": "string", "description": "URL to download from"},
"destination": {"type": "string", "description": "Local path to save file"},
"headers": {"type": "object", "description": "HTTP headers"}
},
"required": ["url", "destination"]
},
server="web"
)
]
servers.append(ServerDefinition("web", "HTTP operations for making requests and scraping web content", web_tools))
# Analytics Server
analytics_tools = [
ToolDefinition(
name="analytics_summarize",
description="Calculate summary statistics for a dataset",
parameters={
"type": "object",
"properties": {
"data": {"type": "array", "description": "Array of numeric values"},
"metrics": {"type": "array", "items": {"type": "string"}, "description": "Metrics to calculate (mean, median, std, etc.)"}
},
"required": ["data"]
},
server="analytics"
),
ToolDefinition(
name="analytics_visualize",
description="Create visualizations from data (charts, graphs)",
parameters={
"type": "object",
"properties": {
"data": {"type": "object", "description": "Data to visualize"},
"chart_type": {"type": "string", "enum": ["line", "bar", "scatter", "pie"], "description": "Type of chart"},
"title": {"type": "string", "description": "Chart title"},
"output_path": {"type": "string", "description": "Path to save chart image"}
},
"required": ["data", "chart_type"]
},
server="analytics"
),
ToolDefinition(
name="analytics_correlation",
description="Calculate correlation between variables in a dataset",
parameters={
"type": "object",
"properties": {
"data": {"type": "object", "description": "Dataset with variables as keys"},
"method": {"type": "string", "enum": ["pearson", "spearman"], "description": "Correlation method"}
},
"required": ["data"]
},
server="analytics"
),
ToolDefinition(
name="analytics_predict",
description="Make predictions using machine learning models",
parameters={
"type": "object",
"properties": {
"model_type": {"type": "string", "description": "Type of ML model (linear, tree, etc.)"},
"features": {"type": "array", "description": "Feature values for prediction"},
"trained_model_path": {"type": "string", "description": "Path to trained model file"}
},
"required": ["features"]
},
server="analytics"
)
]
servers.append(ServerDefinition("analytics", "Data analysis and visualization tools for statistical operations", analytics_tools))
# Communication Server
communication_tools = [
ToolDefinition(
name="comm_send_email",
description="Send an email message to recipients",
parameters={
"type": "object",
"properties": {
"to": {"type": "array", "items": {"type": "string"}, "description": "Recipient email addresses"},
"subject": {"type": "string", "description": "Email subject"},
"body": {"type": "string", "description": "Email body content"},
"attachments": {"type": "array", "items": {"type": "string"}, "description": "File paths to attach"}
},
"required": ["to", "subject", "body"]
},
server="communication"
),
ToolDefinition(
name="comm_send_slack",
description="Send a message to a Slack channel or user",
parameters={
"type": "object",
"properties": {
"channel": {"type": "string", "description": "Channel name or user ID"},
"message": {"type": "string", "description": "Message content"},
"thread_ts": {"type": "string", "description": "Thread timestamp for replies"}
},
"required": ["channel", "message"]
},
server="communication"
),
ToolDefinition(
name="comm_read_email",
description="Read emails from inbox with filtering options",
parameters={
"type": "object",
"properties": {
"folder": {"type": "string", "description": "Email folder (inbox, sent, etc.)"},
"unread_only": {"type": "boolean", "description": "Only return unread emails"},
"limit": {"type": "integer", "description": "Maximum number of emails to return"}
},
"required": []
},
server="communication"
),
ToolDefinition(
name="comm_schedule_meeting",
description="Schedule a meeting in calendar",
parameters={
"type": "object",
"properties": {
"title": {"type": "string", "description": "Meeting title"},
"start_time": {"type": "string", "description": "Start time (ISO format)"},
"duration": {"type": "integer", "description": "Duration in minutes"},
"attendees": {"type": "array", "items": {"type": "string"}, "description": "Attendee emails"}
},
"required": ["title", "start_time", "attendees"]
},
server="communication"
)
]
servers.append(ServerDefinition("communication", "Email, messaging, and calendar tools for communication", communication_tools))
# DevOps Server
devops_tools = [
ToolDefinition(
name="devops_deploy",
description="Deploy application to specified environment",
parameters={
"type": "object",
"properties": {
"environment": {"type": "string", "enum": ["dev", "staging", "production"], "description": "Target environment"},
"version": {"type": "string", "description": "Version to deploy"},
"rollback": {"type": "boolean", "description": "Enable auto-rollback on failure"}
},
"required": ["environment", "version"]
},
server="devops"
),
ToolDefinition(
name="devops_monitor",
description="Get monitoring metrics for services and infrastructure",
parameters={
"type": "object",
"properties": {
"service": {"type": "string", "description": "Service name to monitor"},
"metrics": {"type": "array", "items": {"type": "string"}, "description": "Metrics to retrieve (cpu, memory, etc.)"},
"timerange": {"type": "string", "description": "Time range (e.g., '1h', '24h')"}
},
"required": ["service"]
},
server="devops"
),
ToolDefinition(
name="devops_logs",
description="Query and filter application logs",
parameters={
"type": "object",
"properties": {
"service": {"type": "string", "description": "Service name"},
"level": {"type": "string", "enum": ["debug", "info", "warning", "error"], "description": "Log level filter"},
"query": {"type": "string", "description": "Text to search in logs"},
"limit": {"type": "integer", "description": "Maximum number of log entries"}
},
"required": ["service"]
},
server="devops"
),
ToolDefinition(
name="devops_run_pipeline",
description="Trigger a CI/CD pipeline execution",
parameters={
"type": "object",
"properties": {
"pipeline": {"type": "string", "description": "Pipeline name or ID"},
"branch": {"type": "string", "description": "Git branch to build"},
"parameters": {"type": "object", "description": "Pipeline parameters"}
},
"required": ["pipeline"]
},
server="devops"
)
]
servers.append(ServerDefinition("devops", "DevOps tools for deployment, monitoring, and CI/CD operations", devops_tools))
# Cloud Server
cloud_tools = [
ToolDefinition(
name="cloud_create_vm",
description="Create a virtual machine in the cloud",
parameters={
"type": "object",
"properties": {
"instance_type": {"type": "string", "description": "VM instance type (e.g., 't2.micro')"},
"region": {"type": "string", "description": "Cloud region"},
"image_id": {"type": "string", "description": "OS image ID"},
"tags": {"type": "object", "description": "Tags for the VM"}
},
"required": ["instance_type", "region"]
},
server="cloud"
),
ToolDefinition(
name="cloud_list_resources",
description="List cloud resources (VMs, storage, databases)",
parameters={
"type": "object",
"properties": {
"resource_type": {"type": "string", "enum": ["vm", "storage", "database", "network"], "description": "Type of resource"},
"region": {"type": "string", "description": "Cloud region"},
"filters": {"type": "object", "description": "Filter criteria"}
},
"required": ["resource_type"]
},
server="cloud"
),
ToolDefinition(
name="cloud_upload_storage",
description="Upload files to cloud storage",
parameters={
"type": "object",
"properties": {
"bucket": {"type": "string", "description": "Storage bucket name"},
"file_path": {"type": "string", "description": "Local file path"},
"destination": {"type": "string", "description": "Destination path in bucket"},
"public": {"type": "boolean", "description": "Make file publicly accessible"}
},
"required": ["bucket", "file_path"]
},
server="cloud"
),
ToolDefinition(
name="cloud_manage_firewall",
description="Configure cloud firewall rules",
parameters={
"type": "object",
"properties": {
"resource_id": {"type": "string", "description": "Resource ID to configure"},
"action": {"type": "string", "enum": ["add", "remove"], "description": "Action to perform"},
"rule": {"type": "object", "description": "Firewall rule specification"}
},
"required": ["resource_id", "action", "rule"]
},
server="cloud"
)
]
servers.append(ServerDefinition("cloud", "Cloud infrastructure management for VMs, storage, and networking", cloud_tools))
return servers
def get_all_tools(servers: List[ServerDefinition]) -> List[ToolDefinition]:
"""Get flat list of all tools from all servers."""
all_tools = []
for server in servers:
all_tools.extend(server.tools)
return all_tools
def count_tokens_in_schema(schema: Dict[str, Any]) -> int:
"""Rough estimation of tokens in a tool schema (approximately 1 token per 4 characters)."""
import json
schema_str = json.dumps(schema)
return len(schema_str) // 4
def calculate_total_tokens(tools: List[ToolDefinition]) -> int:
"""Calculate total tokens required to inject all tool schemas."""
total = 0
for tool in tools:
total += count_tokens_in_schema(tool.to_schema())
return total
chapter4/collaboration-tools/.gitignore
+67
View File
@@ -0,0 +1,67 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Virtual Environment
venv/
env/
ENV/
.venv
# Environment Variables
.env
.env.local
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Logs
*.log
logs/
# Browser User Data
.config/
# Timer Storage
timers.json
# Screenshots
screenshots/
# Playwright
.playwright/
# Testing
.pytest_cache/
.coverage
htmlcov/
# Temporary Files
*.tmp
temp/
chapter4/collaboration-tools/00_START_HERE.md
+366
View File
@@ -0,0 +1,366 @@
# 🚀 Collaboration Tools MCP Server
> **Start Here** - Complete guide to the Collaboration Tools MCP Server implementation
## 📋 What Is This?
A production-ready Model Context Protocol (MCP) server that provides **19 collaboration tools** for AI agents across 5 categories:
### ✅ Implemented Features
#### 🌐 Browser Automation (5 tools)
- Virtual browser using **browser-use** library (知名虚拟浏览器库)
- Navigate websites, extract content, take screenshots
- AI-powered autonomous browser tasks
- Multi-tab management
#### 👤 Human-in-the-Loop (4 tools)
- Request admin approval for sensitive operations
- Request human input with timeout handling
- Multi-channel admin notifications
- Pending request management
#### 💬 Instant Messaging (3 tools)
- **Telegram** bot integration
- **Slack** webhook messaging
- **Discord** webhook messaging
#### 📧 Email Notifications (1 tool)
- SMTP support (Gmail, etc.)
- SendGrid API support
- HTML emails with attachments
#### ⏰ Timer & Scheduling (5 tools)
- One-time timers
- Recurring timers
- Timer cancellation and management
- Persistent timer storage
- Callback notifications
---
## 🎯 Quick Start
### 1. Installation
```bash
cd projects/week4/collaboration-tools
# Install dependencies
pip install -r requirements.txt
# Install Playwright browsers
playwright install chromium
# Configure environment
cp env.example .env
# Edit .env with your credentials
```
### 2. Run Demo
```bash
# Quick start demo (all tools)
python quickstart.py
# Real-world example
python client_example.py
# Basic tests
python test_basic.py
```
### 3. Start MCP Server
```bash
# Run as MCP server
python src/main.py
# Use with Claude Desktop (add to config)
# See README.md for configuration
```
---
## 📁 Project Structure
```
collaboration-tools/ (Total: 2,331 lines of Python code)
│
├── 📘 Documentation (80KB total)
│ ├── 00_START_HERE.md ← You are here
│ ├── README.md (6.7KB) Main documentation
│ ├── IMPLEMENTATION.md (7.3KB) Technical details
│ ├── ARCHITECTURE.md (23KB) System architecture
│ ├── USAGE_EXAMPLES.md (14KB) 7+ practical examples
│ └── PROJECT_SUMMARY.md (9.2KB) Project overview
│
├── 🔧 Configuration
│ ├── requirements.txt 19 dependencies
│ ├── env.example Configuration template
│ └── .gitignore Git ignore patterns
│
├── 🎯 Demo & Testing
│ ├── quickstart.py (6.1KB) Quick start demo
│ ├── client_example.py (7.2KB) Real-world workflow
│ └── test_basic.py (4.7KB) Basic tests
│
└── 📦 Source Code (src/)
├── main.py (11KB) MCP server (19 tools)
├── config.py (3.5KB) Configuration management
├── browser_tools.py (8.3KB) Browser automation
├── notification_tools.py (11KB) Email & IM notifications
├── hitl_tools.py (11KB) Human-in-the-loop
└── timer_tools.py (14KB) Timer management
```
---
## 🛠️ Technology Stack
| Component | Technology |
|-----------|-----------|
| **MCP Server** | FastMCP (mcp>=0.9.0) |
| **Browser Automation** | browser-use + Playwright |
| **AI Agent** | LangChain + OpenAI |
| **Email** | aiosmtplib (SMTP) + SendGrid |
| **IM** | httpx (Webhooks) + Telegram Bot API |
| **Async** | asyncio (Python 3.11+) |
| **Config** | Pydantic + python-dotenv |
| **Scheduling** | apscheduler + asyncio |
---
## 📚 Documentation Guide
### For Getting Started
1. **00_START_HERE.md** (this file) - Overview and quick start
2. **README.md** - Installation, configuration, and basic usage
### For Implementation
3. **ARCHITECTURE.md** - System architecture and data flows
4. **IMPLEMENTATION.md** - Technical implementation details
### For Usage
5. **USAGE_EXAMPLES.md** - 7+ practical usage examples
6. **quickstart.py** - Runnable demo of all features
7. **client_example.py** - Real-world workflow example
### For Summary
8. **PROJECT_SUMMARY.md** - Complete project overview
---
## 🎨 Key Features
### 1. Browser Automation with AI
```python
# Autonomous browser task using AI
await mcp_browser_execute_task(
task="Search for AI agent tutorials on Google and extract top 5 results",
max_steps=30
)
```
### 2. Human-in-the-Loop Workflow
```python
# Request approval with timeout
result = await mcp_request_admin_approval(
request_message="Delete 1000 database records?",
urgent=True,
timeout_seconds=300
)
if result["approved"]:
# Proceed with action
perform_deletion()
```
### 3. Multi-Channel Notifications
```python
# Send alert via all channels
await mcp_send_email(to_email="admin@example.com", ...)
await mcp_send_slack_message(message="🚨 Alert!")
await mcp_send_telegram_message(message="Alert!")
await mcp_send_discord_message(message="Alert!")
```
### 4. Timer & Scheduling
```python
# Set timer for delayed execution
timer = await mcp_set_timer(
duration_seconds=3600,
callback_message="Time to check website"
)
# Recurring timer
await mcp_set_recurring_timer(
interval_seconds=300, # Every 5 minutes
max_occurrences=10
)
```
---
## 📊 Statistics
- **Total Files**: 17 (7 Python modules + 10 docs/config)
- **Lines of Code**: 2,331 (Python)
- **Documentation**: ~80KB
- **MCP Tools**: 19 tools across 5 categories
- **Dependencies**: 19 packages
- **Test Coverage**: Basic tests included
---
## 🔐 Security Features
✅ Environment-based configuration (no hardcoded secrets)
✅ .env file excluded from git
✅ Isolated browser user data directory
✅ HITL timeout and multi-channel verification
✅ Graceful error handling throughout
✅ Audit trail for admin approvals
---
## 🚦 Usage Patterns
### Pattern 1: Website Monitoring
```python
navigate → screenshot → set_recurring_timer → notify_via_slack
```
### Pattern 2: Admin Approval Flow
```python
request_approval → wait_for_response → notify_decision → execute_action
```
### Pattern 3: Scheduled Task
```python
set_timer → browser_task → extract_data → send_email_report
```
### Pattern 4: Multi-Channel Alert
```python
critical_event → [email, slack, telegram, discord] → admin_approval
```
---
## 📖 Next Steps
### To Use This Project:
1. **Read Documentation**
- Start with `README.md` for setup
- Check `USAGE_EXAMPLES.md` for practical examples
- Review `ARCHITECTURE.md` for technical details
2. **Configure Environment**
- Copy `env.example` to `.env`
- Add your API keys and credentials
- Configure notification channels
3. **Run Demos**
- `python quickstart.py` - See all tools in action
- `python client_example.py` - Real-world workflow
- `python test_basic.py` - Verify installation
4. **Start Using**
- Run as MCP server: `python src/main.py`
- Use with Claude Desktop or custom client
- Integrate into your AI agent application
### To Extend This Project:
1. **Add New Tools**: Create new functions in existing modules
2. **Add New Channels**: Extend `notification_tools.py`
3. **Add Storage**: Replace in-memory state with database
4. **Add Dashboard**: Build web UI for admin management
5. **Add Analytics**: Track tool usage and performance
---
## 🆘 Troubleshooting
### Browser Issues
```bash
# Reinstall Playwright
playwright install chromium --force
```
### Email Issues
- Use Gmail App Passwords (not regular password)
- Check SMTP port and host settings
### Import Errors
```bash
# Reinstall dependencies
pip install -r requirements.txt --force-reinstall
```
### Permission Issues
```bash
# Ensure config directory is writable
mkdir -p ~/.config/collaboration-tools
chmod 755 ~/.config/collaboration-tools
```
---
## 📞 Support
- **Documentation**: Check all .md files in this directory
- **Examples**: See `quickstart.py` and `client_example.py`
- **Tests**: Run `test_basic.py` to verify functionality
- **Issues**: Review error messages and logs
---
## 🎓 Learning Path
1. **Beginner**: Run `quickstart.py` and read `README.md`
2. **Intermediate**: Study `USAGE_EXAMPLES.md` and modify examples
3. **Advanced**: Review `ARCHITECTURE.md` and extend functionality
---
## ✅ Implementation Checklist
✅ Virtual browser (browser-use library)
✅ Human-in-the-loop tools
✅ IM notifications (Telegram, Slack, Discord)
✅ Email notifications (SMTP + SendGrid)
✅ Timer and scheduling tools
✅ Configuration management
✅ Error handling and logging
✅ Comprehensive documentation
✅ Working examples and demos
✅ Basic test suite
✅ Clean architecture
✅ Production-ready code
---
## 🌟 Highlights
- **Production-Ready**: Comprehensive error handling and logging
- **Well-Documented**: 80KB+ of documentation
- **Modular Design**: Easy to extend and maintain
- **Real Examples**: Working demos and use cases
- **Best Practices**: SOLID principles, clean code, async patterns
---
## 📝 License
MIT License - See project root for details
---
**Ready to start?** → Continue to `README.md` for detailed setup instructions!
chapter4/collaboration-tools/ARCHITECTURE.md
+486
View File
@@ -0,0 +1,486 @@
# Architecture Documentation
## System Architecture
```
┌─────────────────────────────────────────────────────────────────┐
│ MCP Client (AI Agent) │
│ (Claude, Custom App, etc.) │
└────────────────────────────┬────────────────────────────────────┘
│
│ MCP Protocol (stdio)
│
┌────────────────────────────▼────────────────────────────────────┐
│ Collaboration Tools MCP Server │
│ (main.py) │
│ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ FastMCP Server Layer │ │
│ │ • Tool Registration │ │
│ │ • Request Routing │ │
│ │ • Response Formatting │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ ┌────────────┬────────────┬────────────┬────────────┐ │
│ │ Browser │ HITL │ Notify │ Timer │ │
│ │ Tools │ Tools │ Tools │ Tools │ │
│ └─────┬──────┴──────┬─────┴──────┬─────┴──────┬─────┘ │
│ │ │ │ │ │
└────────┼─────────────┼────────────┼────────────┼───────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌────────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐
│ browser- │ │ Admin │ │ Email │ │ asyncio │
│ use │ │ Webhook/ │ │ SMTP/ │ │ Timer │
│ (Playwright)│ │ Email/ │ │ SendGrid │ │ Tasks │
│ │ │ IM │ │ │ │ │
│ ┌──────┐ │ └──────────┘ │ ┌────┐ │ └──────────┘
│ │Chrome│ │ │ │ IM │ │
│ └──────┘ │ │ │Webhooks│
└────────────┘ │ └────┘ │
└──────────┘
```
## Component Architecture
### 1. MCP Server Layer (`main.py`)
```python
┌─────────────────────────────────────┐
│ FastMCP Server │
│ │
│ @mcp.tool(...) │
│ async def mcp_tool_name(...) -> str│
│ result = await internal_func() │
│ return str(result) │
│ │
│ @mcp.on_shutdown │
│ async def cleanup() │
└─────────────────────────────────────┘
```
**Responsibilities:**
- Tool registration and exposure
- Request validation
- Response serialization
- Lifecycle management
### 2. Browser Tools Layer (`browser_tools.py`)
```
┌──────────────────────────────────────────┐
│ Browser Tools Module │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Browser Session Manager │ │
│ │ • Singleton pattern │ │
│ │ • Lazy initialization │ │
│ │ • Profile management │ │
│ └────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Navigation & Interaction │ │
│ │ • browser_navigate() │ │
│ │ • browser_get_content() │ │
│ │ • browser_screenshot() │ │
│ │ • browser_list_tabs() │ │
│ └────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────┐ │
│ │ AI Agent Integration │ │
│ │ • browser_execute_task() │ │
│ │ • LangChain + OpenAI │ │
│ │ • Autonomous task execution │ │
│ └────────────────────────────────────┘ │
└──────────────────────────────────────────┘
│
▼
┌────────────┐
│ browser-use│
│ Library │
└────────────┘
```
### 3. Notification Layer (`notification_tools.py`)
```
┌────────────────────────────────────────┐
│ Notification Tools Module │
│ │
│ ┌──────────────────────────────────┐ │
│ │ Email Handler │ │
│ │ ┌────────────┬────────────┐ │ │
│ │ │ SMTP │ SendGrid │ │ │
│ │ │ Fallback │ Primary │ │ │
│ │ └────────────┴────────────┘ │ │
│ └──────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────┐ │
│ │ IM Handler │ │
│ │ ┌──────────┬──────────┬──────┐ │ │
│ │ │ Telegram │ Slack │Discord│ │ │
│ │ │ Bot API │ Webhook │Webhook│ │ │
│ │ └──────────┴──────────┴──────┘ │ │
│ └──────────────────────────────────┘ │
│ │
│ • Async delivery │
│ • Error handling │
│ • Multi-channel support │
└────────────────────────────────────────┘
```
### 4. HITL Layer (`hitl_tools.py`)
```
┌─────────────────────────────────────────┐
│ Human-in-the-Loop Module │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Request Manager │ │
│ │ • Generate unique request IDs │ │
│ │ • Track pending requests │ │
│ │ • Timeout handling │ │
│ └────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Notification Dispatcher │ │
│ │ • Multi-channel alerts │ │
│ │ • Email notifications │ │
│ │ • IM notifications │ │
│ │ • Webhook callbacks │ │
│ └────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Response Handler │ │
│ │ • Wait for admin response │ │
│ │ • Process approval/rejection │ │
│ │ • Update request status │ │
│ └────────────────────────────────────┘ │
│ │
│ In-Memory Storage: │
│ _pending_requests: Dict[str, Request] │
└─────────────────────────────────────────┘
```
### 5. Timer Layer (`timer_tools.py`)
```
┌──────────────────────────────────────────┐
│ Timer Management Module │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Timer Registry │ │
│ │ • Active timers storage │ │
│ │ • Timer metadata tracking │ │
│ │ • Status management │ │
│ └────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Timer Execution Engine │ │
│ │ ┌──────────────┬──────────────┐ │ │
│ │ │ One-time │ Recurring │ │ │
│ │ │ Timers │ Timers │ │ │
│ │ │ │ │ │ │
│ │ │ asyncio.sleep│ While loop │ │ │
│ │ └──────────────┴──────────────┘ │ │
│ └────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Callback System │ │
│ │ • Notification dispatch │ │
│ │ • Custom callback data │ │
│ └────────────────────────────────────┘ │
│ │
│ ┌────────────────────────────────────┐ │
│ │ Persistence Layer │ │
│ │ • JSON file storage │ │
│ │ • State restoration on restart │ │
│ └────────────────────────────────────┘ │
│ │
│ In-Memory Storage: │
│ _active_timers: Dict[str, Timer] │
│ _timer_tasks: Dict[str, asyncio.Task] │
└──────────────────────────────────────────┘
```
## Data Flow
### 1. Browser Automation Flow
```
MCP Client
│
│ call: mcp_browser_execute_task(task="...")
▼
MCP Server (main.py)
│
│ await browser_execute_task()
▼
Browser Tools
│
│ 1. Initialize browser (if needed)
│ 2. Create LangChain agent
│ 3. Execute task
▼
browser-use Library
│
│ • Navigate pages
│ • Interact with elements
│ • Extract content
▼
Playwright (Chrome)
│
│ • Actual browser automation
▼
Result returned to client
```
### 2. HITL Approval Flow
```
Agent Request
│
│ request_admin_approval(message, urgent=True)
▼
HITL Tools
│
│ 1. Create request record
│ 2. Generate unique ID
▼
Notification Dispatcher
│
├─► Email → Admin
├─► Telegram → Admin
├─► Slack → Admin
└─► Webhook → Admin Dashboard
Admin receives notifications
│
│ Reviews request
│ Responds via API/interface
▼
Response Handler
│
│ Update request status
▼
Wait loop completes
│
│ Return approval result
▼
Agent receives response
```
### 3. Timer Execution Flow
```
Agent
│
│ set_timer(duration=300, callback="...")
▼
Timer Tools
│
│ 1. Create timer record
│ 2. Generate timer ID
│ 3. Save to storage
▼
Create asyncio.Task
│
│ async def _run_timer(timer_id, duration):
│ await asyncio.sleep(duration)
│ trigger_callback()
▼
Timer Expires
│
│ 1. Update status to "expired"
│ 2. Execute callback
▼
Callback Handler
│
├─► Send notification (if configured)
├─► Update storage
└─► Log completion
```
## Configuration Flow
```
Environment Variables (.env)
│
▼
config.py
│
│ Pydantic Models:
│ • BrowserConfig
│ • EmailConfig
│ • IMConfig
│ • HITLConfig
│ • TimerConfig
▼
Loaded into Config object
│
├─► browser_tools.py
├─► notification_tools.py
├─► hitl_tools.py
└─► timer_tools.py
```
## Error Handling Pattern
```python
┌──────────────────────────────┐
│ Tool Function Entry │
└──────────┬───────────────────┘
│
▼
┌─────────────┐
│ Try Block │
│ │
│ • Validate │
│ • Execute │
│ • Return │
└──────┬──────┘
│
┌──────▼──────┐
│ Success │
│ Response │
│ │
│ { │
│ success: T │
│ data: ... │
│ message:..│
│ } │
└─────────────┘
Exception ▼
┌─────────────┐
│ Error │
│ Response │
│ │
│ { │
│ success: F │
│ error: ... │
│ message:..│
│ } │
└─────────────┘
```
## State Management
### In-Memory State
```
┌─────────────────────────────────────┐
│ Application Memory │
│ │
│ _browser_session: BrowserSession │
│ _pending_requests: Dict[str, Req] │
│ _active_timers: Dict[str, Timer] │
│ _timer_tasks: Dict[str, Task] │
└─────────────────────────────────────┘
```
### Persistent State
```
┌─────────────────────────────────────┐
│ Filesystem Storage │
│ │
│ ~/.config/collaboration-tools/ │
│ ├── browser/ │
│ │ └── (browser profile data) │
│ ├── timers.json │
│ │ └── (active timers state) │
│ └── screenshots/ │
│ └── (captured screenshots) │
└─────────────────────────────────────┘
```
## Security Considerations
```
┌─────────────────────────────────────┐
│ Security Layers │
│ │
│ ┌───────────────────────────────┐ │
│ │ Configuration Security │ │
│ │ • .env file (gitignored) │ │
│ │ • No hardcoded credentials │ │
│ │ • Environment-based config │ │
│ └───────────────────────────────┘ │
│ │
│ ┌───────────────────────────────┐ │
│ │ Browser Security │ │
│ │ • Isolated user data dir │ │
│ │ • Optional domain whitelist │ │
│ │ • Configurable security │ │
│ └───────────────────────────────┘ │
│ │
│ ┌───────────────────────────────┐ │
│ │ HITL Security │ │
│ │ • Timeout on requests │ │
│ │ • Multi-channel verification │ │
│ │ • Audit trail │ │
│ └───────────────────────────────┘ │
│ │
│ ┌───────────────────────────────┐ │
│ │ API Security │ │
│ │ • API keys in env vars │ │
│ │ • No secrets in logs │ │
│ │ • Webhook validation ready │ │
│ └───────────────────────────────┘ │
└─────────────────────────────────────┘
```
## Scaling Considerations
### Current Architecture (Single Process)
```
┌──────────────────────┐
│ MCP Server │
│ ┌────────────────┐ │
│ │ All Tools │ │
│ │ In-Memory │ │
│ │ State │ │
│ └────────────────┘ │
└──────────────────────┘
```
### Future Distributed Architecture
```
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Browser │ │ HITL │ │ Timer │
│ Service │ │ Service │ │ Service │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
└───────────────────┼────────────────────┘
│
┌──────▼────────┐
│ MCP Server │
│ (Gateway) │
└───────────────┘
│
┌──────▼────────┐
│ Database │
│ (State) │
└───────────────┘
```
## Performance Characteristics
- **Browser Initialization**: 2-5 seconds (one-time)
- **Navigation**: 1-3 seconds per page
- **Email Send**: 1-2 seconds
- **IM Webhook**: <500ms
- **Timer Accuracy**: ±1-2 seconds
- **Memory Usage**: ~100-200MB (with browser)
- **Concurrent Timers**: Thousands (asyncio-based)
## Extension Points
1. **New Tool Categories**: Add new `*_tools.py` modules
2. **New Notification Channels**: Extend `notification_tools.py`
3. **Custom Storage Backends**: Replace JSON persistence
4. **Advanced Browser Features**: Extend `browser_tools.py`
5. **Admin Dashboard**: Web UI for HITL management
6. **Analytics**: Tool usage tracking and monitoring
chapter4/collaboration-tools/Dockerfile
+62
View File
@@ -0,0 +1,62 @@
# Collaboration Tools MCP Server Dockerfile
# Supports browser automation, Excel processing, and human-in-the-loop interactions
# Uses latest stable versions (as of 2025)
FROM ubuntu:24.04
ENV DEBIAN_FRONTEND=noninteractive
ENV PYTHONUNBUFFERED=1
ENV TZ=UTC
# Install system dependencies
RUN apt-get update && apt-get install -y \
curl \
wget \
git \
build-essential \
software-properties-common \
ca-certificates \
# For browser automation
chromium-browser \
chromium-chromedriver \
# For GUI applications (headless)
xvfb \
&& rm -rf /var/lib/apt/lists/*
# Install Python 3.13 (latest stable)
RUN add-apt-repository ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install -y \
python3.13 \
python3.13-dev \
python3.13-venv \
python3-pip \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.13 1 && \
update-alternatives --install /usr/bin/python python /usr/bin/python3.13 1 && \
rm -rf /var/lib/apt/lists/*
# Set up working directory
WORKDIR /app
# Copy requirements and install Python packages
COPY requirements.txt .
RUN python3 -m pip install --upgrade pip setuptools wheel && \
python3 -m pip install -r requirements.txt
# Copy application files
COPY . .
# Set environment variables
ENV DISPLAY=:99
ENV WORKSPACE_DIR=/workspace
RUN mkdir -p /workspace && chmod 777 /workspace
# Create non-root user for security
RUN useradd -m -u 1000 mcpuser && \
chown -R mcpuser:mcpuser /app /workspace
# Switch to non-root user
USER mcpuser
# Run the MCP server (with xvfb for headless browser)
CMD ["sh", "-c", "Xvfb :99 -screen 0 1920x1080x24 & python3 src/main.py"]
chapter4/collaboration-tools/IMPLEMENTATION.md
+280
View File
@@ -0,0 +1,280 @@
# Implementation Details
## Architecture Overview
The Collaboration Tools MCP Server is built with a modular architecture that separates concerns into distinct tool categories:
1. **Browser Automation** - Virtual browser operations using browser-use
2. **Notifications** - Email and instant messaging integrations
3. **Human-in-the-Loop** - Admin approval and input request system
4. **Timers** - Scheduling and delayed task execution
## Core Components
### 1. Browser Tools (`browser_tools.py`)
The browser automation module integrates the `browser-use` library to provide AI-driven web automation capabilities.
**Key Features:**
- Singleton browser session management
- Integration with browser-use Agent for autonomous tasks
- Support for multiple tabs
- Screenshot capture
- Content extraction with CSS selectors
**Implementation Details:**
```python
# Browser session is initialized lazily and reused
_browser_session = None
async def init_browser():
global _browser_session
if _browser_session is not None:
return _browser_session
# Create browser with profile and settings
profile = BrowserProfile(...)
browser = Browser(browser_profile=profile)
await browser.start()
_browser_session = browser
return browser
```
### 2. Notification Tools (`notification_tools.py`)
Provides multi-channel notification capabilities with fallback support.
**Supported Channels:**
- **Email**: SMTP or SendGrid API
- **Telegram**: Bot API integration
- **Slack**: Webhook-based messaging
- **Discord**: Webhook-based messaging
**Implementation Pattern:**
```python
async def send_email(...):
# Check if SendGrid is configured (preferred)
if config.email.sendgrid_api_key:
return await _send_email_sendgrid(...)
# Fall back to SMTP
elif config.email.smtp_username:
return await _send_email_smtp(...)
else:
return {"success": False, "error": "No email service configured"}
```
### 3. Human-in-the-Loop Tools (`hitl_tools.py`)
Enables AI agents to request human assistance when needed.
**Key Features:**
- Async request/response pattern
- Multiple notification channels for admin alerts
- Timeout handling
- Request tracking and status management
**Request Flow:**
1. Agent creates approval request
2. System notifies admin via configured channels
3. System waits for admin response (with timeout)
4. Admin responds through API or interface
5. Result returned to agent
**Storage:**
```python
# In-memory storage of pending requests
_pending_requests: Dict[str, Dict[str, Any]] = {}
# Each request has:
# - request_id: Unique identifier
# - message: What needs approval
# - context: Additional data
# - status: pending/approved/rejected/timeout
# - admin_notes: Admin's response
```
### 4. Timer Tools (`timer_tools.py`)
Provides scheduling capabilities for delayed task execution.
**Timer Types:**
- **One-time timers**: Execute once after delay
- **Recurring timers**: Execute at intervals
**Implementation:**
```python
# Active timers stored in-memory and persisted to disk
_active_timers: Dict[str, Dict[str, Any]] = {}
_timer_tasks: Dict[str, asyncio.Task] = {}
async def _run_timer(timer_id: str, duration_seconds: int):
await asyncio.sleep(duration_seconds)
# Timer expired - trigger callback
await _trigger_timer_callback(timer_data)
```
**Persistence:**
- Timers are saved to JSON file on disk
- Active timers are restored on server restart
- Remaining time is recalculated on restore
### 5. Configuration (`config.py`)
Centralized configuration management using Pydantic models.
**Configuration Hierarchy:**
```python
Config
├── BrowserConfig (browser settings)
├── EmailConfig (email service settings)
├── IMConfig (IM service settings)
├── HITLConfig (HITL settings)
└── TimerConfig (timer storage settings)
```
**Environment Variable Mapping:**
- All settings can be configured via environment variables
- Defaults provided for most settings
- Sensitive credentials loaded from .env file
## MCP Server Implementation
The main server (`main.py`) uses FastMCP to expose all tools via the MCP protocol.
**Server Structure:**
```python
from mcp.server.fastmcp import FastMCP
mcp = FastMCP("collaboration-tools")
@mcp.tool(description="...")
async def mcp_tool_name(...) -> str:
result = await internal_function(...)
return str(result)
```
**Lifecycle Management:**
```python
@mcp.on_shutdown
async def cleanup():
# Close browser sessions
await close_browser()
# Save timer state
await _save_timers()
```
## Error Handling
All tools follow a consistent error handling pattern:
```python
try:
# Perform operation
result = await operation()
return {
"success": True,
"data": result,
"message": "Operation successful"
}
except Exception as e:
logger.error(f"Operation failed: {e}")
return {
"success": False,
"error": str(e),
"message": "Operation failed"
}
```
## Integration Patterns
### Using with Claude Desktop
Add to `claude_desktop_config.json`:
```json
{
"mcpServers": {
"collaboration-tools": {
"command": "python",
"args": ["/path/to/src/main.py"]
}
}
}
```
### Using as Python Client
```python
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
async def use_tools():
server_params = StdioServerParameters(
command="python",
args=["src/main.py"]
)
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# Call tools
result = await session.call_tool("mcp_set_timer", {
"duration_seconds": 60,
"timer_name": "Test"
})
```
## Security Considerations
1. **Browser Security**:
- Option to restrict allowed domains
- Configurable security settings
- Isolated user data directory
2. **Credentials**:
- All secrets loaded from environment variables
- No hardcoded credentials
- .env file excluded from version control
3. **HITL**:
- Timeout on all approval requests
- Admin notification via multiple channels
- Request tracking and audit trail
4. **Timer Persistence**:
- Timers stored in user's home directory
- JSON format for easy inspection
- State recovery on restart
## Performance Considerations
1. **Browser Session**:
- Lazy initialization (only when needed)
- Single shared session (reduces memory)
- Proper cleanup on shutdown
2. **Async Operations**:
- All I/O operations are async
- Non-blocking timer implementation
- Concurrent notification delivery
3. **Resource Management**:
- Browser tabs can be closed individually
- Expired timers cleaned up
- Temporary files managed
## Testing
The implementation includes:
- `quickstart.py` - Functional demo of all tools
- `client_example.py` - Real-world workflow example
- Modular design enables unit testing of individual components
## Future Enhancements
Potential improvements:
1. Database storage for HITL requests and timers
2. Web dashboard for admin management
3. More notification channels (SMS, push notifications)
4. Browser recording/replay capabilities
5. Advanced scheduling (cron-like expressions)
6. Tool usage analytics and monitoring
chapter4/collaboration-tools/INSTALL.md
+82
View File
@@ -0,0 +1,82 @@
# Installation Guide
## Quick Installation
### 1. Install Core Dependencies
```bash
cd projects/week4/collaboration-tools
# Upgrade pip first
pip install --upgrade pip
# Install with correct versions
pip install --upgrade pydantic>=2.8.0 pydantic-settings>=2.4.0 anyio>=4.5.0
pip install -r requirements.txt
```
### 2. Install Browser Dependencies
```bash
# Install Playwright browsers
playwright install chromium
```
### 3. Configure Environment
```bash
# Copy example configuration
cp env.example .env
# Edit .env with your credentials
# At minimum, set OPENAI_API_KEY for browser AI tasks
nano .env # or use your preferred editor
```
### 4. Verify Installation
```bash
# Run basic tests
python test_basic.py
# Or run the quickstart demo
python quickstart.py
```
## Troubleshooting
### Issue: Pydantic Import Errors
**Error:** `ModuleNotFoundError: No module named 'pydantic._internal._signature'`
**Solution:**
```bash
pip install --upgrade pydantic>=2.8.0 pydantic-settings>=2.4.0
```
### Issue: anyio Type Errors
**Error:** `TypeError: 'function' object is not subscriptable`
**Solution:**
```bash
pip install --upgrade anyio>=4.5.0
```
### Issue: Browser Errors
**Error:** Browser fails to start or Playwright not found
**Solution:**
```bash
playwright install chromium --force
```
### Issue: MCP Server Won't Start
**Solution:**
```bash
# Clean install
pip uninstall mcp fastmcp pydantic pydantic-settings anyio -y
pip install -r requirements.txt
```
chapter4/collaboration-tools/PROJECT_SUMMARY.md
+355
View File
@@ -0,0 +1,355 @@
# Collaboration Tools MCP Server - Project Summary
## Overview
A comprehensive MCP (Model Context Protocol) server implementation that provides collaboration tools for AI agents, including browser automation, human-in-the-loop capabilities, multi-channel notifications, and timer management.
## Project Structure
```
collaboration-tools/
├── src/
│ ├── __init__.py # Package initialization
│ ├── main.py # MCP server entry point (19 tools)
│ ├── config.py # Configuration management with Pydantic
│ ├── browser_tools.py # Browser automation using browser-use
│ ├── notification_tools.py # Email & IM notifications
│ ├── hitl_tools.py # Human-in-the-loop tools
│ └── timer_tools.py # Timer and scheduling tools
│
├── README.md # Main documentation
├── IMPLEMENTATION.md # Technical implementation details
├── USAGE_EXAMPLES.md # Practical usage examples
├── PROJECT_SUMMARY.md # This file
│
├── requirements.txt # Python dependencies
├── env.example # Environment configuration template
├── .gitignore # Git ignore patterns
│
├── quickstart.py # Quick start demo
├── client_example.py # Real-world workflow example
└── test_basic.py # Basic functionality tests
```
## Features Implemented
### ✅ 1. Browser Automation (5 tools)
- `mcp_browser_navigate` - Navigate to URLs
- `mcp_browser_get_content` - Extract page content
- `mcp_browser_execute_task` - AI-driven autonomous browser tasks
- `mcp_browser_screenshot` - Capture screenshots
- `mcp_browser_list_tabs` - List all open tabs
**Implementation:**
- Uses `browser-use` library (知名虚拟浏览器库)
- Singleton browser session management
- Support for autonomous AI agents via LangChain + OpenAI
- Full Playwright-based automation
### ✅ 2. Human-in-the-Loop (4 tools)
- `mcp_request_admin_approval` - Request admin approval
- `mcp_request_admin_input` - Request admin input
- `mcp_respond_to_request` - Admin response handling
- `mcp_list_pending_requests` - List pending requests
**Implementation:**
- Async request/response pattern
- Multi-channel admin notifications (Email, Telegram, Slack)
- Configurable timeouts
- In-memory request tracking with webhook support
### ✅ 3. Instant Messaging (3 tools)
- `mcp_send_telegram_message` - Send Telegram messages
- `mcp_send_slack_message` - Send Slack webhooks
- `mcp_send_discord_message` - Send Discord webhooks
**Implementation:**
- Telegram Bot API integration
- Webhook-based messaging for Slack/Discord
- Configurable default channels
- Async message delivery
### ✅ 4. Email Notifications (1 tool)
- `mcp_send_email` - Send email notifications
**Implementation:**
- SMTP support (Gmail, etc.)
- SendGrid API support
- HTML and plain text emails
- CC recipients and attachments support
### ✅ 5. Timer & Scheduling (5 tools)
- `mcp_set_timer` - Set one-time timers
- `mcp_set_recurring_timer` - Set recurring timers
- `mcp_cancel_timer` - Cancel timers
- `mcp_list_timers` - List all timers
- `mcp_get_timer_status` - Check timer status
**Implementation:**
- Async timer execution using asyncio
- Persistent storage (JSON file)
- Timer restoration on restart
- Callback notifications via IM/Email
## Total Tools Implemented
**19 MCP Tools** across 5 categories:
- Browser: 5 tools
- HITL: 4 tools
- IM: 3 tools
- Email: 1 tool
- Timer: 5 tools
- Management: 1 tool (shutdown)
## Key Technologies
- **MCP Protocol**: FastMCP for server implementation
- **Browser Automation**: browser-use (Playwright-based)
- **AI Integration**: LangChain + OpenAI for autonomous tasks
- **Async Framework**: asyncio for non-blocking operations
- **Configuration**: Pydantic models + python-dotenv
- **Notifications**:
- Email: aiosmtplib (SMTP) + sendgrid
- IM: httpx for webhook APIs
- Telegram: Bot API via httpx
## Configuration
All tools are configurable via environment variables:
```env
# Browser
BROWSER_HEADLESS=false
BROWSER_USER_DATA_DIR=~/.config/collaboration-tools/browser
# Email
SMTP_HOST=smtp.gmail.com
SMTP_USERNAME=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SENDGRID_API_KEY=your-key
# IM
TELEGRAM_BOT_TOKEN=your-token
SLACK_WEBHOOK_URL=your-webhook
DISCORD_WEBHOOK_URL=your-webhook
# HITL
HITL_ADMIN_EMAIL=admin@example.com
HITL_TIMEOUT_SECONDS=3600
# Timer
TIMER_STORAGE_PATH=~/.config/collaboration-tools/timers.json
# AI (for browser tasks)
OPENAI_API_KEY=your-key
OPENAI_MODEL=gpt-5.6-luna
```
## Usage
### Start the MCP Server
```bash
cd projects/week4/collaboration-tools
python src/main.py
```
### Run Quick Start Demo
```bash
python quickstart.py
```
### Run Real-World Example
```bash
python client_example.py
```
### Run Tests
```bash
python test_basic.py
```
### Use with Claude Desktop
Add to `claude_desktop_config.json`:
```json
{
"mcpServers": {
"collaboration-tools": {
"command": "python",
"args": ["/path/to/collaboration-tools/src/main.py"]
}
}
}
```
## Example Workflows
### 1. Website Monitoring
```python
# Navigate to website
await mcp_browser_navigate(url="https://example.com")
# Take screenshot
await mcp_browser_screenshot(full_page=True)
# Set recurring check
await mcp_set_recurring_timer(
interval_seconds=3600,
timer_name="Website Check"
)
# Notify via Slack
await mcp_send_slack_message(
message="🌐 Website monitoring started"
)
```
### 2. Admin Approval Flow
```python
# Request approval
result = await mcp_request_admin_approval(
request_message="Delete 1000 database records?",
urgent=True,
timeout_seconds=300
)
if result["approved"]:
# Proceed with action
await mcp_send_email(
to_email="admin@example.com",
subject="✅ Operation Completed",
body="Database cleanup finished successfully"
)
```
### 3. Scheduled Task
```python
# Set timer for delayed execution
timer = await mcp_set_timer(
duration_seconds=3600, # 1 hour
timer_name="Report Generation",
callback_message="Generate daily report"
)
# When timer expires, generate and email report
await mcp_send_email(
to_email="team@example.com",
subject="📊 Daily Report",
body=report_content
)
```
## Architecture Highlights
### Modular Design
- Each tool category in separate module
- Clean separation of concerns
- Easy to extend with new tools
### Error Handling
- Consistent error response format
- Graceful degradation when services unavailable
- Detailed error logging
### Async Operations
- Non-blocking I/O throughout
- Concurrent notification delivery
- Efficient timer management
### State Management
- In-memory state with disk persistence
- Timer restoration on restart
- HITL request tracking
## Testing
### Basic Tests (`test_basic.py`)
- Configuration loading
- Timer functionality
- HITL tools
- Notification tools (mock)
- Browser tools (import check)
### Demo Scripts
- `quickstart.py` - All tools demonstration
- `client_example.py` - Real-world workflow
## Documentation
1. **README.md** - Main documentation with setup and usage
2. **IMPLEMENTATION.md** - Technical implementation details
3. **USAGE_EXAMPLES.md** - 7+ practical usage examples
4. **PROJECT_SUMMARY.md** - This overview document
## Dependencies
Core dependencies:
- `mcp>=0.9.0` - MCP protocol support
- `fastmcp>=0.2.0` - Fast MCP server framework
- `browser-use>=0.1.0` - Browser automation
- `playwright>=1.40.0` - Browser driver
- `pydantic>=2.0.0` - Configuration validation
- `aiosmtplib>=3.0.0` - Async SMTP
- `sendgrid>=6.11.0` - SendGrid API
- `httpx>=0.24.0` - Async HTTP client
- `apscheduler>=3.10.0` - Scheduling support
## Integration Points
### As MCP Server
- Claude Desktop
- MCP-compatible clients
- Any application using MCP protocol
### As Python Library
- Import tools directly
- Use ClientSession for tool calls
- Extend with custom tools
## Future Enhancements
Potential additions:
1. Database storage for persistent state
2. Web dashboard for admin management
3. More IM platforms (WeChat, DingTalk)
4. SMS notifications
5. Advanced scheduling (cron expressions)
6. Tool usage analytics
7. Browser session recording/replay
8. Multi-browser support
9. Distributed timer management
10. Webhook server for HITL responses
## Success Criteria
✅ All required features implemented:
- ✅ Virtual browser (browser-use)
- ✅ Human-in-the-loop tools
- ✅ IM notifications (Telegram, Slack, Discord)
- ✅ Email notifications
- ✅ Timer/scheduling tools
✅ Production-ready code:
- ✅ Comprehensive error handling
- ✅ Configuration management
- ✅ Logging throughout
- ✅ Clean architecture
- ✅ Extensive documentation
- ✅ Working examples
- ✅ Basic tests
## Conclusion
This MCP server provides a complete collaboration toolkit for AI agents, enabling them to:
- Automate web browser tasks
- Request human assistance when needed
- Send notifications across multiple channels
- Schedule and time tasks
- Coordinate complex workflows
The implementation follows best practices with clean architecture, comprehensive error handling, and extensive documentation, making it ready for production use or further extension.
chapter4/collaboration-tools/README.md
+771
View File
@@ -0,0 +1,771 @@
# Collaboration Tools MCP Server / 协作工具 MCP 服务器
> Companion code for *AI Agents in Depth*, Chapter 4 — **Experiment 4-4 ★★**. MCP server: browser automation, sub-agents, HITL, multi-channel notifications, timers.
> 配套《深入理解 AI Agent》第 4 章 **实验 4-4 ★★**。协作 MCP 服务器：浏览器、子 Agent、HITL、多渠道通知、定时器。
← [Chapter 4 index / 返回第 4 章目录](../README.md)
---
## English
A comprehensive Model Context Protocol (MCP) server that provides collaboration tools for AI agents, including browser automation, human-in-the-loop assistance, notifications, and timer management.
### Features
#### Browser Automation (using browser-use)
- Navigate to URLs and manage browser tabs
- Extract content from web pages
- Execute high-level browser tasks using AI agents
- Take screenshots
- Full virtual browser capabilities
#### Sub-Agent Management
- Spawn sub-agents in **sync** (wait for result) or **async** (returns a `task_id`) mode
- Send follow-up messages to a sub-agent and cancel a running one
- **Two context-passing strategies**, made inspectable (context text + token count):
- `minimal` — pass only the task plus an optional hand-picked slice (cheapest, private, may starve the sub-agent)
- `llm_generated` — one extra LLM call synthesizes a compact, privacy-filtered hand-off context from the parent trajectory
- Sub-agent system prompt uses labeled context sources (`[FROM_MAIN_AGENT]` / `[FROM_USER]` / `[TOOL_RESULT]`) and standardized JSON output
#### Human-in-the-Loop (HITL)
- Request admin approval for sensitive actions
- Request input from human administrators
- Manage pending approval requests
- Configurable timeout and notification channels
#### Email Notifications
- Send emails via SMTP or SendGrid
- Support for HTML emails
- CC recipients and attachments
- Flexible configuration
#### Instant Messaging
- Telegram bot integration
- Slack webhook support
- Discord webhook support
- Configurable default channels
#### Timer & Scheduling
- Set one-time timers
- Create recurring timers
- Cancel and manage timers
- Persistent timer storage
- Callback notifications when timers expire
### Installation
1. Install and activate the shared Chapter 4 environment from the repository root:
```bash
# From the repository root: use the shared Chapter 4 environment
uv sync --locked --python 3.12 --extra ch4
# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat
# pip fallback when uv is not installed:
# python -m pip install -e ".[ch4]"
cd chapter4/collaboration-tools
# Exact legacy parity path, including direct Playwright/pydantic-settings/scheduler pins:
# python -m pip install -r requirements.txt
```
2. Copy the example environment file and configure it:
```bash
cp env.example .env
# Edit .env with your configuration
```
3. Install Playwright browsers (for browser automation):
```bash
playwright install chromium
```
### Configuration
Configure the server by setting environment variables in `.env`:
#### Browser Settings
```env
BROWSER_HEADLESS=false
BROWSER_USER_DATA_DIR=~/.config/collaboration-tools/browser
```
#### Email Configuration
```env
# SMTP (Gmail example)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USERNAME=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SMTP_FROM_EMAIL=your-email@gmail.com
# Or use SendGrid
SENDGRID_API_KEY=your-sendgrid-api-key
```
#### Instant Messaging
```env
TELEGRAM_BOT_TOKEN=your-telegram-bot-token
TELEGRAM_DEFAULT_CHAT_ID=your-chat-id
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/YOUR/WEBHOOK
```
#### HITL Settings
```env
HITL_ADMIN_EMAIL=admin@example.com
HITL_TIMEOUT_SECONDS=3600
```
#### For Browser Tasks (AI Agent)
```env
OPENAI_API_KEY=your-openai-api-key
# Or use Alibaba Cloud Model Studio / Bailian (Qwen):
# COLLAB_PROVIDER=dashscope # qwen and bailian are aliases
# DASHSCOPE_API_KEY=your-dashscope-api-key
OPENAI_MODEL=gpt-5.6-luna
```
> **Universal OpenRouter fallback**: all LLM entry points (`spawn_subagent`,
> intelligence tools, browser-use) resolve credentials via `src/llm_fallback.py`.
> When `OPENAI_API_KEY` is absent but `OPENROUTER_API_KEY` is set, they route
> through OpenRouter (`base_url=https://openrouter.ai/api/v1`, model id mapped to
> `provider/model` form, e.g. `gpt-5.6-luna` → `openai/gpt-5.6-luna`). With neither
> key set, sub-agents run in deterministic offline mode (no fabricated output).
### Usage
#### CLI entry (`main.py`)
Without starting the MCP server, use the unified CLI to list tools, call them individually, or run end-to-end demos. Help text is Chinese; `-h` works on any subcommand:
```bash
python main.py --help # overview
python main.py list # list all collaboration tools (sub-agent / HITL / multi-channel notify)
python main.py demo # end-to-end collab demo: support agent handles a refund
python main.py subagent -h # sub-agent subcommand help
python main.py hitl -h # HITL subcommand help
python main.py notify -h # notify subcommand help
```
Common examples:
```bash
# Compare two context-passing strategies (minimal vs llm_generated)
python main.py subagent compare
# Spawn sub-agent (sync, minimal context)
python main.py subagent spawn --task "查询订单 A12345 状态" --strategy minimal --role 订单查询助手
# Sensitive decision needs admin approval; --auto-approve simulates admin reply offline
python main.py hitl approve --message "删除 1000 条记录？" --timeout 5 --auto-approve
# Multi-channel notification
python main.py notify slack --message "部署完成"
```
The formal Experiment 4-4 runner defaults to credential-free notification
preflights. Use `--interactive-human` to pause on a real pending MCP approval
and accept exactly one live `APPROVE` or `REJECT` line from standard input. Use
`--real-notifications` only when email, Telegram, and Slack are all configured;
the runner fails before creating a run directory if any channel is missing and
redacts credentials and delivery identifiers from retained receipts. The
context comparison deliberately retains a hard-coded, non-secret privacy canary
in its input receipt so the validator can prove that it is absent from both
prepared handoffs. `publication_authorized` records only whether MCP accepted a
live approval to publish that run's validation artifact; it does not imply that
the experiment passed or that `official_complete` is true.
```bash
python run_experiment_4_4.py \
--campaign-id real_mcp_human_example \
--interactive-human \
--human-timeout-seconds 14400
python validate_experiment_4_4.py \
validation/experiment_4_4/real_mcp_human_example
```
`demo` chains three collaboration tool types: (1) delegate a sub-agent for refund approval and compare context strategies; (2) large action triggers HITL (approve-before-timeout vs conservative default-on-timeout); (3) multi-channel notify collaborators. **HITL and notify paths run fully offline**; real sub-agent execution and `llm_generated` need `OPENAI_API_KEY` (if unset, the command still parses and runs with a clear prompt).
#### Running the MCP Server
Start the server using stdio transport:
```bash
python src/main.py
```
Or use it as an MCP server with any MCP-compatible client.
#### Quick Start Demo
Run the quickstart demo to see all features in action:
```bash
python quickstart.py
```
#### Sub-Agent Context Strategy Comparison
Spawn a sub-agent under **both** context-passing strategies on the same task and
print the difference (context tokens handed off, extra preparation cost, whether
private data leaked, and each sub-agent's result). Requires `OPENAI_API_KEY`
(default model `gpt-5.6-luna`, override with `OPENAI_MODEL`):
```bash
export OPENAI_API_KEY=your-openai-api-key
python subagent_comparison.py
```
Typically `minimal` uses far fewer tokens and never leaks private fields, but the
sub-agent may return `need_info`; `llm_generated` spends one extra LLM call to
hand off richer, privacy-filtered context so the sub-agent can complete the task.
#### Using with Claude Desktop
Add to your Claude Desktop configuration (`claude_desktop_config.json`):
```json
{
"mcpServers": {
"collaboration-tools": {
"command": "python",
"args": ["/path/to/collaboration-tools/src/main.py"],
"env": {
"OPENAI_API_KEY": "your-key-here"
}
}
}
}
```
### Available Tools
#### Browser Tools
- `mcp_browser_navigate` - Navigate to a URL
- `mcp_browser_get_content` - Get page content
- `mcp_browser_execute_task` - Execute AI-driven browser task
- `mcp_browser_screenshot` - Take a screenshot
- `mcp_browser_list_tabs` - List all open tabs
#### Notification Tools
- `mcp_send_email` - Send email notification
- `mcp_send_telegram_message` - Send Telegram message
- `mcp_send_slack_message` - Send Slack message
- `mcp_send_discord_message` - Send Discord message
#### Sub-Agent Tools
- `mcp_spawn_subagent` - Spawn a sub-agent (sync/async, `minimal`/`llm_generated` context)
- `mcp_send_message_to_subagent` - Send a follow-up message to a sub-agent
- `mcp_cancel_subagent` - Cancel a sub-agent
- `mcp_get_subagent_status` - Get a sub-agent's status/result (for async)
#### Human-in-the-Loop Tools
- `mcp_request_admin_approval` - Request admin approval
- `mcp_request_admin_input` - Request admin input
- `mcp_respond_to_request` - Respond to approval request (admin)
- `mcp_list_pending_requests` - List pending requests
#### Timer Tools
- `mcp_set_timer` - Set a one-time timer
- `mcp_set_recurring_timer` - Set a recurring timer
- `mcp_cancel_timer` - Cancel a timer
- `mcp_list_timers` - List all timers
- `mcp_get_timer_status` - Get timer status
### Example Usage
#### Browser Automation
```python
# Navigate to a website
await mcp_browser_navigate(url="https://example.com")
# Execute a complex task
await mcp_browser_execute_task(
task="Search for AI agent tutorials on Google and extract the top 5 results"
)
# Take a screenshot
await mcp_browser_screenshot(full_page=True)
```
#### Notifications
```python
# Send email
await mcp_send_email(
to_email="user@example.com",
subject="Task Completed",
body="Your task has finished successfully!"
)
# Send Slack message
await mcp_send_slack_message(
message="🎉 Deployment successful!"
)
```
#### Human-in-the-Loop
```python
# Request approval for sensitive action
result = await mcp_request_admin_approval(
request_message="Delete 1000 records from database?",
urgent=True,
timeout_seconds=300
)
if result["approved"]:
# Proceed with action
pass
```
#### Timers
```python
# Set a timer
await mcp_set_timer(
duration_seconds=300,
timer_name="Check website",
callback_message="Time to check the website status"
)
# Set recurring timer
await mcp_set_recurring_timer(
interval_seconds=3600,
max_occurrences=24,
timer_name="Hourly health check"
)
```
### Architecture
The server is organized into modular components:
```
collaboration-tools/
├── src/
│ ├── main.py # MCP server entry point
│ ├── config.py # Configuration management
│ ├── browser_tools.py # Browser automation
│ ├── notification_tools.py # Email & IM notifications
│ ├── hitl_tools.py # Human-in-the-loop
│ └── timer_tools.py # Timer management
├── requirements.txt # Python dependencies
├── env.example # Example configuration
└── README.md # This file
```
### Requirements
- Python 3.12 for the root `ch4` install (`browser-use` requires Python 3.11+)
- OpenAI API key (for browser AI agent tasks)
- Optional: Email/IM service credentials
- Playwright browsers for browser automation
### Troubleshooting
#### Browser Issues
If browser automation fails:
```bash
# Reinstall Playwright browsers
playwright install chromium --force
```
#### Email Issues
- For Gmail, use an [App Password](https://support.google.com/accounts/answer/185833)
- Ensure "Less secure app access" is NOT enabled (use App Passwords instead)
#### Telegram Issues
- Create a bot via [@BotFather](https://t.me/botfather)
- Get your chat ID from [@userinfobot](https://t.me/userinfobot)
#### LangChain/Pydantic Issues
If you see errors like "`ChatOpenAI` is not fully defined" or Pydantic validation errors:
- This is a known compatibility issue between LangChain and Pydantic v2
- The fix: ChatOpenAI is now initialized on-demand only when needed (in `browser_execute_task`)
- Simple browser navigation doesn't require OpenAI API key
- Only autonomous browser tasks (`browser_execute_task`) require `OPENAI_API_KEY`
### License
MIT License
### Contributing
Contributions are welcome! Please feel free to submit issues or pull requests.
---
## 中文
为 AI Agent 提供协作能力的综合 Model Context Protocol（MCP）服务器，涵盖浏览器自动化、人机协同、通知与定时器管理。
### 功能
#### 浏览器自动化（browser-use）
- 导航 URL、管理标签页
- 抽取网页内容
- 用 AI Agent 执行高层浏览器任务
- 截图
- 完整虚拟浏览器能力
#### 子 Agent 管理
- 以 **sync**（等待结果）或 **async**（返回 `task_id`）模式 spawn 子 Agent
- 向子 Agent 发送后续消息、取消运行中的子 Agent
- **两种上下文传递策略**（可检查上下文文本与 token 数）：
- `minimal` — 只传任务 + 可选手选片段（最省、隐私好，可能饿死子 Agent）
- `llm_generated` — 额外一次 LLM 调用，从父轨迹合成紧凑、隐私过滤的交接上下文
- 子 Agent system prompt 使用带标签的上下文来源（`[FROM_MAIN_AGENT]` / `[FROM_USER]` / `[TOOL_RESULT]`）与标准化 JSON 输出
#### 人机协同（HITL）
- 敏感操作请求管理员审批
- 向人类管理员请求输入
- 管理待处理审批
- 可配置超时与通知渠道
#### 邮件通知
- 经 SMTP 或 SendGrid 发信
- 支持 HTML
- 抄送与附件
- 灵活配置
#### 即时通讯
- Telegram bot
- Slack webhook
- Discord webhook
- 可配置默认频道
#### 定时器与调度
- 一次性定时器
- 循环定时器
- 取消与管理
- 持久化存储
- 到期回调通知
### 安装
1. 从仓库根目录安装并激活统一的第 4 章环境：
```bash
# 在仓库根目录使用统一的第 4 章环境
uv sync --locked --python 3.12 --extra ch4
# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat
# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch4]"
cd chapter4/collaboration-tools
# 精确复现旧版单项目环境，含直接 Playwright/pydantic-settings/scheduler 约束：
# python -m pip install -r requirements.txt
```
2. 复制环境模板并配置：
```bash
cp env.example .env
# Edit .env with your configuration
```
3. 安装 Playwright 浏览器（浏览器自动化）：
```bash
playwright install chromium
```
### 配置
在 `.env` 中设置环境变量：
#### 浏览器
```env
BROWSER_HEADLESS=false
BROWSER_USER_DATA_DIR=~/.config/collaboration-tools/browser
```
#### 邮件
```env
# SMTP (Gmail example)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USERNAME=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SMTP_FROM_EMAIL=your-email@gmail.com
# Or use SendGrid
SENDGRID_API_KEY=your-sendgrid-api-key
```
#### 即时通讯
```env
TELEGRAM_BOT_TOKEN=your-telegram-bot-token
TELEGRAM_DEFAULT_CHAT_ID=your-chat-id
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/YOUR/WEBHOOK
```
#### HITL
```env
HITL_ADMIN_EMAIL=admin@example.com
HITL_TIMEOUT_SECONDS=3600
```
#### 浏览器任务（AI Agent）
```env
OPENAI_API_KEY=your-openai-api-key
OPENAI_MODEL=gpt-5.6-luna
```
> **OpenRouter 通用兜底**：所有 LLM 入口（`spawn_subagent`、
> intelligence 工具、browser-use）经 `src/llm_fallback.py` 解析凭据。
> 未设置 `OPENAI_API_KEY` 但设置了 `OPENROUTER_API_KEY` 时，走
> OpenRouter（`base_url=https://openrouter.ai/api/v1`，模型 id 映射为
> `provider/model`，如 `gpt-5.6-luna` → `openai/gpt-5.6-luna`）。两者皆无时，
> 子 Agent 以确定性离线模式运行（不编造输出）。
### 使用
#### 命令行入口（`main.py`）
不启动 MCP 服务器，也可以用统一的命令行入口列出、单独调用协作工具，或运行端到端演示。
帮助信息为中文，`-h` 可查看任意子命令的参数：
```bash
python main.py --help # 总览
python main.py list # 列出全部协作工具（子 Agent / HITL / 多渠道通知）
python main.py demo # 端到端协作演示：客服协调 Agent 处理一笔退款
python main.py subagent -h # 子 Agent 子命令帮助
python main.py hitl -h # HITL 子命令帮助
python main.py notify -h # 通知子命令帮助
```
常用示例：
```bash
# 对比两种上下文传递策略（minimal vs llm_generated）
python main.py subagent compare
# 创建子 Agent（同步、最小化上下文）
python main.py subagent spawn --task "查询订单 A12345 状态" --strategy minimal --role 订单查询助手
# 关键决策请求管理员批准；--auto-approve 在后台模拟管理员应答，便于离线演示闭环
python main.py hitl approve --message "删除 1000 条记录？" --timeout 5 --auto-approve
# 多渠道通知
python main.py notify slack --message "部署完成"
```
`demo` 会串联三类协作工具：① 委派子 Agent 审批退款并对比上下文策略；② 大额操作
触发 HITL 审批（演示"超时前批准"与"超时保守默认"两种路径）；③ 向协作者多渠道通知结果。
其中 **HITL 与通知路径完全离线可跑**；子 Agent 的真实执行与 `llm_generated` 策略需要
`OPENAI_API_KEY`（未配置时会明确提示，命令仍可正常解析运行）。
#### 运行 MCP 服务器
使用 stdio 传输启动：
```bash
python src/main.py
```
也可作为 MCP 服务器接入任意兼容客户端。
#### 快速演示
```bash
python quickstart.py
```
#### 子 Agent 上下文策略对比
对同一任务分别用**两种**上下文传递策略 spawn，并打印差异（交接 token、额外准备成本、
是否泄漏隐私字段、各子 Agent 结果）。需要 `OPENAI_API_KEY`
（默认模型 `gpt-5.6-luna`，可用 `OPENAI_MODEL` 覆盖）：
```bash
export OPENAI_API_KEY=your-openai-api-key
python subagent_comparison.py
```
通常 `minimal` token 更少且不泄漏隐私字段，但子 Agent 可能返回 `need_info`；
`llm_generated` 多一次 LLM 调用交接更丰富、经隐私过滤的上下文，便于子 Agent 完成任务。
#### 与 Claude Desktop 联用
在 Claude Desktop 配置（`claude_desktop_config.json`）中加入：
```json
{
"mcpServers": {
"collaboration-tools": {
"command": "python",
"args": ["/path/to/collaboration-tools/src/main.py"],
"env": {
"OPENAI_API_KEY": "your-key-here"
}
}
}
}
```
### 可用工具
#### 浏览器工具
- `mcp_browser_navigate` — 导航到 URL
- `mcp_browser_get_content` — 获取页面内容
- `mcp_browser_execute_task` — 执行 AI 驱动的浏览器任务
- `mcp_browser_screenshot` — 截图
- `mcp_browser_list_tabs` — 列出标签页
#### 通知工具
- `mcp_send_email` — 发送邮件
- `mcp_send_telegram_message` — Telegram 消息
- `mcp_send_slack_message` — Slack 消息
- `mcp_send_discord_message` — Discord 消息
#### 子 Agent 工具
- `mcp_spawn_subagent` — 创建子 Agent（sync/async，`minimal`/`llm_generated` 上下文）
- `mcp_send_message_to_subagent` — 向子 Agent 发后续消息
- `mcp_cancel_subagent` — 取消子 Agent
- `mcp_get_subagent_status` — 查询状态/结果（async）
#### HITL 工具
- `mcp_request_admin_approval` — 请求管理员审批
- `mcp_request_admin_input` — 请求管理员输入
- `mcp_respond_to_request` — 响应审批请求（管理员侧）
- `mcp_list_pending_requests` — 列出待处理请求
#### 定时器工具
- `mcp_set_timer` — 一次性定时器
- `mcp_set_recurring_timer` — 循环定时器
- `mcp_cancel_timer` — 取消定时器
- `mcp_list_timers` — 列出定时器
- `mcp_get_timer_status` — 查询定时器状态
### 使用示例
#### 浏览器自动化
```python
# Navigate to a website
await mcp_browser_navigate(url="https://example.com")
# Execute a complex task
await mcp_browser_execute_task(
task="Search for AI agent tutorials on Google and extract the top 5 results"
)
# Take a screenshot
await mcp_browser_screenshot(full_page=True)
```
#### 通知
```python
# Send email
await mcp_send_email(
to_email="user@example.com",
subject="Task Completed",
body="Your task has finished successfully!"
)
# Send Slack message
await mcp_send_slack_message(
message="🎉 Deployment successful!"
)
```
#### 人机协同
```python
# Request approval for sensitive action
result = await mcp_request_admin_approval(
request_message="Delete 1000 records from database?",
urgent=True,
timeout_seconds=300
)
if result["approved"]:
# Proceed with action
pass
```
#### 定时器
```python
# Set a timer
await mcp_set_timer(
duration_seconds=300,
timer_name="Check website",
callback_message="Time to check the website status"
)
# Set recurring timer
await mcp_set_recurring_timer(
interval_seconds=3600,
max_occurrences=24,
timer_name="Hourly health check"
)
```
### 架构
服务器按模块组织：
```
collaboration-tools/
├── src/
│ ├── main.py # MCP server entry point
│ ├── config.py # Configuration management
│ ├── browser_tools.py # Browser automation
│ ├── notification_tools.py # Email & IM notifications
│ ├── hitl_tools.py # Human-in-the-loop
│ └── timer_tools.py # Timer management
├── requirements.txt # Python dependencies
├── env.example # Example configuration
└── README.md # This file
```
### 依赖要求
- 根目录 `ch4` 安装使用 Python 3.12（`browser-use` 要求 Python 3.11+）
- OpenAI API key（浏览器 AI 任务）
- 可选：邮件/IM 凭据
- Playwright 浏览器（浏览器自动化）
### 故障排除
#### 浏览器问题
若浏览器自动化失败：
```bash
# Reinstall Playwright browsers
playwright install chromium --force
```
#### 邮件问题
- Gmail 请使用 [应用专用密码](https://support.google.com/accounts/answer/185833)
- 不要开启「不够安全的应用访问」（改用应用专用密码）
#### Telegram 问题
- 通过 [@BotFather](https://t.me/botfather) 创建 bot
- 用 [@userinfobot](https://t.me/userinfobot) 获取 chat ID
#### LangChain/Pydantic 问题
若出现 "`ChatOpenAI` is not fully defined" 或 Pydantic 校验错误：
- 这是 LangChain 与 Pydantic v2 的已知兼容问题
- 修复：ChatOpenAI 仅在需要时按需初始化（`browser_execute_task`）
- 简单导航不需要 OpenAI API key
- 仅自主浏览器任务（`browser_execute_task`）需要 `OPENAI_API_KEY`
### 许可证
MIT License
### 贡献
欢迎提交 issue 或 pull request。
---
## Notes / 说明
- HITL + notify paths in `python main.py demo` run offline without API keys.
- `python main.py demo` 中 HITL 与通知路径可离线、无需 API Key。
- Browser AI tasks and `llm_generated` sub-agent strategy need an LLM key.
- 浏览器 AI 任务与 `llm_generated` 子 Agent 策略需要 LLM Key。
chapter4/collaboration-tools/USAGE_EXAMPLES.md
+426
View File
@@ -0,0 +1,426 @@
# Usage Examples
This document provides practical examples of using the Collaboration Tools MCP Server in various scenarios.
## Table of Contents
1. [Web Scraping with Notifications](#web-scraping-with-notifications)
2. [Scheduled Health Checks](#scheduled-health-checks)
3. [Admin Approval Workflow](#admin-approval-workflow)
4. [Multi-Channel Alerting](#multi-channel-alerting)
5. [Browser Automation Pipeline](#browser-automation-pipeline)
---
## Web Scraping with Notifications
Monitor a website and send alerts when specific content appears.
```python
async def monitor_for_keyword(agent, url, keyword, check_interval=3600):
"""Check website for keyword and alert if found."""
# Set up recurring check
timer = await agent.call_tool("mcp_set_recurring_timer", {
"interval_seconds": check_interval,
"timer_name": f"Monitor {keyword} on {url}",
"callback_message": f"Check {url} for {keyword}"
})
# Initial check
await agent.call_tool("mcp_browser_navigate", {"url": url})
content = await agent.call_tool("mcp_browser_get_content", {})
if keyword in content["content"]:
# Keyword found! Alert via multiple channels
await agent.call_tool("mcp_send_email", {
"to_email": "team@example.com",
"subject": f"🔍 Keyword '{keyword}' found on {url}",
"body": f"The keyword '{keyword}' was detected on {url}"
})
await agent.call_tool("mcp_send_slack_message", {
"message": f"🎯 Found '{keyword}' on {url}!"
})
# Take screenshot as evidence
await agent.call_tool("mcp_browser_screenshot", {
"full_page": True
})
```
---
## Scheduled Health Checks
Perform regular health checks with escalation.
```python
async def health_check_workflow(agent, service_url):
"""Monitor service health and escalate issues."""
# Check every 5 minutes
await agent.call_tool("mcp_set_recurring_timer", {
"interval_seconds": 300,
"timer_name": "Health Check",
"callback_message": "Perform health check"
})
# Navigate to health endpoint
result = await agent.call_tool("mcp_browser_navigate", {
"url": f"{service_url}/health"
})
if not result["success"]:
# Service down - escalate to admin
approval = await agent.call_tool("mcp_request_admin_approval", {
"request_message": f"Service {service_url} is down. Restart service?",
"context": {"service": service_url, "error": result["error"]},
"timeout_seconds": 300,
"urgent": True
})
if approval["approved"]:
# Admin approved restart
await agent.call_tool("mcp_send_telegram_message", {
"message": f"🔧 Restarting {service_url}..."
})
# ... perform restart ...
else:
# Notify team of ongoing issue
await agent.call_tool("mcp_send_email", {
"to_email": "oncall@example.com",
"subject": f"🚨 Service Down: {service_url}",
"body": "Service is down and restart was not approved."
})
```
---
## Admin Approval Workflow
Request human approval for sensitive operations.
```python
async def database_maintenance(agent):
"""Perform database maintenance with admin approval."""
# Step 1: Analyze database
print("Analyzing database...")
# ... analysis code ...
records_to_delete = 50000
# Step 2: Request approval
approval = await agent.call_tool("mcp_request_admin_approval", {
"request_message": f"Delete {records_to_delete} old records from database?",
"context": {
"operation": "delete",
"table": "logs",
"count": records_to_delete,
"estimated_time": "5 minutes"
},
"timeout_seconds": 600,
"urgent": False
})
if not approval["approved"]:
print("❌ Operation cancelled by admin")
return
# Step 3: Perform deletion with progress updates
await agent.call_tool("mcp_send_slack_message", {
"message": f"🗑️ Starting deletion of {records_to_delete} records..."
})
# Set timer to check progress
await agent.call_tool("mcp_set_timer", {
"duration_seconds": 300,
"timer_name": "Deletion timeout",
"callback_message": "Check if deletion completed"
})
# ... perform deletion ...
# Step 4: Notify completion
await agent.call_tool("mcp_send_email", {
"to_email": approval["admin_email"],
"subject": "✅ Database Maintenance Complete",
"body": f"Successfully deleted {records_to_delete} records.\n\n"
f"Notes: {approval['admin_notes']}"
})
```
---
## Multi-Channel Alerting
Send alerts across multiple communication channels.
```python
async def critical_alert(agent, title, message, severity="high"):
"""Send critical alert via all available channels."""
emoji = "🚨" if severity == "high" else "⚠️"
full_message = f"{emoji} {title}\n\n{message}"
# Send to all channels in parallel
tasks = []
# Email
tasks.append(agent.call_tool("mcp_send_email", {
"to_email": "alerts@example.com",
"subject": f"{emoji} {title}",
"body": message,
"cc": ["oncall@example.com"]
}))
# Slack
tasks.append(agent.call_tool("mcp_send_slack_message", {
"message": full_message,
"channel": "#alerts"
}))
# Telegram
tasks.append(agent.call_tool("mcp_send_telegram_message", {
"message": full_message,
"parse_mode": None
}))
# Discord
tasks.append(agent.call_tool("mcp_send_discord_message", {
"message": full_message
}))
# Wait for all to complete
results = await asyncio.gather(*tasks)
success_count = sum(1 for r in results if r.get("success"))
print(f"Alert sent via {success_count}/{len(tasks)} channels")
# If high severity and email/Slack failed, request admin intervention
if severity == "high" and success_count < 2:
await agent.call_tool("mcp_request_admin_approval", {
"request_message": "Alert delivery partially failed. Manual notification needed?",
"context": {"title": title, "channels_failed": len(tasks) - success_count},
"urgent": True
})
```
---
## Browser Automation Pipeline
Complex multi-step browser automation workflow.
```python
async def competitor_research(agent, competitor_url):
"""Research competitor and compile report."""
print("🔍 Starting competitor research...")
# Step 1: Navigate and take initial screenshot
await agent.call_tool("mcp_browser_navigate", {
"url": competitor_url
})
screenshot1 = await agent.call_tool("mcp_browser_screenshot", {
"full_page": True
})
# Step 2: Extract pricing information
print("📊 Extracting pricing...")
pricing_result = await agent.call_tool("mcp_browser_execute_task", {
"task": f"Go to {competitor_url} and extract all pricing plans with their features",
"max_steps": 30
})
# Step 3: Check their blog for recent posts
print("📝 Checking blog...")
await agent.call_tool("mcp_browser_execute_task", {
"task": "Find the blog and extract titles of the 5 most recent posts",
"max_steps": 20
})
blog_screenshot = await agent.call_tool("mcp_browser_screenshot", {
"full_page": False
})
# Step 4: Request admin review of findings
print("👤 Requesting admin review...")
review = await agent.call_tool("mcp_request_admin_input", {
"prompt": "Review competitor research findings. Any additional areas to investigate?",
"input_type": "text",
"timeout_seconds": 7200 # 2 hours
})
# Step 5: If admin provided additional areas, research them
if review["success"] and review["input"]:
print(f"🔍 Investigating additional area: {review['input']}")
await agent.call_tool("mcp_browser_execute_task", {
"task": f"Research: {review['input']}",
"max_steps": 25
})
# Step 6: Compile and send report
print("📧 Sending report...")
await agent.call_tool("mcp_send_email", {
"to_email": "team@example.com",
"subject": f"Competitor Research: {competitor_url}",
"body": f"""
Competitor Research Report
URL: {competitor_url}
Screenshots: {screenshot1['path']}, {blog_screenshot['path']}
Pricing Info:
{pricing_result['result']}
Admin Notes:
{review.get('input', 'None')}
""",
"html": False
})
# Schedule follow-up research in 30 days
await agent.call_tool("mcp_set_timer", {
"duration_seconds": 30 * 24 * 3600, # 30 days
"timer_name": f"Follow-up: {competitor_url}",
"callback_message": f"Time to re-check {competitor_url}"
})
print("✅ Research complete!")
```
---
## Delayed Task Execution
Use timers for delayed or scheduled operations.
```python
async def scheduled_report(agent, report_type, delay_hours=24):
"""Generate and send report after a delay."""
# Schedule report generation
timer = await agent.call_tool("mcp_set_timer", {
"duration_seconds": delay_hours * 3600,
"timer_name": f"{report_type} Report",
"callback_message": f"Generate {report_type} report",
"callback_data": {"report_type": report_type}
})
print(f"📅 Report scheduled for {delay_hours} hours from now")
print(f" Timer ID: {timer['timer_id']}")
# Send confirmation
await agent.call_tool("mcp_send_slack_message", {
"message": f"📊 {report_type} report scheduled for "
f"{delay_hours} hours from now\n"
f"Timer: {timer['timer_id']}"
})
return timer
async def recurring_backup_notification(agent):
"""Send backup reminders every week."""
await agent.call_tool("mcp_set_recurring_timer", {
"interval_seconds": 7 * 24 * 3600, # 1 week
"timer_name": "Weekly Backup Reminder",
"callback_message": "Time to verify backups!",
"max_occurrences": None # Run indefinitely
})
print("✅ Weekly backup reminder configured")
```
---
## Error Recovery Workflow
Handle errors with admin escalation.
```python
async def resilient_task(agent, task_description):
"""Execute task with automatic retry and admin escalation."""
max_retries = 3
retry_count = 0
while retry_count < max_retries:
try:
# Attempt task
result = await agent.call_tool("mcp_browser_execute_task", {
"task": task_description,
"max_steps": 30
})
if result["success"]:
# Success! Notify and return
await agent.call_tool("mcp_send_slack_message", {
"message": f"✅ Task completed: {task_description}"
})
return result
retry_count += 1
if retry_count < max_retries:
# Wait before retry
wait_seconds = 60 * retry_count
print(f"⏳ Retry {retry_count}/{max_retries} in {wait_seconds}s...")
await agent.call_tool("mcp_set_timer", {
"duration_seconds": wait_seconds,
"timer_name": f"Retry {retry_count}"
})
# Actual wait
await asyncio.sleep(wait_seconds)
except Exception as e:
print(f"❌ Error: {e}")
retry_count += 1
# All retries failed - escalate to admin
print("🚨 All retries failed, requesting admin assistance...")
admin_help = await agent.call_tool("mcp_request_admin_approval", {
"request_message": f"Task failed after {max_retries} retries. Manual intervention needed?",
"context": {
"task": task_description,
"retries": retry_count,
"last_error": str(result.get("error", "Unknown"))
},
"urgent": True,
"timeout_seconds": 1800
})
if admin_help["approved"]:
# Admin will handle manually
await agent.call_tool("mcp_send_email", {
"to_email": "admin@example.com",
"subject": "Task Requires Manual Intervention",
"body": f"Task: {task_description}\n"
f"Failed after {max_retries} retries\n"
f"Admin notes: {admin_help.get('admin_notes', 'None')}"
})
return None
```
---
## Tips for Effective Usage
1. **Combine Tools**: Use multiple tools together for powerful workflows
2. **Error Handling**: Always check `success` field in results
3. **Timeouts**: Set appropriate timeouts for HITL requests
4. **Notifications**: Use multiple channels for critical alerts
5. **Timers**: Leverage timers for retries and scheduled tasks
6. **Screenshots**: Take screenshots for audit trail
7. **Admin Context**: Provide rich context in HITL requests
---
For more examples, see `client_example.py` and `quickstart.py`.
chapter4/collaboration-tools/client_example.py
+205
View File
@@ -0,0 +1,205 @@
"""Example client showing how to use Collaboration Tools MCP Server.
This example demonstrates a real-world use case: monitoring a website
and notifying administrators when changes are detected.
"""
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from mcp.types import TextContent
import sys
from result_parsing import parse_mapping
class CollaborationAgent:
"""An AI agent that uses collaboration tools."""
def __init__(self):
self.session = None
async def connect(self):
"""Connect to the MCP server."""
server_params = StdioServerParameters(
command=sys.executable,
args=["src/main.py"]
)
print("🔌 Connecting to Collaboration Tools MCP Server...")
self.read, self.write = await stdio_client(server_params).__aenter__()
self.session = ClientSession(self.read, self.write)
await self.session.__aenter__()
await self.session.initialize()
print("✅ Connected successfully\n")
async def disconnect(self):
"""Disconnect from the server."""
if self.session:
await self.session.__aexit__(None, None, None)
print("\n📴 Disconnected from server")
async def call_tool(self, tool_name: str, arguments: dict):
"""Call a tool and return the result."""
result = await self.session.call_tool(tool_name, arguments)
text_content = [c.text for c in result.content if isinstance(c, TextContent)]
return parse_mapping(text_content[0]) if text_content else {}
async def monitor_website_workflow(self, url: str, check_interval: int = 300):
"""Monitor a website and notify on changes.
Args:
url: Website URL to monitor
check_interval: Check interval in seconds
"""
print(f"🔍 Starting website monitoring workflow for: {url}")
print(f" Check interval: {check_interval} seconds\n")
# Step 1: Set up recurring timer for checks
print("⏰ Setting up recurring monitoring timer...")
timer_result = await self.call_tool(
"mcp_set_recurring_timer",
{
"interval_seconds": check_interval,
"max_occurrences": 5, # Check 5 times for demo
"timer_name": f"Monitor {url}",
"callback_message": f"Time to check {url}"
}
)
if timer_result.get("success"):
print(f"✅ Timer set: {timer_result['timer_id']}")
timer_id = timer_result['timer_id']
else:
print(f"❌ Failed to set timer: {timer_result}")
return
# Step 2: Take initial screenshot
print("\n📸 Taking initial screenshot of the website...")
await self.call_tool("mcp_browser_navigate", {"url": url})
screenshot_result = await self.call_tool(
"mcp_browser_screenshot",
{"full_page": True}
)
if screenshot_result.get("success"):
initial_screenshot = screenshot_result['path']
print(f"✅ Screenshot saved: {initial_screenshot}")
else:
print(f"⚠️ Screenshot failed: {screenshot_result}")
initial_screenshot = None
# Step 3: Request admin approval for monitoring
print("\n👤 Requesting admin approval to continue monitoring...")
approval_result = await self.call_tool(
"mcp_request_admin_approval",
{
"request_message": f"Approve continuous monitoring of {url}?",
"context": {
"url": url,
"interval": check_interval,
"initial_screenshot": initial_screenshot
},
"timeout_seconds": 30, # Short timeout for demo
"urgent": False
}
)
if approval_result.get("approved"):
print("✅ Admin approved monitoring")
elif approval_result.get("timeout"):
print("⏱️ Admin approval timeout - proceeding anyway for demo")
else:
print("❌ Admin rejected monitoring - stopping")
await self.call_tool("mcp_cancel_timer", {"timer_id": timer_id})
return
# Step 4: Send notification that monitoring started
print("\n📧 Sending start notification...")
await self.call_tool(
"mcp_send_slack_message",
{
"message": f"🚀 Started monitoring {url}\nInterval: {check_interval}s",
"username": "Monitor Bot"
}
)
print("\n✨ Monitoring workflow initialized!")
print(f" Timer will check {url} every {check_interval} seconds")
print(f" Timer ID: {timer_id}")
# Step 5: Simulate monitoring loop
print("\n⏳ Monitoring in progress...")
print(" (In a real application, timer callbacks would trigger checks)")
# Wait a bit to show timer is active
await asyncio.sleep(10)
# Check timer status
status = await self.call_tool("mcp_get_timer_status", {"timer_id": timer_id})
print(f"\n📊 Timer status: {status.get('timer', {}).get('status')}")
# List all active timers
timers = await self.call_tool("mcp_list_timers", {"status": "active"})
print(f" Active timers: {timers.get('count', 0)}")
async def main():
"""Run the example client."""
print("=" * 70)
print("Collaboration Tools MCP Client Example")
print("Website Monitoring Workflow Demo")
print("=" * 70)
print()
agent = CollaborationAgent()
try:
await agent.connect()
# Run the monitoring workflow
await agent.monitor_website_workflow(
url="https://example.com",
check_interval=60 # Check every 60 seconds
)
# Additional examples
print("\n" + "=" * 70)
print("Additional Features Demo")
print("=" * 70)
# Example: Send email notification
print("\n📧 Sending email notification example...")
email_result = await agent.call_tool(
"mcp_send_email",
{
"to_email": "admin@example.com",
"subject": "Monitoring Report",
"body": "Website monitoring is active and running smoothly.",
"html": False
}
)
print(f" Result: {'✅ Sent' if email_result.get('success') else '⚠️ Not configured'}")
# Example: Request admin input
print("\n❓ Requesting admin input example...")
print(" (This would normally wait for admin response)")
print("\n✨ Demo complete!")
except Exception as e:
print(f"\n❌ Error: {e}")
import traceback
traceback.print_exc()
finally:
await agent.disconnect()
if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
print("\n\n⚠️ Interrupted by user")
chapter4/collaboration-tools/env.example
+52
View File
@@ -0,0 +1,52 @@
# LLM Configuration (for sub-agents, intelligence tools, and browser-use)
# Set COLLAB_PROVIDER=dashscope (or qwen/bailian) for Alibaba Cloud Model Studio.
# COLLAB_PROVIDER=dashscope
# DASHSCOPE_API_KEY=your-dashscope-api-key
# DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
# Direct OpenAI (preferred when set):
OPENAI_API_KEY=your-openai-api-key
# OPENAI_MODEL=gpt-5.6-luna
# OPENAI_BASE_URL=https://your-gateway/v1 # optional custom gateway
#
# Universal OpenRouter fallback: if OPENAI_API_KEY is absent but OPENROUTER_API_KEY
# is set, all LLM entry points (spawn_subagent, intelligence_tools, browser_tools)
# route through OpenRouter (base_url=https://openrouter.ai/api/v1) with the model
# id mapped to provider/model form (gpt-* -> openai/…). With neither key set, the
# sub-agent runs in deterministic offline mode.
# OPENROUTER_API_KEY=your-openrouter-api-key
# Browser Settings
BROWSER_HEADLESS=false
BROWSER_USER_DATA_DIR=~/.config/collaboration-tools/browser
# Email Configuration (SMTP)
SMTP_HOST=smtp.gmail.com
SMTP_PORT=587
SMTP_USERNAME=your-email@gmail.com
SMTP_PASSWORD=your-app-password
SMTP_FROM_EMAIL=your-email@gmail.com
SMTP_USE_TLS=true
# SendGrid (Alternative to SMTP)
SENDGRID_API_KEY=your-sendgrid-api-key
# Telegram Bot
TELEGRAM_BOT_TOKEN=your-telegram-bot-token
TELEGRAM_DEFAULT_CHAT_ID=your-default-chat-id
# Slack Webhook
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
# Discord Webhook
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/YOUR/WEBHOOK/URL
# Human-in-the-Loop Settings
HITL_ADMIN_EMAIL=admin@example.com
HITL_WEBHOOK_URL=http://localhost:8080/hitl
HITL_TIMEOUT_SECONDS=3600
# Timer Storage
TIMER_STORAGE_PATH=~/.config/collaboration-tools/timers.json
# Logging
LOG_LEVEL=INFO

Some files were not shown because too many files have changed in this diff Show More