ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,983 @@
|
||||
# Context-Aware AI Agent with Ablation Studies / 上下文感知 Agent 与消融实验
|
||||
|
||||
> Multi-provider context-aware agent with systematic ablation of context components (history, reasoning, tool calls, tool results).
|
||||
> 配套《深入理解 AI Agent》第 1 章 **实验 1-1 ★★：上下文的关键作用**。
|
||||
|
||||
← [Chapter 1 index / 返回第 1 章目录](../README.md) · 📖 [Read the chapter / 读本章正文](../../book/chapter1.md)（[EN](../../book-en/chapter1.md)）
|
||||
|
||||
---
|
||||
|
||||
## Code map
|
||||
|
||||
- **Run first:** python main.py --mode interactive (after provider setup).
|
||||
- **Start here:** main.py builds the selected provider and agent loop.
|
||||
- **Core behavior:** agent.py assembles history, reasoning, tool calls and tool results.
|
||||
- **State / protocol:** AgentTrajectory and the provider adapter messages.
|
||||
- **Verifier:** the ablation runner and tests under tests/; compare behavior, not just final text.
|
||||
- **Experiment variable:** context modes (full, no history, no reasoning, no tool calls, no tool results).
|
||||
- **Skip on first pass:** provider-specific clients, plotting and credential checks.
|
||||
|
||||
## English
|
||||
|
||||
### Overview
|
||||
|
||||
This project implements a context-aware AI agent with multiple tools (PDF parsing, currency conversion, calculator, code interpreter) and provides comprehensive ablation testing to explore how different context components affect agent behavior and performance. It supports multiple LLM providers, including Qwen directly through Alibaba Cloud Model Studio (Bailian), SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi, and DeepSeek.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Multi-provider Support**: Works with Alibaba Cloud Model Studio (Qwen), SiliconFlow (Qwen), Doubao (ByteDance), Kimi (Moonshot), and DeepSeek LLMs
|
||||
- **Multi-tool Agent**: PDF parsing, currency conversion, calculations, and Python code execution
|
||||
- **Context Modes**: Five different context configurations for ablation studies
|
||||
- **Interactive & Batch Modes**: Run single tasks or comprehensive test suites
|
||||
- **Conversation History**: Maintains context across multiple queries in a session
|
||||
- **Detailed Analytics**: Performance metrics, visualizations, and comprehensive reports
|
||||
|
||||
### Supported LLM Providers
|
||||
|
||||
#### Doubao (ByteDance) - Default
|
||||
|
||||
- **Model**: `doubao-seed-1-6-thinking-250715` (customizable)
|
||||
- **API**: OpenAI-compatible via Volcano Engine
|
||||
- **Best for**: Advanced reasoning, faster responses, both English and Chinese tasks
|
||||
|
||||
#### SiliconFlow
|
||||
|
||||
- **Model**: `Qwen/Qwen3.5-397B-A17B` (customizable)
|
||||
- **API**: OpenAI-compatible
|
||||
- **Best for**: Complex reasoning tasks, detailed analysis
|
||||
|
||||
#### Alibaba Cloud Model Studio / Bailian (Qwen)
|
||||
|
||||
- **Model**: `qwen3.7-plus` (customizable with `--model`)
|
||||
- **API**: Direct OpenAI-compatible DashScope endpoint; no SiliconFlow account required
|
||||
- **Provider names**: `dashscope` (canonical), with `qwen` and `bailian` aliases
|
||||
- **Region note**: API keys are region-bound. Mainland keys use the default endpoint; international keys must set `DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1`
|
||||
|
||||
#### Kimi (Moonshot AI)
|
||||
|
||||
- **Model**: `kimi-k3` (K3 reasoning model; temperature is forced to 1 and max_tokens is large enough for its thinking output)
|
||||
- **API**: OpenAI-compatible via Moonshot platform
|
||||
- **Best for**: Advanced reasoning, multi-turn conversations, both English and Chinese tasks
|
||||
- **Features**: Context caching for cost optimization
|
||||
|
||||
#### DeepSeek
|
||||
|
||||
- **Model**: `deepseek-v4-flash` (default; use `--model deepseek-v4-pro` for the stronger tier)
|
||||
- **API**: OpenAI-compatible via [DeepSeek Platform](https://platform.deepseek.com/)
|
||||
- **Best for**: Cost-effective tool-calling agents; thinking mode enabled so the `no_reasoning` ablation can strip `reasoning_content`
|
||||
- **Note**: Legacy aliases `deepseek-chat` / `deepseek-reasoner` are deprecated (2026-07-24); prefer the V4 ids
|
||||
|
||||
### Architecture
|
||||
|
||||
#### Context Components
|
||||
|
||||
1. **Full Context** — Complete agent with all components
|
||||
2. **No History** — Lacks historical tool call tracking
|
||||
3. **No Reasoning** — Operates without strategic planning
|
||||
4. **No Tool Calls** — Cannot execute external tools
|
||||
5. **No Tool Results** — Blind to tool execution outcomes
|
||||
|
||||
#### Available Tools
|
||||
|
||||
- **`parse_pdf(url)`** — Download and extract text from PDF documents
|
||||
- **`convert_currency(amount, from, to)`** — Real-time currency conversion
|
||||
- **`calculate(expression)`** — Simple mathematical expression evaluation
|
||||
- **`code_interpreter(code)`** — Execute Python code for complex calculations, totals, and data processing
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.10+
|
||||
- API key for one of the supported providers:
|
||||
- **Alibaba Cloud Model Studio / Bailian**: Get from [Model Studio](https://bailian.console.aliyun.com/)
|
||||
- **SiliconFlow**: Get from [SiliconFlow](https://siliconflow.cn)
|
||||
- **Doubao (ByteDance)**: Get from [Volcano Engine](https://www.volcengine.com/)
|
||||
- **Kimi (Moonshot)**: Get from [Moonshot Platform](https://platform.moonshot.cn/)
|
||||
- **DeepSeek**: Get from [DeepSeek Platform](https://platform.deepseek.com/api_keys)
|
||||
|
||||
### Sample Tasks
|
||||
|
||||
The system includes 5 pre-defined sample tasks demonstrating different capabilities:
|
||||
|
||||
1. **Simple Currency Conversion** — Basic multi-currency calculations
|
||||
2. **Multi-Currency Budget Analysis** — Complex expense analysis across offices
|
||||
3. **PDF Financial Analysis** — Parse and analyze financial documents
|
||||
4. **Investment Growth Calculation** — Compound interest with currency conversion
|
||||
5. **Comprehensive Financial Report** — Complete workflow using all tools
|
||||
|
||||
These samples are designed to showcase the agent's capabilities and the impact of context ablation.
|
||||
|
||||
### Quick Start
|
||||
|
||||
#### 1. Installation
|
||||
|
||||
```bash
|
||||
# Recommended from the repository root: use the shared Chapter 1 environment
|
||||
uv sync --locked --extra ch1
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch1]"
|
||||
|
||||
# Enter this experiment directory for the commands below
|
||||
cd chapter1/context
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
# Copy and configure environment
|
||||
cp env.example .env
|
||||
# Edit .env and add one provider key (for example DASHSCOPE_API_KEY or ARK_API_KEY)
|
||||
```
|
||||
|
||||
#### 2. Configure Provider
|
||||
|
||||
```bash
|
||||
# For Doubao (ByteDance) - Default
|
||||
export ARK_API_KEY=your_key_here
|
||||
python main.py # Uses Doubao by default
|
||||
|
||||
# For SiliconFlow (Qwen)
|
||||
export SILICONFLOW_API_KEY=your_key_here
|
||||
python main.py --provider siliconflow
|
||||
|
||||
# For Qwen directly through Alibaba Cloud Model Studio / Bailian
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope
|
||||
# --provider qwen and --provider bailian are equivalent aliases.
|
||||
# For an international-region key, also set:
|
||||
export DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
|
||||
# For Kimi (Moonshot)
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python main.py --provider kimi
|
||||
|
||||
# For DeepSeek
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python main.py --provider deepseek
|
||||
# Optional stronger model:
|
||||
python main.py --provider deepseek --model deepseek-v4-pro
|
||||
|
||||
# Or specify a custom model
|
||||
python main.py --model doubao-seed-1-6-thinking-250715
|
||||
|
||||
# Universal OpenRouter fallback: if the provider key above is missing/invalid
|
||||
# but OPENROUTER_API_KEY is set, requests are routed through OpenRouter and the
|
||||
# model id is mapped automatically (bare gpt-*/o1-* -> openai/*, claude-* ->
|
||||
# anthropic/*, deepseek-* -> deepseek/*, other native ids -> OPENROUTER_MODEL
|
||||
# or openai/gpt-5.6-luna).
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python main.py # falls back to OpenRouter when ARK_API_KEY is unset
|
||||
python main.py --provider openrouter # or use OpenRouter directly
|
||||
```
|
||||
|
||||
#### 3. Testing Qwen / Kimi / DeepSeek Integration
|
||||
|
||||
```bash
|
||||
# Run the ablation study directly on Alibaba Cloud Qwen
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope --mode ablation
|
||||
|
||||
# Quick test of Kimi K3 model
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python tests/manual/check_kimi.py
|
||||
|
||||
# Use Kimi in main script
|
||||
python main.py --provider kimi --mode interactive
|
||||
|
||||
# Run ablation study with Kimi
|
||||
python main.py --provider kimi --mode ablation
|
||||
|
||||
# Quick test of DeepSeek V4
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python tests/manual/check_deepseek.py
|
||||
# or: python tests/manual/check_deepseek_quick.py
|
||||
|
||||
# Use DeepSeek in main script / ablation study
|
||||
python main.py --provider deepseek --mode interactive
|
||||
python main.py --provider deepseek --mode ablation
|
||||
```
|
||||
|
||||
#### 4. Run Interactive Mode (Recommended)
|
||||
|
||||
```bash
|
||||
# Default (Doubao)
|
||||
python main.py --mode interactive
|
||||
|
||||
# With SiliconFlow provider
|
||||
python main.py --mode interactive --provider siliconflow
|
||||
|
||||
# In interactive mode, you can:
|
||||
# - Type 'samples' to see pre-defined tasks
|
||||
# - Type 'sample 2' to test PDF parsing
|
||||
# - Type 'providers' to list available providers
|
||||
# - Type 'provider kimi' to switch providers
|
||||
# - Type 'status' to see current configuration
|
||||
# - Type 'help' for all commands
|
||||
```
|
||||
|
||||
#### 5. Run Sample Tasks
|
||||
|
||||
```bash
|
||||
# Run without arguments to select from samples
|
||||
python main.py --mode single
|
||||
|
||||
# With specific provider
|
||||
python main.py --mode single --provider doubao
|
||||
|
||||
# Or provide your own task
|
||||
python main.py --mode single \
|
||||
--task "Convert $1000 USD to EUR, GBP, and JPY. Calculate the average." \
|
||||
--context-mode full \
|
||||
--provider siliconflow
|
||||
```
|
||||
|
||||
#### 6. Run Ablation Study
|
||||
|
||||
```bash
|
||||
# With default provider (single case, all five context modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# With Doubao provider
|
||||
python main.py --mode ablation --provider doubao
|
||||
|
||||
# Multi-case comparison across modes (stronger evidence for the book's point)
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# Compare only two modes and save raw results to a custom path
|
||||
python main.py --mode ablation --ablation-modes full no_history --output my_ablation.json
|
||||
```
|
||||
|
||||
`main.py` is the single CLI entry point. Run `python main.py --help` for the full (Chinese) flag reference.
|
||||
|
||||
Key flags:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--mode` | `single` / `ablation` / `interactive` (default) |
|
||||
| `--task` | Task text for `single` mode |
|
||||
| `--context-mode` | Context mode for `single` mode (`full`, `no_history`, `no_reasoning`, `no_tool_calls`, `no_tool_results`) |
|
||||
| `--ablation-modes` | Subset of modes to test in `ablation` mode (default: all five) |
|
||||
| `--cases` | Number of cases each mode is run against in `ablation` mode (default: 1) |
|
||||
| `--provider` / `--model` | LLM provider and optional model override |
|
||||
| `--output` | Output path for the JSON result (single) or raw results (ablation) |
|
||||
|
||||
### Ablation Studies
|
||||
|
||||
#### Accepted real Kimi K3 execution (2026-07-29)
|
||||
|
||||
`run_experiment_1_1.py` executes the exact five arms from the manuscript and
|
||||
persists every credential-free provider request/response, rather than only a
|
||||
summary table:
|
||||
|
||||
```bash
|
||||
python run_experiment_1_1.py --provider kimi --model kimi-k3 --max-iterations 5
|
||||
```
|
||||
|
||||
The same evidence runner can use a Bailian key directly:
|
||||
|
||||
```bash
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python run_experiment_1_1.py --provider dashscope --model qwen3.7-plus --max-iterations 5
|
||||
```
|
||||
|
||||
The accepted artifact is [validation/latest.json](validation/latest.json). All
|
||||
five request-shape contracts passed on the direct Moonshot API. The full arm
|
||||
produced the correct USD total and average; removing tool definitions produced
|
||||
zero tool actions; removing tool results and removing history both caused
|
||||
repeated actions. The no-reasoning arm still answered correctly in this run, so
|
||||
the manuscript's categorical contradiction claim was **not reproduced** and is
|
||||
reported separately from execution acceptance.
|
||||
|
||||
Observed results (these are not the expected-behavior labels below):
|
||||
|
||||
| Arm | Iterations | Tool actions | Repeated action | Correct numerical answer |
|
||||
|---|---:|---:|---|---|
|
||||
| full | 3 | 4 | no | yes |
|
||||
| no history | 5 (ceiling) | 15 | yes | no answer |
|
||||
| no reasoning | 3 | 4 | no | **yes — negative result for the manuscript claim** |
|
||||
| no tool definitions | 1 | 0 | no | no; the model explicitly declined to invent rates |
|
||||
| no tool results | 5 | 7 | yes | no; the model eventually reported that observations were hidden |
|
||||
|
||||
In an individual raw arm, `completed` means that the API/agent loop returned a
|
||||
terminal response. It does not mean that the requested task was correct. The
|
||||
legacy `success` field is retained as an alias for `completed` so older result
|
||||
readers continue to work; new readers should use `completed` explicitly.
|
||||
`task_success` is the task-specific correctness result. For this experiment it
|
||||
is computed by the canonical numeric rubric, while the generic agent cannot
|
||||
infer correctness from arbitrary natural-language prompts. The canonical
|
||||
behavioral booleans are under `analysis.manuscript_behavior_claims`;
|
||||
`all_manuscript_behavior_claims_observed` is false. This separation prevents a
|
||||
graceful refusal or hallucinated tool markup in an ablated arm from being
|
||||
mislabeled as task success, without forcing any ablation outcome in advance.
|
||||
|
||||
The ablation studies systematically remove context components to understand their importance.
|
||||
|
||||
#### Test Scenario
|
||||
|
||||
A complex financial analysis task requiring:
|
||||
|
||||
1. PDF document parsing
|
||||
2. Multiple currency conversions
|
||||
3. Mathematical calculations
|
||||
4. Result aggregation
|
||||
|
||||
#### Expected Behaviors
|
||||
|
||||
| Context Mode | Removed Component (book §实验 1.1) | Expected Behavior | Impact |
|
||||
|-------------|-----------------------------------|-------------------|---------|
|
||||
| **full** | none (baseline) | Complete successful execution | Baseline performance |
|
||||
| **no_history** | 历史消息 (history) | Redundant operations, inefficiency | May repeat tool calls |
|
||||
| **no_reasoning** | 思考过程 (reasoning) | Unstructured approach, potential errors | Lacks strategic planning |
|
||||
| **no_tool_calls** | 工具定义 (tool definitions) | Complete failure | Cannot interact with external world |
|
||||
| **no_tool_results** | 工具执行结果 (tool results) | Incorrect conclusions | Makes decisions without feedback |
|
||||
|
||||
**How each ablation is applied** (see `agent.py`):
|
||||
|
||||
- **no_tool_calls** — the `tools` parameter is omitted from the request, so the model has no tool definitions to call.
|
||||
- **no_tool_results** — every tool result is replaced with a `[Tool result hidden]` placeholder.
|
||||
- **no_reasoning** — `reasoning_content` is stripped from each assistant message before it is added back to the trajectory.
|
||||
- **no_history** — `_prepare_messages_for_api()` sends only a sliding window (system prompt + current task + the most recent ReAct step) to the model, so earlier steps are forgotten and the agent tends to repeat tool calls. Full mode always sends the complete trajectory.
|
||||
|
||||
#### Running Tests
|
||||
|
||||
```bash
|
||||
# Run the full ablation study (single case, all five modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# Run across multiple cases for a stronger comparison
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# This will generate:
|
||||
# - ablation_study_results.png (visualization, if matplotlib is installed)
|
||||
# - ablation_study_report.md (detailed report)
|
||||
# - ablation_results.json (raw data; override path with --output)
|
||||
```
|
||||
|
||||
The console prints two tables: a per-run **ablation study results** table and a **comparison matrix** (context mode x case) for reading the effect of each component at a glance.
|
||||
|
||||
#### Automated Regression Tests
|
||||
|
||||
```bash
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
Manual provider/API smoke scripts live under `tests/manual/` and require the corresponding API keys.
|
||||
|
||||
### Understanding Results
|
||||
|
||||
#### Performance Metrics
|
||||
|
||||
- **Terminal Response Rate**: Whether the agent returned a terminal response
|
||||
- **Task Success**: Correctness under the task-specific rubric (when one is available)
|
||||
- **Execution Time**: Total time to complete the task
|
||||
- **Iterations**: Number of agent-model interactions
|
||||
- **Tool Calls**: Number of external tool invocations
|
||||
- **Reasoning Steps**: Strategic planning iterations
|
||||
|
||||
#### Sample Output
|
||||
|
||||
```
|
||||
ABLATION STUDY RESULTS
|
||||
================================================================================
|
||||
| Test Name | Success | Time | Iterations | Tool Calls |
|
||||
|--------------------------------|---------|--------|------------|------------|
|
||||
| Baseline - Full Context | ✓ | 12.3s | 5 | 8 |
|
||||
| No Historical Tool Calls | ✓ | 18.7s | 8 | 12 |
|
||||
| No Reasoning Process | ✗ | 25.4s | 10 | 15 |
|
||||
| No Tool Call Commands | ✗ | 3.2s | 2 | 0 |
|
||||
| No Tool Call Results | ✗ | 15.6s | 10 | 10 |
|
||||
```
|
||||
|
||||
### Key Insights
|
||||
|
||||
1. **Tool Calls Are Fundamental** — Without tool call capability, the agent cannot interact with external systems, making task completion impossible.
|
||||
2. **Tool Results Provide Critical Feedback** — Without seeing results, the agent operates blind, leading to incorrect conclusions and infinite loops.
|
||||
3. **Reasoning Enables Efficiency** — Strategic planning reduces iterations and tool calls, improving both speed and accuracy.
|
||||
4. **History Prevents Redundancy** — Historical context prevents repeated operations and maintains task coherence across iterations.
|
||||
|
||||
### Advanced Usage
|
||||
|
||||
#### Interactive Mode Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `samples` | Display all available sample tasks |
|
||||
| `sample <n>` | Run sample task number n |
|
||||
| `providers` | List all available LLM providers |
|
||||
| `provider <name>` | Switch to a different provider (e.g., `provider kimi`) |
|
||||
| `modes` | List available context modes for ablation testing |
|
||||
| `mode <name>` | Switch context mode (e.g., `mode no_history`) |
|
||||
| `status` | Show current configuration (provider, model, mode, etc.) |
|
||||
| `reset` | Reset agent trajectory (clear history) |
|
||||
| `create_pdfs` | Generate sample PDF files for testing |
|
||||
| `quit` | Exit interactive mode |
|
||||
|
||||
**Note:** The prompt shows the current provider in brackets, e.g., `[KIMI]>` or `[DOUBAO]>`
|
||||
|
||||
#### Conversation History
|
||||
|
||||
The agent maintains conversation history throughout interactive sessions:
|
||||
|
||||
- **Persistent Context**: The agent remembers previous queries and responses within a session
|
||||
- **Multi-turn Conversations**: You can reference information from earlier in the conversation
|
||||
- **Tool Call Memory**: Previous tool executions are remembered and can be referenced
|
||||
- **Reset on Demand**: Use the `reset` command to clear history and start fresh
|
||||
|
||||
Example conversation flow:
|
||||
|
||||
```
|
||||
[DOUBAO]> Remember that our budget is $10,000. Calculate 15% of it.
|
||||
# Agent calculates and remembers the budget
|
||||
|
||||
[DOUBAO]> Now convert that 15% amount to EUR
|
||||
# Agent uses the previously calculated amount without re-asking
|
||||
|
||||
[DOUBAO]> What was our original budget?
|
||||
# Agent recalls the $10,000 mentioned earlier
|
||||
```
|
||||
|
||||
#### Custom Tasks
|
||||
|
||||
```python
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
result = agent.execute_task("""
|
||||
Download the PDF from https://example.com/report.pdf,
|
||||
extract all monetary values, convert them to EUR,
|
||||
and calculate the total.
|
||||
""")
|
||||
```
|
||||
|
||||
#### Creating Test PDFs
|
||||
|
||||
```bash
|
||||
python create_sample_pdf.py
|
||||
# Creates fixtures/pdfs/ with sample financial reports
|
||||
```
|
||||
|
||||
#### Configuration
|
||||
|
||||
Edit `config.py` or set environment variables:
|
||||
|
||||
```bash
|
||||
export MODEL_TEMPERATURE=0.5
|
||||
export MAX_ITERATIONS=15
|
||||
export LOG_LEVEL=DEBUG
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
context/
|
||||
├── README.md # This file
|
||||
├── main.py # Single CLI entry point (single / ablation / interactive)
|
||||
├── agent.py # Core agent implementation + context modes
|
||||
├── config.py # Configuration management
|
||||
├── create_sample_pdf.py # PDF generation utility
|
||||
├── fixtures/
|
||||
│ └── pdfs/ # Sample PDFs used by local demos/tests
|
||||
├── tests/
|
||||
│ ├── test_agent.py
|
||||
│ ├── test_code_interpreter.py
|
||||
│ ├── test_malformed_tool_json.py
|
||||
│ └── manual/ # Provider/API smoke scripts; require real keys
|
||||
├── requirements.txt # Dependencies
|
||||
└── env.example # Environment template
|
||||
```
|
||||
|
||||
> Note: the ablation study lives in `main.py` (`AblationTestSuite`), run via `python main.py --mode ablation`. There is no separate `ablation_tests.py`.
|
||||
|
||||
### Research Applications
|
||||
|
||||
- **AI Safety Research**: Understanding failure modes
|
||||
- **System Design**: Identifying critical components
|
||||
- **Optimization**: Finding minimal viable configurations
|
||||
- **Education**: Teaching agent architecture principles
|
||||
|
||||
### Limitations
|
||||
|
||||
- Currency rates are fixed (production should use real-time APIs)
|
||||
- PDF parsing may fail on complex layouts
|
||||
- Model token limits may affect very large documents
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 概述
|
||||
|
||||
本项目实现一个上下文感知 AI Agent，配备多种工具（PDF 解析、货币换算、计算器、代码解释器），并通过系统化的**消融实验**（Ablation Study）检验不同上下文组件对 Agent 行为与性能的影响。支持通过阿里云百炼直连 Qwen，也支持 SiliconFlow Qwen、字节跳动 Doubao、月之暗面 Kimi、DeepSeek。对应书中**实验 1-1 ★★：上下文的关键作用**。
|
||||
|
||||
### 主要特性
|
||||
|
||||
- **多提供商支持**：阿里云百炼（Qwen 直连）、SiliconFlow（Qwen）、Doubao（字节）、Kimi（月之暗面）、DeepSeek
|
||||
- **多工具 Agent**：PDF 解析、货币换算、计算与 Python 代码执行
|
||||
- **上下文模式**：五种配置，用于消融对照
|
||||
- **交互与批处理**：单任务运行或完整测试套件
|
||||
- **对话历史**：同一会话内跨多轮查询保持上下文
|
||||
- **详细分析**：性能指标、可视化与综合报告
|
||||
|
||||
### 支持的 LLM 提供商
|
||||
|
||||
#### Doubao（字节跳动）— 默认
|
||||
|
||||
- **模型**：`doubao-seed-1-6-thinking-250715`（可自定义）
|
||||
- **API**：火山引擎上的 OpenAI 兼容接口
|
||||
- **适合**：深度推理、较快响应，中英文任务均可
|
||||
|
||||
#### SiliconFlow
|
||||
|
||||
- **模型**：`Qwen/Qwen3.5-397B-A17B`（可自定义）
|
||||
- **API**：OpenAI 兼容
|
||||
- **适合**：复杂推理与细致分析
|
||||
|
||||
#### 阿里云百炼（Qwen 直连）
|
||||
|
||||
- **模型**：`qwen3.7-plus`（可通过 `--model` 自定义）
|
||||
- **API**：直连 DashScope 的 OpenAI 兼容接口，无需 SiliconFlow 账号
|
||||
- **提供商名称**：规范名称为 `dashscope`，也可使用别名 `qwen` 或 `bailian`
|
||||
- **区域说明**：API Key 与区域绑定。中国内地 Key 默认直连内地端点；国际站 Key 必须设置 `DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1`
|
||||
|
||||
#### Kimi（月之暗面）
|
||||
|
||||
- **模型**：`kimi-k3`（K3 推理模型；temperature 强制为 1，max_tokens 足够容纳思考输出）
|
||||
- **API**：Moonshot 平台 OpenAI 兼容接口
|
||||
- **适合**：深度推理、多轮对话，中英文任务均可
|
||||
- **特性**：上下文缓存以优化成本
|
||||
|
||||
#### DeepSeek
|
||||
|
||||
- **模型**：`deepseek-v4-flash`（默认；更强档可用 `--model deepseek-v4-pro`）
|
||||
- **API**：[DeepSeek Platform](https://platform.deepseek.com/) 的 OpenAI 兼容接口
|
||||
- **适合**：性价比高的工具调用；开启 thinking，便于 `no_reasoning` 消融剥离 `reasoning_content`
|
||||
- **说明**：旧别名 `deepseek-chat` / `deepseek-reasoner` 已弃用（2026-07-24），请优先使用 V4 id
|
||||
|
||||
### 架构
|
||||
|
||||
#### 上下文组件
|
||||
|
||||
1. **Full Context** — 完整 Agent，保留全部组件
|
||||
2. **No History** — 缺少历史工具调用追踪
|
||||
3. **No Reasoning** — 无战略规划/思考过程
|
||||
4. **No Tool Calls** — 无法执行外部工具
|
||||
5. **No Tool Results** — 看不到工具执行结果
|
||||
|
||||
#### 可用工具
|
||||
|
||||
- **`parse_pdf(url)`** — 下载并抽取 PDF 文本
|
||||
- **`convert_currency(amount, from, to)`** — 货币换算
|
||||
- **`calculate(expression)`** — 简单数学表达式求值
|
||||
- **`code_interpreter(code)`** — 执行 Python，用于复杂计算、汇总与数据处理
|
||||
|
||||
### 前置条件
|
||||
|
||||
- Python 3.10+
|
||||
- 任一支持提供商的 API Key：
|
||||
- **阿里云百炼**：[百炼控制台](https://bailian.console.aliyun.com/)
|
||||
- **SiliconFlow**：[SiliconFlow](https://siliconflow.cn)
|
||||
- **Doubao（字节）**：[火山引擎](https://www.volcengine.com/)
|
||||
- **Kimi（月之暗面）**：[Moonshot Platform](https://platform.moonshot.cn/)
|
||||
- **DeepSeek**：[DeepSeek Platform](https://platform.deepseek.com/api_keys)
|
||||
|
||||
### 示例任务
|
||||
|
||||
系统预置 5 个样例任务：
|
||||
|
||||
1. **简单货币换算** — 基础多币种计算
|
||||
2. **多币种预算分析** — 跨办公室费用分析
|
||||
3. **PDF 财务分析** — 解析并分析财务文档
|
||||
4. **投资增长计算** — 复利与货币换算
|
||||
5. **综合财务报告** — 串联全部工具的完整流程
|
||||
|
||||
用于展示 Agent 能力与上下文消融的影响。
|
||||
|
||||
### 快速开始
|
||||
|
||||
#### 1. 安装
|
||||
|
||||
```bash
|
||||
# 推荐在仓库根目录使用统一的第 1 章环境
|
||||
uv sync --locked --extra ch1
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch1]"
|
||||
|
||||
# 进入本实验目录，后续命令都在这里运行
|
||||
cd chapter1/context
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
# 复制并配置环境变量
|
||||
cp env.example .env
|
||||
# 编辑 .env 并填入一个提供商的 API Key（例如 DASHSCOPE_API_KEY 或 ARK_API_KEY）
|
||||
```
|
||||
|
||||
#### 2. 配置提供商
|
||||
|
||||
```bash
|
||||
# For Doubao (ByteDance) - Default
|
||||
export ARK_API_KEY=your_key_here
|
||||
python main.py # Uses Doubao by default
|
||||
|
||||
# For SiliconFlow (Qwen)
|
||||
export SILICONFLOW_API_KEY=your_key_here
|
||||
python main.py --provider siliconflow
|
||||
|
||||
# 通过阿里云百炼直连 Qwen
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope
|
||||
# --provider qwen 与 --provider bailian 是等价别名。
|
||||
# 如果使用国际站 Key，还需设置：
|
||||
export DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
|
||||
# For Kimi (Moonshot)
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python main.py --provider kimi
|
||||
|
||||
# For DeepSeek
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python main.py --provider deepseek
|
||||
# Optional stronger model:
|
||||
python main.py --provider deepseek --model deepseek-v4-pro
|
||||
|
||||
# Or specify a custom model
|
||||
python main.py --model doubao-seed-1-6-thinking-250715
|
||||
|
||||
# Universal OpenRouter fallback: if the provider key above is missing/invalid
|
||||
# but OPENROUTER_API_KEY is set, requests are routed through OpenRouter and the
|
||||
# model id is mapped automatically (bare gpt-*/o1-* -> openai/*, claude-* ->
|
||||
# anthropic/*, deepseek-* -> deepseek/*, other native ids -> OPENROUTER_MODEL
|
||||
# or openai/gpt-5.6-luna).
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python main.py # falls back to OpenRouter when ARK_API_KEY is unset
|
||||
python main.py --provider openrouter # or use OpenRouter directly
|
||||
```
|
||||
|
||||
#### 3. 测试 Qwen / Kimi / DeepSeek 集成
|
||||
|
||||
```bash
|
||||
# 通过阿里云百炼 Qwen 直接运行消融实验
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope --mode ablation
|
||||
|
||||
# Quick test of Kimi K3 model
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python tests/manual/check_kimi.py
|
||||
|
||||
# Use Kimi in main script
|
||||
python main.py --provider kimi --mode interactive
|
||||
|
||||
# Run ablation study with Kimi
|
||||
python main.py --provider kimi --mode ablation
|
||||
|
||||
# Quick test of DeepSeek V4
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python tests/manual/check_deepseek.py
|
||||
# or: python tests/manual/check_deepseek_quick.py
|
||||
|
||||
# Use DeepSeek in main script / ablation study
|
||||
python main.py --provider deepseek --mode interactive
|
||||
python main.py --provider deepseek --mode ablation
|
||||
```
|
||||
|
||||
#### 4. 交互模式（推荐）
|
||||
|
||||
```bash
|
||||
# Default (Doubao)
|
||||
python main.py --mode interactive
|
||||
|
||||
# With SiliconFlow provider
|
||||
python main.py --mode interactive --provider siliconflow
|
||||
|
||||
# In interactive mode, you can:
|
||||
# - Type 'samples' to see pre-defined tasks
|
||||
# - Type 'sample 2' to test PDF parsing
|
||||
# - Type 'providers' to list available providers
|
||||
# - Type 'provider kimi' to switch providers
|
||||
# - Type 'status' to see current configuration
|
||||
# - Type 'help' for all commands
|
||||
```
|
||||
|
||||
#### 5. 运行样例任务
|
||||
|
||||
```bash
|
||||
# Run without arguments to select from samples
|
||||
python main.py --mode single
|
||||
|
||||
# With specific provider
|
||||
python main.py --mode single --provider doubao
|
||||
|
||||
# Or provide your own task
|
||||
python main.py --mode single \
|
||||
--task "Convert $1000 USD to EUR, GBP, and JPY. Calculate the average." \
|
||||
--context-mode full \
|
||||
--provider siliconflow
|
||||
```
|
||||
|
||||
#### 6. 运行消融实验
|
||||
|
||||
```bash
|
||||
# With default provider (single case, all five context modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# With Doubao provider
|
||||
python main.py --mode ablation --provider doubao
|
||||
|
||||
# Multi-case comparison across modes (stronger evidence for the book's point)
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# Compare only two modes and save raw results to a custom path
|
||||
python main.py --mode ablation --ablation-modes full no_history --output my_ablation.json
|
||||
```
|
||||
|
||||
`main.py` 是唯一 CLI 入口。运行 `python main.py --help` 查看完整（中文）参数说明。
|
||||
|
||||
关键参数：
|
||||
|
||||
| 参数 | 说明 |
|
||||
|------|------|
|
||||
| `--mode` | `single` / `ablation` / `interactive`（默认） |
|
||||
| `--task` | `single` 模式的任务文本 |
|
||||
| `--context-mode` | `single` 模式的上下文模式（`full`、`no_history`、`no_reasoning`、`no_tool_calls`、`no_tool_results`） |
|
||||
| `--ablation-modes` | `ablation` 模式下要测的模式子集（默认全部五种） |
|
||||
| `--cases` | `ablation` 模式下每种模式跑的用例数（默认 1） |
|
||||
| `--provider` / `--model` | LLM 提供商与可选模型覆盖 |
|
||||
| `--output` | 单次结果或消融原始结果的 JSON 输出路径 |
|
||||
|
||||
### 消融实验
|
||||
|
||||
#### 已验收的 Kimi K3 真实执行（2026-07-29）
|
||||
|
||||
`run_experiment_1_1.py` 会按正文运行五个精确实验组，并保存每轮真实 API 的无凭据
|
||||
请求与响应，而不只是汇总表：
|
||||
|
||||
```bash
|
||||
python run_experiment_1_1.py --provider kimi --model kimi-k3 --max-iterations 5
|
||||
```
|
||||
|
||||
同一证据运行器也支持直接使用百炼 Key：
|
||||
|
||||
```bash
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python run_experiment_1_1.py --provider dashscope --model qwen3.7-plus --max-iterations 5
|
||||
```
|
||||
|
||||
验收产物见 [validation/latest.json](validation/latest.json)。五组上下文契约全部通过；
|
||||
完整组算出了正确结果，移除工具定义后没有工具行动，移除工具结果或历史后都出现重复行动。
|
||||
但本次“移除思考过程”仍得到正确答案，因此正文关于必然出现矛盾决策的断言**没有复现**；
|
||||
产物把“实验执行通过”和“正文行为结论复现”分开记录。
|
||||
|
||||
系统性地移除上下文组件，以理解其重要性。
|
||||
|
||||
#### 测试场景
|
||||
|
||||
需要以下能力的复杂财务分析任务：
|
||||
|
||||
1. PDF 文档解析
|
||||
2. 多次货币换算
|
||||
3. 数学计算
|
||||
4. 结果汇总
|
||||
|
||||
#### 预期行为
|
||||
|
||||
| 上下文模式 | 移除组件（书中 §实验 1.1） | 预期行为 | 影响 |
|
||||
|-------------|---------------------------|----------|------|
|
||||
| **full** | 无（基线） | 完整成功执行 | 基线性能 |
|
||||
| **no_history** | 历史消息 (history) | 冗余操作、效率下降 | 可能重复调用工具 |
|
||||
| **no_reasoning** | 思考过程 (reasoning) | 方法无结构、易出错 | 缺少战略规划 |
|
||||
| **no_tool_calls** | 工具定义 (tool definitions) | 完全失败 | 无法与外部世界交互 |
|
||||
| **no_tool_results** | 工具执行结果 (tool results) | 错误结论 | 无反馈下做决策 |
|
||||
|
||||
**各消融如何落地**（见 `agent.py`）：
|
||||
|
||||
- **no_tool_calls** — 请求中省略 `tools` 参数，模型没有可调用的工具定义。
|
||||
- **no_tool_results** — 每个工具结果替换为 `[Tool result hidden]` 占位符。
|
||||
- **no_reasoning** — 写回轨迹前，从每条 assistant 消息中剥离 `reasoning_content`。
|
||||
- **no_history** — `_prepare_messages_for_api()` 只发送滑动窗口（系统提示 + 当前任务 + 最近一步 ReAct），早期步骤被遗忘，易重复调工具。完整模式始终发送完整轨迹。
|
||||
|
||||
#### 运行测试
|
||||
|
||||
```bash
|
||||
# Run the full ablation study (single case, all five modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# Run across multiple cases for a stronger comparison
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# This will generate:
|
||||
# - ablation_study_results.png (visualization, if matplotlib is installed)
|
||||
# - ablation_study_report.md (detailed report)
|
||||
# - ablation_results.json (raw data; override path with --output)
|
||||
```
|
||||
|
||||
控制台会打印两张表：逐次运行的 **ablation study results**，以及 **comparison matrix**（上下文模式 × 用例），便于一眼对比各组件的作用。
|
||||
|
||||
#### 自动化回归测试
|
||||
|
||||
```bash
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
需要真实 API Key 的手动提供商/API 冒烟脚本放在 `tests/manual/`。
|
||||
|
||||
### 结果解读
|
||||
|
||||
#### 性能指标
|
||||
|
||||
- **Terminal Response Rate**：Agent 是否返回了终止响应
|
||||
- **Task Success**：在存在任务专用评分标准时，任务是否正确完成
|
||||
- **Execution Time**：完成任务总耗时
|
||||
- **Iterations**：Agent 与模型交互次数
|
||||
- **Tool Calls**：外部工具调用次数
|
||||
- **Reasoning Steps**：战略规划迭代次数
|
||||
|
||||
#### 输出示例
|
||||
|
||||
```
|
||||
ABLATION STUDY RESULTS
|
||||
================================================================================
|
||||
| Test Name | Success | Time | Iterations | Tool Calls |
|
||||
|--------------------------------|---------|--------|------------|------------|
|
||||
| Baseline - Full Context | ✓ | 12.3s | 5 | 8 |
|
||||
| No Historical Tool Calls | ✓ | 18.7s | 8 | 12 |
|
||||
| No Reasoning Process | ✗ | 25.4s | 10 | 15 |
|
||||
| No Tool Call Commands | ✗ | 3.2s | 2 | 0 |
|
||||
| No Tool Call Results | ✗ | 15.6s | 10 | 10 |
|
||||
```
|
||||
|
||||
### 关键洞察
|
||||
|
||||
1. **工具调用是基础** — 没有工具调用能力，Agent 无法与外部系统交互，任务无法完成。
|
||||
2. **工具结果提供关键反馈** — 看不到结果等于盲目行动，易导致错误结论与死循环。
|
||||
3. **推理提升效率** — 战略规划减少迭代与工具调用，兼顾速度与准确。
|
||||
4. **历史避免冗余** — 历史上下文防止重复操作，并在多轮中保持任务连贯。
|
||||
|
||||
### 进阶用法
|
||||
|
||||
#### 交互模式命令
|
||||
|
||||
| 命令 | 说明 |
|
||||
|------|------|
|
||||
| `samples` | 显示全部样例任务 |
|
||||
| `sample <n>` | 运行第 n 个样例任务 |
|
||||
| `providers` | 列出可用 LLM 提供商 |
|
||||
| `provider <name>` | 切换提供商（如 `provider kimi`） |
|
||||
| `modes` | 列出可用于消融的上下文模式 |
|
||||
| `mode <name>` | 切换上下文模式（如 `mode no_history`） |
|
||||
| `status` | 显示当前配置（提供商、模型、模式等） |
|
||||
| `reset` | 重置 Agent 轨迹（清空历史） |
|
||||
| `create_pdfs` | 生成测试用样例 PDF |
|
||||
| `quit` | 退出交互模式 |
|
||||
|
||||
**说明：** 提示符会以括号显示当前提供商，如 `[KIMI]>` 或 `[DOUBAO]>`
|
||||
|
||||
#### 对话历史
|
||||
|
||||
交互会话中 Agent 会维护对话历史：
|
||||
|
||||
- **持久上下文**：会话内记住先前查询与回复
|
||||
- **多轮对话**：可引用更早提到的信息
|
||||
- **工具调用记忆**：先前工具执行结果可被引用
|
||||
- **按需重置**：使用 `reset` 清空历史重新开始
|
||||
|
||||
示例对话流程：
|
||||
|
||||
```
|
||||
[DOUBAO]> Remember that our budget is $10,000. Calculate 15% of it.
|
||||
# Agent calculates and remembers the budget
|
||||
|
||||
[DOUBAO]> Now convert that 15% amount to EUR
|
||||
# Agent uses the previously calculated amount without re-asking
|
||||
|
||||
[DOUBAO]> What was our original budget?
|
||||
# Agent recalls the $10,000 mentioned earlier
|
||||
```
|
||||
|
||||
#### 自定义任务
|
||||
|
||||
```python
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
result = agent.execute_task("""
|
||||
Download the PDF from https://example.com/report.pdf,
|
||||
extract all monetary values, convert them to EUR,
|
||||
and calculate the total.
|
||||
""")
|
||||
```
|
||||
|
||||
#### 生成测试 PDF
|
||||
|
||||
```bash
|
||||
python create_sample_pdf.py
|
||||
# Creates fixtures/pdfs/ with sample financial reports
|
||||
```
|
||||
|
||||
#### 配置
|
||||
|
||||
编辑 `config.py` 或设置环境变量：
|
||||
|
||||
```bash
|
||||
export MODEL_TEMPERATURE=0.5
|
||||
export MAX_ITERATIONS=15
|
||||
export LOG_LEVEL=DEBUG
|
||||
```
|
||||
|
||||
### 项目结构
|
||||
|
||||
```
|
||||
context/
|
||||
├── README.md # 本文件
|
||||
├── main.py # 单一 CLI 入口（single / ablation / interactive）
|
||||
├── agent.py # Core agent implementation + context modes
|
||||
├── config.py # Configuration management
|
||||
├── create_sample_pdf.py # PDF generation utility
|
||||
├── fixtures/
|
||||
│ └── pdfs/ # 本地 demo/tests 使用的样例 PDF
|
||||
├── tests/
|
||||
│ ├── test_agent.py
|
||||
│ ├── test_code_interpreter.py
|
||||
│ ├── test_malformed_tool_json.py
|
||||
│ └── manual/ # 需真实 Key 的提供商/API 冒烟脚本
|
||||
├── requirements.txt # Dependencies
|
||||
└── env.example # Environment template
|
||||
```
|
||||
|
||||
> 说明：消融实验逻辑在 `main.py` 的 `AblationTestSuite` 中，通过 `python main.py --mode ablation` 运行，没有单独的 `ablation_tests.py`。
|
||||
|
||||
### 研究用途
|
||||
|
||||
- **AI 安全研究**：理解失败模式
|
||||
- **系统设计**：识别关键组件
|
||||
- **优化**：寻找最小可用配置
|
||||
- **教学**：讲解 Agent 架构原理
|
||||
|
||||
### 局限
|
||||
|
||||
- 货币汇率为固定值（生产环境应使用实时 API）
|
||||
- 复杂版式 PDF 解析可能失败
|
||||
- 模型 token 上限可能影响超大文档
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Educational project for context ablation; for production, add proper error handling, rate limiting, and security.
|
||||
本项目为教学向消融实验；生产使用请补齐错误处理、限流与安全措施。
|
||||
- OpenRouter is a universal fallback when the direct provider key is missing.
|
||||
未配置直连提供商 Key 时，可走 `OPENROUTER_API_KEY` 通用兜底。
|
||||
- License: MIT. Contributions welcome (extra tools, scenarios, ablation strategies, performance).
|
||||
许可证：MIT。欢迎贡献（更多工具、场景、消融策略、性能优化）。
|
||||
@@ -0,0 +1,966 @@
|
||||
"""
|
||||
Context-Aware AI Agent with Tool Calls
|
||||
An agent using Qwen model from SiliconFlow with document parsing, currency conversion, and calculator tools.
|
||||
Designed to demonstrate the importance of context through ablation studies.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
import PyPDF2
|
||||
from io import BytesIO
|
||||
import math
|
||||
from datetime import datetime
|
||||
from concurrent.futures import TimeoutError
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _reasoning_safe_temperature(model, requested=1.0):
|
||||
"""Reasoning models (Kimi K3, GPT-5, ...) only accept temperature=1.
|
||||
Return 1 for those; otherwise the requested value so non-reasoning
|
||||
providers (Doubao, DeepSeek, older Moonshot) are unchanged."""
|
||||
m = str(model or "").lower().replace("/", "-")
|
||||
return 1 if ("kimi-k3" in m or "gpt-5" in m) else requested
|
||||
|
||||
|
||||
class ContextMode(Enum):
|
||||
"""Different context modes for ablation studies"""
|
||||
FULL = "full" # Complete context with all components
|
||||
NO_HISTORY = "no_history" # No historical tool calls
|
||||
NO_REASONING = "no_reasoning" # No reasoning/thinking process
|
||||
NO_TOOL_CALLS = "no_tool_calls" # No tool call commands
|
||||
NO_TOOL_RESULTS = "no_tool_results" # No tool call results
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolCall:
|
||||
"""Represents a single tool call"""
|
||||
tool_name: str
|
||||
arguments: Dict[str, Any]
|
||||
result: Optional[Any] = None
|
||||
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentTrajectory:
|
||||
"""Tracks the agent's execution trajectory"""
|
||||
reasoning_steps: List[str] = field(default_factory=list)
|
||||
tool_calls: List[ToolCall] = field(default_factory=list)
|
||||
# Exact, credential-free request/response evidence for every real model
|
||||
# turn. This is deliberately part of the trajectory: Experiment 1-1 is
|
||||
# about what the model could see at decision time, so reconstructing the
|
||||
# request after the fact is not acceptable evidence.
|
||||
api_turns: List[Dict[str, Any]] = field(default_factory=list)
|
||||
context_mode: ContextMode = ContextMode.FULL
|
||||
|
||||
|
||||
class ToolRegistry:
|
||||
"""Registry for available tools"""
|
||||
|
||||
@staticmethod
|
||||
def parse_pdf(url: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Download and parse a PDF from URL or local file
|
||||
|
||||
Args:
|
||||
url: URL or file path of the PDF to parse
|
||||
|
||||
Returns:
|
||||
Dictionary containing parsed text and metadata
|
||||
"""
|
||||
try:
|
||||
# Check if it's a local file
|
||||
if url.startswith('file://'):
|
||||
# Extract the file path from file:// URL
|
||||
file_path = url.replace('file://', '')
|
||||
logger.info(f"Reading local PDF from {file_path}")
|
||||
|
||||
# Read the file directly
|
||||
with open(file_path, 'rb') as f:
|
||||
pdf_content = f.read()
|
||||
|
||||
elif url.startswith('/') or url.startswith('./') or url.startswith('../') or ':\\' in url or ':/' in url[1:3]:
|
||||
# Direct file path (absolute or relative)
|
||||
logger.info(f"Reading local PDF from {url}")
|
||||
|
||||
# Read the file directly
|
||||
with open(url, 'rb') as f:
|
||||
pdf_content = f.read()
|
||||
|
||||
else:
|
||||
# It's a remote URL, download it
|
||||
logger.info(f"Downloading PDF from {url}")
|
||||
response = requests.get(url, timeout=30)
|
||||
response.raise_for_status()
|
||||
pdf_content = response.content
|
||||
|
||||
# Parse the PDF content
|
||||
pdf_file = BytesIO(pdf_content)
|
||||
pdf_reader = PyPDF2.PdfReader(pdf_file)
|
||||
|
||||
text_content = []
|
||||
for page_num, page in enumerate(pdf_reader.pages, 1):
|
||||
text = page.extract_text()
|
||||
text_content.append({
|
||||
"page": page_num,
|
||||
"text": text
|
||||
})
|
||||
|
||||
result = {
|
||||
"url": url,
|
||||
"num_pages": len(pdf_reader.pages),
|
||||
"content": text_content,
|
||||
"metadata": pdf_reader.metadata if hasattr(pdf_reader, 'metadata') else {}
|
||||
}
|
||||
|
||||
logger.info(f"Successfully parsed PDF with {len(pdf_reader.pages)} pages")
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error parsing PDF: {str(e)}")
|
||||
return {"error": str(e)}
|
||||
|
||||
@staticmethod
|
||||
def convert_currency(amount: float, from_currency: str, to_currency: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert currency using live exchange rates
|
||||
|
||||
Args:
|
||||
amount: Amount to convert
|
||||
from_currency: Source currency code (e.g., 'USD')
|
||||
to_currency: Target currency code (e.g., 'EUR')
|
||||
|
||||
Returns:
|
||||
Dictionary with conversion result
|
||||
"""
|
||||
try:
|
||||
if isinstance(amount, str):
|
||||
clean_amt = amount.replace(",", "").strip()
|
||||
symbols_to_strip = sorted(
|
||||
[
|
||||
"USD$", "U.S.$", "US$", "$",
|
||||
"SGD$", "SG$", "S$",
|
||||
"AUD$", "AU$", "A$",
|
||||
"CAD$", "CA$", "C$",
|
||||
"€", "£", "₹",
|
||||
],
|
||||
key=len,
|
||||
reverse=True,
|
||||
)
|
||||
for sym in symbols_to_strip:
|
||||
clean_amt = clean_amt.replace(sym, "")
|
||||
amount = float(clean_amt.strip())
|
||||
else:
|
||||
amount = float(amount)
|
||||
exchange_rates = {
|
||||
"USD": 1.0,
|
||||
"EUR": 0.92,
|
||||
"GBP": 0.79,
|
||||
"JPY": 149.50,
|
||||
"CNY": 7.24,
|
||||
"CAD": 1.36,
|
||||
"AUD": 1.53,
|
||||
"CHF": 0.88,
|
||||
"INR": 83.12,
|
||||
"SGD": 1.34
|
||||
}
|
||||
|
||||
def _normalize_code(code: str) -> str:
|
||||
if not isinstance(code, str):
|
||||
return str(code or "")
|
||||
c = code.strip().upper()
|
||||
symbols = {
|
||||
"$": "USD",
|
||||
"US$": "USD",
|
||||
"U.S.$": "USD",
|
||||
"USD$": "USD",
|
||||
"S$": "SGD",
|
||||
"SG$": "SGD",
|
||||
"SGD$": "SGD",
|
||||
"A$": "AUD",
|
||||
"AU$": "AUD",
|
||||
"AUD$": "AUD",
|
||||
"C$": "CAD",
|
||||
"CA$": "CAD",
|
||||
"CAD$": "CAD",
|
||||
"€": "EUR",
|
||||
"£": "GBP",
|
||||
"₹": "INR",
|
||||
}
|
||||
if c in symbols:
|
||||
return symbols[c]
|
||||
if c.endswith("$"):
|
||||
prefix = c[:-1].strip()
|
||||
if prefix in exchange_rates:
|
||||
return prefix
|
||||
if prefix in ("US", "U.S."):
|
||||
return "USD"
|
||||
if prefix in ("AU", "A"):
|
||||
return "AUD"
|
||||
if prefix in ("CA", "C"):
|
||||
return "CAD"
|
||||
return c
|
||||
|
||||
from_currency = _normalize_code(from_currency)
|
||||
to_currency = _normalize_code(to_currency)
|
||||
|
||||
logger.info(f"Converting {amount} {from_currency} to {to_currency}")
|
||||
|
||||
if from_currency not in exchange_rates or to_currency not in exchange_rates:
|
||||
return {"error": f"Unsupported currency: {from_currency} or {to_currency}"}
|
||||
|
||||
# Convert to USD first, then to target currency
|
||||
usd_amount = amount / exchange_rates[from_currency]
|
||||
converted_amount = usd_amount * exchange_rates[to_currency]
|
||||
|
||||
result = {
|
||||
"original_amount": amount,
|
||||
"from_currency": from_currency,
|
||||
"to_currency": to_currency,
|
||||
"converted_amount": round(converted_amount, 2),
|
||||
"exchange_rate": round(exchange_rates[to_currency] / exchange_rates[from_currency], 4),
|
||||
"timestamp": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
logger.info(f"Conversion result: {result['converted_amount']} {to_currency}")
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error converting currency: {str(e)}")
|
||||
return {"error": str(e)}
|
||||
|
||||
@staticmethod
|
||||
def calculate(expression: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Evaluate a mathematical expression
|
||||
|
||||
Args:
|
||||
expression: Mathematical expression to evaluate
|
||||
|
||||
Returns:
|
||||
Dictionary with calculation result
|
||||
"""
|
||||
try:
|
||||
logger.info(f"Calculating: {expression}")
|
||||
|
||||
# Sanitize expression - only allow safe mathematical operations
|
||||
allowed_names = {
|
||||
k: v for k, v in math.__dict__.items() if not k.startswith("__")
|
||||
}
|
||||
allowed_names.update({"abs": abs, "round": round, "min": min, "max": max})
|
||||
|
||||
# Replace common operations for clarity
|
||||
expression = expression.replace("^", "**")
|
||||
|
||||
# Evaluate the expression
|
||||
result = eval(expression, {"__builtins__": {}}, allowed_names)
|
||||
|
||||
return {
|
||||
"expression": expression,
|
||||
"result": result,
|
||||
"type": type(result).__name__
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error calculating expression: {str(e)}")
|
||||
return {"error": str(e)}
|
||||
|
||||
@staticmethod
|
||||
def code_interpreter(code: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute Python code for complex calculations and data processing
|
||||
|
||||
Args:
|
||||
code: Python code to execute
|
||||
|
||||
Returns:
|
||||
Dictionary with execution results and any output
|
||||
"""
|
||||
try:
|
||||
logger.info(f"Executing Python code: {code[:100]}...")
|
||||
|
||||
# Create a restricted namespace with safe built-ins
|
||||
safe_namespace = {
|
||||
'__builtins__': {
|
||||
'abs': abs,
|
||||
'all': all,
|
||||
'any': any,
|
||||
'sum': sum,
|
||||
'min': min,
|
||||
'max': max,
|
||||
'round': round,
|
||||
'len': len,
|
||||
'list': list,
|
||||
'dict': dict,
|
||||
'set': set,
|
||||
'tuple': tuple,
|
||||
'enumerate': enumerate,
|
||||
'zip': zip,
|
||||
'map': map,
|
||||
'filter': filter,
|
||||
'sorted': sorted,
|
||||
'reversed': reversed,
|
||||
'range': range,
|
||||
'int': int,
|
||||
'float': float,
|
||||
'str': str,
|
||||
'bool': bool,
|
||||
'print': print,
|
||||
}
|
||||
}
|
||||
|
||||
# Add math module
|
||||
safe_namespace['math'] = math
|
||||
|
||||
# Capture printed output
|
||||
import io
|
||||
import contextlib
|
||||
|
||||
output_buffer = io.StringIO()
|
||||
|
||||
with contextlib.redirect_stdout(output_buffer):
|
||||
# Execute the code
|
||||
exec(code, safe_namespace)
|
||||
|
||||
# Get printed output
|
||||
printed_output = output_buffer.getvalue()
|
||||
|
||||
# Try to extract a result if it's assigned to 'result' variable
|
||||
result = safe_namespace.get('result', None)
|
||||
|
||||
# Also check for common variable names
|
||||
if result is None:
|
||||
for var_name in ['total', 'sum', 'output', 'answer', 'final']:
|
||||
if var_name in safe_namespace:
|
||||
result = safe_namespace[var_name]
|
||||
break
|
||||
|
||||
# Get all variables defined (excluding built-ins and modules)
|
||||
variables = {
|
||||
k: v for k, v in safe_namespace.items()
|
||||
if not k.startswith('__') and k not in ['math'] and not callable(v)
|
||||
}
|
||||
|
||||
return {
|
||||
"code": code,
|
||||
"result": result,
|
||||
"output": printed_output if printed_output else None,
|
||||
"variables": variables,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error executing code: {str(e)}")
|
||||
return {
|
||||
"code": code,
|
||||
"error": str(e),
|
||||
"success": False
|
||||
}
|
||||
|
||||
|
||||
class ContextAwareAgent:
|
||||
"""
|
||||
AI Agent with configurable LLM providers and context modes for ablation studies
|
||||
"""
|
||||
|
||||
def __init__(self, api_key: str, context_mode: ContextMode = ContextMode.FULL,
|
||||
provider: str = "siliconflow", model: Optional[str] = None,
|
||||
verbose: bool = True):
|
||||
"""
|
||||
Initialize the agent
|
||||
|
||||
Args:
|
||||
api_key: API key for the LLM provider
|
||||
context_mode: Context mode for ablation studies
|
||||
provider: Any provider registered in ``agentbook.providers`` (for
|
||||
example ``dashscope``/``qwen``, ``siliconflow``, ``doubao``,
|
||||
``kimi``, ``deepseek``, or ``openrouter``)
|
||||
model: Optional model override
|
||||
verbose: If True, log full HTTP requests and responses (default: True)
|
||||
"""
|
||||
self.provider = provider.lower()
|
||||
self.verbose = verbose
|
||||
|
||||
# Base URLs, default models and key lookup all live in the shared
|
||||
# registry (agentbook/providers.py), so adding a provider there makes it
|
||||
# usable here with no change. resolve_backend also applies the universal
|
||||
# OpenRouter fallback: when the provider's own key is missing but
|
||||
# OPENROUTER_API_KEY is set, the request routes through OpenRouter with a
|
||||
# mapped model id. Behaviour is unchanged when the provider key is set.
|
||||
from config import resolve_backend
|
||||
backend = resolve_backend(self.provider, model=model, api_key=api_key)
|
||||
resolved_key = backend.api_key
|
||||
resolved_base_url = backend.base_url
|
||||
self.model = backend.model
|
||||
self.using_openrouter = backend.using_openrouter
|
||||
if self.using_openrouter:
|
||||
logger.info(
|
||||
f"{self.provider} API key not set; routing via OpenRouter "
|
||||
f"(model: {self.model})"
|
||||
)
|
||||
self.client = OpenAI(
|
||||
api_key=resolved_key,
|
||||
base_url=resolved_base_url
|
||||
)
|
||||
self.base_url = resolved_base_url
|
||||
|
||||
self.context_mode = context_mode
|
||||
self.trajectory = AgentTrajectory(context_mode=context_mode)
|
||||
self.tools = ToolRegistry()
|
||||
|
||||
# Initialize conversation history
|
||||
self.conversation_history = []
|
||||
self._init_system_prompt()
|
||||
|
||||
logger.info(f"Agent initialized with provider: {self.provider}, model: {self.model}, context mode: {context_mode.value}, verbose: {self.verbose}")
|
||||
|
||||
def _init_system_prompt(self):
|
||||
"""Initialize the system prompt for the conversation"""
|
||||
self.conversation_history = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": """You are an intelligent assistant with access to tools.
|
||||
|
||||
Your task is to solve the given problems using the available tools. Think step by step and use tools as needed.
|
||||
|
||||
Important: When you have gathered all necessary information and computed the final answer, clearly state "FINAL ANSWER:" followed by your answer."""
|
||||
}
|
||||
]
|
||||
|
||||
def _get_tools_description(self) -> List[Dict[str, Any]]:
|
||||
"""Get tool descriptions for the model"""
|
||||
return [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "parse_pdf",
|
||||
"description": "Download and parse a PDF document from a URL or a file path to extract text content",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"url": {
|
||||
"type": "string",
|
||||
"description": "The URL or file path of the PDF document to parse"
|
||||
}
|
||||
},
|
||||
"required": ["url"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "convert_currency",
|
||||
"description": "Convert an amount from one currency to another using current exchange rates",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"amount": {
|
||||
"type": "number",
|
||||
"description": "The amount to convert"
|
||||
},
|
||||
"from_currency": {
|
||||
"type": "string",
|
||||
"description": "The source currency code (e.g., USD, EUR)"
|
||||
},
|
||||
"to_currency": {
|
||||
"type": "string",
|
||||
"description": "The target currency code (e.g., USD, EUR)"
|
||||
}
|
||||
},
|
||||
"required": ["amount", "from_currency", "to_currency"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "calculate",
|
||||
"description": "Evaluate a simple mathematical expression",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"expression": {
|
||||
"type": "string",
|
||||
"description": "The mathematical expression to evaluate (e.g., '2 + 2 * 3')"
|
||||
}
|
||||
},
|
||||
"required": ["expression"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "code_interpreter",
|
||||
"description": "Execute Python code for complex calculations, data processing, and computing totals. Use this for tasks like: summing lists of values, calculating percentages, aggregating financial data, performing multi-step calculations, or any computation requiring variables and intermediate steps.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "Python code to execute. Can use variables, loops, and mathematical operations. Example: 'amounts = [2500000, 2278481, 2541806, 2282609, 2388060]; total = sum(amounts); print(f\"Total: ${total:,.2f}\")"
|
||||
}
|
||||
},
|
||||
"required": ["code"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
def _prepare_assistant_message(self, message) -> Dict[str, Any]:
|
||||
"""
|
||||
Prepare assistant message for adding to messages list,
|
||||
filtering out reasoning_content if in NO_REASONING mode
|
||||
|
||||
Args:
|
||||
message: The assistant message object
|
||||
|
||||
Returns:
|
||||
Dictionary representation of the message
|
||||
"""
|
||||
msg_dict = message.dict() if hasattr(message, 'dict') else message.model_dump()
|
||||
|
||||
# Remove reasoning_content if in NO_REASONING mode
|
||||
if self.context_mode == ContextMode.NO_REASONING and 'reasoning_content' in msg_dict:
|
||||
msg_dict.pop('reasoning_content')
|
||||
|
||||
return msg_dict
|
||||
|
||||
@staticmethod
|
||||
def _reasoning_content(message) -> Optional[str]:
|
||||
"""Return provider reasoning text without assuming one SDK shape."""
|
||||
value = getattr(message, "reasoning_content", None)
|
||||
if value:
|
||||
return str(value)
|
||||
extra = getattr(message, "model_extra", None) or {}
|
||||
value = extra.get("reasoning_content") or extra.get("reasoning")
|
||||
if isinstance(value, dict):
|
||||
value = value.get("content") or value.get("text")
|
||||
return str(value) if value else None
|
||||
|
||||
@staticmethod
|
||||
def _json_snapshot(value: Any) -> Any:
|
||||
"""Detach an API evidence object from later in-memory mutations."""
|
||||
return json.loads(json.dumps(value, ensure_ascii=False, default=str))
|
||||
|
||||
def _build_context(self) -> str:
|
||||
"""
|
||||
Build a human-readable summary of the trajectory (legacy helper, kept
|
||||
for inspection/debugging only).
|
||||
|
||||
NOTE: The message list sent to the model is assembled by
|
||||
``_prepare_messages_for_api`` -- that is where the NO_HISTORY ablation
|
||||
actually takes effect. This method is not part of the request path.
|
||||
|
||||
Returns:
|
||||
Context string for the model
|
||||
"""
|
||||
context_parts = []
|
||||
|
||||
# Add reasoning steps if not disabled
|
||||
if self.context_mode != ContextMode.NO_REASONING and self.trajectory.reasoning_steps:
|
||||
context_parts.append("## Previous Reasoning Steps:")
|
||||
for step in self.trajectory.reasoning_steps:
|
||||
context_parts.append(f"- {step}")
|
||||
context_parts.append("")
|
||||
|
||||
# Add tool call history if not disabled
|
||||
if self.context_mode not in [ContextMode.NO_HISTORY, ContextMode.NO_TOOL_CALLS] and self.trajectory.tool_calls:
|
||||
context_parts.append("## Tool Call History:")
|
||||
for call in self.trajectory.tool_calls:
|
||||
if self.context_mode != ContextMode.NO_TOOL_CALLS:
|
||||
context_parts.append(f"- Called {call.tool_name} with args: {json.dumps(call.arguments)}")
|
||||
if self.context_mode != ContextMode.NO_TOOL_RESULTS and call.result:
|
||||
context_parts.append(f" Result: {json.dumps(call.result, indent=2)}")
|
||||
context_parts.append("")
|
||||
|
||||
return "\n".join(context_parts) if context_parts else ""
|
||||
|
||||
def _log_request_response(self, request_data: Dict[str, Any], response_data: Any, iteration: int):
|
||||
"""
|
||||
Log full request and response when in verbose mode
|
||||
|
||||
Args:
|
||||
request_data: The request payload sent to the API
|
||||
response_data: The response received from the API
|
||||
iteration: Current iteration number
|
||||
"""
|
||||
if not self.verbose:
|
||||
return
|
||||
|
||||
if request_data:
|
||||
print("\n" + "="*80)
|
||||
print(f"📤 ITERATION {iteration} - FULL REQUEST JSON:")
|
||||
print("-"*80)
|
||||
print(json.dumps(request_data, indent=2, ensure_ascii=False))
|
||||
|
||||
if response_data:
|
||||
print("\n" + "="*80)
|
||||
print(f"📥 ITERATION {iteration} - FULL RESPONSE:")
|
||||
print("-"*80)
|
||||
|
||||
# Convert response to dict for display
|
||||
if hasattr(response_data, 'model_dump'):
|
||||
response_dict = response_data.model_dump()
|
||||
elif hasattr(response_data, 'dict'):
|
||||
response_dict = response_data.dict()
|
||||
else:
|
||||
response_dict = {"raw_response": str(response_data)}
|
||||
|
||||
print(json.dumps(response_dict, indent=2, ensure_ascii=False))
|
||||
print("="*80 + "\n")
|
||||
|
||||
def _execute_tool(self, tool_name: str, arguments: Dict[str, Any]) -> Any:
|
||||
"""
|
||||
Execute a tool and return the result
|
||||
|
||||
Args:
|
||||
tool_name: Name of the tool to execute
|
||||
arguments: Arguments for the tool
|
||||
|
||||
Returns:
|
||||
Tool execution result
|
||||
"""
|
||||
tool_map = {
|
||||
"parse_pdf": self.tools.parse_pdf,
|
||||
"convert_currency": self.tools.convert_currency,
|
||||
"calculate": self.tools.calculate,
|
||||
"code_interpreter": self.tools.code_interpreter
|
||||
}
|
||||
|
||||
if tool_name not in tool_map:
|
||||
return {"error": f"Unknown tool: {tool_name}"}
|
||||
|
||||
return tool_map[tool_name](**arguments)
|
||||
|
||||
def _prepare_messages_for_api(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Build the message list actually sent to the model for the current
|
||||
iteration, applying the NO_HISTORY ablation.
|
||||
|
||||
For every mode except NO_HISTORY the full conversation history (the
|
||||
accumulated trajectory) is returned unchanged. For NO_HISTORY the
|
||||
request contains only the static system prompt and the current user
|
||||
task. No assistant decision, tool call, or tool result from a previous
|
||||
round is retained. This is the literal Experiment 1-1 ablation: the
|
||||
model restarts the task on every inference and therefore tends to issue
|
||||
the same first action repeatedly. A one-step sliding window would still
|
||||
be history and would materially narrow the experiment described in the
|
||||
manuscript.
|
||||
|
||||
Returns:
|
||||
The message list to send to the model for this iteration.
|
||||
"""
|
||||
messages = self.conversation_history
|
||||
if self.context_mode != ContextMode.NO_HISTORY:
|
||||
return messages
|
||||
|
||||
# System prompt(s) are always kept as the static prefix.
|
||||
windowed = [m for m in messages if m.get("role") == "system"]
|
||||
|
||||
# Anchor on the latest user task. Nothing after it is retained: those
|
||||
# messages are precisely the previous-round history being ablated.
|
||||
user_indices = [i for i, m in enumerate(messages) if m.get("role") == "user"]
|
||||
if not user_indices:
|
||||
return windowed
|
||||
last_user_idx = user_indices[-1]
|
||||
windowed.append(messages[last_user_idx])
|
||||
return windowed
|
||||
|
||||
@staticmethod
|
||||
def _extract_final_answer(content: str) -> Optional[str]:
|
||||
"""Extract text after FINAL ANSWER: if present; otherwise None."""
|
||||
if not content or "FINAL ANSWER:" not in content:
|
||||
return None
|
||||
return content.split("FINAL ANSWER:", 1)[1].strip()
|
||||
|
||||
def execute_task(self, task: str, max_iterations: Optional[int] = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute a task using available tools (ReAct loop).
|
||||
|
||||
Stops when:
|
||||
1. The model emits a text-only reply (no tool_calls) — conversational
|
||||
or task complete, including plain replies like "hi" that omit the
|
||||
FINAL ANSWER: marker; or
|
||||
2. max_iterations is hit (safety cap for tool-call loops, e.g. the
|
||||
no_tool_results ablation).
|
||||
|
||||
Args:
|
||||
task: The task to execute
|
||||
max_iterations: Maximum ReAct steps (default: Config.MAX_ITERATIONS
|
||||
or 10). This is a safety ceiling, not a target round count.
|
||||
|
||||
Returns:
|
||||
Task execution result
|
||||
|
||||
Result semantics:
|
||||
- ``completed`` means the loop received a non-empty terminal text
|
||||
response. It does not claim that the requested task was correct.
|
||||
- ``task_success`` is ``None`` here because correctness is
|
||||
task-specific and cannot be inferred from arbitrary natural
|
||||
language prompts. Callers with a known rubric should compute it
|
||||
from the final answer and trajectory.
|
||||
- ``success`` is retained as a backwards-compatible alias for
|
||||
``completed``. New consumers should use ``completed`` or their
|
||||
task-specific ``task_success`` value instead.
|
||||
"""
|
||||
if max_iterations is None:
|
||||
try:
|
||||
from config import Config
|
||||
max_iterations = Config.MAX_ITERATIONS
|
||||
except Exception:
|
||||
max_iterations = 10
|
||||
|
||||
# Add user message to conversation history
|
||||
self.conversation_history.append({"role": "user", "content": task})
|
||||
|
||||
# Use conversation history directly (no copy needed)
|
||||
messages = self.conversation_history
|
||||
|
||||
iteration = 0
|
||||
final_answer = None
|
||||
|
||||
while iteration < max_iterations:
|
||||
iteration += 1
|
||||
logger.info(f"Iteration {iteration}/{max_iterations}")
|
||||
|
||||
try:
|
||||
# Build the message list actually sent to the model. For every
|
||||
# mode except NO_HISTORY this equals the full trajectory; for
|
||||
# NO_HISTORY it is a sliding window that drops earlier steps.
|
||||
api_messages = self._prepare_messages_for_api()
|
||||
|
||||
# Prepare request data for logging
|
||||
request_data = {
|
||||
"model": self.model,
|
||||
"messages": api_messages,
|
||||
"temperature": _reasoning_safe_temperature(self.model, 0.3),
|
||||
"max_tokens": 8192
|
||||
}
|
||||
|
||||
if self.context_mode != ContextMode.NO_TOOL_CALLS:
|
||||
request_data["tools"] = self._get_tools_description()
|
||||
request_data["tool_choice"] = "auto"
|
||||
|
||||
# DeepSeek V4: enable thinking so reasoning_content is present
|
||||
# for the no_reasoning ablation (parity with thinking defaults of
|
||||
# Doubao/Kimi). Skip when routed via OpenRouter, which may not
|
||||
# accept the same extra body shape.
|
||||
create_kwargs = {
|
||||
"model": self.model,
|
||||
"messages": api_messages,
|
||||
"tools": self._get_tools_description() if self.context_mode != ContextMode.NO_TOOL_CALLS else None,
|
||||
"tool_choice": "auto" if self.context_mode != ContextMode.NO_TOOL_CALLS else None,
|
||||
"temperature": _reasoning_safe_temperature(self.model, 0.3),
|
||||
"max_tokens": 8192,
|
||||
"timeout": 180, # 180 second timeout for main execution
|
||||
}
|
||||
if self.provider == "deepseek" and not getattr(self, "using_openrouter", False):
|
||||
create_kwargs["extra_body"] = {"thinking": {"type": "enabled"}}
|
||||
request_data["thinking"] = {"type": "enabled"}
|
||||
|
||||
logger.info(f"Sending request to {self.provider} API")
|
||||
|
||||
# Call the model with tools
|
||||
response = self.client.chat.completions.create(**create_kwargs)
|
||||
|
||||
response_dict = (
|
||||
response.model_dump() if hasattr(response, "model_dump")
|
||||
else response.dict() if hasattr(response, "dict")
|
||||
else {"raw_response": str(response)}
|
||||
)
|
||||
self.trajectory.api_turns.append({
|
||||
"iteration": iteration,
|
||||
"provider": self.provider,
|
||||
"resolved_model": self.model,
|
||||
"base_url": self.base_url,
|
||||
"using_openrouter": bool(getattr(self, "using_openrouter", False)),
|
||||
"request": self._json_snapshot(request_data),
|
||||
"response": self._json_snapshot(response_dict),
|
||||
})
|
||||
|
||||
# Log response if verbose
|
||||
if self.verbose:
|
||||
self._log_request_response(request_data, response, iteration)
|
||||
|
||||
message = response.choices[0].message
|
||||
has_tool_calls = bool(getattr(message, "tool_calls", None))
|
||||
reasoning_content = self._reasoning_content(message)
|
||||
if reasoning_content:
|
||||
self.trajectory.reasoning_steps.append(reasoning_content)
|
||||
|
||||
# --- Terminal path: text reply with no tool calls ---
|
||||
# A normal chat turn ("hi" -> "Hello!") or a task answer without
|
||||
# the FINAL ANSWER: marker must end the ReAct loop. Previously
|
||||
# only "FINAL ANSWER:" broke the loop, so plain replies were
|
||||
# re-sent for up to max_iterations (wasted API calls).
|
||||
if not has_tool_calls:
|
||||
assistant_msg = self._prepare_assistant_message(message)
|
||||
messages.append(assistant_msg)
|
||||
content = (message.content or "").strip()
|
||||
if content:
|
||||
marked = self._extract_final_answer(content)
|
||||
final_answer = marked if marked is not None else content
|
||||
logger.info(
|
||||
"Terminal text response (no tool calls); "
|
||||
f"stopping after iteration {iteration}"
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"Empty model response with no tool calls; "
|
||||
"stopping to avoid burning remaining iterations"
|
||||
)
|
||||
break
|
||||
|
||||
# --- Continue path: model requested tool execution ---
|
||||
assistant_msg = self._prepare_assistant_message(message)
|
||||
messages.append(assistant_msg)
|
||||
for tool_call in message.tool_calls:
|
||||
function_name = tool_call.function.name
|
||||
raw_args = tool_call.function.arguments or "{}"
|
||||
try:
|
||||
function_args = json.loads(raw_args)
|
||||
except json.JSONDecodeError as exc:
|
||||
# Keep the turn alive on bad tool-arg JSON.
|
||||
err = (
|
||||
f"Invalid tool arguments (not valid JSON): {exc}. "
|
||||
f"Raw arguments: {raw_args[:500]}"
|
||||
)
|
||||
logger.warning(err)
|
||||
self.trajectory.tool_calls.append(ToolCall(
|
||||
tool_name=function_name,
|
||||
arguments={},
|
||||
result={"error": err},
|
||||
))
|
||||
messages.append({
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
"content": json.dumps({"error": err}),
|
||||
})
|
||||
continue
|
||||
|
||||
logger.info(f"Executing tool: {function_name} with args: {function_args}")
|
||||
|
||||
result = self._execute_tool(function_name, function_args)
|
||||
|
||||
tool_call_record = ToolCall(
|
||||
tool_name=function_name,
|
||||
arguments=function_args,
|
||||
result=result
|
||||
)
|
||||
self.trajectory.tool_calls.append(tool_call_record)
|
||||
|
||||
if self.context_mode != ContextMode.NO_TOOL_RESULTS:
|
||||
tool_msg = {
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
# default=str: code_interpreter returns the raw
|
||||
# namespace in `variables`, which can hold sets,
|
||||
# dict views etc. that json can't encode — that
|
||||
# must not abort the whole task.
|
||||
"content": json.dumps(result, default=str)
|
||||
}
|
||||
else:
|
||||
tool_msg = {
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
"content": "[Tool result hidden due to context mode]"
|
||||
}
|
||||
messages.append(tool_msg)
|
||||
|
||||
# If the same turn also tagged FINAL ANSWER: (unusual with tools),
|
||||
# still prefer extracting it after tools are recorded.
|
||||
if message.content and "FINAL ANSWER:" in message.content:
|
||||
final_answer = self._extract_final_answer(message.content)
|
||||
logger.info(f"Final answer found alongside tool calls: {final_answer}")
|
||||
break
|
||||
|
||||
# Note: We do NOT modify the system prompt anymore.
|
||||
# The context is already built into the conversation through tool history
|
||||
|
||||
except TimeoutError:
|
||||
logger.error("Request timed out after 60 seconds")
|
||||
return {
|
||||
"error": "Request timed out. The model is taking too long to respond. Try a simpler task or different provider.",
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": False,
|
||||
"task_success": False,
|
||||
"success": False,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Error during task execution: {str(e)}")
|
||||
self.trajectory.api_turns.append({
|
||||
"iteration": iteration,
|
||||
"provider": self.provider,
|
||||
"resolved_model": self.model,
|
||||
"base_url": self.base_url,
|
||||
"using_openrouter": bool(getattr(self, "using_openrouter", False)),
|
||||
"error": {"class": type(e).__name__, "message": str(e)},
|
||||
})
|
||||
# Check if it's a timeout-related error
|
||||
if "timeout" in str(e).lower() or "timed out" in str(e).lower():
|
||||
return {
|
||||
"error": "Request timed out. The model is taking too long to respond. Try a simpler task or different provider.",
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": False,
|
||||
"task_success": False,
|
||||
"success": False,
|
||||
}
|
||||
return {
|
||||
"error": str(e),
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": False,
|
||||
"task_success": False,
|
||||
"success": False,
|
||||
}
|
||||
completed = bool(final_answer and str(final_answer).strip())
|
||||
return {
|
||||
"final_answer": final_answer,
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": completed,
|
||||
"task_success": None,
|
||||
# Backwards-compatible alias. This is terminal-response status,
|
||||
# not a correctness judgment.
|
||||
"success": completed,
|
||||
"provider": self.provider,
|
||||
"model": self.model,
|
||||
"base_url": self.base_url,
|
||||
"using_openrouter": bool(getattr(self, "using_openrouter", False)),
|
||||
}
|
||||
|
||||
def reset(self):
|
||||
"""Reset the agent's trajectory and conversation history"""
|
||||
self.trajectory = AgentTrajectory(context_mode=self.context_mode)
|
||||
self._init_system_prompt() # Reinitialize conversation with system prompt
|
||||
logger.info("Agent trajectory and conversation history reset")
|
||||
|
||||
def process(self, query: str, max_iterations: Optional[int] = None) -> str:
|
||||
"""
|
||||
Process a query and return the final answer as a string
|
||||
|
||||
Args:
|
||||
query: The query to process
|
||||
max_iterations: Maximum ReAct steps (default from Config)
|
||||
|
||||
Returns:
|
||||
The final answer as a string
|
||||
"""
|
||||
result = self.execute_task(query, max_iterations)
|
||||
if result.get('final_answer'):
|
||||
return result['final_answer']
|
||||
elif result.get('error'):
|
||||
return f"Error: {result['error']}"
|
||||
else:
|
||||
return "No answer found"
|
||||
@@ -0,0 +1,218 @@
|
||||
"""
|
||||
Configuration module for Context-Aware Agent
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Optional
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def _reasoning_safe_temperature(model, requested=1.0):
|
||||
"""Reasoning models (Kimi K3, GPT-5, ...) only accept temperature=1.
|
||||
Return 1 for those; otherwise the requested value so non-reasoning
|
||||
providers (Doubao, DeepSeek, older Moonshot) are unchanged."""
|
||||
m = str(model or "").lower().replace("/", "-")
|
||||
return 1 if ("kimi-k3" in m or "gpt-5" in m) else requested
|
||||
|
||||
|
||||
# Provider resolution lives in the shared agentbook package so every chapter
|
||||
# stays consistent; see agentbook/providers.py. The fallback keeps this
|
||||
# experiment runnable from a checkout where agentbook is not installed.
|
||||
try:
|
||||
from agentbook.providers import (
|
||||
PROVIDERS,
|
||||
SUPPORTED_PROVIDERS,
|
||||
canonical_provider,
|
||||
canonical_provider as _canonical_provider,
|
||||
map_model_to_openrouter,
|
||||
resolve_backend,
|
||||
resolve_llm_backend,
|
||||
)
|
||||
except ImportError: # pragma: no cover - exercised only without the package
|
||||
import sys as _sys
|
||||
|
||||
_sys.path.insert(
|
||||
0, str(__import__("pathlib").Path(__file__).resolve().parents[2])
|
||||
)
|
||||
from agentbook.providers import (
|
||||
PROVIDERS,
|
||||
SUPPORTED_PROVIDERS,
|
||||
canonical_provider,
|
||||
canonical_provider as _canonical_provider,
|
||||
map_model_to_openrouter,
|
||||
resolve_backend,
|
||||
resolve_llm_backend,
|
||||
)
|
||||
|
||||
|
||||
class Config:
|
||||
"""Configuration settings for the agent"""
|
||||
|
||||
# Provider Configuration
|
||||
LLM_PROVIDER: str = os.getenv("LLM_PROVIDER", "doubao").lower()
|
||||
|
||||
# API Configuration
|
||||
DASHSCOPE_API_KEY: str = os.getenv("DASHSCOPE_API_KEY", "")
|
||||
DASHSCOPE_BASE_URL: str = os.getenv(
|
||||
"DASHSCOPE_BASE_URL", "https://dashscope.aliyuncs.com/compatible-mode/v1"
|
||||
)
|
||||
|
||||
SILICONFLOW_API_KEY: str = os.getenv("SILICONFLOW_API_KEY", "")
|
||||
SILICONFLOW_BASE_URL: str = "https://api.siliconflow.cn/v1"
|
||||
|
||||
ARK_API_KEY: str = os.getenv("ARK_API_KEY", "")
|
||||
ARK_BASE_URL: str = "https://ark.cn-beijing.volces.com/api/v3"
|
||||
|
||||
MOONSHOT_API_KEY: str = os.getenv("MOONSHOT_API_KEY", "")
|
||||
MOONSHOT_BASE_URL: str = "https://api.moonshot.cn/v1"
|
||||
|
||||
DEEPSEEK_API_KEY: str = os.getenv("DEEPSEEK_API_KEY", "")
|
||||
DEEPSEEK_BASE_URL: str = os.getenv(
|
||||
"DEEPSEEK_BASE_URL", "https://api.deepseek.com"
|
||||
)
|
||||
|
||||
ZHIPU_API_KEY: str = os.getenv("ZHIPU_API_KEY", "")
|
||||
ZHIPU_BASE_URL: str = "https://open.bigmodel.cn/api/paas/v4"
|
||||
|
||||
# Model Configuration (defaults based on provider)
|
||||
MODEL_NAME: str = os.getenv("MODEL_NAME", "") # Will be set based on provider if not specified
|
||||
MODEL_TEMPERATURE: float = float(os.getenv("MODEL_TEMPERATURE", "0.3"))
|
||||
MODEL_MAX_TOKENS: int = int(os.getenv("MODEL_MAX_TOKENS", "1000"))
|
||||
|
||||
# Agent Configuration
|
||||
MAX_ITERATIONS: int = int(os.getenv("MAX_ITERATIONS", "10"))
|
||||
ENABLE_REASONING: bool = os.getenv("ENABLE_REASONING", "true").lower() == "true"
|
||||
|
||||
# Test Configuration
|
||||
TEST_PDF_URL: str = os.getenv(
|
||||
"TEST_PDF_URL",
|
||||
"https://www.berkshirehathaway.com/qtrly/1stqtr23.pdf"
|
||||
)
|
||||
|
||||
# Currency Configuration (Example rates - in production use real API)
|
||||
EXCHANGE_RATES = {
|
||||
"USD": 1.0,
|
||||
"EUR": 0.92,
|
||||
"GBP": 0.79,
|
||||
"JPY": 149.50,
|
||||
"CNY": 7.24,
|
||||
"CAD": 1.36,
|
||||
"AUD": 1.53,
|
||||
"CHF": 0.88,
|
||||
"INR": 83.12,
|
||||
"SGD": 1.34
|
||||
}
|
||||
|
||||
# Logging Configuration
|
||||
LOG_LEVEL: str = os.getenv("LOG_LEVEL", "INFO")
|
||||
LOG_FILE: Optional[str] = os.getenv("LOG_FILE")
|
||||
LOG_FORMAT: str = "%(asctime)s - %(levelname)s - %(name)s - %(message)s"
|
||||
|
||||
# File paths
|
||||
RESULTS_DIR: str = "results"
|
||||
TEST_PDFS_DIR: str = "fixtures/pdfs"
|
||||
|
||||
@classmethod
|
||||
def get_api_key(cls, provider: str = None) -> str:
|
||||
"""
|
||||
Get API key for the specified provider
|
||||
|
||||
Args:
|
||||
provider: Provider name (defaults to LLM_PROVIDER)
|
||||
|
||||
Returns:
|
||||
API key for the provider
|
||||
"""
|
||||
provider = provider or cls.LLM_PROVIDER
|
||||
# The shared registry knows every provider's key variables, so this
|
||||
# stays correct as providers are added there.
|
||||
try:
|
||||
return PROVIDERS[_canonical_provider(provider)].api_key()
|
||||
except KeyError:
|
||||
return ""
|
||||
|
||||
@classmethod
|
||||
def get_default_model(cls, provider: str = None) -> str:
|
||||
"""
|
||||
Get default model for the specified provider
|
||||
|
||||
Args:
|
||||
provider: Provider name (defaults to LLM_PROVIDER)
|
||||
|
||||
Returns:
|
||||
Default model name for the provider
|
||||
"""
|
||||
provider = provider or cls.LLM_PROVIDER
|
||||
provider = provider.lower()
|
||||
|
||||
if cls.MODEL_NAME:
|
||||
return cls.MODEL_NAME
|
||||
|
||||
try:
|
||||
return PROVIDERS[_canonical_provider(provider)].default_model
|
||||
except KeyError:
|
||||
return ""
|
||||
|
||||
@classmethod
|
||||
def validate(cls, provider: str = None) -> bool:
|
||||
"""
|
||||
Validate required configuration
|
||||
|
||||
Args:
|
||||
provider: Provider to validate (defaults to LLM_PROVIDER)
|
||||
|
||||
Returns:
|
||||
True if configuration is valid
|
||||
"""
|
||||
provider = provider or cls.LLM_PROVIDER
|
||||
# resolve_backend already accounts for providers that need no key
|
||||
# (ollama) and for the OpenRouter fallback, and its error names the
|
||||
# exact variables to set -- so a missing key is not the only signal.
|
||||
try:
|
||||
resolve_backend(provider)
|
||||
except ValueError as exc:
|
||||
print(f"ERROR: {exc}")
|
||||
print("Please set it in .env file or as environment variable")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
@classmethod
|
||||
def create_directories(cls):
|
||||
"""Create necessary directories if they don't exist"""
|
||||
os.makedirs(cls.RESULTS_DIR, exist_ok=True)
|
||||
os.makedirs(cls.TEST_PDFS_DIR, exist_ok=True)
|
||||
|
||||
@classmethod
|
||||
def get_model_config(cls) -> dict:
|
||||
"""
|
||||
Get model configuration as dictionary
|
||||
|
||||
Returns:
|
||||
Model configuration dict
|
||||
"""
|
||||
return {
|
||||
"model": cls.MODEL_NAME,
|
||||
"temperature": _reasoning_safe_temperature(cls.MODEL_NAME, cls.MODEL_TEMPERATURE),
|
||||
"max_tokens": cls.MODEL_MAX_TOKENS
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def print_config(cls):
|
||||
"""Print current configuration (hiding sensitive data)"""
|
||||
provider = canonical_provider(cls.LLM_PROVIDER)
|
||||
api_key = cls.get_api_key(provider)
|
||||
print("\n" + "="*50)
|
||||
print("CONFIGURATION")
|
||||
print("="*50)
|
||||
print(f"Provider: {provider}")
|
||||
print(f"Model: {cls.MODEL_NAME}")
|
||||
print(f"Temperature: {cls.MODEL_TEMPERATURE}")
|
||||
print(f"Max Tokens: {cls.MODEL_MAX_TOKENS}")
|
||||
print(f"Max Iterations: {cls.MAX_ITERATIONS}")
|
||||
print(f"API Key Set: {'Yes' if api_key else 'No'}")
|
||||
print(f"Log Level: {cls.LOG_LEVEL}")
|
||||
print("="*50 + "\n")
|
||||
@@ -0,0 +1,274 @@
|
||||
"""
|
||||
Create Sample PDF for Testing
|
||||
Generates a financial report PDF with various currency amounts and calculations
|
||||
"""
|
||||
|
||||
from reportlab.lib import colors
|
||||
from reportlab.lib.pagesizes import letter, A4
|
||||
from reportlab.platypus import SimpleDocTemplate, Table, TableStyle, Paragraph, Spacer, PageBreak
|
||||
from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle
|
||||
from reportlab.lib.units import inch
|
||||
from reportlab.lib.enums import TA_CENTER, TA_RIGHT
|
||||
import os
|
||||
|
||||
|
||||
def create_financial_report():
|
||||
"""Create a sample financial report PDF for testing"""
|
||||
|
||||
# Create PDF
|
||||
filename = "sample_financial_report_q1_2024.pdf"
|
||||
doc = SimpleDocTemplate(filename, pagesize=letter)
|
||||
|
||||
# Container for the 'Flowable' objects
|
||||
elements = []
|
||||
|
||||
# Define styles
|
||||
styles = getSampleStyleSheet()
|
||||
title_style = ParagraphStyle(
|
||||
'CustomTitle',
|
||||
parent=styles['Heading1'],
|
||||
fontSize=24,
|
||||
textColor=colors.HexColor('#1f4788'),
|
||||
spaceAfter=30,
|
||||
alignment=TA_CENTER
|
||||
)
|
||||
|
||||
heading_style = ParagraphStyle(
|
||||
'CustomHeading',
|
||||
parent=styles['Heading2'],
|
||||
fontSize=16,
|
||||
textColor=colors.HexColor('#1f4788'),
|
||||
spaceAfter=12,
|
||||
)
|
||||
|
||||
# Title
|
||||
elements.append(Paragraph("Global Corporation Financial Report", title_style))
|
||||
elements.append(Paragraph("Q1 2024 - Quarterly Results", styles['Heading2']))
|
||||
elements.append(Spacer(1, 0.5*inch))
|
||||
|
||||
# Executive Summary
|
||||
elements.append(Paragraph("Executive Summary", heading_style))
|
||||
summary_text = """This report presents the financial performance of Global Corporation
|
||||
for the first quarter of 2024. The company operates in multiple regions with
|
||||
transactions in various currencies. Total consolidated revenue for Q1 2024
|
||||
reached $45.8 million USD, representing a 12% increase year-over-year."""
|
||||
elements.append(Paragraph(summary_text, styles['BodyText']))
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Regional Revenue Table
|
||||
elements.append(Paragraph("Regional Revenue Breakdown", heading_style))
|
||||
|
||||
revenue_data = [
|
||||
['Region', 'Local Currency', 'Q1 2024 Revenue', 'Q4 2023 Revenue', 'Growth %'],
|
||||
['North America', 'USD', '$15,250,000', '$14,100,000', '8.16%'],
|
||||
['Europe', 'EUR', '€11,340,000', '€10,800,000', '5.00%'],
|
||||
['United Kingdom', 'GBP', '£8,920,000', '£8,500,000', '4.94%'],
|
||||
['Asia Pacific', 'JPY', '¥1,245,000,000', '¥1,180,000,000', '5.51%'],
|
||||
['Singapore', 'SGD', 'S$4,180,000', 'S$3,950,000', '5.82%'],
|
||||
]
|
||||
|
||||
revenue_table = Table(revenue_data, colWidths=[2*inch, 1.2*inch, 1.5*inch, 1.5*inch, 0.8*inch])
|
||||
revenue_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -1), colors.beige),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(revenue_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Operating Expenses
|
||||
elements.append(Paragraph("Operating Expenses by Department", heading_style))
|
||||
|
||||
expense_data = [
|
||||
['Department', 'Q1 2024 (USD)', 'Q4 2023 (USD)', 'Change'],
|
||||
['Research & Development', '$8,450,000', '$7,900,000', '+$550,000'],
|
||||
['Sales & Marketing', '$6,230,000', '$6,100,000', '+$130,000'],
|
||||
['General & Administrative', '$4,180,000', '$4,050,000', '+$130,000'],
|
||||
['Operations', '$9,870,000', '$9,500,000', '+$370,000'],
|
||||
['Total Operating Expenses', '$28,730,000', '$27,550,000', '+$1,180,000'],
|
||||
]
|
||||
|
||||
expense_table = Table(expense_data, colWidths=[2.5*inch, 1.5*inch, 1.5*inch, 1.2*inch])
|
||||
expense_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -2), colors.lightgrey),
|
||||
('BACKGROUND', (0, -1), (-1, -1), colors.yellow),
|
||||
('FONTNAME', (0, -1), (-1, -1), 'Helvetica-Bold'),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(expense_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Key Financial Metrics
|
||||
elements.append(Paragraph("Key Financial Metrics", heading_style))
|
||||
|
||||
metrics_text = """
|
||||
• Gross Profit Margin: 37.2%<br/>
|
||||
• Operating Profit Margin: 18.4%<br/>
|
||||
• Net Profit Margin: 14.8%<br/>
|
||||
• EBITDA: $10,250,000 USD<br/>
|
||||
• Cash Flow from Operations: $8,930,000 USD<br/>
|
||||
• Total Assets: $125,400,000 USD<br/>
|
||||
• Total Liabilities: $48,200,000 USD<br/>
|
||||
• Shareholders' Equity: $77,200,000 USD<br/>
|
||||
"""
|
||||
elements.append(Paragraph(metrics_text, styles['BodyText']))
|
||||
|
||||
# Add page break
|
||||
elements.append(PageBreak())
|
||||
|
||||
# Currency Exchange Rates Used
|
||||
elements.append(Paragraph("Currency Exchange Rates (as of March 31, 2024)", heading_style))
|
||||
|
||||
exchange_data = [
|
||||
['Currency Pair', 'Exchange Rate', 'Previous Quarter', 'Change'],
|
||||
['USD/EUR', '0.9234', '0.9156', '+0.85%'],
|
||||
['USD/GBP', '0.7891', '0.7823', '+0.87%'],
|
||||
['USD/JPY', '149.85', '147.23', '+1.78%'],
|
||||
['USD/SGD', '1.3452', '1.3389', '+0.47%'],
|
||||
]
|
||||
|
||||
exchange_table = Table(exchange_data, colWidths=[2*inch, 1.5*inch, 1.5*inch, 1.2*inch])
|
||||
exchange_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -1), colors.beige),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(exchange_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Investment Portfolio
|
||||
elements.append(Paragraph("Investment Portfolio Performance", heading_style))
|
||||
|
||||
portfolio_text = """The company's investment portfolio showed strong performance in Q1 2024:
|
||||
|
||||
• Fixed Income Securities: $23,450,000 USD (yielding 4.2% annually)
|
||||
• Equity Investments: $18,750,000 USD (up 8.3% this quarter)
|
||||
• Real Estate Holdings: $31,200,000 USD (appreciation of 3.1%)
|
||||
• Cash and Cash Equivalents: $15,890,000 USD
|
||||
|
||||
Total portfolio value: $89,290,000 USD, representing a 5.7% increase from Q4 2023."""
|
||||
|
||||
elements.append(Paragraph(portfolio_text, styles['BodyText']))
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Future Projections
|
||||
elements.append(Paragraph("Q2 2024 Projections", heading_style))
|
||||
|
||||
projection_data = [
|
||||
['Metric', 'Q1 2024 Actual', 'Q2 2024 Projected', 'Growth'],
|
||||
['Total Revenue', '$45,800,000', '$48,500,000', '+5.9%'],
|
||||
['Operating Expenses', '$28,730,000', '$29,800,000', '+3.7%'],
|
||||
['Net Income', '$6,780,000', '$7,450,000', '+9.9%'],
|
||||
['EPS (Earnings Per Share)', '$2.34', '$2.57', '+9.8%'],
|
||||
]
|
||||
|
||||
projection_table = Table(projection_data, colWidths=[2.5*inch, 1.5*inch, 1.5*inch, 1*inch])
|
||||
projection_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -1), colors.lightblue),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(projection_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Footer
|
||||
footer_text = """
|
||||
<para alignment="center">
|
||||
<b>Note:</b> All financial figures are preliminary and subject to audit.<br/>
|
||||
For more information, please contact: investor.relations@globalcorp.com<br/>
|
||||
Global Corporation © 2024 - Confidential Financial Report
|
||||
</para>
|
||||
"""
|
||||
elements.append(Spacer(1, 0.5*inch))
|
||||
elements.append(Paragraph(footer_text, styles['Normal']))
|
||||
|
||||
# Build PDF
|
||||
doc.build(elements)
|
||||
|
||||
print(f"Sample PDF created: {filename}")
|
||||
return filename
|
||||
|
||||
|
||||
def create_simple_expense_report():
|
||||
"""Create a simpler expense report for quick testing"""
|
||||
|
||||
filename = "simple_expense_report.pdf"
|
||||
doc = SimpleDocTemplate(filename, pagesize=A4)
|
||||
|
||||
elements = []
|
||||
styles = getSampleStyleSheet()
|
||||
|
||||
# Title
|
||||
elements.append(Paragraph("Quarterly Expense Report", styles['Title']))
|
||||
elements.append(Spacer(1, 0.2*inch))
|
||||
|
||||
# Simple expense data
|
||||
elements.append(Paragraph("Q1 2024 Regional Expenses", styles['Heading2']))
|
||||
|
||||
expense_text = """
|
||||
Our company has the following expenses for Q1 2024:
|
||||
|
||||
<b>United States Office:</b> $2,500,000 USD<br/>
|
||||
<b>United Kingdom Office:</b> £1,800,000 GBP<br/>
|
||||
<b>Japan Office:</b> ¥380,000,000 JPY<br/>
|
||||
<b>European Union Office:</b> €2,100,000 EUR<br/>
|
||||
<b>Singapore Office:</b> S$3,200,000 SGD<br/>
|
||||
|
||||
These expenses include salaries, operations, marketing, and R&D costs.
|
||||
|
||||
Additional financial metrics:
|
||||
• Total headcount: 1,250 employees globally
|
||||
• Average expense per employee: varies by region
|
||||
• Projected Q2 expense reduction target: 8% across all regions
|
||||
"""
|
||||
|
||||
elements.append(Paragraph(expense_text, styles['BodyText']))
|
||||
|
||||
# Build PDF
|
||||
doc.build(elements)
|
||||
|
||||
print(f"Simple PDF created: {filename}")
|
||||
return filename
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Create both PDFs
|
||||
create_financial_report()
|
||||
create_simple_expense_report()
|
||||
|
||||
# Create a fixture directory for PDFs if needed
|
||||
os.makedirs("fixtures/pdfs", exist_ok=True)
|
||||
|
||||
# Move PDFs to test directory
|
||||
import shutil
|
||||
for pdf in ["sample_financial_report_q1_2024.pdf", "simple_expense_report.pdf"]:
|
||||
if os.path.exists(pdf):
|
||||
shutil.move(pdf, f"fixtures/pdfs/{pdf}")
|
||||
|
||||
print("\nPDFs created in fixtures/pdfs/ directory")
|
||||
print("You can host these PDFs online or use a local server for testing")
|
||||
@@ -0,0 +1,39 @@
|
||||
# LLM Provider Configuration (dashscope/qwen, siliconflow, doubao, kimi,
|
||||
# moonshot, deepseek, zhipu, openrouter, or ollama)
|
||||
LLM_PROVIDER=doubao
|
||||
|
||||
# API Keys (set the appropriate one for your provider)
|
||||
SILICONFLOW_API_KEY=your_siliconflow_api_key_here
|
||||
# Alibaba Cloud Model Studio (Bailian) / Qwen
|
||||
DASHSCOPE_API_KEY=your_dashscope_api_key_here
|
||||
# Optional: use this instead for an international-region key
|
||||
# DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
ARK_API_KEY=your_ark_api_key_here
|
||||
MOONSHOT_API_KEY=your_moonshot_api_key_here
|
||||
DEEPSEEK_API_KEY=your_deepseek_api_key_here
|
||||
# Optional: override DeepSeek base URL (default: https://api.deepseek.com)
|
||||
# DEEPSEEK_BASE_URL=https://api.deepseek.com
|
||||
ZHIPU_API_KEY=your_zhipu_api_key_here
|
||||
|
||||
# Universal fallback: if the provider key above is missing/invalid but
|
||||
# OPENROUTER_API_KEY is set, requests are routed through OpenRouter and the
|
||||
# model id is mapped automatically (bare gpt-*/o1-* -> openai/*, claude-* ->
|
||||
# anthropic/*, deepseek-* -> deepseek/*, other native ids -> OPENROUTER_MODEL
|
||||
# or openai/gpt-5.6-luna).
|
||||
# You can also pass --provider openrouter to use OpenRouter directly.
|
||||
OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
# OPENROUTER_MODEL=openai/gpt-5.6-luna # model used when falling back for non-OpenRouter native ids
|
||||
|
||||
# Optional: Model Configuration
|
||||
# MODEL_NAME= # Leave empty to use provider default
|
||||
# For DeepSeek: deepseek-v4-flash (default) or deepseek-v4-pro
|
||||
# MODEL_TEMPERATURE=0.3
|
||||
# MODEL_MAX_TOKENS=1000
|
||||
|
||||
# Optional: Test Configuration
|
||||
# MAX_ITERATIONS=10
|
||||
# TEST_PDF_URL=https://example.com/sample.pdf
|
||||
|
||||
# Optional: Logging Configuration
|
||||
# LOG_LEVEL=INFO
|
||||
# LOG_FILE=agent.log
|
||||
@@ -0,0 +1,112 @@
|
||||
%PDF-1.4
|
||||
%“Œ‹ž ReportLab Generated PDF document http://www.reportlab.com
|
||||
1 0 obj
|
||||
<<
|
||||
/F1 2 0 R /F2 3 0 R
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
3 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding /Name /F2 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Contents 10 0 R /MediaBox [ 0 0 612 792 ] /Parent 9 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Contents 11 0 R /MediaBox [ 0 0 612 792 ] /Parent 9 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Contents 12 0 R /MediaBox [ 0 0 612 792 ] /Parent 9 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/PageMode /UseNone /Pages 9 0 R /Type /Catalog
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Author (\(anonymous\)) /CreationDate (D:20250909105803+08'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20250909105803+08'00') /Producer (ReportLab PDF Library - www.reportlab.com)
|
||||
/Subject (\(unspecified\)) /Title (\(anonymous\)) /Trapped /False
|
||||
>>
|
||||
endobj
|
||||
9 0 obj
|
||||
<<
|
||||
/Count 3 /Kids [ 4 0 R 5 0 R 6 0 R ] /Type /Pages
|
||||
>>
|
||||
endobj
|
||||
10 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 1740
|
||||
>>
|
||||
stream
|
||||
Gatm<>?BQ=&:WeDbX:Tf`FpQ#^-0o1E_\:">geBjLDSPS`,0=g[M,*Z^EZBl9Seu%Z>`(SkSn6Md7BlL"r)PipHL/.hG3$j%h';1krg^d"AD%t`UrBW&MqKVL+qQl7N5d+"^qIc8$VP$OMFp^:d5rY5cc()%#6?P?k-G?JN6fLE^kHCSBg'RE`%6X(\"T-QW9TbGu__:]M-L"8*c.&A,@1AhdQ?ca6MJ7Y&!r`)Oh:!2VQi-,7ugFecip"(!+KS"&:SX1UpbfFXuds#%qdRMmfn>m[]8ZH.]A+dD*n#T,\UV@_2=\ZW3Ku-/>Y\,-<f,8-83ibsOVt/hHCekjj3YS`^I3?$9XZb=7e7"79@1I@pD5XRE4$^eYTCHe>Z"fPQ$$VM1X'IgW538mj]H2Xlp3.[FeZJI4nA>M28J6@h_RQOAXi4Q(/f4Qc#Ng,^gT1i#/\U=O%L=/uP2<F&7V9K"kNeRBpE,-"'q_sq@;7($:CE[A?Mm+U#Gc/s$N'+p<CML;[*d=8iXn#7"7p3HH6.;%`[kKM;HOg's^I9*I]IAWTlEtKfJ@S]:GaOjj'&iX*R\fIL?]u:h<bc8^(1orrm0Dp:qJ?gG"dlqVgqGP.4KHtLUn-F(:pMZ'_&lOZ&qDJh/,Qd:n:lWM="E.JgY9Uup"c1P9=9je<@>GZPOZ-eU`!0%mPQK]/5?F7&8mNC*r@ZkA,bf;IA';`bEkdekperQ6S:%f0&.r?mf'0KC*jMN%#O(,!h;+>XK2Fr;Ln\d_5B3IDiOk7pSu!["M*_i4h?1rJ,TQpF/8E$'NsF10P`FfJ'LT7s(j2K?a9^mU$o)2LnU>#!>7pU(dbufZ2'*0D=r-u-`^S(DmPq&%H+l8ZD!4-tpijdsQ[hmn6I$*un3JbA&7pk,'R[b2Al]X!P1J[amF<&oK_Vg+h/VDo?G`6QULt(Qmr1*)Jg=fYUBkjZEGp!0o-R8l.a@P]4<%cI'C&g"a/csF(lF!"JV@OJ!V>fQ$J<0MLn6W3R(Ah>;r\USJ;l9X#fhPa@(3Y'.12`C=FCUsJ1hpIOGY);&N8G78fbcs06(j^^.o-BQk,$&JbY%Nhp##?F,:t[HP%[;@pM5Li3osiA"ugNFF_cPcTMqW9X_Z[6ddl?YVoH1D8QToo0[$*#AZ[Y1m>kt`e9pd[G.+[:PlOGVeqL`a/2?kD0lE:ZgQ4:[cdLtf_]n&NVA5l:Q57K@fKMR8Q%L1MdK+[2L&O([Xn0nM&35pmjb)qO*-'6'Cn>G_kJenF[I%fI]QKiTf_XBp5K<[TCuY5'0B`@qsip#\/=//M!(M_e%I1IDSLkb5p2aJrLHj(]@S:>MpeLu7d6I=Jj!-1GOr1hOT.o8I30!>?8D',"#$.s*+%TUYN;K^Nmm)H-5_@MkL^GC-"l0J*I3M5G[uBR0S;Q["O7Z.;-9TO\*3r%p(_sHlQ9#)YqpXb1^Q>#Ws=:7U>=E4*P=\1:7-*9FZ\q)mo#I]q%ahIq5hZ\]P5ns>`YELL4o([G`HPP>;UIL<Kf-#"!g4_0@>]G3C**4a;MEP:D,=@%Yq_d]5J6ACkFpBq"og6G5^Wbi%abieM$m(p\[Q%@ho\p4=<8b,'Qn@!BBZ*29m&TQg);g`h'"j.'<>Zg_En-*LBj^^X#-tRWh2$>BaXZ>Bbd%>Bcpp9\52>XCG2r5]#s/haoE."//FU/[O(jUm)0Zl5hBcbh-8PV4Zg\n9*.qg#DnXmM@s(oFL`C7hY~>endstream
|
||||
endobj
|
||||
11 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 317
|
||||
>>
|
||||
stream
|
||||
Gas304&<aJ'SbR0^Z$1t>SI?'F_fI.V(G#+\Ys+M0p%;j&l[NjnaCHa@0dPfab09!8!Zh7_U/:0C4Nk5fntBYO9I%*(hAAtN4'>S8M2]j,2dDV2#UQ"-BLr/Am[Vj[tF<_9,p/mc>dQXF1AWTclQt69AN>ti^nH@!r2o=diH)Hc)pQJ/Vf^NKFRHomXHsWL`p!JYN0.N`;q9)3[QR)"S\D)5[>d>@9&e@'\5+u%l(0oTnq#bq`8H(W\JQIpDp*p3+h6$S8%YpC[H9$'hlR1hI'82RocJ9cE01FEr2a47qam^ZO&j>(C&87[.S\m~>endstream
|
||||
endobj
|
||||
12 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 1717
|
||||
>>
|
||||
stream
|
||||
Gatm;;,c4/&:WeDliYJ.D,;N]Z6c*FV1H'r]aKocg^Gj=UC]5<*FoDt^V?%Nfa2%BG<I](Zi7?/Bt`C%_LUX$<I^2i^rI>*5@_JgbSIts&MX(a-F8?:"4eS>"ecD<N&JO4PG&2t,:*Bq6jE&am*,Pl@]aS@[RS"O=BPB!cKj\9CTY^A8!s3JQ>q(Xi%coZ!S=*VJ:%CT@[enl?t8,ZMsE<?@!2pa[uS`;IQC)Y'*V:UL6SK-LQ9UXMPN"\CCNU`Gd([!JMjZVHU)[DnGpq=.cc_RIRh?T%_V)[dB*#WX-;*PjA#VsdLgF28e-ZsUi]1q`i7cSiQ_WS"jngl48,uQnd[8s'!QQ"e=-7kh\L19NGWBU_?>`.VH$9j$F0c1(!`E_C!jX.$;Dd:?+Kc[p$)G9qdn*qnB",C6RgDF&3VDhF<??^f6<180\XNr7u<C!011l"ck6IsR@\nu%Phmoa@0f$aE.[/K41Vde1lr/QqG#Hp!&,Tl.>1WK1m44__Wc&aL"B*j>EeDoGthJp#hQ\O=eeZo2!9M^(*m>+o1W&pV9&Y=.['#Vq=Y?lPEpupU7'^bB*UE_=rV*o8_km$Ebj/D2iY`7QZ^HrPEI=(#Fh7hC:s0`Ck+Y_UaU2hMQ0u>Of#-1/WgJCE&4B4)ma![`0Gkft+RG?9IJAfZIbBTR6$p:[LGXs4&Ir-+c*)\MdRse(g7II$NhJ20P,:FdsbT\#CSY&Md-k\+t+h-*:\F2kskTW5QAO#7iMW/Ia!$H4sqP^D2iRF>b_A@[THJ3>A=7ZQ%P=j:De;X;X;NE6Y*_cI/>@r0!aD"'6i&g9aOJET3$FX-eZ)(^ipUPZ#;`!%6K1C0'p?]aM]<+O790]NRI)^T"nAm^?j*#[OTj'0'N;*:<RSDEs?B#g-^^pirRG\tsaLQR6%TUaU4>^ur?;l$VG$j`K*U]\-DNUiDgbR@CSb@7T1pBj<3Oik2j9"$)D9n]I&ZA#28BZb45d>t,F92$d-,bWFGoV^I3Nd"b7_%$F"KR,c,R\7dC$['9?E/qclE5R-O!^uk>WnI""&J$jMSGkLE3C@bYn\6KdIHk<>Y)u"Ie&uo4D&iTZPLa\99UA+.PJiN].,![?4:"57"Dl0mUdaOqqo6,_YGnJt%;I]1ioP14ZN^UKj5C[;%1G]m>43h^B:bMd3l9NT=<'0j?MS(e?-JS6n,T7<`nV7/r\pd(oV@0E_*OHa.+gbVh#s1'6,gh#B;io'nJA8>_`7;ISq7/4_@>Z5jc)*jgN?WV4H&L7kkOd!3.LAY<-jfG06-tL@5>Bi'`BMSi'iT`K%*+JKj;dT^,Mu7O3I?1KHa=_H&dg9(-IXJ]Kp1/$Y"cL,"BWe!R_>mC]1:d=O*%`(&Ds3'Yo7SpLQg+97H.INJ#-5N+ZraTB^42!e]hb(C7TsbB.=(%cSQ!_SgbA\Fo"MVES]@5'3.PGCSstt94XkpE:)7)4@3'E6At#Y]kjUP[bj74n"Iq0AjhC%%I&/e6E3d`[1qM3Y`eV+e-!C"3$\4&>/;E-:_ok3]=Q\T,Qk+`^lc&tf6GJ&<b&Jgrc$['arKF.4>#k:k+:im(H?s1pnP1N_E,/$A5mlmNL)A9<c/\I@`cbP.8LkuP$<N(.S0+,bpr`*YEI[Y#ek2YF8T>^WTJ/po*PSt#g.!Ijfc423qtG`ZUHp!Eu72UmAV^t)r\L8O#9r`Y8oO]%M&I=!%GTi~>endstream
|
||||
endobj
|
||||
xref
|
||||
0 13
|
||||
0000000000 65535 f
|
||||
0000000073 00000 n
|
||||
0000000114 00000 n
|
||||
0000000221 00000 n
|
||||
0000000333 00000 n
|
||||
0000000527 00000 n
|
||||
0000000721 00000 n
|
||||
0000000915 00000 n
|
||||
0000000983 00000 n
|
||||
0000001266 00000 n
|
||||
0000001337 00000 n
|
||||
0000003169 00000 n
|
||||
0000003577 00000 n
|
||||
trailer
|
||||
<<
|
||||
/ID
|
||||
[<2dbcccf9df210d48854ffa3169a0c96f><2dbcccf9df210d48854ffa3169a0c96f>]
|
||||
% ReportLab generated PDF document -- digest (http://www.reportlab.com)
|
||||
|
||||
/Info 8 0 R
|
||||
/Root 7 0 R
|
||||
/Size 13
|
||||
>>
|
||||
startxref
|
||||
5386
|
||||
%%EOF
|
||||
@@ -0,0 +1,74 @@
|
||||
%PDF-1.4
|
||||
%“Œ‹ž ReportLab Generated PDF document http://www.reportlab.com
|
||||
1 0 obj
|
||||
<<
|
||||
/F1 2 0 R /F2 3 0 R
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
3 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding /Name /F2 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Contents 8 0 R /MediaBox [ 0 0 595.2756 841.8898 ] /Parent 7 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/PageMode /UseNone /Pages 7 0 R /Type /Catalog
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Author (\(anonymous\)) /CreationDate (D:20250909105803+08'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20250909105803+08'00') /Producer (ReportLab PDF Library - www.reportlab.com)
|
||||
/Subject (\(unspecified\)) /Title (\(anonymous\)) /Trapped /False
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/Count 1 /Kids [ 4 0 R ] /Type /Pages
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 657
|
||||
>>
|
||||
stream
|
||||
GasJOd;IYl'Sc(%MZ9E?Z70:l$"2CcX0P@RUd>A(PBu79-DX:Dk*U>)rD$%ZUSFA(+d@'GrL`Hj%Rq!'^I\`:`;'7qK+#Ujn>I9Lrf0S*q:4G@k>5V)nZJ*$8LpaTI)0D8>0@F9>$)D0JBJ-C6j)FBFpBR]<0'r\Z]27cgL?)_n$T<5NEa(SepTg]JG:?Q*VYc;N1OMo_7NRFntIrX4dZ.d)>O`b`\Np$#UUb$a`:=Z@]rl%`D2injlinXUD,_%@^nW(J%D]fgD_<GfiTMs>cVUtQ)`eD)G=!CjsdJhkVer;=gVpRGSP/b@DS\\SPCn`b#.KLEg\;$gc"O5".Y4PcWJ7*_d\)ZbKYM8ja<iae<I.gmR5RbTkmk5(Xl<aCIMHR$dnE7&+G\Hl&Sc)ai9dW-4ZH3[qL_WOLM=5iq=(%::jC\@JX!Jr2^]l(,EXN5+s**Bq!Ac?0#!`Cf66r(!>[M%RH7Y3Du[QV6W=nUTb<09!NFDhHt`*CT^t!`Hdp7DDteC]M_amX(":u6)gF"PBr0L4rbb)Nk/7pV2<l)D@@SD5(bK>L"U3=aU3*m$MoR+qtVINA-Z%peY.2G7k]etm30^M);hQe>4\DNk5&`M!3eN?,`O=EZC*$?q$nmZVQN4ifE#<-)X.)":S]IY~>endstream
|
||||
endobj
|
||||
xref
|
||||
0 9
|
||||
0000000000 65535 f
|
||||
0000000073 00000 n
|
||||
0000000114 00000 n
|
||||
0000000221 00000 n
|
||||
0000000333 00000 n
|
||||
0000000536 00000 n
|
||||
0000000604 00000 n
|
||||
0000000887 00000 n
|
||||
0000000946 00000 n
|
||||
trailer
|
||||
<<
|
||||
/ID
|
||||
[<6a67c647f1d47536a5416ec836e1ab9c><6a67c647f1d47536a5416ec836e1ab9c>]
|
||||
% ReportLab generated PDF document -- digest (http://www.reportlab.com)
|
||||
|
||||
/Info 6 0 R
|
||||
/Root 5 0 R
|
||||
/Size 9
|
||||
>>
|
||||
startxref
|
||||
1693
|
||||
%%EOF
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,22 @@
|
||||
# Core dependencies
|
||||
openai>=1.0.0
|
||||
requests>=2.31.0
|
||||
PyPDF2>=3.0.0
|
||||
|
||||
# Data processing and visualization
|
||||
numpy>=1.24.0
|
||||
matplotlib>=3.7.0
|
||||
tabulate>=0.9.0
|
||||
|
||||
# PDF generation for sample data
|
||||
reportlab>=4.0.0
|
||||
|
||||
# Optional: for better PDF parsing (uncomment if needed)
|
||||
# pdfplumber>=0.10.0
|
||||
# pymupdf>=1.23.0
|
||||
|
||||
# Development tools (optional)
|
||||
python-dotenv>=1.0.0
|
||||
pytest>=7.4.0
|
||||
black>=23.0.0
|
||||
flake8>=6.0.0
|
||||
@@ -0,0 +1,463 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the exact five-arm context ablation from book/chapter1.md.
|
||||
|
||||
Unlike the legacy demo table, this runner persists every credential-free API
|
||||
request and response. That makes it possible to prove which context component
|
||||
was removed on every inference instead of inferring the ablation from a CLI
|
||||
flag after the fact.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
|
||||
EXPERIMENT_ID = "1-1"
|
||||
MODES = list(ContextMode)
|
||||
CANONICAL_TASK = """According to the company's quarterly revenue:
|
||||
- Q1: 2.5 million USD
|
||||
- Q2: 2.1 million EUR
|
||||
- Q3: 1.8 million GBP
|
||||
- Q4: 380 million JPY
|
||||
|
||||
Use the available currency-conversion and calculation tools to convert every
|
||||
non-USD quarter to USD, then calculate the annual total and quarterly average.
|
||||
Report both values rounded to two decimal places. Do not estimate exchange
|
||||
rates yourself; use the tool observations."""
|
||||
|
||||
EXPECTED_NUMBERS = ("9602895.73", "2400723.93")
|
||||
KEY_ENV = {
|
||||
"dashscope": ("DASHSCOPE_API_KEY",),
|
||||
"qwen": ("DASHSCOPE_API_KEY",),
|
||||
"bailian": ("DASHSCOPE_API_KEY",),
|
||||
"kimi": ("MOONSHOT_API_KEY", "KIMI_API_KEY"),
|
||||
"moonshot": ("MOONSHOT_API_KEY", "KIMI_API_KEY"),
|
||||
"doubao": ("ARK_API_KEY",),
|
||||
"siliconflow": ("SILICONFLOW_API_KEY",),
|
||||
"deepseek": ("DEEPSEEK_API_KEY",),
|
||||
"openrouter": ("OPENROUTER_API_KEY",),
|
||||
}
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def git_value(*args: str) -> str | None:
|
||||
try:
|
||||
return subprocess.check_output(
|
||||
["git", *args], text=True, stderr=subprocess.DEVNULL
|
||||
).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def package_version(distribution: str) -> str | None:
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
|
||||
return version(distribution)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def resolve_key(provider: str) -> tuple[str, str]:
|
||||
names = KEY_ENV.get(provider, ())
|
||||
for name in names:
|
||||
value = os.getenv(name)
|
||||
if value:
|
||||
return value, name
|
||||
raise RuntimeError(
|
||||
f"No direct credential for {provider}; expected one of {', '.join(names)}"
|
||||
)
|
||||
|
||||
|
||||
def tool_call_dict(call: Any) -> Dict[str, Any]:
|
||||
return {
|
||||
"tool_name": call.tool_name,
|
||||
"arguments": call.arguments,
|
||||
"result": call.result,
|
||||
"timestamp": call.timestamp,
|
||||
}
|
||||
|
||||
|
||||
def call_signatures(tool_calls: Iterable[Dict[str, Any]]) -> List[str]:
|
||||
signatures = []
|
||||
for call in tool_calls:
|
||||
signatures.append(
|
||||
f"{call['tool_name']}:"
|
||||
+ json.dumps(call.get("arguments", {}), sort_keys=True, ensure_ascii=False)
|
||||
)
|
||||
return signatures
|
||||
|
||||
|
||||
def response_message(turn: Dict[str, Any]) -> Dict[str, Any]:
|
||||
choices = turn.get("response", {}).get("choices") or []
|
||||
return (choices[0].get("message") or {}) if choices else {}
|
||||
|
||||
|
||||
def request_roles(turn: Dict[str, Any]) -> List[str]:
|
||||
return [message.get("role") for message in turn.get("request", {}).get("messages", [])]
|
||||
|
||||
|
||||
def evaluate_context_contract(mode: str, turns: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""Verify the actual provider request, not the requested CLI mode."""
|
||||
requests = [turn.get("request", {}) for turn in turns if turn.get("request")]
|
||||
real_responses = [turn for turn in turns if turn.get("response", {}).get("id")]
|
||||
details: Dict[str, Any] = {
|
||||
"has_provider_response_ids": len(real_responses) == len(turns) and bool(turns),
|
||||
"turn_count": len(turns),
|
||||
"request_roles": [request_roles(turn) for turn in turns],
|
||||
}
|
||||
|
||||
if mode == ContextMode.FULL.value:
|
||||
details.update(
|
||||
{
|
||||
"tools_present_every_turn": all(bool(r.get("tools")) for r in requests),
|
||||
"history_present_after_first_turn": len(requests) > 1
|
||||
and all(
|
||||
"assistant" in [m.get("role") for m in r.get("messages", [])]
|
||||
and "tool" in [m.get("role") for m in r.get("messages", [])]
|
||||
for r in requests[1:]
|
||||
),
|
||||
"reasoning_retained_after_first_turn": len(requests) > 1
|
||||
and any(
|
||||
bool(m.get("reasoning_content"))
|
||||
for m in requests[1].get("messages", [])
|
||||
if m.get("role") == "assistant"
|
||||
),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"tools_present_every_turn",
|
||||
"history_present_after_first_turn",
|
||||
"reasoning_retained_after_first_turn",
|
||||
)
|
||||
elif mode == ContextMode.NO_TOOL_CALLS.value:
|
||||
details.update(
|
||||
{
|
||||
"tools_absent_every_turn": all(
|
||||
"tools" not in r and "tool_choice" not in r for r in requests
|
||||
),
|
||||
}
|
||||
)
|
||||
required = ("has_provider_response_ids", "tools_absent_every_turn")
|
||||
elif mode == ContextMode.NO_TOOL_RESULTS.value:
|
||||
tool_messages = [
|
||||
m
|
||||
for r in requests[1:]
|
||||
for m in r.get("messages", [])
|
||||
if m.get("role") == "tool"
|
||||
]
|
||||
details.update(
|
||||
{
|
||||
"tool_calls_retained": any(
|
||||
m.get("role") == "assistant" and m.get("tool_calls")
|
||||
for r in requests[1:]
|
||||
for m in r.get("messages", [])
|
||||
),
|
||||
"tool_results_hidden": bool(tool_messages)
|
||||
and all(
|
||||
m.get("content") == "[Tool result hidden due to context mode]"
|
||||
for m in tool_messages
|
||||
),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"tool_calls_retained",
|
||||
"tool_results_hidden",
|
||||
)
|
||||
elif mode == ContextMode.NO_REASONING.value:
|
||||
assistant_history = [
|
||||
m
|
||||
for r in requests[1:]
|
||||
for m in r.get("messages", [])
|
||||
if m.get("role") == "assistant"
|
||||
]
|
||||
provider_reasoning = [
|
||||
response_message(turn).get("reasoning_content") for turn in turns
|
||||
]
|
||||
details.update(
|
||||
{
|
||||
"provider_generated_reasoning": any(provider_reasoning),
|
||||
"reasoning_removed_from_history": bool(assistant_history)
|
||||
and all(not m.get("reasoning_content") for m in assistant_history),
|
||||
"tool_and_result_history_retained": any(
|
||||
"tool" in request_roles(turn) for turn in turns[1:]
|
||||
),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"provider_generated_reasoning",
|
||||
"reasoning_removed_from_history",
|
||||
"tool_and_result_history_retained",
|
||||
)
|
||||
elif mode == ContextMode.NO_HISTORY.value:
|
||||
details.update(
|
||||
{
|
||||
"only_static_prefix_and_user_every_turn": bool(requests)
|
||||
and all(
|
||||
[m.get("role") for m in r.get("messages", [])]
|
||||
== ["system", "user"]
|
||||
for r in requests
|
||||
),
|
||||
"tools_still_present": all(bool(r.get("tools")) for r in requests),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"only_static_prefix_and_user_every_turn",
|
||||
"tools_still_present",
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unknown mode: {mode}")
|
||||
|
||||
details["required_checks"] = list(required)
|
||||
details["passed"] = all(details[name] is True for name in required)
|
||||
return details
|
||||
|
||||
|
||||
def normalized_number_text(value: str | None) -> str:
|
||||
return (value or "").replace(",", "").replace("$", "").replace(" ", "")
|
||||
|
||||
|
||||
def canonical_answer_correct(final_answer: str | None) -> bool:
|
||||
"""Evaluate the known numeric rubric for the canonical Experiment 1-1 task.
|
||||
|
||||
This is deliberately kept outside ``ContextAwareAgent``. A generic agent
|
||||
cannot infer correctness from an arbitrary natural-language task, while
|
||||
this experiment has an explicit answer rubric.
|
||||
"""
|
||||
normalized = normalized_number_text(final_answer)
|
||||
return bool(final_answer) and all(number in normalized for number in EXPECTED_NUMBERS)
|
||||
|
||||
|
||||
def summarize_arm(mode: ContextMode, result: Dict[str, Any], elapsed: float) -> Dict[str, Any]:
|
||||
trajectory = result["trajectory"]
|
||||
tool_calls = [tool_call_dict(call) for call in trajectory.tool_calls]
|
||||
signatures = call_signatures(tool_calls)
|
||||
repeats = len(signatures) - len(set(signatures))
|
||||
final_answer = result.get("final_answer")
|
||||
completed = bool(result.get("completed", result.get("success", False)))
|
||||
task_success = canonical_answer_correct(final_answer)
|
||||
arm = {
|
||||
"mode": mode.value,
|
||||
"provider": result.get("provider"),
|
||||
"model": result.get("model"),
|
||||
"base_url": result.get("base_url"),
|
||||
"using_openrouter": result.get("using_openrouter", False),
|
||||
"started_at": None,
|
||||
"elapsed_seconds": round(elapsed, 6),
|
||||
# ``success`` is retained for compatibility with existing evidence;
|
||||
# it means terminal response/completion, not task correctness.
|
||||
"success": completed,
|
||||
"completed": completed,
|
||||
"task_success": task_success,
|
||||
"iterations": result.get("iterations", 0),
|
||||
"error": result.get("error"),
|
||||
"final_answer": final_answer,
|
||||
"tool_calls": tool_calls,
|
||||
"tool_call_signatures": signatures,
|
||||
"repeated_tool_calls": repeats,
|
||||
"reasoning_steps": trajectory.reasoning_steps,
|
||||
"api_turns": trajectory.api_turns,
|
||||
}
|
||||
arm["context_contract"] = evaluate_context_contract(mode.value, trajectory.api_turns)
|
||||
arm["behavior"] = {
|
||||
"tool_action_count": len(tool_calls),
|
||||
"has_repeated_tool_action": repeats > 0,
|
||||
"hit_iteration_ceiling": result.get("iterations") >= 5 and not completed,
|
||||
"canonical_answer_correct": task_success,
|
||||
}
|
||||
return arm
|
||||
|
||||
|
||||
def token_usage(arms: List[Dict[str, Any]]) -> Dict[str, int]:
|
||||
prompt = completion = cached = reasoning = 0
|
||||
for arm in arms:
|
||||
for turn in arm["api_turns"]:
|
||||
usage = turn.get("response", {}).get("usage") or {}
|
||||
prompt += int(usage.get("prompt_tokens") or usage.get("input_tokens") or 0)
|
||||
completion += int(
|
||||
usage.get("completion_tokens") or usage.get("output_tokens") or 0
|
||||
)
|
||||
prompt_details = usage.get("prompt_tokens_details") or usage.get(
|
||||
"input_tokens_details"
|
||||
) or {}
|
||||
completion_details = usage.get("completion_tokens_details") or usage.get(
|
||||
"output_tokens_details"
|
||||
) or {}
|
||||
cached += int(prompt_details.get("cached_tokens") or 0)
|
||||
reasoning += int(completion_details.get("reasoning_tokens") or 0)
|
||||
return {
|
||||
"prompt_tokens": prompt,
|
||||
"completion_tokens": completion,
|
||||
"total_tokens": prompt + completion,
|
||||
"cached_prompt_tokens": cached,
|
||||
"reasoning_tokens": reasoning,
|
||||
}
|
||||
|
||||
|
||||
def analyze(arms: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
by_mode = {arm["mode"]: arm for arm in arms}
|
||||
exact_five_arms = set(by_mode) == {mode.value for mode in MODES}
|
||||
contracts_pass = exact_five_arms and all(
|
||||
arm["context_contract"]["passed"] for arm in arms
|
||||
)
|
||||
direct_real_api = all(
|
||||
not arm["using_openrouter"]
|
||||
and arm["api_turns"]
|
||||
and all(turn.get("response", {}).get("id") for turn in arm["api_turns"])
|
||||
for arm in arms
|
||||
)
|
||||
behavior = {
|
||||
"full_baseline_correct": by_mode.get("full", {}).get("behavior", {}).get(
|
||||
"canonical_answer_correct", by_mode.get("full", {}).get("task_success", False)
|
||||
),
|
||||
"without_tool_definitions_no_tool_action": by_mode.get(
|
||||
"no_tool_calls", {}
|
||||
).get("behavior", {}).get("tool_action_count")
|
||||
== 0,
|
||||
"without_tool_results_repeated_action": by_mode.get(
|
||||
"no_tool_results", {}
|
||||
).get("behavior", {}).get("has_repeated_tool_action", False),
|
||||
"without_history_repeated_action": by_mode.get("no_history", {}).get(
|
||||
"behavior", {}
|
||||
).get("has_repeated_tool_action", False),
|
||||
# Contradiction is an empirical outcome, not something the harness can
|
||||
# legitimately force. We report whether the no-reasoning answer lost
|
||||
# canonical correctness and keep this separate from execution validity.
|
||||
"without_reasoning_degraded": not by_mode.get("no_reasoning", {}).get(
|
||||
"behavior", {}
|
||||
).get("canonical_answer_correct", False),
|
||||
}
|
||||
behavior["all_manuscript_behavior_claims_observed"] = all(behavior.values())
|
||||
return {
|
||||
"exact_five_arms_present": exact_five_arms,
|
||||
"all_context_contracts_passed": contracts_pass,
|
||||
"direct_real_api_evidence": direct_real_api,
|
||||
"experiment_execution_accepted": bool(
|
||||
exact_five_arms
|
||||
and contracts_pass
|
||||
and direct_real_api
|
||||
and behavior["full_baseline_correct"]
|
||||
),
|
||||
"manuscript_behavior_claims": behavior,
|
||||
"usage": token_usage(arms),
|
||||
}
|
||||
|
||||
|
||||
def write_json(path: Path, payload: Dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--provider", default="kimi", choices=sorted(KEY_ENV))
|
||||
parser.add_argument("--model", default="kimi-k3")
|
||||
parser.add_argument("--max-iterations", type=int, default=5)
|
||||
parser.add_argument("--output-dir", type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.max_iterations < 2:
|
||||
parser.error("--max-iterations must be at least 2")
|
||||
|
||||
key, key_env = resolve_key(args.provider)
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
output_dir = args.output_dir or Path("validation") / f"real_{stamp}"
|
||||
command = [
|
||||
sys.executable,
|
||||
Path(__file__).name,
|
||||
"--provider",
|
||||
args.provider,
|
||||
"--model",
|
||||
args.model,
|
||||
"--max-iterations",
|
||||
str(args.max_iterations),
|
||||
"--output-dir",
|
||||
str(output_dir),
|
||||
]
|
||||
|
||||
arms = []
|
||||
for mode in MODES:
|
||||
started = utc_now()
|
||||
agent = ContextAwareAgent(
|
||||
key,
|
||||
context_mode=mode,
|
||||
provider=args.provider,
|
||||
model=args.model,
|
||||
verbose=False,
|
||||
)
|
||||
begin = time.monotonic()
|
||||
result = agent.execute_task(CANONICAL_TASK, max_iterations=args.max_iterations)
|
||||
arm = summarize_arm(mode, result, time.monotonic() - begin)
|
||||
arm["started_at"] = started
|
||||
# Recompute the configured ceiling rather than retaining the default in
|
||||
# the pure summarizer (which is also exercised by unit tests).
|
||||
arm["behavior"]["hit_iteration_ceiling"] = (
|
||||
result.get("iterations") >= args.max_iterations and not result.get("success")
|
||||
)
|
||||
arms.append(arm)
|
||||
|
||||
evidence: Dict[str, Any] = {
|
||||
"schema_version": "1.0",
|
||||
"experiment_id": EXPERIMENT_ID,
|
||||
"evidence_mode": "real_api",
|
||||
"created_at": utc_now(),
|
||||
"canonical_source": "book/chapter1.md#实验-1-1-上下文的关键作用",
|
||||
"task": CANONICAL_TASK,
|
||||
"expected_numbers": list(EXPECTED_NUMBERS),
|
||||
"command": command,
|
||||
"credential_source_env": key_env,
|
||||
"credential_value_recorded": False,
|
||||
"host": {
|
||||
"platform": platform.platform(),
|
||||
"python": sys.version,
|
||||
"machine": platform.machine(),
|
||||
},
|
||||
"dependencies": {
|
||||
"openai": package_version("openai"),
|
||||
"requests": package_version("requests"),
|
||||
},
|
||||
"repository": {
|
||||
"commit": git_value("rev-parse", "HEAD"),
|
||||
"branch": git_value("branch", "--show-current"),
|
||||
"worktree_dirty": bool(git_value("status", "--porcelain")),
|
||||
},
|
||||
"arms": arms,
|
||||
}
|
||||
evidence["analysis"] = analyze(arms)
|
||||
evidence_path = output_dir / "evidence.json"
|
||||
write_json(evidence_path, evidence)
|
||||
digest = hashlib.sha256(evidence_path.read_bytes()).hexdigest()
|
||||
(output_dir / "evidence.sha256").write_text(
|
||||
f"{digest} evidence.json\n", encoding="utf-8"
|
||||
)
|
||||
latest = Path("validation/latest.json")
|
||||
latest.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(evidence_path, latest)
|
||||
|
||||
print(json.dumps(evidence["analysis"], ensure_ascii=False, indent=2))
|
||||
print(f"Evidence: {evidence_path}")
|
||||
print(f"SHA-256: {digest}")
|
||||
return 0 if evidence["analysis"]["experiment_execution_accepted"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,125 @@
|
||||
from agent import AgentTrajectory, ContextMode
|
||||
from run_experiment_1_1 import (
|
||||
canonical_answer_correct,
|
||||
evaluate_context_contract,
|
||||
summarize_arm,
|
||||
)
|
||||
|
||||
|
||||
def turn(messages, *, tools=True, reasoning="reason"):
|
||||
request = {"messages": messages}
|
||||
if tools:
|
||||
request.update({"tools": [{"type": "function"}], "tool_choice": "auto"})
|
||||
return {
|
||||
"request": request,
|
||||
"response": {
|
||||
"id": "real-response-id",
|
||||
"choices": [{"message": {"reasoning_content": reasoning}}],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
SYSTEM = {"role": "system", "content": "system"}
|
||||
USER = {"role": "user", "content": "task"}
|
||||
ASSISTANT = {
|
||||
"role": "assistant",
|
||||
"reasoning_content": "reason",
|
||||
"tool_calls": [{"id": "call"}],
|
||||
}
|
||||
TOOL = {"role": "tool", "content": '{"result": 4}'}
|
||||
|
||||
|
||||
def test_full_contract_uses_raw_followup_context():
|
||||
result = evaluate_context_contract(
|
||||
"full", [turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, TOOL])]
|
||||
)
|
||||
assert result["passed"] is True
|
||||
|
||||
|
||||
def test_no_history_contract_rejects_sliding_window():
|
||||
exact = evaluate_context_contract(
|
||||
"no_history", [turn([SYSTEM, USER]), turn([SYSTEM, USER])]
|
||||
)
|
||||
sliding = evaluate_context_contract(
|
||||
"no_history", [turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, TOOL])]
|
||||
)
|
||||
assert exact["passed"] is True
|
||||
assert sliding["passed"] is False
|
||||
|
||||
|
||||
def test_no_reasoning_requires_provider_reasoning_but_stripped_history():
|
||||
stripped_assistant = {k: v for k, v in ASSISTANT.items() if k != "reasoning_content"}
|
||||
result = evaluate_context_contract(
|
||||
"no_reasoning",
|
||||
[turn([SYSTEM, USER]), turn([SYSTEM, USER, stripped_assistant, TOOL])],
|
||||
)
|
||||
assert result["passed"] is True
|
||||
|
||||
|
||||
def test_no_tool_results_requires_literal_hidden_observations():
|
||||
hidden = {"role": "tool", "content": "[Tool result hidden due to context mode]"}
|
||||
result = evaluate_context_contract(
|
||||
"no_tool_results",
|
||||
[turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, hidden])],
|
||||
)
|
||||
assert result["passed"] is True
|
||||
leaked = evaluate_context_contract(
|
||||
"no_tool_results", [turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, TOOL])]
|
||||
)
|
||||
assert leaked["passed"] is False
|
||||
|
||||
|
||||
def test_no_tool_definitions_requires_absent_request_fields():
|
||||
result = evaluate_context_contract("no_tool_calls", [turn([SYSTEM, USER], tools=False)])
|
||||
assert result["passed"] is True
|
||||
|
||||
|
||||
def _arm_result(final_answer, *, mode=ContextMode.NO_TOOL_CALLS, iterations=1):
|
||||
completed = final_answer is not None
|
||||
return {
|
||||
"trajectory": AgentTrajectory(context_mode=mode),
|
||||
"final_answer": final_answer,
|
||||
"completed": completed,
|
||||
"success": completed,
|
||||
"iterations": iterations,
|
||||
"provider": "test",
|
||||
"model": "test-model",
|
||||
}
|
||||
|
||||
|
||||
def test_canonical_answer_rubric_rejects_refusal_and_hallucinated_markup():
|
||||
refusal = "I cannot compute the exchange rates without tools."
|
||||
hallucinated = "<request_tool>currency_converter(...)</request_tool>"
|
||||
assert canonical_answer_correct(refusal) is False
|
||||
assert canonical_answer_correct(hallucinated) is False
|
||||
|
||||
|
||||
def test_summarize_arm_separates_completion_from_task_success():
|
||||
result = summarize_arm(
|
||||
ContextMode.NO_TOOL_CALLS,
|
||||
_arm_result("I cannot compute the exchange rates without tools."),
|
||||
elapsed=0.1,
|
||||
)
|
||||
|
||||
# The model did return a terminal response, but it did not complete the
|
||||
# canonical financial task. A mode-independent evaluator must preserve
|
||||
# that distinction instead of forcing the mode to fail.
|
||||
assert result["completed"] is True
|
||||
assert result["success"] is True # compatibility alias
|
||||
assert result["task_success"] is False
|
||||
assert result["behavior"]["canonical_answer_correct"] is False
|
||||
|
||||
|
||||
def test_summarize_arm_accepts_correct_answer_even_in_an_ablated_arm():
|
||||
answer = "Annual total: $9,602,895.73; quarterly average: $2,400,723.93"
|
||||
result = summarize_arm(
|
||||
ContextMode.NO_TOOL_RESULTS,
|
||||
_arm_result(answer, mode=ContextMode.NO_TOOL_RESULTS),
|
||||
elapsed=0.1,
|
||||
)
|
||||
|
||||
# Correctness is an observed task result. The experiment may separately
|
||||
# report that tool feedback was hidden; it must not manufacture failure.
|
||||
assert result["completed"] is True
|
||||
assert result["task_success"] is True
|
||||
assert result["behavior"]["canonical_answer_correct"] is True
|
||||
@@ -0,0 +1,11 @@
|
||||
from main import _completed
|
||||
|
||||
|
||||
def test_completed_field_is_authoritative_over_legacy_success_alias():
|
||||
assert _completed({"completed": False, "success": True}) is False
|
||||
assert _completed({"completed": True, "success": False}) is True
|
||||
|
||||
|
||||
def test_completed_falls_back_for_old_result_artifacts():
|
||||
assert _completed({"success": True}) is True
|
||||
assert _completed({"success": False}) is False
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Pytest bootstrap for the context experiment tests."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Helpers for running manual smoke scripts from tests/manual."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
def add_project_root() -> Path:
|
||||
project_root = Path(__file__).resolve().parents[2]
|
||||
if str(project_root) not in sys.path:
|
||||
sys.path.insert(0, str(project_root))
|
||||
return project_root
|
||||
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify conversation history persistence
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
import json
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def test_conversation_history():
|
||||
"""Test that conversation history persists between tasks"""
|
||||
print("🧪 Testing Conversation History Persistence")
|
||||
print("=" * 50)
|
||||
|
||||
# Get API key (use any available provider)
|
||||
api_key = (
|
||||
os.getenv("ARK_API_KEY")
|
||||
or os.getenv("DASHSCOPE_API_KEY")
|
||||
or os.getenv("MOONSHOT_API_KEY")
|
||||
or os.getenv("SILICONFLOW_API_KEY")
|
||||
)
|
||||
provider = (
|
||||
"doubao"
|
||||
if os.getenv("ARK_API_KEY")
|
||||
else (
|
||||
"dashscope"
|
||||
if os.getenv("DASHSCOPE_API_KEY")
|
||||
else ("kimi" if os.getenv("MOONSHOT_API_KEY") else "siliconflow")
|
||||
)
|
||||
)
|
||||
|
||||
if not api_key:
|
||||
print("❌ No API key found. Please set one of:")
|
||||
print(" - ARK_API_KEY")
|
||||
print(" - DASHSCOPE_API_KEY")
|
||||
print(" - MOONSHOT_API_KEY")
|
||||
print(" - SILICONFLOW_API_KEY")
|
||||
return False
|
||||
|
||||
print(f"Using provider: {provider}")
|
||||
print("-" * 50)
|
||||
|
||||
try:
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider=provider,
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test 1: First query
|
||||
print("\n📝 Test 1: First query")
|
||||
query1 = "Remember that my favorite number is 42. What is 10 + 5?"
|
||||
result1 = agent.execute_task(query1)
|
||||
print(f"Query: {query1}")
|
||||
print(f"Response: {result1.get('final_answer', 'No answer')}")
|
||||
|
||||
# Check conversation history
|
||||
print(f"\n📚 Conversation history after first query:")
|
||||
print(f" Total messages: {len(agent.conversation_history)}")
|
||||
|
||||
# Print message roles
|
||||
for i, msg in enumerate(agent.conversation_history):
|
||||
role = msg.get('role', 'unknown')
|
||||
content_preview = str(msg.get('content', ''))[:50] + "..." if len(str(msg.get('content', ''))) > 50 else str(msg.get('content', ''))
|
||||
print(f" Message {i}: Role={role}, Content={content_preview}")
|
||||
|
||||
# Test 2: Second query that references first
|
||||
print("\n📝 Test 2: Second query (should remember context)")
|
||||
query2 = "What was my favorite number that I mentioned earlier?"
|
||||
result2 = agent.execute_task(query2)
|
||||
print(f"Query: {query2}")
|
||||
print(f"Response: {result2.get('final_answer', 'No answer')}")
|
||||
|
||||
# Check if 42 is mentioned in the response
|
||||
if "42" in str(result2.get('final_answer', '')):
|
||||
print("✅ SUCCESS: Agent remembered the favorite number from conversation history!")
|
||||
else:
|
||||
print("⚠️ WARNING: Agent might not have remembered the number. Check response above.")
|
||||
|
||||
# Check conversation history growth
|
||||
print(f"\n📚 Conversation history after second query:")
|
||||
print(f" Total messages: {len(agent.conversation_history)}")
|
||||
|
||||
# Test 3: Verify system prompt unchanged
|
||||
print("\n📝 Test 3: Verify system prompt remains unchanged")
|
||||
system_prompt = agent.conversation_history[0].get('content', '')
|
||||
if "favorite number" not in system_prompt and "42" not in system_prompt:
|
||||
print("✅ SUCCESS: System prompt remains unchanged!")
|
||||
else:
|
||||
print("❌ FAILURE: System prompt was modified!")
|
||||
|
||||
# Test 4: Reset and verify history cleared
|
||||
print("\n📝 Test 4: Test reset functionality")
|
||||
agent.reset()
|
||||
print(f" Messages after reset: {len(agent.conversation_history)}")
|
||||
|
||||
if len(agent.conversation_history) == 1 and agent.conversation_history[0]['role'] == 'system':
|
||||
print("✅ SUCCESS: Reset properly cleared history and kept system prompt!")
|
||||
else:
|
||||
print("❌ FAILURE: Reset did not work correctly!")
|
||||
|
||||
# Test 5: New conversation after reset
|
||||
print("\n📝 Test 5: New conversation after reset")
|
||||
query3 = "What was my favorite number?"
|
||||
result3 = agent.execute_task(query3)
|
||||
print(f"Query: {query3}")
|
||||
print(f"Response: {result3.get('final_answer', 'No answer')}")
|
||||
|
||||
if "42" not in str(result3.get('final_answer', '')) and "don't" in str(result3.get('final_answer', '').lower()):
|
||||
print("✅ SUCCESS: Agent correctly doesn't remember after reset!")
|
||||
else:
|
||||
print("⚠️ Check if agent properly forgot the previous conversation")
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("Conversation history tests complete!")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = test_conversation_history()
|
||||
exit(0 if success else 1)
|
||||
@@ -0,0 +1,226 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for DeepSeek model integration.
|
||||
Tests deepseek-v4-flash (default) with conversation and tool calling.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def test_basic_conversation():
|
||||
"""Test basic conversation capabilities"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 1: Basic Conversation")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set in environment")
|
||||
print("Please set it in your .env file or as environment variable")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
query = "What is 25 * 4 + 10? Reply with FINAL ANSWER: and the number."
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
if "110" in response:
|
||||
print("\n✅ Basic conversation test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n❌ Test failed - incorrect answer")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_tool_usage():
|
||||
"""Test tool calling capabilities"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 2: Tool Usage (Calculator)")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
query = (
|
||||
"Calculate: (123.45 * 67.89) / 12.34 + sqrt(144) - 2^8. "
|
||||
"Use the calculate tool. End with FINAL ANSWER:"
|
||||
)
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
if agent.trajectory.tool_calls:
|
||||
print(f"\n🔧 Tools used: {len(agent.trajectory.tool_calls)}")
|
||||
for call in agent.trajectory.tool_calls:
|
||||
print(f" - {call.tool_name}: {call.arguments}")
|
||||
print("\n✅ Tool usage test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ No tools were used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_currency_conversion():
|
||||
"""Test currency conversion tool"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 3: Currency Conversion")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
query = "Convert 100 USD to EUR and JPY. Use convert_currency. FINAL ANSWER: the amounts."
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
tool_names = [call.tool_name for call in agent.trajectory.tool_calls]
|
||||
if "convert_currency" in tool_names:
|
||||
print("\n🔧 Currency converter was used")
|
||||
print("\n✅ Currency conversion test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ Currency converter was not used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_model_info():
|
||||
"""Test and display model information"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 4: Model Information")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
expected = Config.get_default_model("deepseek")
|
||||
print("\n📊 Model Configuration:")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Expected default: {expected}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
print(f" Context Mode: {agent.context_mode.value}")
|
||||
|
||||
if agent.provider != "deepseek" or agent.model != expected:
|
||||
print("\n❌ Model config mismatch")
|
||||
return False
|
||||
|
||||
print("\n✅ Model info test completed!")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all tests"""
|
||||
print("\n" + "=" * 60)
|
||||
print("DEEPSEEK MODEL INTEGRATION TEST SUITE")
|
||||
print("=" * 60)
|
||||
print("\nModel: deepseek-v4-flash (default)")
|
||||
print("Provider: DeepSeek")
|
||||
print("API: https://api.deepseek.com")
|
||||
|
||||
if not os.getenv("DEEPSEEK_API_KEY"):
|
||||
print("\n❌ ERROR: DEEPSEEK_API_KEY not found in environment")
|
||||
print("\nPlease set up your .env file with:")
|
||||
print(" DEEPSEEK_API_KEY=your_api_key_here")
|
||||
print("\nYou can get an API key from: https://platform.deepseek.com/api_keys")
|
||||
sys.exit(1)
|
||||
|
||||
results = []
|
||||
results.append(("Model Information", test_model_info()))
|
||||
results.append(("Basic Conversation", test_basic_conversation()))
|
||||
results.append(("Tool Usage", test_tool_usage()))
|
||||
results.append(("Currency Conversion", test_currency_conversion()))
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST SUMMARY")
|
||||
print("=" * 60)
|
||||
|
||||
passed = sum(1 for _, result in results if result)
|
||||
total = len(results)
|
||||
|
||||
for test_name, result in results:
|
||||
status = "✅ PASSED" if result else "❌ FAILED"
|
||||
print(f" {test_name}: {status}")
|
||||
|
||||
print(f"\nTotal: {passed}/{total} tests passed")
|
||||
|
||||
if passed == total:
|
||||
print("\n🎉 All tests passed! DeepSeek integration is working correctly.")
|
||||
else:
|
||||
print(f"\n⚠️ {total - passed} test(s) failed. Please check the errors above.")
|
||||
|
||||
return passed == total
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = main()
|
||||
sys.exit(0 if success else 1)
|
||||
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick smoke test for DeepSeek provider (deepseek-v4-flash).
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
task = "What is 10 + 5? Provide FINAL ANSWER with just the number."
|
||||
|
||||
print("=" * 60)
|
||||
print("QUICK TEST - DeepSeek Provider")
|
||||
print("=" * 60)
|
||||
|
||||
deepseek_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not deepseek_key:
|
||||
print("❌ DEEPSEEK_API_KEY not set")
|
||||
print("Set it in .env or: export DEEPSEEK_API_KEY=your_key")
|
||||
print("Get a key at: https://platform.deepseek.com/api_keys")
|
||||
sys.exit(1)
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
agent = ContextAwareAgent(deepseek_key, ContextMode.FULL, provider="deepseek")
|
||||
print(f"✅ Using: {agent.provider} / {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
print(f"\n📝 Task: {task}")
|
||||
print("-" * 40)
|
||||
|
||||
start = time.time()
|
||||
print("Processing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task, max_iterations=3)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"\n✅ Completed in {elapsed:.2f} seconds")
|
||||
|
||||
if result.get("success"):
|
||||
print("Success: True")
|
||||
if result.get("final_answer"):
|
||||
print(f"Answer: {result['final_answer']}")
|
||||
else:
|
||||
print("Success: False")
|
||||
if result.get("error"):
|
||||
print(f"Error: {result['error']}")
|
||||
|
||||
print(f"Iterations: {result.get('iterations', 0)}")
|
||||
print(f"Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted")
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {str(e)}")
|
||||
sys.exit(1)
|
||||
|
||||
print("=" * 60)
|
||||
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test that Doubao is the default provider
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
# Test without any arguments - should use Doubao
|
||||
print("Testing default provider...")
|
||||
|
||||
# Check if ARK_API_KEY is available
|
||||
ark_key = os.getenv("ARK_API_KEY")
|
||||
sf_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
|
||||
print(f"ARK_API_KEY available: {'Yes' if ark_key else 'No'}")
|
||||
print(f"SILICONFLOW_API_KEY available: {'Yes' if sf_key else 'No'}")
|
||||
|
||||
if ark_key:
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Check config default
|
||||
print(f"\nConfig default provider: {Config.LLM_PROVIDER}")
|
||||
|
||||
# Create agent with default provider from config
|
||||
agent = ContextAwareAgent(ark_key, ContextMode.FULL, provider=Config.LLM_PROVIDER)
|
||||
|
||||
print(f"\n✅ Default agent created successfully!")
|
||||
print(f"Provider: {agent.provider}")
|
||||
print(f"Model: {agent.model}")
|
||||
print(f"Base URL: {agent.client.base_url}")
|
||||
|
||||
if agent.provider == "doubao":
|
||||
print("\n🎉 SUCCESS: Doubao is the default provider!")
|
||||
else:
|
||||
print(f"\n❌ ERROR: Expected doubao, got {agent.provider}")
|
||||
sys.exit(1)
|
||||
else:
|
||||
print("\n⚠️ ARK_API_KEY not set. Cannot test default provider.")
|
||||
print("Please set: export ARK_API_KEY=your_key_here")
|
||||
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick test for Doubao provider
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_doubao():
|
||||
"""Test Doubao provider with a simple task"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 DOUBAO PROVIDER TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check for API key
|
||||
api_key = os.getenv("ARK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ARK_API_KEY not found. Please set it to test Doubao provider.")
|
||||
print(" export ARK_API_KEY=your_key_here")
|
||||
return
|
||||
|
||||
print("✅ ARK API key found")
|
||||
|
||||
# Create agent with Doubao provider
|
||||
try:
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL, provider="doubao")
|
||||
print(f"✅ Agent created with Doubao provider")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
|
||||
# Simple test task (minimal to save tokens)
|
||||
print("\n📝 Running simple test task...")
|
||||
task = "Calculate: What is 15 + 27? Provide FINAL ANSWER with the result."
|
||||
|
||||
result = agent.execute_task(task, max_iterations=3)
|
||||
|
||||
if result.get('success'):
|
||||
print("✅ Task executed successfully!")
|
||||
if result.get('final_answer'):
|
||||
print(f" Answer: {result['final_answer'][:100]}...")
|
||||
else:
|
||||
print(f"⚠️ Task did not complete successfully")
|
||||
if result.get('error'):
|
||||
print(f" Error: {result['error']}")
|
||||
|
||||
print(f"\n📊 Execution stats:")
|
||||
print(f" Iterations: {result.get('iterations', 0)}")
|
||||
print(f" Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {str(e)}")
|
||||
print("\nNote: Make sure your ARK_API_KEY is valid and has access to the doubao model.")
|
||||
|
||||
print("\n" + "="*60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_doubao()
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick test with Doubao as default
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
# Set a very simple task to test quickly
|
||||
task = "What is 10 + 5? Provide FINAL ANSWER with just the number."
|
||||
|
||||
print("="*60)
|
||||
print("QUICK TEST - Doubao Default Provider")
|
||||
print("="*60)
|
||||
|
||||
ark_key = os.getenv("ARK_API_KEY")
|
||||
if not ark_key:
|
||||
print("❌ ARK_API_KEY not set")
|
||||
sys.exit(1)
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
# Create agent with default Doubao
|
||||
agent = ContextAwareAgent(ark_key, ContextMode.FULL, provider="doubao")
|
||||
print(f"✅ Using: {agent.provider} / {agent.model}")
|
||||
print(f"\n📝 Task: {task}")
|
||||
print("-"*40)
|
||||
|
||||
start = time.time()
|
||||
print("Processing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task, max_iterations=2)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"\n✅ Completed in {elapsed:.2f} seconds")
|
||||
|
||||
if result.get('success'):
|
||||
print(f"Success: True")
|
||||
if result.get('final_answer'):
|
||||
print(f"Answer: {result['final_answer']}")
|
||||
else:
|
||||
print(f"Success: False")
|
||||
if result.get('error'):
|
||||
print(f"Error: {result['error']}")
|
||||
|
||||
print(f"Iterations: {result.get('iterations', 0)}")
|
||||
print(f"Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted")
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {str(e)}")
|
||||
|
||||
print("="*60)
|
||||
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for Kimi K3 model integration
|
||||
Tests the Kimi K3 model (kimi-k3) with various tasks
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def test_basic_conversation():
|
||||
"""Test basic conversation capabilities"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 1: Basic Conversation")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set in environment")
|
||||
print("Please set it in your .env file or as environment variable")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test basic conversation
|
||||
query = "What is 25 * 4 + 10?"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
# Verify response contains correct answer
|
||||
if "110" in response:
|
||||
print("\n✅ Basic conversation test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n❌ Test failed - incorrect answer")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_tool_usage():
|
||||
"""Test tool calling capabilities"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 2: Tool Usage (Calculator)")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test complex calculation requiring calculator tool
|
||||
query = "Calculate: (123.45 * 67.89) / 12.34 + sqrt(144) - 2^8"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
# Check if calculator was used
|
||||
if agent.trajectory.tool_calls:
|
||||
print(f"\n🔧 Tools used: {len(agent.trajectory.tool_calls)}")
|
||||
for call in agent.trajectory.tool_calls:
|
||||
print(f" - {call.tool_name}: {call.arguments}")
|
||||
print("\n✅ Tool usage test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ No tools were used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_currency_conversion():
|
||||
"""Test currency conversion tool"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 3: Currency Conversion")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test currency conversion
|
||||
query = "Convert 100 USD to EUR and JPY"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
# Check if currency converter was used
|
||||
tool_names = [call.tool_name for call in agent.trajectory.tool_calls]
|
||||
if "convert_currency" in tool_names:
|
||||
print(f"\n🔧 Currency converter was used")
|
||||
print("\n✅ Currency conversion test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ Currency converter was not used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_model_info():
|
||||
"""Test and display model information"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 4: Model Information")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
print(f"\n📊 Model Configuration:")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
print(f" Context Mode: {agent.context_mode.value}")
|
||||
|
||||
# Test model identification
|
||||
query = "What model are you?"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
print("\n✅ Model info test completed!")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all tests"""
|
||||
print("\n" + "="*60)
|
||||
print("KIMI K3 MODEL INTEGRATION TEST SUITE")
|
||||
print("="*60)
|
||||
print("\nModel: kimi-k3")
|
||||
print("Provider: Moonshot AI")
|
||||
print("API: https://api.moonshot.cn/v1")
|
||||
|
||||
# Check environment
|
||||
if not os.getenv("MOONSHOT_API_KEY"):
|
||||
print("\n❌ ERROR: MOONSHOT_API_KEY not found in environment")
|
||||
print("\nPlease set up your .env file with:")
|
||||
print(" MOONSHOT_API_KEY=your_api_key_here")
|
||||
print("\nYou can get an API key from: https://platform.moonshot.cn/")
|
||||
sys.exit(1)
|
||||
|
||||
# Run tests
|
||||
results = []
|
||||
|
||||
# Test 1: Basic conversation
|
||||
results.append(("Basic Conversation", test_basic_conversation()))
|
||||
|
||||
# Test 2: Tool usage
|
||||
results.append(("Tool Usage", test_tool_usage()))
|
||||
|
||||
# Test 3: Currency conversion
|
||||
results.append(("Currency Conversion", test_currency_conversion()))
|
||||
|
||||
# Test 4: Model information
|
||||
results.append(("Model Information", test_model_info()))
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*60)
|
||||
print("TEST SUMMARY")
|
||||
print("="*60)
|
||||
|
||||
passed = sum(1 for _, result in results if result)
|
||||
total = len(results)
|
||||
|
||||
for test_name, result in results:
|
||||
status = "✅ PASSED" if result else "❌ FAILED"
|
||||
print(f" {test_name}: {status}")
|
||||
|
||||
print(f"\nTotal: {passed}/{total} tests passed")
|
||||
|
||||
if passed == total:
|
||||
print("\n🎉 All tests passed! Kimi K3 integration is working correctly.")
|
||||
else:
|
||||
print(f"\n⚠️ {total - passed} test(s) failed. Please check the errors above.")
|
||||
|
||||
return passed == total
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = main()
|
||||
sys.exit(0 if success else 1)
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick test script to verify Kimi K3 model integration
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def main():
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
print("Please add to your .env file:")
|
||||
print(" MOONSHOT_API_KEY=your_api_key_here")
|
||||
return
|
||||
|
||||
print("🚀 Testing Kimi K3 Model (kimi-k3)")
|
||||
print("=" * 50)
|
||||
|
||||
try:
|
||||
# Create agent with Kimi provider
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
print(f"✅ Agent created successfully")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
|
||||
# Test simple query
|
||||
print("\n📝 Testing basic query...")
|
||||
query = "What is 2 + 2?"
|
||||
response = agent.process(query)
|
||||
print(f" Query: {query}")
|
||||
print(f" Response: {response}")
|
||||
|
||||
print("\n✅ Kimi K3 integration is working!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify PDF parsing and currency conversion
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_pdf_with_currencies():
|
||||
"""Test PDF parsing with currency conversion"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 PDF PARSING & CURRENCY CONVERSION TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ No API key found. Set SILICONFLOW_API_KEY environment variable.")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
|
||||
# Test task
|
||||
task = """
|
||||
Analyze the expense report at fixtures/pdfs/simple_expense_report.pdf
|
||||
|
||||
Extract the following expenses mentioned in the document:
|
||||
- US Office: $2,500,000 USD
|
||||
- UK Office: £1,800,000 GBP
|
||||
- Japan Office: ¥380,000,000 JPY
|
||||
- EU Office: €2,100,000 EUR
|
||||
- Singapore Office: S$3,200,000 SGD
|
||||
|
||||
Convert all amounts to USD and calculate the total.
|
||||
|
||||
FINAL ANSWER: Provide the total expenses in USD.
|
||||
"""
|
||||
|
||||
print("📋 Task: Parse PDF and convert multiple currencies to USD")
|
||||
print("-"*40)
|
||||
|
||||
try:
|
||||
# Execute task
|
||||
result = agent.execute_task(task, max_iterations=5)
|
||||
|
||||
print("\n" + "="*40)
|
||||
print("RESULTS:")
|
||||
print("="*40)
|
||||
print(f"Success: {result.get('success', False)}")
|
||||
print(f"Iterations: {result.get('iterations', 0)}")
|
||||
print(f"Tool Calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
# Show tool calls made
|
||||
print("\n📊 Tool Calls Made:")
|
||||
for i, tc in enumerate(result['trajectory'].tool_calls, 1):
|
||||
print(f"{i}. {tc.tool_name}")
|
||||
if tc.tool_name == "parse_pdf":
|
||||
print(f" - PDF: {tc.arguments.get('url', 'N/A')}")
|
||||
if tc.result and 'num_pages' in tc.result:
|
||||
print(f" - Pages: {tc.result['num_pages']}")
|
||||
elif tc.tool_name == "convert_currency":
|
||||
print(f" - {tc.arguments.get('amount', 0)} {tc.arguments.get('from_currency', '')} → {tc.arguments.get('to_currency', '')}")
|
||||
if tc.result and 'converted_amount' in tc.result:
|
||||
print(f" - Result: {tc.result['converted_amount']}")
|
||||
elif tc.tool_name == "calculate":
|
||||
print(f" - Expression: {tc.arguments.get('expression', '')}")
|
||||
if tc.result and 'result' in tc.result:
|
||||
print(f" - Result: {tc.result['result']}")
|
||||
|
||||
if result.get('final_answer'):
|
||||
print("\n✅ Final Answer:")
|
||||
print("-"*40)
|
||||
print(result['final_answer'])
|
||||
|
||||
if result.get('error'):
|
||||
print(f"\n❌ Error: {result['error']}")
|
||||
|
||||
return result.get('success', False)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Exception: {str(e)}")
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Ensure PDFs exist
|
||||
if not os.path.exists("fixtures/pdfs/simple_expense_report.pdf"):
|
||||
print("⚠️ Creating sample PDFs...")
|
||||
os.system("python create_sample_pdf.py")
|
||||
|
||||
# Run test
|
||||
success = test_pdf_with_currencies()
|
||||
sys.exit(0 if success else 1)
|
||||
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify provider configuration
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_providers():
|
||||
"""Test different provider configurations"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 PROVIDER CONFIGURATION TEST")
|
||||
print("="*60)
|
||||
|
||||
# Test Alibaba Cloud Model Studio / Bailian
|
||||
dashscope_key = os.getenv("DASHSCOPE_API_KEY")
|
||||
if dashscope_key:
|
||||
print("\n✅ Alibaba Cloud Model Studio API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(
|
||||
dashscope_key, ContextMode.FULL, provider="dashscope"
|
||||
)
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ Alibaba Cloud Model Studio API key not found (DASHSCOPE_API_KEY)")
|
||||
|
||||
# Test SiliconFlow
|
||||
sf_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if sf_key:
|
||||
print("\n✅ SiliconFlow API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(sf_key, ContextMode.FULL, provider="siliconflow")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ SiliconFlow API key not found (SILICONFLOW_API_KEY)")
|
||||
|
||||
# Test Doubao
|
||||
ark_key = os.getenv("ARK_API_KEY")
|
||||
if ark_key:
|
||||
print("\n✅ Doubao/ARK API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(ark_key, ContextMode.FULL, provider="doubao")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ Doubao/ARK API key not found (ARK_API_KEY)")
|
||||
|
||||
# Test DeepSeek
|
||||
deepseek_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if deepseek_key:
|
||||
print("\n✅ DeepSeek API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(deepseek_key, ContextMode.FULL, provider="deepseek")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ DeepSeek API key not found (DEEPSEEK_API_KEY)")
|
||||
|
||||
# Test custom model
|
||||
if sf_key:
|
||||
print("\n🔧 Testing custom model specification:")
|
||||
try:
|
||||
agent = ContextAwareAgent(sf_key, ContextMode.FULL,
|
||||
provider="siliconflow",
|
||||
model="Qwen/QwQ-32B")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Custom Model: {agent.model}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Test complete!")
|
||||
|
||||
# Show usage examples
|
||||
print("\n📖 Usage Examples:")
|
||||
print("-"*40)
|
||||
|
||||
if dashscope_key:
|
||||
print("\n# Using Qwen directly through Alibaba Cloud Model Studio:")
|
||||
print("python main.py --provider dashscope")
|
||||
print("python main.py --provider dashscope --model qwen3.7-plus")
|
||||
|
||||
if sf_key:
|
||||
print("\n# Using SiliconFlow:")
|
||||
print("python main.py --provider siliconflow")
|
||||
print("python main.py --provider siliconflow --model Qwen/QwQ-32B")
|
||||
|
||||
if ark_key:
|
||||
print("\n# Using Doubao:")
|
||||
print("python main.py --provider doubao")
|
||||
print("python main.py --provider doubao --model doubao-seed-1-6-thinking-250715")
|
||||
|
||||
if deepseek_key:
|
||||
print("\n# Using DeepSeek:")
|
||||
print("python main.py --provider deepseek")
|
||||
print("python main.py --provider deepseek --model deepseek-v4-pro")
|
||||
|
||||
if not dashscope_key and not sf_key and not ark_key and not deepseek_key:
|
||||
print("\n⚠️ No API keys found. Please set one of:")
|
||||
print(" export DASHSCOPE_API_KEY=your_key")
|
||||
print(" export SILICONFLOW_API_KEY=your_key")
|
||||
print(" export ARK_API_KEY=your_key")
|
||||
print(" export DEEPSEEK_API_KEY=your_key")
|
||||
|
||||
print("\n" + "="*60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_providers()
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify provider switching functionality
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def test_provider_switching():
|
||||
"""Test switching between different providers"""
|
||||
print("🧪 Testing Provider Switching")
|
||||
print("=" * 50)
|
||||
|
||||
providers_to_test = []
|
||||
|
||||
# Check which providers have API keys configured
|
||||
if os.getenv("DASHSCOPE_API_KEY"):
|
||||
providers_to_test.append(("dashscope", os.getenv("DASHSCOPE_API_KEY")))
|
||||
print("✅ Alibaba Cloud Model Studio API key found")
|
||||
else:
|
||||
print("⏭️ Skipping Alibaba Cloud Model Studio (no API key)")
|
||||
|
||||
if os.getenv("SILICONFLOW_API_KEY"):
|
||||
providers_to_test.append(("siliconflow", os.getenv("SILICONFLOW_API_KEY")))
|
||||
print("✅ SiliconFlow API key found")
|
||||
else:
|
||||
print("⏭️ Skipping SiliconFlow (no API key)")
|
||||
|
||||
if os.getenv("ARK_API_KEY"):
|
||||
providers_to_test.append(("doubao", os.getenv("ARK_API_KEY")))
|
||||
print("✅ Doubao API key found")
|
||||
else:
|
||||
print("⏭️ Skipping Doubao (no API key)")
|
||||
|
||||
if os.getenv("MOONSHOT_API_KEY"):
|
||||
providers_to_test.append(("kimi", os.getenv("MOONSHOT_API_KEY")))
|
||||
print("✅ Kimi API key found")
|
||||
else:
|
||||
print("⏭️ Skipping Kimi (no API key)")
|
||||
|
||||
if os.getenv("DEEPSEEK_API_KEY"):
|
||||
providers_to_test.append(("deepseek", os.getenv("DEEPSEEK_API_KEY")))
|
||||
print("✅ DeepSeek API key found")
|
||||
else:
|
||||
print("⏭️ Skipping DeepSeek (no API key)")
|
||||
|
||||
if not providers_to_test:
|
||||
print("\n❌ No API keys configured. Please set at least one:")
|
||||
print(" - DASHSCOPE_API_KEY")
|
||||
print(" - SILICONFLOW_API_KEY")
|
||||
print(" - ARK_API_KEY")
|
||||
print(" - MOONSHOT_API_KEY")
|
||||
print(" - DEEPSEEK_API_KEY")
|
||||
return
|
||||
|
||||
print(f"\nTesting {len(providers_to_test)} provider(s)...")
|
||||
print("-" * 50)
|
||||
|
||||
# Test each available provider
|
||||
for provider_name, api_key in providers_to_test:
|
||||
print(f"\n📌 Testing {provider_name.upper()}")
|
||||
|
||||
try:
|
||||
# Create agent with provider
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider=provider_name,
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Get default model from config
|
||||
default_model = Config.get_default_model(provider_name)
|
||||
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Expected: {default_model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
|
||||
# Test with a simple query
|
||||
query = "What is 5 + 3?"
|
||||
print(f" Testing query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
|
||||
if "8" in response:
|
||||
print(f" ✅ {provider_name} working correctly!")
|
||||
else:
|
||||
print(f" ⚠️ {provider_name} response didn't contain expected answer")
|
||||
print(f" Response: {response[:100]}...")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ Error with {provider_name}: {e}")
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("Provider switching test complete!")
|
||||
|
||||
# Show summary
|
||||
print("\n📊 Summary:")
|
||||
print(f" Providers tested: {len(providers_to_test)}")
|
||||
print(" Available providers include: dashscope (qwen/bailian), siliconflow, doubao, kimi, moonshot, deepseek")
|
||||
|
||||
if len(providers_to_test) < 3:
|
||||
print("\n💡 Tip: Configure more API keys to test all providers")
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_provider_switching()
|
||||
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test with a simpler task to diagnose the issue
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_simple_task():
|
||||
"""Test with a very simple task to check if the agent is working"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 SIMPLE TASK TEST")
|
||||
print("="*60)
|
||||
|
||||
# Get API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ SILICONFLOW_API_KEY not found")
|
||||
return
|
||||
|
||||
print("✅ API key found")
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL, provider="siliconflow")
|
||||
print(f"✅ Agent created")
|
||||
print(f" Model: {agent.model}")
|
||||
|
||||
# Very simple task - no tools needed
|
||||
print("\n📝 Test 1: Simple question (no tools)")
|
||||
task1 = "What is 2 + 2? Just tell me the answer. FINAL ANSWER: provide the result."
|
||||
|
||||
start = time.time()
|
||||
print("Executing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task1, max_iterations=1)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"✅ Completed in {elapsed:.2f} seconds")
|
||||
if result.get('final_answer'):
|
||||
print(f" Answer: {result['final_answer'][:100]}")
|
||||
print(f" Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {str(e)}")
|
||||
return
|
||||
|
||||
# Task with a single tool
|
||||
print("\n📝 Test 2: Simple calculation (with tool)")
|
||||
task2 = "Use the calculate tool to compute 15 * 3. FINAL ANSWER: provide the result."
|
||||
|
||||
start = time.time()
|
||||
print("Executing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task2, max_iterations=2)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"✅ Completed in {elapsed:.2f} seconds")
|
||||
if result.get('final_answer'):
|
||||
print(f" Answer: {result['final_answer'][:100]}")
|
||||
print(f" Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
print("The model might be taking too long to respond.")
|
||||
print("\nSuggestions:")
|
||||
print("1. Try using --provider doubao for faster responses")
|
||||
print("2. Check your internet connection")
|
||||
print("3. The model might be overloaded - try again later")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {str(e)}")
|
||||
|
||||
print("\n" + "="*60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_simple_task()
|
||||
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Demo script showing conversation history persistence
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def main():
|
||||
# Get API key (use any available provider)
|
||||
if os.getenv("ARK_API_KEY"):
|
||||
api_key, provider = os.getenv("ARK_API_KEY"), "doubao"
|
||||
elif os.getenv("DASHSCOPE_API_KEY"):
|
||||
api_key, provider = os.getenv("DASHSCOPE_API_KEY"), "dashscope"
|
||||
elif os.getenv("MOONSHOT_API_KEY"):
|
||||
api_key, provider = os.getenv("MOONSHOT_API_KEY"), "kimi"
|
||||
elif os.getenv("DEEPSEEK_API_KEY"):
|
||||
api_key, provider = os.getenv("DEEPSEEK_API_KEY"), "deepseek"
|
||||
elif os.getenv("SILICONFLOW_API_KEY"):
|
||||
api_key, provider = os.getenv("SILICONFLOW_API_KEY"), "siliconflow"
|
||||
else:
|
||||
api_key, provider = None, None
|
||||
|
||||
if not api_key:
|
||||
print("❌ No API key found. Please set one of:")
|
||||
print(" - ARK_API_KEY")
|
||||
print(" - DASHSCOPE_API_KEY")
|
||||
print(" - MOONSHOT_API_KEY")
|
||||
print(" - DEEPSEEK_API_KEY")
|
||||
print(" - SILICONFLOW_API_KEY")
|
||||
return
|
||||
|
||||
print("🎭 Conversation History Demo")
|
||||
print("=" * 50)
|
||||
print(f"Provider: {provider.upper()}")
|
||||
print("-" * 50)
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider=provider,
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Conversation 1: Set context
|
||||
print("\n💬 Turn 1: Setting context...")
|
||||
result = agent.execute_task("My name is Alice and I have a budget of $5,000. What is 20% of my budget?")
|
||||
print(f"Agent: {result.get('final_answer', 'No answer')}")
|
||||
|
||||
# Conversation 2: Reference previous context
|
||||
print("\n💬 Turn 2: Referencing previous context...")
|
||||
result = agent.execute_task("Convert that 20% amount to EUR please.")
|
||||
print(f"Agent: {result.get('final_answer', 'No answer')}")
|
||||
|
||||
# Conversation 3: Recall information
|
||||
print("\n💬 Turn 3: Recalling information...")
|
||||
result = agent.execute_task("What was my name and total budget that I mentioned?")
|
||||
print(f"Agent: {result.get('final_answer', 'No answer')}")
|
||||
|
||||
print("\n" + "-" * 50)
|
||||
print(f"📊 Final Statistics:")
|
||||
print(f" Total messages in history: {len(agent.conversation_history)}")
|
||||
print(f" Total tool calls made: {len(agent.trajectory.tool_calls)}")
|
||||
|
||||
# Show that system prompt is unchanged
|
||||
system_prompt = agent.conversation_history[0]['content']
|
||||
if "Alice" not in system_prompt and "5000" not in system_prompt:
|
||||
print(" ✅ System prompt remained unchanged")
|
||||
else:
|
||||
print(" ❌ System prompt was modified")
|
||||
|
||||
print("\n✨ Demo complete!")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick Start Script for Context-Aware Agent
|
||||
Run this to test the agent with a simple example
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
def main():
|
||||
"""Quick start demonstration"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("CONTEXT-AWARE AGENT - QUICK START")
|
||||
print("="*60)
|
||||
|
||||
# Check for API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("\n❌ ERROR: SILICONFLOW_API_KEY not found!")
|
||||
print("\nPlease set your API key:")
|
||||
print("1. Copy env.example to .env")
|
||||
print("2. Add your API key to .env")
|
||||
print("3. Or export SILICONFLOW_API_KEY=your_key_here")
|
||||
sys.exit(1)
|
||||
|
||||
print("\n✅ API key found!")
|
||||
|
||||
# Simple demonstration task
|
||||
demo_task = """
|
||||
Please help me with the following financial calculation:
|
||||
|
||||
1. I have $10,000 USD that I want to convert to EUR, GBP, and JPY
|
||||
2. Calculate the average amount across all three currencies (converted back to USD)
|
||||
3. If I invest this average amount with a 5% annual return, what will it be worth in 2 years?
|
||||
|
||||
Show all your calculations step by step.
|
||||
"""
|
||||
|
||||
print("\n📋 Demo Task:")
|
||||
print("-"*40)
|
||||
print(demo_task)
|
||||
print("-"*40)
|
||||
|
||||
# Run with full context (baseline)
|
||||
print("\n🚀 Running agent with FULL context...")
|
||||
agent_full = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
result_full = agent_full.execute_task(demo_task)
|
||||
|
||||
print("\n✨ Results with FULL Context:")
|
||||
print(f"Success: {result_full.get('success', False)}")
|
||||
print(f"Tool calls made: {len(result_full['trajectory'].tool_calls)}")
|
||||
print(f"Iterations: {result_full.get('iterations', 0)}")
|
||||
|
||||
if result_full.get('final_answer'):
|
||||
print(f"\nFinal Answer:")
|
||||
print("-"*40)
|
||||
print(result_full['final_answer'])
|
||||
|
||||
# Demonstrate context ablation effect
|
||||
print("\n" + "="*60)
|
||||
print("DEMONSTRATING CONTEXT ABLATION")
|
||||
print("="*60)
|
||||
|
||||
print("\n🔬 Running same task with NO TOOL RESULTS context...")
|
||||
print("(Agent won't see the results of its tool calls)")
|
||||
|
||||
agent_ablated = ContextAwareAgent(api_key, ContextMode.NO_TOOL_RESULTS)
|
||||
result_ablated = agent_ablated.execute_task(demo_task)
|
||||
|
||||
print("\n⚠️ Results with NO TOOL RESULTS:")
|
||||
print(f"Success: {result_ablated.get('success', False)}")
|
||||
print(f"Tool calls made: {len(result_ablated['trajectory'].tool_calls)}")
|
||||
print(f"Iterations: {result_ablated.get('iterations', 0)}")
|
||||
|
||||
if result_ablated.get('final_answer'):
|
||||
print(f"\nFinal Answer (likely incorrect):")
|
||||
print("-"*40)
|
||||
print(result_ablated['final_answer'][:500] + "...")
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*60)
|
||||
print("COMPARISON SUMMARY")
|
||||
print("="*60)
|
||||
|
||||
print("\n📊 Key Observations:")
|
||||
print(f"1. Full Context: {'✅ Success' if result_full.get('success') else '❌ Failed'}")
|
||||
print(f"2. No Tool Results: {'✅ Success' if result_ablated.get('success') else '❌ Failed'}")
|
||||
print(f"3. Efficiency difference: {result_ablated.get('iterations', 0) - result_full.get('iterations', 0)} more iterations without tool results")
|
||||
|
||||
print("\n💡 Insight:")
|
||||
print("Without seeing tool results, the agent operates blind and may:")
|
||||
print("- Make incorrect calculations")
|
||||
print("- Repeat operations unnecessarily")
|
||||
print("- Fail to validate its work")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Quick start complete! 🎉")
|
||||
print("\nNext steps:")
|
||||
print("1. Run full ablation study: python main.py --mode ablation")
|
||||
print("2. Try interactive mode: python main.py --mode interactive")
|
||||
print("3. Read the README.md for more details")
|
||||
print("="*60 + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Demo script to showcase sample tasks with PDF functionality
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from main import get_sample_tasks, ensure_sample_pdfs
|
||||
|
||||
def main():
|
||||
"""Demo the sample tasks"""
|
||||
print("\n" + "="*60)
|
||||
print("🎯 CONTEXT-AWARE AGENT - SAMPLE TASKS DEMO")
|
||||
print("="*60)
|
||||
|
||||
# Ensure PDFs exist
|
||||
print("\n📄 Checking for sample PDFs...")
|
||||
if ensure_sample_pdfs():
|
||||
print("✅ Sample PDFs are ready!")
|
||||
else:
|
||||
print("⚠️ Could not create sample PDFs, will use online alternatives")
|
||||
|
||||
# Get sample tasks
|
||||
tasks = get_sample_tasks()
|
||||
|
||||
print(f"\n📋 Found {len(tasks)} sample tasks:")
|
||||
print("-"*60)
|
||||
|
||||
for i, task in enumerate(tasks, 1):
|
||||
print(f"\n{i}. {task['name']}")
|
||||
print(f" 📝 {task['description']}")
|
||||
print(f" 📊 Complexity: {'⭐' * (i if i <= 3 else 3)}")
|
||||
|
||||
# Show a preview of the task
|
||||
task_preview = task['task'].replace('\n', ' ')[:100] + "..."
|
||||
print(f" 💬 Preview: {task_preview}")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("💡 USAGE TIPS:")
|
||||
print("-"*60)
|
||||
print("1. Run 'python main.py' to enter interactive mode")
|
||||
print("2. Type 'sample 2' to test PDF parsing capabilities")
|
||||
print("3. Type 'sample 5' for the most comprehensive test")
|
||||
print("4. Switch modes with 'mode no_reasoning' to see ablation effects")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🔬 ABLATION TESTING:")
|
||||
print("-"*60)
|
||||
print("Try running the same task in different modes:")
|
||||
print(" • full - Everything works perfectly")
|
||||
print(" • no_history - Agent forgets what it did")
|
||||
print(" • no_reasoning - No planning, chaotic execution")
|
||||
print(" • no_tool_calls - Can't do anything!")
|
||||
print(" • no_tool_results - Works blind, gets confused")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("📊 PDF TASKS:")
|
||||
print("-"*60)
|
||||
|
||||
# Check if local PDFs exist
|
||||
pdf_dir = Path("fixtures/pdfs")
|
||||
if pdf_dir.exists():
|
||||
pdfs = list(pdf_dir.glob("*.pdf"))
|
||||
if pdfs:
|
||||
print(f"✅ Found {len(pdfs)} local PDF files:")
|
||||
for pdf in pdfs:
|
||||
print(f" • {pdf.name}")
|
||||
print("\nTask #2 will use these local PDFs for testing.")
|
||||
else:
|
||||
print("⚠️ No PDFs found in fixtures/pdfs/")
|
||||
else:
|
||||
print("📥 PDF directory not found. Run 'create_pdfs' command to generate samples.")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Ready to test! Run 'python main.py' to start.")
|
||||
print("="*60 + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,305 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for Context-Aware Agent
|
||||
Validates installation and basic functionality
|
||||
"""
|
||||
|
||||
import sys
|
||||
from agent import ContextAwareAgent, ContextMode, ToolRegistry
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestToolRegistry(unittest.TestCase):
|
||||
"""Test the tool registry functions"""
|
||||
|
||||
def test_calculator(self):
|
||||
"""Test calculator tool"""
|
||||
tools = ToolRegistry()
|
||||
|
||||
# Basic arithmetic
|
||||
result = tools.calculate("2 + 2")
|
||||
self.assertEqual(result["result"], 4)
|
||||
|
||||
# Complex expression
|
||||
result = tools.calculate("(10 * 5) + (20 / 4)")
|
||||
self.assertEqual(result["result"], 55.0)
|
||||
|
||||
# With math functions
|
||||
result = tools.calculate("sqrt(16) + abs(-5)")
|
||||
self.assertEqual(result["result"], 9.0)
|
||||
|
||||
def test_currency_converter(self):
|
||||
"""Test currency conversion tool"""
|
||||
tools = ToolRegistry()
|
||||
|
||||
# USD to EUR
|
||||
result = tools.convert_currency(100, "USD", "EUR")
|
||||
self.assertIn("converted_amount", result)
|
||||
self.assertIn("exchange_rate", result)
|
||||
self.assertGreater(result["converted_amount"], 0)
|
||||
|
||||
# Currency symbol normalization (US$, S$, A$, C$, $)
|
||||
result_us = tools.convert_currency(100, "US$", "EUR")
|
||||
self.assertEqual(result_us["from_currency"], "USD")
|
||||
self.assertEqual(result_us["converted_amount"], 92.0)
|
||||
|
||||
result_s = tools.convert_currency(100, "S$", "USD")
|
||||
self.assertEqual(result_s["from_currency"], "SGD")
|
||||
self.assertIn("converted_amount", result_s)
|
||||
|
||||
result_a = tools.convert_currency(100, "A$", "USD")
|
||||
self.assertEqual(result_a["from_currency"], "AUD")
|
||||
self.assertIn("converted_amount", result_a)
|
||||
|
||||
result_c = tools.convert_currency(100, "C$", "USD")
|
||||
self.assertEqual(result_c["from_currency"], "CAD")
|
||||
self.assertIn("converted_amount", result_c)
|
||||
# Invalid currency
|
||||
result = tools.convert_currency(100, "XXX", "YYY")
|
||||
self.assertIn("error", result)
|
||||
result_invalid_s = tools.convert_currency(100, "S$INVALID", "USD")
|
||||
self.assertIn("error", result_invalid_s)
|
||||
|
||||
def test_convert_currency_string_and_formatted_amounts(self):
|
||||
"""
|
||||
Prove that convert_currency accepts string and formatted numeric amounts.
|
||||
|
||||
LLM tool calls frequently pass numeric arguments as strings (e.g., "100", "$1,000.00").
|
||||
Previously, passing a string raised a TypeError during float division. This test locks
|
||||
out regressions by asserting that numeric strings and formatted currency strings convert correctly.
|
||||
"""
|
||||
tools = ToolRegistry()
|
||||
result_str = tools.convert_currency("100", "USD", "EUR")
|
||||
self.assertEqual(result_str["converted_amount"], 92.0)
|
||||
self.assertEqual(result_str["original_amount"], 100.0)
|
||||
|
||||
result_formatted = tools.convert_currency("$1,000.00", "USD", "EUR")
|
||||
self.assertEqual(result_formatted["converted_amount"], 920.0)
|
||||
self.assertEqual(result_formatted["original_amount"], 1000.0)
|
||||
|
||||
result_us_dollar = tools.convert_currency("US$100", "USD", "EUR")
|
||||
self.assertEqual(result_us_dollar["converted_amount"], 92.0)
|
||||
self.assertEqual(result_us_dollar["original_amount"], 100.0)
|
||||
|
||||
result_currency_code = tools.convert_currency("USD$1,000", "USD$", "EUR")
|
||||
self.assertEqual(result_currency_code["converted_amount"], 920.0)
|
||||
self.assertEqual(result_currency_code["original_amount"], 1000.0)
|
||||
|
||||
result_comma_large = tools.convert_currency("1,234,567.89", "USD", "EUR")
|
||||
self.assertEqual(result_comma_large["original_amount"], 1234567.89)
|
||||
|
||||
result_euro_sym = tools.convert_currency("€ 500.25", "EUR", "USD")
|
||||
self.assertIn("converted_amount", result_euro_sym)
|
||||
|
||||
result_invalid_str = tools.convert_currency("invalid_str", "USD", "EUR")
|
||||
self.assertIn("error", result_invalid_str)
|
||||
|
||||
def test_pdf_parser_structure(self):
|
||||
"""Test PDF parser structure (without actual PDF)"""
|
||||
tools = ToolRegistry()
|
||||
|
||||
# Test with invalid URL (should handle gracefully)
|
||||
result = tools.parse_pdf("http://invalid-url-for-testing.com/test.pdf")
|
||||
self.assertIn("error", result)
|
||||
|
||||
|
||||
class TestContextModes(unittest.TestCase):
|
||||
"""Test different context modes"""
|
||||
|
||||
@patch.dict('os.environ', {'SILICONFLOW_API_KEY': 'test_key'})
|
||||
def setUp(self):
|
||||
"""Set up test fixtures"""
|
||||
self.api_key = "test_key"
|
||||
|
||||
def test_context_mode_initialization(self):
|
||||
"""Test agent initialization with different context modes"""
|
||||
for mode in ContextMode:
|
||||
agent = ContextAwareAgent(self.api_key, mode)
|
||||
self.assertEqual(agent.context_mode, mode)
|
||||
self.assertEqual(agent.trajectory.context_mode, mode)
|
||||
|
||||
def test_context_building(self):
|
||||
"""Test context building for different modes"""
|
||||
# Full context mode
|
||||
agent = ContextAwareAgent(self.api_key, ContextMode.FULL)
|
||||
agent.trajectory.reasoning_steps = ["Step 1", "Step 2"]
|
||||
agent.trajectory.tool_calls.append(
|
||||
MagicMock(tool_name="test", arguments={}, result={"test": "result"})
|
||||
)
|
||||
|
||||
context = agent._build_context()
|
||||
self.assertIn("Previous Reasoning Steps", context)
|
||||
self.assertIn("Tool Call History", context)
|
||||
|
||||
# No reasoning mode
|
||||
agent_no_reasoning = ContextAwareAgent(self.api_key, ContextMode.NO_REASONING)
|
||||
agent_no_reasoning.trajectory.reasoning_steps = ["Step 1"]
|
||||
context = agent_no_reasoning._build_context()
|
||||
self.assertNotIn("Previous Reasoning Steps", context)
|
||||
|
||||
# No history mode
|
||||
agent_no_history = ContextAwareAgent(self.api_key, ContextMode.NO_HISTORY)
|
||||
agent_no_history.trajectory.tool_calls.append(
|
||||
MagicMock(tool_name="test", arguments={}, result={"test": "result"})
|
||||
)
|
||||
context = agent_no_history._build_context()
|
||||
self.assertEqual(context, "")
|
||||
|
||||
|
||||
class TestAblationScenarios(unittest.TestCase):
|
||||
"""Test ablation scenarios"""
|
||||
|
||||
def test_tool_execution(self):
|
||||
"""Test tool execution"""
|
||||
agent = ContextAwareAgent("test_key", ContextMode.FULL)
|
||||
|
||||
# Test calculator execution
|
||||
result = agent._execute_tool("calculate", {"expression": "2 + 2"})
|
||||
self.assertEqual(result["result"], 4)
|
||||
|
||||
# Test unknown tool
|
||||
result = agent._execute_tool("unknown_tool", {})
|
||||
self.assertIn("error", result)
|
||||
|
||||
def test_trajectory_reset(self):
|
||||
"""Test trajectory reset"""
|
||||
agent = ContextAwareAgent("test_key", ContextMode.FULL)
|
||||
|
||||
# Add some data to trajectory
|
||||
agent.trajectory.reasoning_steps.append("Test step")
|
||||
agent.trajectory.tool_calls.append(
|
||||
MagicMock(tool_name="test", arguments={})
|
||||
)
|
||||
|
||||
# Reset
|
||||
agent.reset()
|
||||
|
||||
# Check if cleared
|
||||
self.assertEqual(len(agent.trajectory.reasoning_steps), 0)
|
||||
self.assertEqual(len(agent.trajectory.tool_calls), 0)
|
||||
self.assertEqual(agent.trajectory.context_mode, ContextMode.FULL)
|
||||
|
||||
|
||||
def run_integration_test():
|
||||
"""Run a simple integration test"""
|
||||
print("\n" + "="*60)
|
||||
print("INTEGRATION TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check if API key is available
|
||||
import os
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
|
||||
if not api_key:
|
||||
print("⚠️ Skipping integration test (no API key found)")
|
||||
print("Set SILICONFLOW_API_KEY to run integration tests")
|
||||
return False
|
||||
|
||||
print("✅ API key found, running integration test...")
|
||||
|
||||
try:
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
|
||||
# Simple task that doesn't require external PDFs
|
||||
simple_task = "Calculate: What is 15% of $2500? Then convert the result to EUR."
|
||||
|
||||
print(f"\nTest task: {simple_task}")
|
||||
print("Running...")
|
||||
|
||||
# Execute with timeout
|
||||
import signal
|
||||
|
||||
def timeout_handler(signum, frame):
|
||||
raise TimeoutError("Integration test timed out")
|
||||
|
||||
# Set 30 second timeout
|
||||
signal.signal(signal.SIGALRM, timeout_handler)
|
||||
signal.alarm(30)
|
||||
|
||||
try:
|
||||
result = agent.execute_task(simple_task, max_iterations=3)
|
||||
signal.alarm(0) # Cancel alarm
|
||||
|
||||
print("\n✅ Integration test completed!")
|
||||
print(f"Success: {result.get('success', False)}")
|
||||
print(f"Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
if result.get('final_answer'):
|
||||
print(f"Answer preview: {result['final_answer'][:100]}...")
|
||||
|
||||
return True
|
||||
|
||||
except TimeoutError:
|
||||
print("❌ Integration test timed out")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Integration test failed: {str(e)}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Main test runner"""
|
||||
print("\n" + "="*60)
|
||||
print("CONTEXT-AWARE AGENT TEST SUITE")
|
||||
print("="*60)
|
||||
|
||||
# Run unit tests
|
||||
print("\n📋 Running unit tests...")
|
||||
|
||||
# Create test suite
|
||||
loader = unittest.TestLoader()
|
||||
suite = unittest.TestSuite()
|
||||
|
||||
# Add test cases
|
||||
suite.addTests(loader.loadTestsFromTestCase(TestToolRegistry))
|
||||
suite.addTests(loader.loadTestsFromTestCase(TestContextModes))
|
||||
suite.addTests(loader.loadTestsFromTestCase(TestAblationScenarios))
|
||||
|
||||
# Run tests
|
||||
runner = unittest.TextTestRunner(verbosity=2)
|
||||
result = runner.run(suite)
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*60)
|
||||
print("UNIT TEST SUMMARY")
|
||||
print("="*60)
|
||||
print(f"Tests run: {result.testsRun}")
|
||||
print(f"Failures: {len(result.failures)}")
|
||||
print(f"Errors: {len(result.errors)}")
|
||||
|
||||
if result.wasSuccessful():
|
||||
print("✅ All unit tests passed!")
|
||||
else:
|
||||
print("❌ Some tests failed")
|
||||
sys.exit(1)
|
||||
|
||||
# Run integration test if possible
|
||||
print("\n" + "="*60)
|
||||
integration_success = run_integration_test()
|
||||
|
||||
# Final summary
|
||||
print("\n" + "="*60)
|
||||
print("FINAL TEST SUMMARY")
|
||||
print("="*60)
|
||||
|
||||
if result.wasSuccessful():
|
||||
print("✅ Unit tests: PASSED")
|
||||
else:
|
||||
print("❌ Unit tests: FAILED")
|
||||
|
||||
if integration_success:
|
||||
print("✅ Integration test: PASSED")
|
||||
else:
|
||||
print("⚠️ Integration test: SKIPPED or FAILED")
|
||||
|
||||
print("\n🎉 Testing complete!")
|
||||
print("="*60 + "\n")
|
||||
|
||||
return 0 if result.wasSuccessful() else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,125 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test the code_interpreter tool with the agent
|
||||
"""
|
||||
|
||||
import os
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_code_interpreter():
|
||||
"""Test code interpreter integration"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 CODE INTERPRETER TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("⚠️ No API key set, using mock test")
|
||||
# Test just the tool directly
|
||||
from agent import ToolRegistry
|
||||
tools = ToolRegistry()
|
||||
|
||||
code = """
|
||||
# Calculate total expenses
|
||||
expenses_usd = {
|
||||
'US Office': 2500000,
|
||||
'UK Office (converted)': 2278481.01,
|
||||
'Japan Office (converted)': 2541806.02,
|
||||
'EU Office (converted)': 2282608.70,
|
||||
'Singapore Office (converted)': 2388059.70
|
||||
}
|
||||
|
||||
# Calculate total
|
||||
total = sum(expenses_usd.values())
|
||||
|
||||
# Calculate percentages
|
||||
for office, amount in expenses_usd.items():
|
||||
percentage = (amount / total) * 100
|
||||
print(f"{office}: ${amount:,.2f} ({percentage:.2f}%)")
|
||||
|
||||
print(f"\\nTotal Expenses: ${total:,.2f}")
|
||||
|
||||
# Calculate after 12% reduction
|
||||
reduced_total = total * 0.88
|
||||
savings = total - reduced_total
|
||||
print(f"After 12% reduction: ${reduced_total:,.2f}")
|
||||
print(f"Savings: ${savings:,.2f}")
|
||||
|
||||
result = {
|
||||
'total': total,
|
||||
'reduced': reduced_total,
|
||||
'savings': savings
|
||||
}
|
||||
"""
|
||||
|
||||
result = tools.code_interpreter(code)
|
||||
if result['success']:
|
||||
print("✅ Code interpreter executed successfully!")
|
||||
print("\nOutput:")
|
||||
print(result['output'])
|
||||
print(f"\nResult dictionary: {result['result']}")
|
||||
else:
|
||||
print(f"❌ Error: {result['error']}")
|
||||
|
||||
return
|
||||
|
||||
# Test with full agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
|
||||
task = """
|
||||
Calculate the following:
|
||||
|
||||
Given these expenses:
|
||||
- US: $2,500,000
|
||||
- UK: $2,278,481
|
||||
- Japan: $2,541,806
|
||||
- EU: $2,282,609
|
||||
- Singapore: $2,388,060
|
||||
|
||||
Use the code_interpreter tool to:
|
||||
1. Calculate the total expenses
|
||||
2. Calculate what percentage each office represents
|
||||
3. Calculate the new totals if we apply a 12% cost reduction
|
||||
|
||||
FINAL ANSWER: Provide the total, the percentage breakdown, and the reduced total.
|
||||
"""
|
||||
|
||||
print("Running task with agent...")
|
||||
print("Task: Calculate totals and percentages using code_interpreter")
|
||||
print("-"*40)
|
||||
|
||||
result = agent.execute_task(task, max_iterations=3)
|
||||
|
||||
print(f"\nSuccess: {result.get('success', False)}")
|
||||
print(f"Tool calls made: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
# Check if code_interpreter was used
|
||||
code_interpreter_used = any(
|
||||
tc.tool_name == 'code_interpreter'
|
||||
for tc in result['trajectory'].tool_calls
|
||||
)
|
||||
|
||||
if code_interpreter_used:
|
||||
print("✅ Code interpreter was used!")
|
||||
# Show the code that was executed
|
||||
for tc in result['trajectory'].tool_calls:
|
||||
if tc.tool_name == 'code_interpreter':
|
||||
print("\nExecuted code:")
|
||||
print("-"*40)
|
||||
print(tc.arguments.get('code', 'N/A'))
|
||||
print("-"*40)
|
||||
if tc.result and tc.result.get('output'):
|
||||
print("\nOutput:")
|
||||
print(tc.result['output'])
|
||||
else:
|
||||
print("⚠️ Code interpreter was not used")
|
||||
|
||||
if result.get('final_answer'):
|
||||
print("\n📝 Final Answer:")
|
||||
print(result['final_answer'])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_code_interpreter()
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Regression: malformed tool-argument JSON must not abort the ReAct loop."""
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
|
||||
def _choice(*, content=None, tool_calls=None):
|
||||
msg = SimpleNamespace(
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
reasoning_content=None,
|
||||
model_dump=lambda: {
|
||||
"role": "assistant",
|
||||
"content": content,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": tc.id,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": tc.function.name,
|
||||
"arguments": tc.function.arguments,
|
||||
},
|
||||
}
|
||||
for tc in (tool_calls or [])
|
||||
],
|
||||
},
|
||||
)
|
||||
return SimpleNamespace(message=msg)
|
||||
|
||||
|
||||
def test_execute_task_survives_malformed_tool_arguments_json():
|
||||
agent = ContextAwareAgent("test-key", ContextMode.FULL, verbose=False)
|
||||
bad_call = SimpleNamespace(
|
||||
id="call-bad",
|
||||
function=SimpleNamespace(
|
||||
name="calculate",
|
||||
arguments='{"expression": "1+1",}', # trailing comma
|
||||
),
|
||||
)
|
||||
tool_turn = SimpleNamespace(choices=[_choice(tool_calls=[bad_call])])
|
||||
final_turn = SimpleNamespace(
|
||||
choices=[_choice(content="FINAL ANSWER: recovered")]
|
||||
)
|
||||
agent.client = MagicMock()
|
||||
agent.client.chat.completions.create = MagicMock(
|
||||
side_effect=[tool_turn, final_turn]
|
||||
)
|
||||
|
||||
result = agent.execute_task("compute", max_iterations=5)
|
||||
|
||||
assert result.get("error") is None
|
||||
assert result["completed"] is True
|
||||
assert result["task_success"] is None
|
||||
assert result["success"] is True # backwards-compatible completion alias
|
||||
assert "recovered" in (result.get("final_answer") or result.get("answer") or "")
|
||||
tool_roles = [m for m in agent.conversation_history if m.get("role") == "tool"]
|
||||
assert tool_roles
|
||||
assert "Invalid tool arguments" in tool_roles[0]["content"]
|
||||
assert agent.client.chat.completions.create.call_count == 2
|
||||
|
||||
|
||||
def test_execute_task_does_not_complete_on_empty_terminal_content():
|
||||
agent = ContextAwareAgent("test-key", ContextMode.NO_TOOL_CALLS, verbose=False)
|
||||
empty_turn = SimpleNamespace(choices=[_choice(content="")])
|
||||
agent.client = MagicMock()
|
||||
agent.client.chat.completions.create = MagicMock(return_value=empty_turn)
|
||||
|
||||
result = agent.execute_task("say something", max_iterations=5)
|
||||
|
||||
assert result["final_answer"] is None
|
||||
assert result["completed"] is False
|
||||
assert result["task_success"] is None
|
||||
assert result["success"] is False
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
1bd60e9548d7732820e6c8f73b565ee397b42c8d480f1ee2a120b6f81833913b evidence.json
|
||||
@@ -0,0 +1,251 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Serve an HTML report visualizing experiment arms from validation/latest.json.
|
||||
|
||||
Reads validation/latest.json (next to this script), then serves a report over
|
||||
HTTP and opens it in the browser. The report shows:
|
||||
1. A summary table with task_success / iterations / repeated_tool_calls per arm.
|
||||
2. One section per arm showing tool_call_signatures and reasoning_steps,
|
||||
with every reasoning step collapsed by default.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
DEFAULT_JSON = SCRIPT_DIR / "validation" / "latest.json"
|
||||
|
||||
|
||||
def esc(value):
|
||||
return html.escape(str(value))
|
||||
|
||||
|
||||
def fmt(value):
|
||||
if value is None:
|
||||
return "—"
|
||||
if isinstance(value, bool):
|
||||
return str(value)
|
||||
return html.escape(str(value))
|
||||
|
||||
|
||||
def status_badge(value):
|
||||
if value is True:
|
||||
return '<span class="badge ok">✓</span>'
|
||||
if value is False:
|
||||
return '<span class="badge fail">✗</span>'
|
||||
return '<span class="badge na">n/a</span>'
|
||||
|
||||
|
||||
def render_arm_section(arm):
|
||||
mode = esc(arm.get("mode", "?"))
|
||||
model = esc(arm.get("model", ""))
|
||||
provider = esc(arm.get("provider", ""))
|
||||
elapsed = arm.get("elapsed_seconds")
|
||||
elapsed_s = f"{elapsed:.2f} s" if isinstance(elapsed, (int, float)) else "—"
|
||||
final_answer = arm.get("final_answer")
|
||||
|
||||
parts = [f'<section class="arm" id="arm-{esc(arm.get("mode", "unknown"))}">']
|
||||
parts.append(f"<h2>Mode: {mode}</h2>")
|
||||
parts.append(
|
||||
f'<p class="meta">{provider} / {model} · {elapsed_s} '
|
||||
f'· completed={fmt(arm.get("completed"))} '
|
||||
f'success={fmt(arm.get("success"))}</p>'
|
||||
)
|
||||
|
||||
if final_answer:
|
||||
parts.append(
|
||||
f'<h3>Final answer</h3><div class="final-answer"><pre>{esc(final_answer)}</pre></div>'
|
||||
)
|
||||
|
||||
signatures = arm.get("tool_call_signatures") or []
|
||||
parts.append(f"<h3>Tool call signatures ({len(signatures)})</h3>")
|
||||
if signatures:
|
||||
parts.append('<ol class="signatures">')
|
||||
for i, sig in enumerate(signatures, start=1):
|
||||
parts.append(
|
||||
f'<li><span class="idx">{i}</span>'
|
||||
f'<code>{esc(sig)}</code></li>'
|
||||
)
|
||||
parts.append("</ol>")
|
||||
else:
|
||||
parts.append("<p class=\"muted\">No tool calls.</p>")
|
||||
|
||||
reasoning = arm.get("reasoning_steps") or []
|
||||
parts.append(f"<h3>Reasoning steps ({len(reasoning)})</h3>")
|
||||
if reasoning:
|
||||
parts.append('<div class="reasoning">')
|
||||
for i, step in enumerate(reasoning, start=1):
|
||||
if isinstance(step, str):
|
||||
body = esc(step)
|
||||
else:
|
||||
body = esc(json.dumps(step, ensure_ascii=False, indent=2))
|
||||
parts.append(
|
||||
f"<details class=\"step\">"
|
||||
f"<summary>Step {i}</summary>"
|
||||
f"<pre>{body}</pre>"
|
||||
f"</details>"
|
||||
)
|
||||
parts.append("</div>")
|
||||
else:
|
||||
parts.append("<p class=\"muted\">No reasoning steps recorded.</p>")
|
||||
|
||||
parts.append("</section>")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def render(data):
|
||||
arms = data.get("arms", [])
|
||||
expected = data.get("expected_numbers", [])
|
||||
|
||||
rows = []
|
||||
for arm in arms:
|
||||
behavior = arm.get("behavior") or {}
|
||||
elapsed = arm.get("elapsed_seconds")
|
||||
elapsed_s = f"{elapsed:.2f}" if isinstance(elapsed, (int, float)) else "—"
|
||||
rows.append(
|
||||
"".join(
|
||||
[
|
||||
"<tr>",
|
||||
f'<td class="mode"><a href="#arm-{esc(arm.get("mode", ""))}">{esc(arm.get("mode", "?"))}</a></td>',
|
||||
f"<td>{status_badge(arm.get('task_success'))}</td>",
|
||||
f"<td>{fmt(arm.get('iterations'))}</td>",
|
||||
f"<td>{fmt(arm.get('repeated_tool_calls'))}</td>",
|
||||
f"<td>{fmt(behavior.get('tool_action_count'))}</td>",
|
||||
f"<td>{elapsed_s}</td>",
|
||||
"</tr>",
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
table = (
|
||||
"<table>"
|
||||
"<thead><tr>"
|
||||
"<th>mode</th>"
|
||||
"<th>task_success</th>"
|
||||
"<th>iterations</th>"
|
||||
"<th>repeated_tool_calls</th>"
|
||||
"<th>tool actions</th>"
|
||||
"<th>elapsed (s)</th>"
|
||||
"</tr></thead>"
|
||||
f"<tbody>{''.join(rows)}</tbody>"
|
||||
"</table>"
|
||||
)
|
||||
|
||||
sections = "\n".join(render_arm_section(arm) for arm in arms)
|
||||
|
||||
expected_html = ""
|
||||
if expected:
|
||||
items = "".join(f"<li><code>{esc(n)}</code></li>" for n in expected)
|
||||
expected_html = (
|
||||
'<h3>Expected numbers</h3>'
|
||||
f'<ul class="expected">{items}</ul>'
|
||||
)
|
||||
|
||||
return f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Experiment {esc(data.get('experiment_id', ''))} — arm comparison</title>
|
||||
<style>
|
||||
body {{ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
max-width: 1000px; margin: 0 auto; padding: 24px; line-height: 1.55;
|
||||
color: #1c2733; background: #fff; }}
|
||||
h1, h2, h3 {{ color: #0b3d66; }}
|
||||
h2 {{ border-bottom: 2px solid #e3e8ee; padding-bottom: 6px; margin-top: 40px; }}
|
||||
table {{ border-collapse: collapse; width: 100%; margin: 12px 0 8px; }}
|
||||
th, td {{ border: 1px solid #d5dce3; padding: 8px 10px; text-align: left; }}
|
||||
th {{ background: #f2f5f8; }}
|
||||
tbody tr:nth-child(even) {{ background: #f8fafc; }}
|
||||
td.mode a {{ font-weight: 600; color: #0b3d66; }}
|
||||
.badge {{ display: inline-block; padding: 1px 8px; border-radius: 10px;
|
||||
font-size: 13px; color: #fff; }}
|
||||
.badge.ok {{ background: #1a7f37; }}
|
||||
.badge.fail {{ background: #c0392b; }}
|
||||
.badge.na {{ background: #9aa7b4; }}
|
||||
.meta {{ color: #5a6b7b; font-size: 13px; }}
|
||||
.final-answer pre {{ background: #f4f6f8; border-left: 4px solid #0b3d66;
|
||||
padding: 10px 12px; overflow-x: auto; }}
|
||||
.signatures {{ padding-left: 0; list-style: none; }}
|
||||
.signatures li {{ display: flex; align-items: baseline; margin: 4px 0; }}
|
||||
.signatures .idx {{ display: inline-block; min-width: 26px; color: #9aa7b4;
|
||||
font-size: 12px; }}
|
||||
.signatures code, .reasoning pre {{ font-family: ui-monospace, SFMono-Regular,
|
||||
Menlo, monospace; font-size: 13px; }}
|
||||
.reasoning .step {{ margin: 6px 0; border: 1px solid #e3e8ee; border-radius: 6px;
|
||||
background: #fafbfc; }}
|
||||
.reasoning summary {{ cursor: pointer; padding: 8px 12px; font-weight: 600;
|
||||
color: #0b3d66; user-select: none; }}
|
||||
.reasoning pre {{ margin: 0; padding: 10px 12px; border-top: 1px solid #e3e8ee;
|
||||
white-space: pre-wrap; word-break: break-word; overflow-x: auto; }}
|
||||
.muted {{ color: #9aa7b4; }}
|
||||
.expected li {{ margin: 2px 0; }}
|
||||
.task {{ background: #f2f5f8; border: 1px solid #d5dce3; border-left: 4px solid
|
||||
#0b3d66; border-radius: 6px; padding: 12px 16px; margin: 16px 0;
|
||||
white-space: pre-wrap; }}
|
||||
.task-label {{ font-weight: 700; color: #0b3d66; margin-bottom: 6px; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Experiment {esc(data.get('experiment_id', ''))}</h1>
|
||||
<p class="meta">source: {esc(data.get('canonical_source', ''))} · created: {esc(data.get('created_at', ''))}</p>
|
||||
<div class="task"><div class="task-label">Task</div>{esc(data.get('task', ''))}</div>
|
||||
{expected_html}
|
||||
<h2>Summary</h2>
|
||||
{table}
|
||||
{sections}
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
|
||||
def serve(data, port):
|
||||
import http.server
|
||||
import threading
|
||||
import webbrowser
|
||||
|
||||
html_body = render(data)
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.path not in ("/", "/index.html"):
|
||||
self.send_error(404)
|
||||
return
|
||||
body = html_body.encode("utf-8")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
print("[http] %s - %s" % (self.address_string(), fmt % args))
|
||||
|
||||
server = http.server.HTTPServer(("127.0.0.1", port), Handler)
|
||||
print(f"Serving on {url} (Ctrl+C to stop)")
|
||||
threading.Timer(0.5, lambda: webbrowser.open(url)).start()
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
print("\nStopped.")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("json_path", nargs="?", default=str(DEFAULT_JSON),
|
||||
help="Path to the validation JSON (default: %(default)s)")
|
||||
parser.add_argument("-p", "--port", type=int, default=8000,
|
||||
help="Port to serve on (default: 8000)")
|
||||
args = parser.parse_args()
|
||||
|
||||
json_file = Path(args.json_path)
|
||||
with open(json_file, encoding="utf-8") as fh:
|
||||
data = json.load(fh)
|
||||
|
||||
serve(data, args.port)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user