ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,15 @@
|
||||
# Chapter 1 experiment requirement/evidence ledger
|
||||
|
||||
The Chinese manuscript is authoritative. Runtime completion and manuscript
|
||||
behavior claims are kept separate: a real, correctly controlled ablation may
|
||||
finish even when one claimed degradation is not observed.
|
||||
|
||||
| Experiment | Exact manuscript gate | Status | Canonical evidence / qualification |
|
||||
|---|---|---|---|
|
||||
| 1-1 | One complete baseline plus removal of tool definitions, tool results, assistant reasoning, and prior history on the same multi-currency ReAct task | Exact five-arm real run completed; one prose behavior not reproduced | `context/validation/latest.json` retains direct Kimi K3 requests/responses, all tool observations, expected totals, and context contracts. Baseline was correct; no tools removed action; hidden tool results and history caused repeated calls. Removing reasoning still completed correctly, so the manuscript’s “contradictory decisions” claim is explicitly false for this run rather than fabricated. |
|
||||
| 1-2 | Exact Kimi K3 with provider-hosted Formula web search, model-directed multiple search rounds, reasoning, current answer, and authoritative links | Passed | `web-search-agent/validation/latest.json`: direct Moonshot endpoint, exact model, 15 succeeded distinct Formula fibers over multiple rounds, official ASEAN/Indonesian sources, real response IDs and usage. |
|
||||
| 1-3 | Multi-provider policy (author-mandated 2026-07-31): model-directed multi-round hosted web search + hosted code execution closed server-side on a Responses API, clarification-before-tools, and current answers with authoritative sources; the official OpenAI GPT-5.6 Sol path stays canonical, but acceptance is not gated on it — any provider with genuinely equivalent hosted tools is eligible | Passed on DashScope `qwen3.7-plus`; official OpenAI still quota-blocked | `search-codegen/validation/latest.json` (+ `validation/runs/real_20260731T170529Z/{evidence,receipts,manifest}.json`, SHA-256 sidecars, credential-free receipts). ASEAN: one hosted `web_search_call` batching 10 model-issued coordinate queries, then a hosted `code_interpreter_call` enumerating all 45 haversine pairs — Kuala Lumpur–Singapore 316.35 km, the same pair as the independent local reference (309.3 km with standard coordinates). Bitcoin: first turn clarified data source/indicators with no tools; continuation via `previous_response_id` ran 3 model-directed search rounds and 4 `code_interpreter_call`s computing MA7/MA20/RSI14/MACD, period return and max drawdown, and plotted a chart in the sandbox. The OpenAI `gpt-5.6-sol` attempt is recorded honestly as `credit_balance_exhausted` before inference. Qualifications: the DashScope sandbox has no outbound network, so daily closes were search-extracted (the model disclosed this in its report); the chart PNG stays inside the sandbox because the API returns logs only; qwen3.7-plus needed an explicit clarify-first system prompt to ask before using tools. |
|
||||
|
||||
Legacy demos and provider-compatible substitutes are teaching aids only. A
|
||||
provider rejection before inference is not converted into a model failure and
|
||||
does not authorize accepting a narrower proxy.
|
||||
@@ -0,0 +1,31 @@
|
||||
# الفصل الأول · أساسيات الوكيل
|
||||
|
||||
> ينطلق الفصل من منظور «النموذج بوصفه وكيلًا»، ويضع المعادلة الأساسية **الوكيل = LLM + السياق + الأدوات**، ثم يقدم هندسة منظومة التشغيل؛ فالقدرات الهندسية المحيطة بالنموذج هي مصدر الميزة التنافسية الحقيقية.
|
||||
|
||||
← [العودة إلى الملف التمهيدي الرئيسي](../docs/ar/README.md) · 📖 [قراءة نص الفصل](../book-ar/chapter1.ar.md)
|
||||
|
||||
## كيفية قراءة التجارب
|
||||
|
||||
يستخدم النص هياكل آلية قصيرة لشرح تدفق التحكم؛ ويحتوي دليل التجارب على محولات SDK الكاملة والسجلات والاختبارات وأدلة القبول. لا حاجة لقراءة كل ملف سطرًا سطرًا.
|
||||
|
||||
- **Starter:** ابدأ بالهدف والأمر الأدنى وشروط القبول؛ وابدأ من [context](context/);
|
||||
- **Builder:** تتبّع نقطة الدخول والحلقة الأساسية ومخطط الحالة/الرسائل والأدوات وأداة التحقق.
|
||||
- **Maintainer:** ثم اقرأ الاختبارات وmanifest الأدلة ومعالجة الأعطال ومسارات التراجع ومحولات المزوّد.
|
||||
|
||||
في القراءة الأولى يمكنك تجاوز بيانات الاعتماد وطبقة العرض وتوافق المزوّد؛ عُد إليها عند إعادة إنتاج رقم.
|
||||
|
||||
## المشاريع المصاحبة
|
||||
|
||||
| التجربة | المشروع | النوع | الوصف |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [السياق](context/) | ✅ | يوضح أهمية مكونات سياق الوكيل عبر دراسات استئصال منهجية. ويدعم عدة موفري نماذج، منهم SiliconFlow Qwen وByteDance Doubao وMoonshot Kimi، بما يتيح تجربة إعدادات سياق مختلفة ومراقبة أثرها في سلوك الوكيل. |
|
||||
| 1-2 | [وكيل بحث الويب](web-search-agent/) | ✅ | يستخدم وكيلًا يتمتع بقدرات البحث العميق الأساسية، وقادر على البحث متعدد الجولات وتكامل المعلومات. |
|
||||
| 1-3 | [بحث-كودجن](search-codegen/) | ✅ | إنشاء وكيل يتمتع بإمكانات البحث العميق الأساسية ووضع الحماية للتعليمات البرمجية، باستخدام أدوات مثل بحث الويب وتنفيذ الشفرة للتحليل المعقد. |
|
||||
| 7-1، 7-2 | [التعلم من الخبرة](learning-from-experience/) | ✅ | يقارن التعلم المعزز التقليدي (Q-learning) مع التعلم في السياق القائم على LLM، ويعيد إنتاج الأفكار الرئيسية من منشور مدونة Shunyu Yao "النصف الثاني". يوضح كيف يمكن لـ نماذج LLM أن يتفوق على RL التقليدي بكفاءة عينة تبلغ 250-400x من خلال لعبة البحث عن الكنز. |
|
||||
## أنواع المشاريع
|
||||
|
||||
| الأيقونة | النوع | المعنى |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **مستقل** | شفرة كاملة قابلة للتشغيل في هذا المستودع بعد إعداد مفتاح API |
|
||||
| 📖 | **دليل إعادة الإنتاج** | وثائق تفصيلية تعتمد على مستودع خارجي يُجلب باستخدام `git clone` |
|
||||
| 🚧 | **وثيقة التصميم** | وثيقة تصميم وخطة تنفيذ؛ أما الشفرة القابلة للتشغيل فما تزال قيد التطوير |
|
||||
@@ -0,0 +1,31 @@
|
||||
# Chapter 1 · Agent Fundamentals
|
||||
|
||||
> Starting from the new paradigm of "Model as Agent," establishes the core formula **Agent = LLM + Context + Tools**, and introduces Harness engineering—all engineering capabilities beyond the model are the true competitive advantage.
|
||||
|
||||
← [Back to main README](../docs/en/README.md) · 📖 [Read chapter text](../book-en/chapter1.md)
|
||||
|
||||
## How to Read the Experiments
|
||||
|
||||
The prose uses short mechanism skeletons to explain control flow; the experiment directory contains complete SDK adapters, logs, tests, and acceptance evidence. You do not need to read every file line by line.
|
||||
|
||||
- **Starter:** Start with the goal, minimum command, and acceptance conditions; begin with [context](context/);
|
||||
- **Builder:** Follow the entry point, core loop, state/message schema, tools, and verifier.
|
||||
- **Maintainer:** Then read tests, evidence manifests, failure handling, rollback paths, and provider adapters.
|
||||
|
||||
On a first pass, skip credential loading, presentation code, and provider-compatibility layers; return when reproducing a number.
|
||||
|
||||
## Companion Projects
|
||||
|
||||
| Exp. | Project | Type | Description |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | Demonstrates the importance of various Agent context components through systematic ablation experiments. Supports direct Alibaba Cloud Model Studio Qwen plus SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi, and other providers. |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Implements an Agent with basic deep search capabilities, capable of multi-round searching and information integration. |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | Builds an Agent with basic deep search and code sandbox capabilities, utilizing tools like web search and code execution for complex analysis. |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | Compares traditional reinforcement learning (Q-learning) with LLM-based in-context learning, reproducing key insights from Shunyu Yao's "The Second Half" blog post. Demonstrates how LLMs can surpass traditional RL with 250-400x sample efficiency through a treasure hunt game. |
|
||||
## Project Types
|
||||
|
||||
| Icon | Type | Meaning |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **Standalone** | Full code in this repo, runs after configuring API Key |
|
||||
| 📖 | **Reproduction Guide** | Detailed doc depending on **external repos** to `git clone` |
|
||||
| 🚧 | **Design Doc** | Architecture/implementation plan only, runnable code still WIP |
|
||||
@@ -0,0 +1,32 @@
|
||||
# Capítulo 1 · Fundamentos de los Agentes de IA
|
||||
|
||||
> **Agente = LLM + Contexto + Herramientas**; La ingeniería del Harness es la verdadera ventaja competitiva
|
||||
|
||||
← [Volver al README principal](../docs/es/README.md) · 📖 [Leer texto del capítulo](../book-es/chapter1.es.md)
|
||||
|
||||
## Cómo leer los experimentos
|
||||
|
||||
El texto usa skeletons breves para explicar el flujo de control; el directorio de experimentos contiene adaptadores SDK completos, registros, pruebas y evidencias de aceptación. No hace falta leer cada archivo línea por línea.
|
||||
|
||||
- **Starter:** Empieza por el objetivo, el comando mínimo y la aceptación; comienza con [context](context/);
|
||||
- **Builder:** Sigue el punto de entrada, el bucle central, el esquema de estado/mensajes, las herramientas y el verificador.
|
||||
- **Maintainer:** Después revisa pruebas, manifiestos, fallos, rollback y adaptadores de proveedores.
|
||||
|
||||
En la primera pasada puedes omitir credenciales, presentación y compatibilidad de proveedores; vuelve al reproducir una cifra.
|
||||
|
||||
## Proyectos Complementarios
|
||||
|
||||
| Exp. | Proyecto | Tipo | Descripción |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | Experimentos de ablación sistemática que muestran la importancia de los componentes del contexto; compatible con SiliconFlow Qwen, ByteDance Doubao y Moonshot Kimi |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Agente Kimi K3 con capacidad de búsqueda profunda básica, capaz de realizar búsquedas multirronda e integración de información |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | Integración de herramientas nativas de GPT-5, utilizando búsqueda web y sandbox de código para análisis complejos |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | Comparación entre Q-learning tradicional y aprendizaje en contexto basado en LLM, reproduciendo la eficiencia de muestra (250–400x) |
|
||||
|
||||
## Tipos de Proyectos
|
||||
|
||||
| Icono | Tipo | Significado |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **Autónomo** | Código completo en este repositorio, se ejecuta tras configurar la Clave API |
|
||||
| 📖 | **Guía de Reproducción** | Documento detallado que depende de **repositorios externos** para realizar `git clone` |
|
||||
| 🚧 | **Documento de Diseño** | Solo arquitectura/plan de implementación, el código ejecutable aún está en desarrollo |
|
||||
@@ -0,0 +1,32 @@
|
||||
# 1. fejezet · Ügynökalapok
|
||||
|
||||
> A „modell mint ágens” paradigmából kiindulva felépíti az **Ágens = LLM + Kontextus + Eszközök** alapképletet, és bemutatja a modellen túli versenyelőnyt jelentő harness-mérnökséget.
|
||||
|
||||
← [Vissza a magyar főoldalhoz](../docs/hu/README.md) · 📖 [A fejezet olvasása](../book-hu/chapter1.md)
|
||||
|
||||
## Hogyan olvassuk a kísérleteket?
|
||||
|
||||
A törzsszöveg rövid mechanizmus-skeletonokkal magyarázza a vezérlési folyamatot; a kísérleti könyvtárakban találhatók a teljes SDK-adapterek, naplók, tesztek és átvételi bizonyítékok. Nem kell minden fájlt sorról sorra elolvasni.
|
||||
|
||||
- **Starter:** Kezdje a céllal, a minimális paranccsal és az átvételi feltételekkel; induljon innen: [context](context/);
|
||||
- **Builder:** Kövesse a belépési pontot, a fő ciklust, az állapot-/üzenetsémát, az eszközöket és az ellenőrzőt.
|
||||
- **Maintainer:** Végül olvassa el a teszteket, a bizonyíték-manifeszteket, a hibakezelést, a visszaállítási útvonalakat és a provider-adaptereket.
|
||||
|
||||
Első olvasáskor átugorható a hitelesítő adatok betöltése, a megjelenítési réteg és a provider-kompatibilitás; a számok reprodukálásakor térjen vissza.
|
||||
|
||||
## Kapcsolódó projektek
|
||||
|
||||
| Kísérlet | Projekt | Típus | Leírás |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | Több LLM-szolgáltatóval végzett ablációs kísérleteken mutatja be a kontextus összetevőinek fontosságát. |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Alapszintű mélykereső ágenst valósít meg többkörös kereséssel és információ-összesítéssel. |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | Webes keresést és kódsandboxot kapcsol össze összetettebb elemzési feladatokhoz. |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | A Q-learninget és az LLM-alapú kontextuson belüli tanulást hasonlítja össze egy kincskereső játékban. |
|
||||
|
||||
## Projekttípusok
|
||||
|
||||
| Ikon | Típus | Jelentés |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **Önálló** | A teljes kód a repository-ban található, és az API-kulcsok beállítása után futtatható. |
|
||||
| 📖 | **Reprodukciós útmutató** | Külső repository szükséges, amelyet külön kell `git clone` paranccsal letölteni. |
|
||||
| 🚧 | **Folyamatban** | Az implementáció vagy az elfogadási bizonyíték még nem teljes. |
|
||||
@@ -0,0 +1,32 @@
|
||||
# Bab 1 · Dasar-Dasar Agent
|
||||
|
||||
> Memulai dari paradigma “Model sebagai Agent”, membangun rumus inti **Agent = LLM + Context + Tool**, dan memperkenalkan rekayasa Harness sebagai keunggulan di luar model.
|
||||
|
||||
← [Kembali ke README utama](../docs/id/README.md) · 📖 [Baca bab](../book-id/chapter1.md)
|
||||
|
||||
## Cara Membaca Eksperimen
|
||||
|
||||
Teks utama memakai skeleton mekanisme singkat untuk menjelaskan alur kontrol; direktori eksperimen berisi adapter SDK lengkap, log, pengujian, dan bukti penerimaan. Anda tidak perlu membaca setiap berkas baris demi baris.
|
||||
|
||||
- **Starter:** Mulai dari tujuan, perintah minimum, dan syarat penerimaan; awali dengan [context](context/);
|
||||
- **Builder:** Telusuri titik masuk, loop inti, skema status/pesan, alat, dan verifier.
|
||||
- **Maintainer:** Terakhir, baca pengujian, manifest bukti, penanganan kegagalan, rollback, dan adapter provider.
|
||||
|
||||
Pada pembacaan pertama, lewati kredensial, presentasi, dan kompatibilitas provider; kembali saat mereproduksi angka.
|
||||
|
||||
## Proyek Pendamping
|
||||
|
||||
| Eksperimen | Proyek | Jenis | Deskripsi |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | Menunjukkan pentingnya komponen context melalui eksperimen ablasi pada beberapa penyedia LLM. |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Menerapkan Agent pencarian mendalam dasar dengan pencarian multi-putaran dan integrasi informasi. |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | Menggabungkan pencarian web dan sandbox kode untuk analisis yang lebih kompleks. |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | Membandingkan Q-learning dengan in-context learning berbasis LLM pada permainan pencarian harta karun. |
|
||||
|
||||
## Jenis Proyek
|
||||
|
||||
| Ikon | Jenis | Arti |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **Mandiri** | Kode lengkap tersedia di repositori dan dapat dijalankan setelah API Key dikonfigurasi. |
|
||||
| 📖 | **Panduan Reproduksi** | Memerlukan repositori eksternal yang harus di-`git clone`. |
|
||||
| 🚧 | **Dalam Proses** | Implementasi atau bukti penerimaan belum lengkap. |
|
||||
@@ -0,0 +1,31 @@
|
||||
# 第1章 · Agent の基礎
|
||||
|
||||
> 「モデルこそが Agent」という新しいパラダイムから出発し、中核となる公式 **Agent = LLM + コンテキスト + ツール** を確立し、Harness エンジニアリングを紹介する。モデルを超えたすべてのエンジニアリング能力こそが、真の競争優位性である。
|
||||
|
||||
← [メイン README に戻る](../docs/ja/README.md) · 📖 [章の本文を読む](../book-ja/chapter1.ja.md)
|
||||
|
||||
## 実験の読み方
|
||||
|
||||
本文では短い mechanism skeleton で制御フローを説明し、実験ディレクトリには完全な SDK アダプター、ログ、テスト、受け入れ証拠を置きます。すべてのファイルを一行ずつ読む必要はありません。
|
||||
|
||||
- **Starter:** 目的・最小コマンド・受け入れ条件から始め、まず [context](context/);
|
||||
- **Builder:** エントリポイント、中心ループ、状態／メッセージ schema、ツール、検証器を追います。
|
||||
- **Maintainer:** 最後にテスト、証拠 manifest、失敗処理、rollback 経路、provider adapter を読みます。
|
||||
|
||||
初読では認証情報、表示層、provider 互換層を飛ばし、数値を再現するときに戻ってください。
|
||||
|
||||
## 付随プロジェクト
|
||||
|
||||
| 実験 | プロジェクト | 種類 | 説明 |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | 体系的なアブレーション（実験）を通じて、Agent のさまざまなコンテキスト構成要素の重要性を示す。複数の LLM プロバイダー（SiliconFlow Qwen、ByteDance Doubao、Moonshot Kimi）をサポートし、異なるコンテキストモードを設定して Agent の挙動の変化を観察できる。 |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | 基本的なディープ検索能力を持つ Agent を実装し、複数ラウンドの検索と情報統合を可能にする。 |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | 基本的なディープ検索とコードサンドボックス能力を備えた Agent を構築し、Web 検索やコード実行などのツールを活用して複雑な分析を行う。 |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | 従来の強化学習（Q 学習）と LLM ベースのインコンテキスト学習を比較し、Shunyu Yao のブログ記事「The Second Half」の重要な洞察を再現する。宝探しゲームを通じて、LLM が 250〜400 倍のサンプル効率で従来の RL を上回る様子を示す。 |
|
||||
## プロジェクトの種類
|
||||
|
||||
| アイコン | 種類 | 意味 |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **単独実行** | このリポジトリに完全なコードがあり、API キーを設定すれば実行できる |
|
||||
| 📖 | **再現ガイド** | `git clone` が必要な**外部リポジトリ**に依存する詳細ドキュメント |
|
||||
| 🚧 | **設計ドキュメント** | アーキテクチャ/実装計画のみで、実行可能なコードは未完成 |
|
||||
@@ -0,0 +1,32 @@
|
||||
# 제1장 · 에이전트 기초
|
||||
|
||||
> ‘모델이 곧 에이전트’라는 새로운 패러다임에서 출발해 **에이전트 = LLM + 컨텍스트 + 도구**라는 핵심 공식을 세우고, 모델 밖의 모든 엔지니어링 역량이 진정한 경쟁력이라는 하네스 엔지니어링을 소개합니다.
|
||||
|
||||
← [한국어 메인 README로 돌아가기](../docs/ko/README.md) · 📖 [제1장 본문 읽기](../book-ko/chapter1.ko.md)
|
||||
|
||||
## 실험 읽는 방법
|
||||
|
||||
본문은 짧은 메커니즘 skeleton으로 제어 흐름을 설명하고, 실험 디렉터리에는 완전한 SDK 어댑터·로그·테스트·검수 증거를 둡니다. 모든 파일을 줄 단위로 읽을 필요는 없습니다.
|
||||
|
||||
- **Starter:** 목표, 최소 명령, 검수 조건부터 시작하고 다음에서 출발하세요: [context](context/);
|
||||
- **Builder:** 진입점, 핵심 루프, 상태/메시지 스키마, 도구와 verifier를 따라갑니다.
|
||||
- **Maintainer:** 마지막으로 테스트, 증거 manifest, 실패 처리, rollback 경로와 provider adapter를 읽습니다.
|
||||
|
||||
첫 읽기에서는 credential, UI, provider 호환 계층을 건너뛰고 수치를 재현할 때 돌아오세요.
|
||||
|
||||
## 연계 프로젝트
|
||||
|
||||
| 실험 | 프로젝트 | 유형 | 설명 |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | 체계적인 구성 요소 제거 실험을 통해 에이전트 컨텍스트를 이루는 각 요소의 중요성을 보여 줍니다. 여러 LLM 제공자(SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi)를 지원하며, 컨텍스트 모드를 바꾸어 에이전트 행동이 어떻게 달라지는지 관찰할 수 있습니다. |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | 여러 차례 검색하고 정보를 종합하는 기본적인 심층 검색 에이전트를 구현합니다. |
|
||||
| 1-3 | [search-codegen](search-codegen/) | 🚧 | GPT-5 공식 Responses API의 호스팅 검색과 코드 인터프리터 경로를 완전히 구현했지만, 공식 실측 두 번 모두 할당량 부족으로 429 응답을 받았습니다. OpenRouter는 인터페이스 진단에만 사용했으며 본문 검수를 대신하지 않습니다. |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | Q-learning 10,000회와 평가 100회, 공식 Kimi K3의 첫 에피소드까지 두 방식을 실측해 검수를 마쳤습니다. [증거](learning-from-experience/validation/20260730_011704/evidence.json)에는 Kimi가 17단계 만에 성공하고 fallback을 사용하지 않은 사실과 과거 점 추정치와의 차이가 기록돼 있습니다. |
|
||||
|
||||
## 프로젝트 유형
|
||||
|
||||
| 아이콘 | 유형 | 의미 |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **독립 실행** | 전체 코드가 이 저장소에 있으며, API 키를 설정하면 실행할 수 있습니다. |
|
||||
| 📖 | **재현 가이드** | **외부 저장소**를 `git clone`해야 하는 상세 안내 문서입니다. |
|
||||
| 🚧 | **설계 문서** | 아키텍처와 구현 계획만 있으며, 실행 가능한 코드는 아직 작성 중입니다. |
|
||||
@@ -0,0 +1,38 @@
|
||||
# 第 1 章 · Agent 基础知识
|
||||
|
||||
> **Agent = LLM + 上下文 + 工具**；Harness 工程才是竞争力
|
||||
|
||||
← [返回主目录](../README.md) · 📖 [读本章正文](../book/chapter1.md)
|
||||
|
||||
逐项正文验收、真实 API 状态与证据路径见
|
||||
[EXPERIMENT_LEDGER.md](EXPERIMENT_LEDGER.md)。其中实验 1-1 的五臂正式运行已完成，
|
||||
但“去掉 reasoning 必然退化”没有在该次运行中复现；实验 1-3 已按作者批准的多提供商
|
||||
政策验收：官方 OpenAI 路径保留但仍受配额阻塞，阿里云百炼 qwen3.7-plus 的
|
||||
Responses API（hosted web_search + code_interpreter）实测通过全部验收门。
|
||||
|
||||
## 如何阅读实验
|
||||
|
||||
正文用短 skeleton 解释控制流；实验目录承载完整 SDK 适配、日志和验收。无需逐行读完每个文件，建议按三层推进：
|
||||
|
||||
- **Starter**：先读目标、最小命令和验收条件，推荐从 [context](context/) 开始；
|
||||
- **Builder**：沿 main.py 的入口追踪上下文、工具调用和消融变量，再看 [web-search-agent](web-search-agent/) 的多轮循环；
|
||||
- **Maintainer**：最后阅读 tests/、证据 manifest、失败回退和 provider adapter。
|
||||
|
||||
首次阅读可跳过凭据加载、展示层和 provider 兼容代码；当你要复现实验数字时，再回到 [验收台账](EXPERIMENT_LEDGER.md)。
|
||||
|
||||
## 配套项目
|
||||
|
||||
| 编号 | 项目 | 类型 | 一句话说明 |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | 系统性消融实验展示 Agent 上下文各组件的重要性；支持阿里云百炼直连 Qwen、SiliconFlow Qwen、字节 Doubao、月之暗面 Kimi 等多提供商 |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Kimi K3 模型即 Agent，具备基础深度搜索能力，能进行多轮搜索和信息整合 |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | 模型自主多轮搜索 + 服务端代码执行的 Deep Research 闭环，先澄清意图再执行；官方 GPT-5.6 路径保留，阿里云百炼 qwen3.7-plus（hosted web_search + code_interpreter）实测通过东盟首都距离与比特币技术分析全部验收门 |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | 10,000 局 Q-learning + 100 局评估与官方 Kimi K3 第一局双臂实测已验收；[证据](learning-from-experience/validation/20260730_011704/evidence.json)记录 Kimi 17 步成功、零 fallback 及历史点估计差异 |
|
||||
|
||||
## 项目类型说明
|
||||
|
||||
| 图标 | 类型 | 含义 |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **可独立运行** | 本仓库自带完整代码，配置好 API Key 即可运行 |
|
||||
| 📖 | **复现指南** | 依赖需自行 `git clone` 的**外部仓库**（训练框架、评测基准等） |
|
||||
| 🚧 | **设计文档** | 仅包含架构与实现方案，可运行代码仍在完善中 |
|
||||
@@ -0,0 +1,31 @@
|
||||
# Глава 1 · Основы агентов
|
||||
|
||||
> Отталкиваясь от новой парадигмы «модель как агент», формулирует базовую формулу **Агент = LLM + Контекст + Инструменты** и вводит harness-инженерию — все инженерные возможности за пределами модели и есть настоящее конкурентное преимущество.
|
||||
|
||||
← [К оглавлению](../docs/ru/README.md) · 📖 [Читать главу](../book-ru/chapter1.md)
|
||||
|
||||
## Как читать эксперименты
|
||||
|
||||
В основном тексте короткие скелеты механизмов объясняют поток управления; в каталогах экспериментов находятся полные адаптеры SDK, журналы, тесты и приёмочные доказательства. Читать каждый файл построчно не требуется.
|
||||
|
||||
- **Starter:** Начните с цели, минимальной команды и условий приёмки; начните с [context](context/);
|
||||
- **Builder:** Проследите точку входа, основной цикл, схему состояния/сообщений, инструменты и проверяющий модуль.
|
||||
- **Maintainer:** Затем изучите тесты, манифесты доказательств, обработку сбоев, откат и адаптеры провайдеров.
|
||||
|
||||
При первом чтении можно пропустить ключи, слой представления и совместимость провайдеров; вернитесь при воспроизведении чисел.
|
||||
|
||||
## Сопутствующие проекты
|
||||
|
||||
| Эксп. | Проект | Тип | Описание |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | На систематических абляционных экспериментах показывает важность различных компонент контекста агента. Поддерживает несколько провайдеров LLM (SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi), позволяя настраивать разные режимы контекста и наблюдать изменение поведения агента. |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Реализует агента с базовой возможностью глубокого поиска — многораундовый поиск и интеграция информации. |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | Строит агента с базовыми возможностями глубокого поиска и песочницы кода, применяя веб-поиск и исполнение кода для сложного анализа. |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | Сравнивает традиционное обучение с подкреплением (Q-learning) с обучением в контексте на основе LLM, воспроизводя ключевые выводы из эссе Shunyu Yao «The Second Half». На игре-поиске сокровищ показывает, как LLM превосходит традиционный RL по эффективности выборки в 250–400 раз. |
|
||||
## Типы проектов
|
||||
|
||||
| Значок | Тип | Значение |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **Автономный** | Полный код в этом репозитории, запускается после настройки API-ключа |
|
||||
| 📖 | **Гайд по воспроизведению** | Подробный документ, зависящий от **внешних репозиториев** через `git clone` |
|
||||
| 🚧 | **Проектный документ** | Только архитектура/план реализации, рабочий код ещё в разработке |
|
||||
@@ -0,0 +1,32 @@
|
||||
# அத்தியாயம் 1 · ஏஜென்ட் அடிப்படைகள்
|
||||
|
||||
> "மாதிரியே ஏஜென்ட்" என்ற புதிய முன்னுதாரணத்திலிருந்து தொடங்கி, **ஏஜென்ட் = LLM + சூழல் + கருவிகள்** என்ற முக்கிய சூத்திரத்தை நிறுவுகிறது; மேலும் Harness பொறியியலையும் அறிமுகப்படுத்துகிறது — மாதிரிக்கு அப்பாற்பட்ட அனைத்துப் பொறியியல் திறன்களே உண்மையான போட்டித் திறன் அமைந்த இடமாகும்.
|
||||
|
||||
← [முக்கிய README க்குத் திரும்பு](../docs/ta/README.md) · 📖 [அத்தியாய உரையைப் படி](../book-ta/chapter1.ta.md)
|
||||
|
||||
## சோதனைகளை எப்படிப் படிப்பது
|
||||
|
||||
முதன்மை உரை குறுகிய mechanism skeleton-களால் control flow-ஐ விளக்குகிறது; முழு SDK adapters, logs, tests, acceptance evidence ஆகியவை experiment கோப்பகத்தில் உள்ளன. ஒவ்வொரு கோப்பையும் வரி வரியாகப் படிக்க வேண்டியதில்லை.
|
||||
|
||||
- **Starter:** இலக்கு, குறைந்தபட்ச கட்டளை, ஏற்றுக்கொள்ளும் நிபந்தனைகளில் தொடங்குங்கள்; முதலில் [context](context/);
|
||||
- **Builder:** நுழைவுப் புள்ளி, மையச் சுழற்சி, state/message schema, கருவிகள், verifier ஆகியவற்றைப் பின்தொடருங்கள்.
|
||||
- **Maintainer:** பின்னர் tests, evidence manifest, தோல்வி கையாளல், rollback பாதை, provider adapter ஆகியவற்றைப் படியுங்கள்.
|
||||
|
||||
முதல் வாசிப்பில் credentials, UI, provider-compatibility அடுக்குகளைத் தவிர்க்கலாம்; முடிவுகளை மீண்டும் உருவாக்கும்போது திரும்பிப் பாருங்கள்.
|
||||
|
||||
## துணை திட்டங்கள்
|
||||
|
||||
| சோதனை | Project | Type | Description |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | முறையான அப்லேஷன் பரிசோதனைகள் மூலம் ஏஜென்ட் சூழலின் ஒவ்வொரு கூறின் முக்கியத்துவத்தையும் காட்டுகிறது. பல LLM வழங்குநர்களை (SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi) ஆதரிக்கிறது; வெவ்வேறு சூழல் பயன்முறைகளை உள்ளமைத்து ஏஜென்ட் நடத்தை மாற்றங்களைக் கவனிக்கலாம். |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | அடிப்படை ஆழமான தேடல் திறன் கொண்ட ஏஜென்டை உருவாக்குகிறது; பல சுற்று தேடல் மற்றும் தகவல் ஒருங்கிணைப்பைச் செய்ய முடியும். |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | அடிப்படை ஆழமான தேடல் திறன் மற்றும் குறியீடு மணற்பெட்டி (sandbox) திறன் கொண்ட ஏஜென்டை உருவாக்கி, இணையத் தேடல், குறியீடு செயலாக்கம் போன்ற கருவிகளை ஒருங்கிணைத்துச் சிக்கலான பகுப்பாய்வுகளைச் செய்கிறது. |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | பாரம்பரிய வலுவூட்டல் கற்றல் (Q-learning) மற்றும் LLM அடிப்படையிலான சூழல் கற்றலை ஒப்பிட்டு, Shunyu Yao இன் "The Second Half" வலைப்பதிவின் முக்கிய நுண்ணறிவுகளை மீண்டும் உருவாக்குகிறது. புதையல் தேடும் விளையாட்டின் மூலம் LLM எவ்வாறு 250-400 மடங்கு மாதிரி செயல்திறனுடன் (sample efficiency) பாரம்பரிய RL ஐ விஞ்சுகிறது என்பதைக் காட்டுகிறது. |
|
||||
|
||||
## திட்ட வகைகள்
|
||||
|
||||
| சின்னம் | வகை | பொருள் |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **தனித்து இயங்கும்** | முழு குறியீடு இந்த களஞ்சியத்தில், API Key உள்ளமைத்தவுடன் இயங்கும் |
|
||||
| 📖 | **மறு உருவாக்க வழிகாட்டி** | **வெளிப்புற களஞ்சியங்களை** `git clone` செய்ய வேண்டிய விரிவான ஆவணம் |
|
||||
| 🚧 | **வடிவமைப்பு ஆவணம்** | கட்டமைப்பு/செயலாக்கத் திட்டம் மட்டும், இயங்கும் குறியீடு இன்னும் WIP |
|
||||
@@ -0,0 +1,32 @@
|
||||
# Bölüm 1 · Agent Temelleri
|
||||
|
||||
> "Ajan Olarak Model" yeni paradigmasından başlayarak **Agent = LLM + Context + Tools** temel formülünü kurar ve Harness mühendisliğini tanıtır—modelin ötesindeki tüm mühendislik yetenekleri gerçek rekabet avantajıdır.
|
||||
|
||||
← [Ana README'ye dön](../README.tr.md) · 📖 [Bölüm metnini oku](../book-tr/chapter1.tr.md)
|
||||
|
||||
## Deneyler nasıl okunur
|
||||
|
||||
Metin, kontrol akışını açıklamak için kısa mekanizma skeleton'ları kullanır; deney dizininde tam SDK adaptörleri, günlükler, testler ve kabul kanıtı bulunur. Her dosyayı satır satır okumanız gerekmez.
|
||||
|
||||
- **Starter:** Hedef, en kısa komut ve kabul koşullarıyla başlayın; önce [context](context/);
|
||||
- **Builder:** Giriş noktasını, ana döngüyü, durum/mesaj şemasını, araçları ve doğrulayıcıyı izleyin.
|
||||
- **Maintainer:** Son olarak testleri, kanıt manifestlerini, hata işlemeyi, rollback yollarını ve sağlayıcı adaptörlerini okuyun.
|
||||
|
||||
İlk okumada kimlik bilgisi yükleme, sunum katmanı ve sağlayıcı uyumluluğunu atlayıp sayıları yeniden üretirken dönün.
|
||||
|
||||
## Eşlik Eden Projeler
|
||||
|
||||
| Proje | Tür | Açıklama |
|
||||
| --- | :--: | --- |
|
||||
| [learning-from-experience](learning-from-experience/) | ✅ | Geleneksel pekiştirmeli öğrenmeyi (Q-learning) LLM tabanlı bağlam içi öğrenmeyle karşılaştırır, Shunyu Yao'nun "The Second Half" blog yazısındaki temel içgörüleri yeniden üretir. Bir hazine avı oyunu üzerinden LLM'lerin geleneksel RL'yi 250-400 kat örnek verimliliğiyle nasıl geçebildiğini gösterir. |
|
||||
| [web-search-agent](web-search-agent/) | ✅ | Temel derin arama yeteneklerine sahip, çok turlu arama ve bilgi entegrasyonu yapabilen bir Agent uygular. |
|
||||
| [search-codegen](search-codegen/) | ✅ | Temel derin arama ve kod sandbox yeteneklerine sahip bir Agent inşa eder; karmaşık analiz için web araması ve kod yürütme gibi araçları kullanır. |
|
||||
| [context](context/) | ✅ | Sistematik ablasyon deneyleriyle çeşitli Agent context bileşenlerinin önemini gösterir. Birden çok LLM sağlayıcısını destekler (SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi), farklı context modlarını yapılandırıp Agent davranışındaki değişimleri gözlemlemeye olanak tanır. |
|
||||
|
||||
## Proje Türleri
|
||||
|
||||
| İkon | Tür | Anlamı |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **Bağımsız** | Bu depoda tam kod, API Key yapılandırıldıktan sonra çalışır |
|
||||
| 📖 | **Yeniden Üretim Rehberi** | `git clone` ile **harici depolara** bağımlı ayrıntılı belge |
|
||||
| 🚧 | **Tasarım Belgesi** | Yalnızca mimari/uygulama planı, çalıştırılabilir kod henüz hazır değil |
|
||||
@@ -0,0 +1,32 @@
|
||||
# Chương 1 · Kiến thức nền tảng về Agent
|
||||
|
||||
> xuất phát từ mô hình mới “model as Agent”, xây dựng công thức cốt lõi **Agent = LLM + context + tools**, đồng thời giới thiệu kỹ thuật Harness — mọi năng lực kỹ thuật nằm ngoài mô hình mới là lợi thế cạnh tranh thực sự.
|
||||
|
||||
← [Về README chính](../docs/vi/README.md) · 📖 [Đọc nội dung chương](../book-vi/chapter1.vi.md)
|
||||
|
||||
## Cách đọc các thí nghiệm
|
||||
|
||||
Phần văn bản dùng skeleton cơ chế ngắn để giải thích luồng điều khiển; thư mục thí nghiệm chứa adapter SDK đầy đủ, log, kiểm thử và bằng chứng nghiệm thu. Không cần đọc từng tệp theo từng dòng.
|
||||
|
||||
- **Starter:** Bắt đầu từ mục tiêu, lệnh tối thiểu và điều kiện nghiệm thu; hãy bắt đầu với [context](context/);
|
||||
- **Builder:** Lần theo điểm vào, vòng lặp lõi, schema trạng thái/tin nhắn, công cụ và verifier.
|
||||
- **Maintainer:** Sau đó đọc test, manifest bằng chứng, xử lý lỗi, đường rollback và adapter nhà cung cấp.
|
||||
|
||||
Lần đầu có thể bỏ qua credential, lớp trình bày và tương thích provider; quay lại khi cần tái tạo số liệu.
|
||||
|
||||
## Dự án đi kèm
|
||||
|
||||
| Thí nghiệm | Project | Type | Description |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | Thông qua thí nghiệm ablation có hệ thống để cho thấy tầm quan trọng của từng thành phần trong ngữ cảnh Agent. Hỗ trợ nhiều nhà cung cấp LLM (SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi), cấu hình các chế độ ngữ cảnh khác nhau để quan sát thay đổi hành vi của Agent. |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Triển khai Agent có khả năng tìm kiếm chuyên sâu cơ bản, có thể tìm kiếm nhiều vòng và tổng hợp thông tin. |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | Xây dựng Agent có năng lực tìm kiếm chuyên sâu cơ bản và sandbox chạy mã, tổng hợp sử dụng tìm kiếm web, thực thi mã và các công cụ khác để phân tích phức tạp. |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | So sánh học tăng cường truyền thống (Q-learning) với học trong ngữ cảnh dựa trên LLM, tái hiện các insight then chốt trong bài viết “The Second Half” của Shunyu Yao. Thông qua trò chơi săn kho báu, dự án cho thấy LLM có thể vượt RL truyền thống về hiệu quả mẫu tới 250–400 lần. |
|
||||
|
||||
## Phân loại dự án
|
||||
|
||||
| Biểu tượng | Loại | Ý nghĩa |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **Chạy độc lập** | Có mã đầy đủ trong kho, chạy được sau khi cấu hình API Key |
|
||||
| 📖 | **Hướng dẫn tái hiện** | Tài liệu chi tiết, cần `git clone` **kho ngoài** |
|
||||
| 🚧 | **Tài liệu thiết kế** | Chỉ có kiến trúc/phương án, mã chạy được đang hoàn thiện |
|
||||
@@ -0,0 +1,32 @@
|
||||
# 第 1 章 · Agent 基礎知識
|
||||
|
||||
> **Agent = LLM + 上下文 + 工具**；Harness 工程才是競爭力
|
||||
|
||||
← [返回主目錄](../docs/zh-TW/README.md) · 📖 [讀本章正文](../book/chapter1.md)
|
||||
|
||||
## 如何閱讀實驗
|
||||
|
||||
正文用短小的機制 skeleton 說明控制流；實驗目錄放完整的 SDK 適配、日誌、測試與驗收證據，不需要逐行讀完每個檔案。
|
||||
|
||||
- **Starter:** 先讀目標、最小指令與驗收條件；可從 [context](context/);
|
||||
- **Builder:** 沿著入口、核心迴圈、狀態／訊息 schema、工具與驗證器閱讀。
|
||||
- **Maintainer:** 最後再看測試、證據 manifest、失敗處理、回滾路徑與 provider adapter。
|
||||
|
||||
第一次閱讀可先跳過憑證載入、展示層和 provider 相容層；要重現數字時再回來查看。
|
||||
|
||||
## 配套專案
|
||||
|
||||
| 編號 | 專案 | 型別 | 一句話說明 |
|
||||
| :--: | --- | :--: | --- |
|
||||
| 1-1 | [context](context/) | ✅ | 系統性消融實驗展示 Agent 上下文各元件的重要性；支援 SiliconFlow Qwen、字節 Doubao、月之暗面 Kimi 等多提供商 |
|
||||
| 1-2 | [web-search-agent](web-search-agent/) | ✅ | Kimi K3 模型即 Agent，具備基礎深度搜尋能力，能進行多輪搜尋和資訊整合 |
|
||||
| 1-3 | [search-codegen](search-codegen/) | ✅ | GPT-5 原生工具整合，綜合利用網路搜尋與程式碼沙箱實現複雜分析 |
|
||||
| 7-1, 7-2 | [learning-from-experience](learning-from-experience/) | ✅ | 對比 Q-learning 與基於 LLM 的上下文學習，復現 Shunyu Yao 的 "The Second Half"：LLM 以 250–400 倍樣本效率超越傳統 RL |
|
||||
|
||||
## 專案型別說明
|
||||
|
||||
| 圖示 | 型別 | 含義 |
|
||||
| :--: | --- | --- |
|
||||
| ✅ | **可獨立執行** | 本倉庫自帶完整程式碼，配置好 API Key 即可執行 |
|
||||
| 📖 | **復現指南** | 依賴需自行 `git clone` 的**外部倉庫**（訓練框架、評測基準等） |
|
||||
| 🚧 | **設計文件** | 僅包含架構與實現方案，可執行程式碼仍在完善中 |
|
||||
@@ -0,0 +1,983 @@
|
||||
# Context-Aware AI Agent with Ablation Studies / 上下文感知 Agent 与消融实验
|
||||
|
||||
> Multi-provider context-aware agent with systematic ablation of context components (history, reasoning, tool calls, tool results).
|
||||
> 配套《深入理解 AI Agent》第 1 章 **实验 1-1 ★★：上下文的关键作用**。
|
||||
|
||||
← [Chapter 1 index / 返回第 1 章目录](../README.md) · 📖 [Read the chapter / 读本章正文](../../book/chapter1.md)（[EN](../../book-en/chapter1.md)）
|
||||
|
||||
---
|
||||
|
||||
## Code map
|
||||
|
||||
- **Run first:** python main.py --mode interactive (after provider setup).
|
||||
- **Start here:** main.py builds the selected provider and agent loop.
|
||||
- **Core behavior:** agent.py assembles history, reasoning, tool calls and tool results.
|
||||
- **State / protocol:** AgentTrajectory and the provider adapter messages.
|
||||
- **Verifier:** the ablation runner and tests under tests/; compare behavior, not just final text.
|
||||
- **Experiment variable:** context modes (full, no history, no reasoning, no tool calls, no tool results).
|
||||
- **Skip on first pass:** provider-specific clients, plotting and credential checks.
|
||||
|
||||
## English
|
||||
|
||||
### Overview
|
||||
|
||||
This project implements a context-aware AI agent with multiple tools (PDF parsing, currency conversion, calculator, code interpreter) and provides comprehensive ablation testing to explore how different context components affect agent behavior and performance. It supports multiple LLM providers, including Qwen directly through Alibaba Cloud Model Studio (Bailian), SiliconFlow Qwen, ByteDance Doubao, Moonshot Kimi, and DeepSeek.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Multi-provider Support**: Works with Alibaba Cloud Model Studio (Qwen), SiliconFlow (Qwen), Doubao (ByteDance), Kimi (Moonshot), and DeepSeek LLMs
|
||||
- **Multi-tool Agent**: PDF parsing, currency conversion, calculations, and Python code execution
|
||||
- **Context Modes**: Five different context configurations for ablation studies
|
||||
- **Interactive & Batch Modes**: Run single tasks or comprehensive test suites
|
||||
- **Conversation History**: Maintains context across multiple queries in a session
|
||||
- **Detailed Analytics**: Performance metrics, visualizations, and comprehensive reports
|
||||
|
||||
### Supported LLM Providers
|
||||
|
||||
#### Doubao (ByteDance) - Default
|
||||
|
||||
- **Model**: `doubao-seed-1-6-thinking-250715` (customizable)
|
||||
- **API**: OpenAI-compatible via Volcano Engine
|
||||
- **Best for**: Advanced reasoning, faster responses, both English and Chinese tasks
|
||||
|
||||
#### SiliconFlow
|
||||
|
||||
- **Model**: `Qwen/Qwen3.5-397B-A17B` (customizable)
|
||||
- **API**: OpenAI-compatible
|
||||
- **Best for**: Complex reasoning tasks, detailed analysis
|
||||
|
||||
#### Alibaba Cloud Model Studio / Bailian (Qwen)
|
||||
|
||||
- **Model**: `qwen3.7-plus` (customizable with `--model`)
|
||||
- **API**: Direct OpenAI-compatible DashScope endpoint; no SiliconFlow account required
|
||||
- **Provider names**: `dashscope` (canonical), with `qwen` and `bailian` aliases
|
||||
- **Region note**: API keys are region-bound. Mainland keys use the default endpoint; international keys must set `DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1`
|
||||
|
||||
#### Kimi (Moonshot AI)
|
||||
|
||||
- **Model**: `kimi-k3` (K3 reasoning model; temperature is forced to 1 and max_tokens is large enough for its thinking output)
|
||||
- **API**: OpenAI-compatible via Moonshot platform
|
||||
- **Best for**: Advanced reasoning, multi-turn conversations, both English and Chinese tasks
|
||||
- **Features**: Context caching for cost optimization
|
||||
|
||||
#### DeepSeek
|
||||
|
||||
- **Model**: `deepseek-v4-flash` (default; use `--model deepseek-v4-pro` for the stronger tier)
|
||||
- **API**: OpenAI-compatible via [DeepSeek Platform](https://platform.deepseek.com/)
|
||||
- **Best for**: Cost-effective tool-calling agents; thinking mode enabled so the `no_reasoning` ablation can strip `reasoning_content`
|
||||
- **Note**: Legacy aliases `deepseek-chat` / `deepseek-reasoner` are deprecated (2026-07-24); prefer the V4 ids
|
||||
|
||||
### Architecture
|
||||
|
||||
#### Context Components
|
||||
|
||||
1. **Full Context** — Complete agent with all components
|
||||
2. **No History** — Lacks historical tool call tracking
|
||||
3. **No Reasoning** — Operates without strategic planning
|
||||
4. **No Tool Calls** — Cannot execute external tools
|
||||
5. **No Tool Results** — Blind to tool execution outcomes
|
||||
|
||||
#### Available Tools
|
||||
|
||||
- **`parse_pdf(url)`** — Download and extract text from PDF documents
|
||||
- **`convert_currency(amount, from, to)`** — Real-time currency conversion
|
||||
- **`calculate(expression)`** — Simple mathematical expression evaluation
|
||||
- **`code_interpreter(code)`** — Execute Python code for complex calculations, totals, and data processing
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Python 3.10+
|
||||
- API key for one of the supported providers:
|
||||
- **Alibaba Cloud Model Studio / Bailian**: Get from [Model Studio](https://bailian.console.aliyun.com/)
|
||||
- **SiliconFlow**: Get from [SiliconFlow](https://siliconflow.cn)
|
||||
- **Doubao (ByteDance)**: Get from [Volcano Engine](https://www.volcengine.com/)
|
||||
- **Kimi (Moonshot)**: Get from [Moonshot Platform](https://platform.moonshot.cn/)
|
||||
- **DeepSeek**: Get from [DeepSeek Platform](https://platform.deepseek.com/api_keys)
|
||||
|
||||
### Sample Tasks
|
||||
|
||||
The system includes 5 pre-defined sample tasks demonstrating different capabilities:
|
||||
|
||||
1. **Simple Currency Conversion** — Basic multi-currency calculations
|
||||
2. **Multi-Currency Budget Analysis** — Complex expense analysis across offices
|
||||
3. **PDF Financial Analysis** — Parse and analyze financial documents
|
||||
4. **Investment Growth Calculation** — Compound interest with currency conversion
|
||||
5. **Comprehensive Financial Report** — Complete workflow using all tools
|
||||
|
||||
These samples are designed to showcase the agent's capabilities and the impact of context ablation.
|
||||
|
||||
### Quick Start
|
||||
|
||||
#### 1. Installation
|
||||
|
||||
```bash
|
||||
# Recommended from the repository root: use the shared Chapter 1 environment
|
||||
uv sync --locked --extra ch1
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch1]"
|
||||
|
||||
# Enter this experiment directory for the commands below
|
||||
cd chapter1/context
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
# Copy and configure environment
|
||||
cp env.example .env
|
||||
# Edit .env and add one provider key (for example DASHSCOPE_API_KEY or ARK_API_KEY)
|
||||
```
|
||||
|
||||
#### 2. Configure Provider
|
||||
|
||||
```bash
|
||||
# For Doubao (ByteDance) - Default
|
||||
export ARK_API_KEY=your_key_here
|
||||
python main.py # Uses Doubao by default
|
||||
|
||||
# For SiliconFlow (Qwen)
|
||||
export SILICONFLOW_API_KEY=your_key_here
|
||||
python main.py --provider siliconflow
|
||||
|
||||
# For Qwen directly through Alibaba Cloud Model Studio / Bailian
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope
|
||||
# --provider qwen and --provider bailian are equivalent aliases.
|
||||
# For an international-region key, also set:
|
||||
export DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
|
||||
# For Kimi (Moonshot)
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python main.py --provider kimi
|
||||
|
||||
# For DeepSeek
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python main.py --provider deepseek
|
||||
# Optional stronger model:
|
||||
python main.py --provider deepseek --model deepseek-v4-pro
|
||||
|
||||
# Or specify a custom model
|
||||
python main.py --model doubao-seed-1-6-thinking-250715
|
||||
|
||||
# Universal OpenRouter fallback: if the provider key above is missing/invalid
|
||||
# but OPENROUTER_API_KEY is set, requests are routed through OpenRouter and the
|
||||
# model id is mapped automatically (bare gpt-*/o1-* -> openai/*, claude-* ->
|
||||
# anthropic/*, deepseek-* -> deepseek/*, other native ids -> OPENROUTER_MODEL
|
||||
# or openai/gpt-5.6-luna).
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python main.py # falls back to OpenRouter when ARK_API_KEY is unset
|
||||
python main.py --provider openrouter # or use OpenRouter directly
|
||||
```
|
||||
|
||||
#### 3. Testing Qwen / Kimi / DeepSeek Integration
|
||||
|
||||
```bash
|
||||
# Run the ablation study directly on Alibaba Cloud Qwen
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope --mode ablation
|
||||
|
||||
# Quick test of Kimi K3 model
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python tests/manual/check_kimi.py
|
||||
|
||||
# Use Kimi in main script
|
||||
python main.py --provider kimi --mode interactive
|
||||
|
||||
# Run ablation study with Kimi
|
||||
python main.py --provider kimi --mode ablation
|
||||
|
||||
# Quick test of DeepSeek V4
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python tests/manual/check_deepseek.py
|
||||
# or: python tests/manual/check_deepseek_quick.py
|
||||
|
||||
# Use DeepSeek in main script / ablation study
|
||||
python main.py --provider deepseek --mode interactive
|
||||
python main.py --provider deepseek --mode ablation
|
||||
```
|
||||
|
||||
#### 4. Run Interactive Mode (Recommended)
|
||||
|
||||
```bash
|
||||
# Default (Doubao)
|
||||
python main.py --mode interactive
|
||||
|
||||
# With SiliconFlow provider
|
||||
python main.py --mode interactive --provider siliconflow
|
||||
|
||||
# In interactive mode, you can:
|
||||
# - Type 'samples' to see pre-defined tasks
|
||||
# - Type 'sample 2' to test PDF parsing
|
||||
# - Type 'providers' to list available providers
|
||||
# - Type 'provider kimi' to switch providers
|
||||
# - Type 'status' to see current configuration
|
||||
# - Type 'help' for all commands
|
||||
```
|
||||
|
||||
#### 5. Run Sample Tasks
|
||||
|
||||
```bash
|
||||
# Run without arguments to select from samples
|
||||
python main.py --mode single
|
||||
|
||||
# With specific provider
|
||||
python main.py --mode single --provider doubao
|
||||
|
||||
# Or provide your own task
|
||||
python main.py --mode single \
|
||||
--task "Convert $1000 USD to EUR, GBP, and JPY. Calculate the average." \
|
||||
--context-mode full \
|
||||
--provider siliconflow
|
||||
```
|
||||
|
||||
#### 6. Run Ablation Study
|
||||
|
||||
```bash
|
||||
# With default provider (single case, all five context modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# With Doubao provider
|
||||
python main.py --mode ablation --provider doubao
|
||||
|
||||
# Multi-case comparison across modes (stronger evidence for the book's point)
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# Compare only two modes and save raw results to a custom path
|
||||
python main.py --mode ablation --ablation-modes full no_history --output my_ablation.json
|
||||
```
|
||||
|
||||
`main.py` is the single CLI entry point. Run `python main.py --help` for the full (Chinese) flag reference.
|
||||
|
||||
Key flags:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--mode` | `single` / `ablation` / `interactive` (default) |
|
||||
| `--task` | Task text for `single` mode |
|
||||
| `--context-mode` | Context mode for `single` mode (`full`, `no_history`, `no_reasoning`, `no_tool_calls`, `no_tool_results`) |
|
||||
| `--ablation-modes` | Subset of modes to test in `ablation` mode (default: all five) |
|
||||
| `--cases` | Number of cases each mode is run against in `ablation` mode (default: 1) |
|
||||
| `--provider` / `--model` | LLM provider and optional model override |
|
||||
| `--output` | Output path for the JSON result (single) or raw results (ablation) |
|
||||
|
||||
### Ablation Studies
|
||||
|
||||
#### Accepted real Kimi K3 execution (2026-07-29)
|
||||
|
||||
`run_experiment_1_1.py` executes the exact five arms from the manuscript and
|
||||
persists every credential-free provider request/response, rather than only a
|
||||
summary table:
|
||||
|
||||
```bash
|
||||
python run_experiment_1_1.py --provider kimi --model kimi-k3 --max-iterations 5
|
||||
```
|
||||
|
||||
The same evidence runner can use a Bailian key directly:
|
||||
|
||||
```bash
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python run_experiment_1_1.py --provider dashscope --model qwen3.7-plus --max-iterations 5
|
||||
```
|
||||
|
||||
The accepted artifact is [validation/latest.json](validation/latest.json). All
|
||||
five request-shape contracts passed on the direct Moonshot API. The full arm
|
||||
produced the correct USD total and average; removing tool definitions produced
|
||||
zero tool actions; removing tool results and removing history both caused
|
||||
repeated actions. The no-reasoning arm still answered correctly in this run, so
|
||||
the manuscript's categorical contradiction claim was **not reproduced** and is
|
||||
reported separately from execution acceptance.
|
||||
|
||||
Observed results (these are not the expected-behavior labels below):
|
||||
|
||||
| Arm | Iterations | Tool actions | Repeated action | Correct numerical answer |
|
||||
|---|---:|---:|---|---|
|
||||
| full | 3 | 4 | no | yes |
|
||||
| no history | 5 (ceiling) | 15 | yes | no answer |
|
||||
| no reasoning | 3 | 4 | no | **yes — negative result for the manuscript claim** |
|
||||
| no tool definitions | 1 | 0 | no | no; the model explicitly declined to invent rates |
|
||||
| no tool results | 5 | 7 | yes | no; the model eventually reported that observations were hidden |
|
||||
|
||||
In an individual raw arm, `completed` means that the API/agent loop returned a
|
||||
terminal response. It does not mean that the requested task was correct. The
|
||||
legacy `success` field is retained as an alias for `completed` so older result
|
||||
readers continue to work; new readers should use `completed` explicitly.
|
||||
`task_success` is the task-specific correctness result. For this experiment it
|
||||
is computed by the canonical numeric rubric, while the generic agent cannot
|
||||
infer correctness from arbitrary natural-language prompts. The canonical
|
||||
behavioral booleans are under `analysis.manuscript_behavior_claims`;
|
||||
`all_manuscript_behavior_claims_observed` is false. This separation prevents a
|
||||
graceful refusal or hallucinated tool markup in an ablated arm from being
|
||||
mislabeled as task success, without forcing any ablation outcome in advance.
|
||||
|
||||
The ablation studies systematically remove context components to understand their importance.
|
||||
|
||||
#### Test Scenario
|
||||
|
||||
A complex financial analysis task requiring:
|
||||
|
||||
1. PDF document parsing
|
||||
2. Multiple currency conversions
|
||||
3. Mathematical calculations
|
||||
4. Result aggregation
|
||||
|
||||
#### Expected Behaviors
|
||||
|
||||
| Context Mode | Removed Component (book §实验 1.1) | Expected Behavior | Impact |
|
||||
|-------------|-----------------------------------|-------------------|---------|
|
||||
| **full** | none (baseline) | Complete successful execution | Baseline performance |
|
||||
| **no_history** | 历史消息 (history) | Redundant operations, inefficiency | May repeat tool calls |
|
||||
| **no_reasoning** | 思考过程 (reasoning) | Unstructured approach, potential errors | Lacks strategic planning |
|
||||
| **no_tool_calls** | 工具定义 (tool definitions) | Complete failure | Cannot interact with external world |
|
||||
| **no_tool_results** | 工具执行结果 (tool results) | Incorrect conclusions | Makes decisions without feedback |
|
||||
|
||||
**How each ablation is applied** (see `agent.py`):
|
||||
|
||||
- **no_tool_calls** — the `tools` parameter is omitted from the request, so the model has no tool definitions to call.
|
||||
- **no_tool_results** — every tool result is replaced with a `[Tool result hidden]` placeholder.
|
||||
- **no_reasoning** — `reasoning_content` is stripped from each assistant message before it is added back to the trajectory.
|
||||
- **no_history** — `_prepare_messages_for_api()` sends only a sliding window (system prompt + current task + the most recent ReAct step) to the model, so earlier steps are forgotten and the agent tends to repeat tool calls. Full mode always sends the complete trajectory.
|
||||
|
||||
#### Running Tests
|
||||
|
||||
```bash
|
||||
# Run the full ablation study (single case, all five modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# Run across multiple cases for a stronger comparison
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# This will generate:
|
||||
# - ablation_study_results.png (visualization, if matplotlib is installed)
|
||||
# - ablation_study_report.md (detailed report)
|
||||
# - ablation_results.json (raw data; override path with --output)
|
||||
```
|
||||
|
||||
The console prints two tables: a per-run **ablation study results** table and a **comparison matrix** (context mode x case) for reading the effect of each component at a glance.
|
||||
|
||||
#### Automated Regression Tests
|
||||
|
||||
```bash
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
Manual provider/API smoke scripts live under `tests/manual/` and require the corresponding API keys.
|
||||
|
||||
### Understanding Results
|
||||
|
||||
#### Performance Metrics
|
||||
|
||||
- **Terminal Response Rate**: Whether the agent returned a terminal response
|
||||
- **Task Success**: Correctness under the task-specific rubric (when one is available)
|
||||
- **Execution Time**: Total time to complete the task
|
||||
- **Iterations**: Number of agent-model interactions
|
||||
- **Tool Calls**: Number of external tool invocations
|
||||
- **Reasoning Steps**: Strategic planning iterations
|
||||
|
||||
#### Sample Output
|
||||
|
||||
```
|
||||
ABLATION STUDY RESULTS
|
||||
================================================================================
|
||||
| Test Name | Success | Time | Iterations | Tool Calls |
|
||||
|--------------------------------|---------|--------|------------|------------|
|
||||
| Baseline - Full Context | ✓ | 12.3s | 5 | 8 |
|
||||
| No Historical Tool Calls | ✓ | 18.7s | 8 | 12 |
|
||||
| No Reasoning Process | ✗ | 25.4s | 10 | 15 |
|
||||
| No Tool Call Commands | ✗ | 3.2s | 2 | 0 |
|
||||
| No Tool Call Results | ✗ | 15.6s | 10 | 10 |
|
||||
```
|
||||
|
||||
### Key Insights
|
||||
|
||||
1. **Tool Calls Are Fundamental** — Without tool call capability, the agent cannot interact with external systems, making task completion impossible.
|
||||
2. **Tool Results Provide Critical Feedback** — Without seeing results, the agent operates blind, leading to incorrect conclusions and infinite loops.
|
||||
3. **Reasoning Enables Efficiency** — Strategic planning reduces iterations and tool calls, improving both speed and accuracy.
|
||||
4. **History Prevents Redundancy** — Historical context prevents repeated operations and maintains task coherence across iterations.
|
||||
|
||||
### Advanced Usage
|
||||
|
||||
#### Interactive Mode Commands
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `samples` | Display all available sample tasks |
|
||||
| `sample <n>` | Run sample task number n |
|
||||
| `providers` | List all available LLM providers |
|
||||
| `provider <name>` | Switch to a different provider (e.g., `provider kimi`) |
|
||||
| `modes` | List available context modes for ablation testing |
|
||||
| `mode <name>` | Switch context mode (e.g., `mode no_history`) |
|
||||
| `status` | Show current configuration (provider, model, mode, etc.) |
|
||||
| `reset` | Reset agent trajectory (clear history) |
|
||||
| `create_pdfs` | Generate sample PDF files for testing |
|
||||
| `quit` | Exit interactive mode |
|
||||
|
||||
**Note:** The prompt shows the current provider in brackets, e.g., `[KIMI]>` or `[DOUBAO]>`
|
||||
|
||||
#### Conversation History
|
||||
|
||||
The agent maintains conversation history throughout interactive sessions:
|
||||
|
||||
- **Persistent Context**: The agent remembers previous queries and responses within a session
|
||||
- **Multi-turn Conversations**: You can reference information from earlier in the conversation
|
||||
- **Tool Call Memory**: Previous tool executions are remembered and can be referenced
|
||||
- **Reset on Demand**: Use the `reset` command to clear history and start fresh
|
||||
|
||||
Example conversation flow:
|
||||
|
||||
```
|
||||
[DOUBAO]> Remember that our budget is $10,000. Calculate 15% of it.
|
||||
# Agent calculates and remembers the budget
|
||||
|
||||
[DOUBAO]> Now convert that 15% amount to EUR
|
||||
# Agent uses the previously calculated amount without re-asking
|
||||
|
||||
[DOUBAO]> What was our original budget?
|
||||
# Agent recalls the $10,000 mentioned earlier
|
||||
```
|
||||
|
||||
#### Custom Tasks
|
||||
|
||||
```python
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
result = agent.execute_task("""
|
||||
Download the PDF from https://example.com/report.pdf,
|
||||
extract all monetary values, convert them to EUR,
|
||||
and calculate the total.
|
||||
""")
|
||||
```
|
||||
|
||||
#### Creating Test PDFs
|
||||
|
||||
```bash
|
||||
python create_sample_pdf.py
|
||||
# Creates fixtures/pdfs/ with sample financial reports
|
||||
```
|
||||
|
||||
#### Configuration
|
||||
|
||||
Edit `config.py` or set environment variables:
|
||||
|
||||
```bash
|
||||
export MODEL_TEMPERATURE=0.5
|
||||
export MAX_ITERATIONS=15
|
||||
export LOG_LEVEL=DEBUG
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
context/
|
||||
├── README.md # This file
|
||||
├── main.py # Single CLI entry point (single / ablation / interactive)
|
||||
├── agent.py # Core agent implementation + context modes
|
||||
├── config.py # Configuration management
|
||||
├── create_sample_pdf.py # PDF generation utility
|
||||
├── fixtures/
|
||||
│ └── pdfs/ # Sample PDFs used by local demos/tests
|
||||
├── tests/
|
||||
│ ├── test_agent.py
|
||||
│ ├── test_code_interpreter.py
|
||||
│ ├── test_malformed_tool_json.py
|
||||
│ └── manual/ # Provider/API smoke scripts; require real keys
|
||||
├── requirements.txt # Dependencies
|
||||
└── env.example # Environment template
|
||||
```
|
||||
|
||||
> Note: the ablation study lives in `main.py` (`AblationTestSuite`), run via `python main.py --mode ablation`. There is no separate `ablation_tests.py`.
|
||||
|
||||
### Research Applications
|
||||
|
||||
- **AI Safety Research**: Understanding failure modes
|
||||
- **System Design**: Identifying critical components
|
||||
- **Optimization**: Finding minimal viable configurations
|
||||
- **Education**: Teaching agent architecture principles
|
||||
|
||||
### Limitations
|
||||
|
||||
- Currency rates are fixed (production should use real-time APIs)
|
||||
- PDF parsing may fail on complex layouts
|
||||
- Model token limits may affect very large documents
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 概述
|
||||
|
||||
本项目实现一个上下文感知 AI Agent，配备多种工具（PDF 解析、货币换算、计算器、代码解释器），并通过系统化的**消融实验**（Ablation Study）检验不同上下文组件对 Agent 行为与性能的影响。支持通过阿里云百炼直连 Qwen，也支持 SiliconFlow Qwen、字节跳动 Doubao、月之暗面 Kimi、DeepSeek。对应书中**实验 1-1 ★★：上下文的关键作用**。
|
||||
|
||||
### 主要特性
|
||||
|
||||
- **多提供商支持**：阿里云百炼（Qwen 直连）、SiliconFlow（Qwen）、Doubao（字节）、Kimi（月之暗面）、DeepSeek
|
||||
- **多工具 Agent**：PDF 解析、货币换算、计算与 Python 代码执行
|
||||
- **上下文模式**：五种配置，用于消融对照
|
||||
- **交互与批处理**：单任务运行或完整测试套件
|
||||
- **对话历史**：同一会话内跨多轮查询保持上下文
|
||||
- **详细分析**：性能指标、可视化与综合报告
|
||||
|
||||
### 支持的 LLM 提供商
|
||||
|
||||
#### Doubao（字节跳动）— 默认
|
||||
|
||||
- **模型**：`doubao-seed-1-6-thinking-250715`（可自定义）
|
||||
- **API**：火山引擎上的 OpenAI 兼容接口
|
||||
- **适合**：深度推理、较快响应，中英文任务均可
|
||||
|
||||
#### SiliconFlow
|
||||
|
||||
- **模型**：`Qwen/Qwen3.5-397B-A17B`（可自定义）
|
||||
- **API**：OpenAI 兼容
|
||||
- **适合**：复杂推理与细致分析
|
||||
|
||||
#### 阿里云百炼（Qwen 直连）
|
||||
|
||||
- **模型**：`qwen3.7-plus`（可通过 `--model` 自定义）
|
||||
- **API**：直连 DashScope 的 OpenAI 兼容接口，无需 SiliconFlow 账号
|
||||
- **提供商名称**：规范名称为 `dashscope`，也可使用别名 `qwen` 或 `bailian`
|
||||
- **区域说明**：API Key 与区域绑定。中国内地 Key 默认直连内地端点；国际站 Key 必须设置 `DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1`
|
||||
|
||||
#### Kimi（月之暗面）
|
||||
|
||||
- **模型**：`kimi-k3`（K3 推理模型；temperature 强制为 1，max_tokens 足够容纳思考输出）
|
||||
- **API**：Moonshot 平台 OpenAI 兼容接口
|
||||
- **适合**：深度推理、多轮对话，中英文任务均可
|
||||
- **特性**：上下文缓存以优化成本
|
||||
|
||||
#### DeepSeek
|
||||
|
||||
- **模型**：`deepseek-v4-flash`（默认；更强档可用 `--model deepseek-v4-pro`）
|
||||
- **API**：[DeepSeek Platform](https://platform.deepseek.com/) 的 OpenAI 兼容接口
|
||||
- **适合**：性价比高的工具调用；开启 thinking，便于 `no_reasoning` 消融剥离 `reasoning_content`
|
||||
- **说明**：旧别名 `deepseek-chat` / `deepseek-reasoner` 已弃用（2026-07-24），请优先使用 V4 id
|
||||
|
||||
### 架构
|
||||
|
||||
#### 上下文组件
|
||||
|
||||
1. **Full Context** — 完整 Agent，保留全部组件
|
||||
2. **No History** — 缺少历史工具调用追踪
|
||||
3. **No Reasoning** — 无战略规划/思考过程
|
||||
4. **No Tool Calls** — 无法执行外部工具
|
||||
5. **No Tool Results** — 看不到工具执行结果
|
||||
|
||||
#### 可用工具
|
||||
|
||||
- **`parse_pdf(url)`** — 下载并抽取 PDF 文本
|
||||
- **`convert_currency(amount, from, to)`** — 货币换算
|
||||
- **`calculate(expression)`** — 简单数学表达式求值
|
||||
- **`code_interpreter(code)`** — 执行 Python，用于复杂计算、汇总与数据处理
|
||||
|
||||
### 前置条件
|
||||
|
||||
- Python 3.10+
|
||||
- 任一支持提供商的 API Key：
|
||||
- **阿里云百炼**：[百炼控制台](https://bailian.console.aliyun.com/)
|
||||
- **SiliconFlow**：[SiliconFlow](https://siliconflow.cn)
|
||||
- **Doubao（字节）**：[火山引擎](https://www.volcengine.com/)
|
||||
- **Kimi（月之暗面）**：[Moonshot Platform](https://platform.moonshot.cn/)
|
||||
- **DeepSeek**：[DeepSeek Platform](https://platform.deepseek.com/api_keys)
|
||||
|
||||
### 示例任务
|
||||
|
||||
系统预置 5 个样例任务：
|
||||
|
||||
1. **简单货币换算** — 基础多币种计算
|
||||
2. **多币种预算分析** — 跨办公室费用分析
|
||||
3. **PDF 财务分析** — 解析并分析财务文档
|
||||
4. **投资增长计算** — 复利与货币换算
|
||||
5. **综合财务报告** — 串联全部工具的完整流程
|
||||
|
||||
用于展示 Agent 能力与上下文消融的影响。
|
||||
|
||||
### 快速开始
|
||||
|
||||
#### 1. 安装
|
||||
|
||||
```bash
|
||||
# 推荐在仓库根目录使用统一的第 1 章环境
|
||||
uv sync --locked --extra ch1
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch1]"
|
||||
|
||||
# 进入本实验目录，后续命令都在这里运行
|
||||
cd chapter1/context
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
# 复制并配置环境变量
|
||||
cp env.example .env
|
||||
# 编辑 .env 并填入一个提供商的 API Key（例如 DASHSCOPE_API_KEY 或 ARK_API_KEY）
|
||||
```
|
||||
|
||||
#### 2. 配置提供商
|
||||
|
||||
```bash
|
||||
# For Doubao (ByteDance) - Default
|
||||
export ARK_API_KEY=your_key_here
|
||||
python main.py # Uses Doubao by default
|
||||
|
||||
# For SiliconFlow (Qwen)
|
||||
export SILICONFLOW_API_KEY=your_key_here
|
||||
python main.py --provider siliconflow
|
||||
|
||||
# 通过阿里云百炼直连 Qwen
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope
|
||||
# --provider qwen 与 --provider bailian 是等价别名。
|
||||
# 如果使用国际站 Key，还需设置：
|
||||
export DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
|
||||
# For Kimi (Moonshot)
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python main.py --provider kimi
|
||||
|
||||
# For DeepSeek
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python main.py --provider deepseek
|
||||
# Optional stronger model:
|
||||
python main.py --provider deepseek --model deepseek-v4-pro
|
||||
|
||||
# Or specify a custom model
|
||||
python main.py --model doubao-seed-1-6-thinking-250715
|
||||
|
||||
# Universal OpenRouter fallback: if the provider key above is missing/invalid
|
||||
# but OPENROUTER_API_KEY is set, requests are routed through OpenRouter and the
|
||||
# model id is mapped automatically (bare gpt-*/o1-* -> openai/*, claude-* ->
|
||||
# anthropic/*, deepseek-* -> deepseek/*, other native ids -> OPENROUTER_MODEL
|
||||
# or openai/gpt-5.6-luna).
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python main.py # falls back to OpenRouter when ARK_API_KEY is unset
|
||||
python main.py --provider openrouter # or use OpenRouter directly
|
||||
```
|
||||
|
||||
#### 3. 测试 Qwen / Kimi / DeepSeek 集成
|
||||
|
||||
```bash
|
||||
# 通过阿里云百炼 Qwen 直接运行消融实验
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python main.py --provider dashscope --mode ablation
|
||||
|
||||
# Quick test of Kimi K3 model
|
||||
export MOONSHOT_API_KEY=your_key_here
|
||||
python tests/manual/check_kimi.py
|
||||
|
||||
# Use Kimi in main script
|
||||
python main.py --provider kimi --mode interactive
|
||||
|
||||
# Run ablation study with Kimi
|
||||
python main.py --provider kimi --mode ablation
|
||||
|
||||
# Quick test of DeepSeek V4
|
||||
export DEEPSEEK_API_KEY=your_key_here
|
||||
python tests/manual/check_deepseek.py
|
||||
# or: python tests/manual/check_deepseek_quick.py
|
||||
|
||||
# Use DeepSeek in main script / ablation study
|
||||
python main.py --provider deepseek --mode interactive
|
||||
python main.py --provider deepseek --mode ablation
|
||||
```
|
||||
|
||||
#### 4. 交互模式（推荐）
|
||||
|
||||
```bash
|
||||
# Default (Doubao)
|
||||
python main.py --mode interactive
|
||||
|
||||
# With SiliconFlow provider
|
||||
python main.py --mode interactive --provider siliconflow
|
||||
|
||||
# In interactive mode, you can:
|
||||
# - Type 'samples' to see pre-defined tasks
|
||||
# - Type 'sample 2' to test PDF parsing
|
||||
# - Type 'providers' to list available providers
|
||||
# - Type 'provider kimi' to switch providers
|
||||
# - Type 'status' to see current configuration
|
||||
# - Type 'help' for all commands
|
||||
```
|
||||
|
||||
#### 5. 运行样例任务
|
||||
|
||||
```bash
|
||||
# Run without arguments to select from samples
|
||||
python main.py --mode single
|
||||
|
||||
# With specific provider
|
||||
python main.py --mode single --provider doubao
|
||||
|
||||
# Or provide your own task
|
||||
python main.py --mode single \
|
||||
--task "Convert $1000 USD to EUR, GBP, and JPY. Calculate the average." \
|
||||
--context-mode full \
|
||||
--provider siliconflow
|
||||
```
|
||||
|
||||
#### 6. 运行消融实验
|
||||
|
||||
```bash
|
||||
# With default provider (single case, all five context modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# With Doubao provider
|
||||
python main.py --mode ablation --provider doubao
|
||||
|
||||
# Multi-case comparison across modes (stronger evidence for the book's point)
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# Compare only two modes and save raw results to a custom path
|
||||
python main.py --mode ablation --ablation-modes full no_history --output my_ablation.json
|
||||
```
|
||||
|
||||
`main.py` 是唯一 CLI 入口。运行 `python main.py --help` 查看完整（中文）参数说明。
|
||||
|
||||
关键参数：
|
||||
|
||||
| 参数 | 说明 |
|
||||
|------|------|
|
||||
| `--mode` | `single` / `ablation` / `interactive`（默认） |
|
||||
| `--task` | `single` 模式的任务文本 |
|
||||
| `--context-mode` | `single` 模式的上下文模式（`full`、`no_history`、`no_reasoning`、`no_tool_calls`、`no_tool_results`） |
|
||||
| `--ablation-modes` | `ablation` 模式下要测的模式子集（默认全部五种） |
|
||||
| `--cases` | `ablation` 模式下每种模式跑的用例数（默认 1） |
|
||||
| `--provider` / `--model` | LLM 提供商与可选模型覆盖 |
|
||||
| `--output` | 单次结果或消融原始结果的 JSON 输出路径 |
|
||||
|
||||
### 消融实验
|
||||
|
||||
#### 已验收的 Kimi K3 真实执行（2026-07-29）
|
||||
|
||||
`run_experiment_1_1.py` 会按正文运行五个精确实验组，并保存每轮真实 API 的无凭据
|
||||
请求与响应，而不只是汇总表：
|
||||
|
||||
```bash
|
||||
python run_experiment_1_1.py --provider kimi --model kimi-k3 --max-iterations 5
|
||||
```
|
||||
|
||||
同一证据运行器也支持直接使用百炼 Key：
|
||||
|
||||
```bash
|
||||
export DASHSCOPE_API_KEY=your_key_here
|
||||
python run_experiment_1_1.py --provider dashscope --model qwen3.7-plus --max-iterations 5
|
||||
```
|
||||
|
||||
验收产物见 [validation/latest.json](validation/latest.json)。五组上下文契约全部通过；
|
||||
完整组算出了正确结果，移除工具定义后没有工具行动，移除工具结果或历史后都出现重复行动。
|
||||
但本次“移除思考过程”仍得到正确答案，因此正文关于必然出现矛盾决策的断言**没有复现**；
|
||||
产物把“实验执行通过”和“正文行为结论复现”分开记录。
|
||||
|
||||
系统性地移除上下文组件，以理解其重要性。
|
||||
|
||||
#### 测试场景
|
||||
|
||||
需要以下能力的复杂财务分析任务：
|
||||
|
||||
1. PDF 文档解析
|
||||
2. 多次货币换算
|
||||
3. 数学计算
|
||||
4. 结果汇总
|
||||
|
||||
#### 预期行为
|
||||
|
||||
| 上下文模式 | 移除组件（书中 §实验 1.1） | 预期行为 | 影响 |
|
||||
|-------------|---------------------------|----------|------|
|
||||
| **full** | 无（基线） | 完整成功执行 | 基线性能 |
|
||||
| **no_history** | 历史消息 (history) | 冗余操作、效率下降 | 可能重复调用工具 |
|
||||
| **no_reasoning** | 思考过程 (reasoning) | 方法无结构、易出错 | 缺少战略规划 |
|
||||
| **no_tool_calls** | 工具定义 (tool definitions) | 完全失败 | 无法与外部世界交互 |
|
||||
| **no_tool_results** | 工具执行结果 (tool results) | 错误结论 | 无反馈下做决策 |
|
||||
|
||||
**各消融如何落地**（见 `agent.py`）：
|
||||
|
||||
- **no_tool_calls** — 请求中省略 `tools` 参数，模型没有可调用的工具定义。
|
||||
- **no_tool_results** — 每个工具结果替换为 `[Tool result hidden]` 占位符。
|
||||
- **no_reasoning** — 写回轨迹前，从每条 assistant 消息中剥离 `reasoning_content`。
|
||||
- **no_history** — `_prepare_messages_for_api()` 只发送滑动窗口（系统提示 + 当前任务 + 最近一步 ReAct），早期步骤被遗忘，易重复调工具。完整模式始终发送完整轨迹。
|
||||
|
||||
#### 运行测试
|
||||
|
||||
```bash
|
||||
# Run the full ablation study (single case, all five modes)
|
||||
python main.py --mode ablation
|
||||
|
||||
# Run across multiple cases for a stronger comparison
|
||||
python main.py --mode ablation --cases 3
|
||||
|
||||
# This will generate:
|
||||
# - ablation_study_results.png (visualization, if matplotlib is installed)
|
||||
# - ablation_study_report.md (detailed report)
|
||||
# - ablation_results.json (raw data; override path with --output)
|
||||
```
|
||||
|
||||
控制台会打印两张表：逐次运行的 **ablation study results**，以及 **comparison matrix**（上下文模式 × 用例），便于一眼对比各组件的作用。
|
||||
|
||||
#### 自动化回归测试
|
||||
|
||||
```bash
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
需要真实 API Key 的手动提供商/API 冒烟脚本放在 `tests/manual/`。
|
||||
|
||||
### 结果解读
|
||||
|
||||
#### 性能指标
|
||||
|
||||
- **Terminal Response Rate**：Agent 是否返回了终止响应
|
||||
- **Task Success**：在存在任务专用评分标准时，任务是否正确完成
|
||||
- **Execution Time**：完成任务总耗时
|
||||
- **Iterations**：Agent 与模型交互次数
|
||||
- **Tool Calls**：外部工具调用次数
|
||||
- **Reasoning Steps**：战略规划迭代次数
|
||||
|
||||
#### 输出示例
|
||||
|
||||
```
|
||||
ABLATION STUDY RESULTS
|
||||
================================================================================
|
||||
| Test Name | Success | Time | Iterations | Tool Calls |
|
||||
|--------------------------------|---------|--------|------------|------------|
|
||||
| Baseline - Full Context | ✓ | 12.3s | 5 | 8 |
|
||||
| No Historical Tool Calls | ✓ | 18.7s | 8 | 12 |
|
||||
| No Reasoning Process | ✗ | 25.4s | 10 | 15 |
|
||||
| No Tool Call Commands | ✗ | 3.2s | 2 | 0 |
|
||||
| No Tool Call Results | ✗ | 15.6s | 10 | 10 |
|
||||
```
|
||||
|
||||
### 关键洞察
|
||||
|
||||
1. **工具调用是基础** — 没有工具调用能力，Agent 无法与外部系统交互，任务无法完成。
|
||||
2. **工具结果提供关键反馈** — 看不到结果等于盲目行动，易导致错误结论与死循环。
|
||||
3. **推理提升效率** — 战略规划减少迭代与工具调用，兼顾速度与准确。
|
||||
4. **历史避免冗余** — 历史上下文防止重复操作，并在多轮中保持任务连贯。
|
||||
|
||||
### 进阶用法
|
||||
|
||||
#### 交互模式命令
|
||||
|
||||
| 命令 | 说明 |
|
||||
|------|------|
|
||||
| `samples` | 显示全部样例任务 |
|
||||
| `sample <n>` | 运行第 n 个样例任务 |
|
||||
| `providers` | 列出可用 LLM 提供商 |
|
||||
| `provider <name>` | 切换提供商（如 `provider kimi`） |
|
||||
| `modes` | 列出可用于消融的上下文模式 |
|
||||
| `mode <name>` | 切换上下文模式（如 `mode no_history`） |
|
||||
| `status` | 显示当前配置（提供商、模型、模式等） |
|
||||
| `reset` | 重置 Agent 轨迹（清空历史） |
|
||||
| `create_pdfs` | 生成测试用样例 PDF |
|
||||
| `quit` | 退出交互模式 |
|
||||
|
||||
**说明：** 提示符会以括号显示当前提供商，如 `[KIMI]>` 或 `[DOUBAO]>`
|
||||
|
||||
#### 对话历史
|
||||
|
||||
交互会话中 Agent 会维护对话历史：
|
||||
|
||||
- **持久上下文**：会话内记住先前查询与回复
|
||||
- **多轮对话**：可引用更早提到的信息
|
||||
- **工具调用记忆**：先前工具执行结果可被引用
|
||||
- **按需重置**：使用 `reset` 清空历史重新开始
|
||||
|
||||
示例对话流程：
|
||||
|
||||
```
|
||||
[DOUBAO]> Remember that our budget is $10,000. Calculate 15% of it.
|
||||
# Agent calculates and remembers the budget
|
||||
|
||||
[DOUBAO]> Now convert that 15% amount to EUR
|
||||
# Agent uses the previously calculated amount without re-asking
|
||||
|
||||
[DOUBAO]> What was our original budget?
|
||||
# Agent recalls the $10,000 mentioned earlier
|
||||
```
|
||||
|
||||
#### 自定义任务
|
||||
|
||||
```python
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
result = agent.execute_task("""
|
||||
Download the PDF from https://example.com/report.pdf,
|
||||
extract all monetary values, convert them to EUR,
|
||||
and calculate the total.
|
||||
""")
|
||||
```
|
||||
|
||||
#### 生成测试 PDF
|
||||
|
||||
```bash
|
||||
python create_sample_pdf.py
|
||||
# Creates fixtures/pdfs/ with sample financial reports
|
||||
```
|
||||
|
||||
#### 配置
|
||||
|
||||
编辑 `config.py` 或设置环境变量：
|
||||
|
||||
```bash
|
||||
export MODEL_TEMPERATURE=0.5
|
||||
export MAX_ITERATIONS=15
|
||||
export LOG_LEVEL=DEBUG
|
||||
```
|
||||
|
||||
### 项目结构
|
||||
|
||||
```
|
||||
context/
|
||||
├── README.md # 本文件
|
||||
├── main.py # 单一 CLI 入口（single / ablation / interactive）
|
||||
├── agent.py # Core agent implementation + context modes
|
||||
├── config.py # Configuration management
|
||||
├── create_sample_pdf.py # PDF generation utility
|
||||
├── fixtures/
|
||||
│ └── pdfs/ # 本地 demo/tests 使用的样例 PDF
|
||||
├── tests/
|
||||
│ ├── test_agent.py
|
||||
│ ├── test_code_interpreter.py
|
||||
│ ├── test_malformed_tool_json.py
|
||||
│ └── manual/ # 需真实 Key 的提供商/API 冒烟脚本
|
||||
├── requirements.txt # Dependencies
|
||||
└── env.example # Environment template
|
||||
```
|
||||
|
||||
> 说明：消融实验逻辑在 `main.py` 的 `AblationTestSuite` 中，通过 `python main.py --mode ablation` 运行，没有单独的 `ablation_tests.py`。
|
||||
|
||||
### 研究用途
|
||||
|
||||
- **AI 安全研究**：理解失败模式
|
||||
- **系统设计**：识别关键组件
|
||||
- **优化**：寻找最小可用配置
|
||||
- **教学**：讲解 Agent 架构原理
|
||||
|
||||
### 局限
|
||||
|
||||
- 货币汇率为固定值（生产环境应使用实时 API）
|
||||
- 复杂版式 PDF 解析可能失败
|
||||
- 模型 token 上限可能影响超大文档
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Educational project for context ablation; for production, add proper error handling, rate limiting, and security.
|
||||
本项目为教学向消融实验；生产使用请补齐错误处理、限流与安全措施。
|
||||
- OpenRouter is a universal fallback when the direct provider key is missing.
|
||||
未配置直连提供商 Key 时，可走 `OPENROUTER_API_KEY` 通用兜底。
|
||||
- License: MIT. Contributions welcome (extra tools, scenarios, ablation strategies, performance).
|
||||
许可证：MIT。欢迎贡献（更多工具、场景、消融策略、性能优化）。
|
||||
@@ -0,0 +1,966 @@
|
||||
"""
|
||||
Context-Aware AI Agent with Tool Calls
|
||||
An agent using Qwen model from SiliconFlow with document parsing, currency conversion, and calculator tools.
|
||||
Designed to demonstrate the importance of context through ablation studies.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import List, Dict, Any, Optional
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
import requests
|
||||
from openai import OpenAI
|
||||
import PyPDF2
|
||||
from io import BytesIO
|
||||
import math
|
||||
from datetime import datetime
|
||||
from concurrent.futures import TimeoutError
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _reasoning_safe_temperature(model, requested=1.0):
|
||||
"""Reasoning models (Kimi K3, GPT-5, ...) only accept temperature=1.
|
||||
Return 1 for those; otherwise the requested value so non-reasoning
|
||||
providers (Doubao, DeepSeek, older Moonshot) are unchanged."""
|
||||
m = str(model or "").lower().replace("/", "-")
|
||||
return 1 if ("kimi-k3" in m or "gpt-5" in m) else requested
|
||||
|
||||
|
||||
class ContextMode(Enum):
|
||||
"""Different context modes for ablation studies"""
|
||||
FULL = "full" # Complete context with all components
|
||||
NO_HISTORY = "no_history" # No historical tool calls
|
||||
NO_REASONING = "no_reasoning" # No reasoning/thinking process
|
||||
NO_TOOL_CALLS = "no_tool_calls" # No tool call commands
|
||||
NO_TOOL_RESULTS = "no_tool_results" # No tool call results
|
||||
|
||||
|
||||
@dataclass
|
||||
class ToolCall:
|
||||
"""Represents a single tool call"""
|
||||
tool_name: str
|
||||
arguments: Dict[str, Any]
|
||||
result: Optional[Any] = None
|
||||
timestamp: str = field(default_factory=lambda: datetime.now().isoformat())
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentTrajectory:
|
||||
"""Tracks the agent's execution trajectory"""
|
||||
reasoning_steps: List[str] = field(default_factory=list)
|
||||
tool_calls: List[ToolCall] = field(default_factory=list)
|
||||
# Exact, credential-free request/response evidence for every real model
|
||||
# turn. This is deliberately part of the trajectory: Experiment 1-1 is
|
||||
# about what the model could see at decision time, so reconstructing the
|
||||
# request after the fact is not acceptable evidence.
|
||||
api_turns: List[Dict[str, Any]] = field(default_factory=list)
|
||||
context_mode: ContextMode = ContextMode.FULL
|
||||
|
||||
|
||||
class ToolRegistry:
|
||||
"""Registry for available tools"""
|
||||
|
||||
@staticmethod
|
||||
def parse_pdf(url: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Download and parse a PDF from URL or local file
|
||||
|
||||
Args:
|
||||
url: URL or file path of the PDF to parse
|
||||
|
||||
Returns:
|
||||
Dictionary containing parsed text and metadata
|
||||
"""
|
||||
try:
|
||||
# Check if it's a local file
|
||||
if url.startswith('file://'):
|
||||
# Extract the file path from file:// URL
|
||||
file_path = url.replace('file://', '')
|
||||
logger.info(f"Reading local PDF from {file_path}")
|
||||
|
||||
# Read the file directly
|
||||
with open(file_path, 'rb') as f:
|
||||
pdf_content = f.read()
|
||||
|
||||
elif url.startswith('/') or url.startswith('./') or url.startswith('../') or ':\\' in url or ':/' in url[1:3]:
|
||||
# Direct file path (absolute or relative)
|
||||
logger.info(f"Reading local PDF from {url}")
|
||||
|
||||
# Read the file directly
|
||||
with open(url, 'rb') as f:
|
||||
pdf_content = f.read()
|
||||
|
||||
else:
|
||||
# It's a remote URL, download it
|
||||
logger.info(f"Downloading PDF from {url}")
|
||||
response = requests.get(url, timeout=30)
|
||||
response.raise_for_status()
|
||||
pdf_content = response.content
|
||||
|
||||
# Parse the PDF content
|
||||
pdf_file = BytesIO(pdf_content)
|
||||
pdf_reader = PyPDF2.PdfReader(pdf_file)
|
||||
|
||||
text_content = []
|
||||
for page_num, page in enumerate(pdf_reader.pages, 1):
|
||||
text = page.extract_text()
|
||||
text_content.append({
|
||||
"page": page_num,
|
||||
"text": text
|
||||
})
|
||||
|
||||
result = {
|
||||
"url": url,
|
||||
"num_pages": len(pdf_reader.pages),
|
||||
"content": text_content,
|
||||
"metadata": pdf_reader.metadata if hasattr(pdf_reader, 'metadata') else {}
|
||||
}
|
||||
|
||||
logger.info(f"Successfully parsed PDF with {len(pdf_reader.pages)} pages")
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error parsing PDF: {str(e)}")
|
||||
return {"error": str(e)}
|
||||
|
||||
@staticmethod
|
||||
def convert_currency(amount: float, from_currency: str, to_currency: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert currency using live exchange rates
|
||||
|
||||
Args:
|
||||
amount: Amount to convert
|
||||
from_currency: Source currency code (e.g., 'USD')
|
||||
to_currency: Target currency code (e.g., 'EUR')
|
||||
|
||||
Returns:
|
||||
Dictionary with conversion result
|
||||
"""
|
||||
try:
|
||||
if isinstance(amount, str):
|
||||
clean_amt = amount.replace(",", "").strip()
|
||||
symbols_to_strip = sorted(
|
||||
[
|
||||
"USD$", "U.S.$", "US$", "$",
|
||||
"SGD$", "SG$", "S$",
|
||||
"AUD$", "AU$", "A$",
|
||||
"CAD$", "CA$", "C$",
|
||||
"€", "£", "₹",
|
||||
],
|
||||
key=len,
|
||||
reverse=True,
|
||||
)
|
||||
for sym in symbols_to_strip:
|
||||
clean_amt = clean_amt.replace(sym, "")
|
||||
amount = float(clean_amt.strip())
|
||||
else:
|
||||
amount = float(amount)
|
||||
exchange_rates = {
|
||||
"USD": 1.0,
|
||||
"EUR": 0.92,
|
||||
"GBP": 0.79,
|
||||
"JPY": 149.50,
|
||||
"CNY": 7.24,
|
||||
"CAD": 1.36,
|
||||
"AUD": 1.53,
|
||||
"CHF": 0.88,
|
||||
"INR": 83.12,
|
||||
"SGD": 1.34
|
||||
}
|
||||
|
||||
def _normalize_code(code: str) -> str:
|
||||
if not isinstance(code, str):
|
||||
return str(code or "")
|
||||
c = code.strip().upper()
|
||||
symbols = {
|
||||
"$": "USD",
|
||||
"US$": "USD",
|
||||
"U.S.$": "USD",
|
||||
"USD$": "USD",
|
||||
"S$": "SGD",
|
||||
"SG$": "SGD",
|
||||
"SGD$": "SGD",
|
||||
"A$": "AUD",
|
||||
"AU$": "AUD",
|
||||
"AUD$": "AUD",
|
||||
"C$": "CAD",
|
||||
"CA$": "CAD",
|
||||
"CAD$": "CAD",
|
||||
"€": "EUR",
|
||||
"£": "GBP",
|
||||
"₹": "INR",
|
||||
}
|
||||
if c in symbols:
|
||||
return symbols[c]
|
||||
if c.endswith("$"):
|
||||
prefix = c[:-1].strip()
|
||||
if prefix in exchange_rates:
|
||||
return prefix
|
||||
if prefix in ("US", "U.S."):
|
||||
return "USD"
|
||||
if prefix in ("AU", "A"):
|
||||
return "AUD"
|
||||
if prefix in ("CA", "C"):
|
||||
return "CAD"
|
||||
return c
|
||||
|
||||
from_currency = _normalize_code(from_currency)
|
||||
to_currency = _normalize_code(to_currency)
|
||||
|
||||
logger.info(f"Converting {amount} {from_currency} to {to_currency}")
|
||||
|
||||
if from_currency not in exchange_rates or to_currency not in exchange_rates:
|
||||
return {"error": f"Unsupported currency: {from_currency} or {to_currency}"}
|
||||
|
||||
# Convert to USD first, then to target currency
|
||||
usd_amount = amount / exchange_rates[from_currency]
|
||||
converted_amount = usd_amount * exchange_rates[to_currency]
|
||||
|
||||
result = {
|
||||
"original_amount": amount,
|
||||
"from_currency": from_currency,
|
||||
"to_currency": to_currency,
|
||||
"converted_amount": round(converted_amount, 2),
|
||||
"exchange_rate": round(exchange_rates[to_currency] / exchange_rates[from_currency], 4),
|
||||
"timestamp": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
logger.info(f"Conversion result: {result['converted_amount']} {to_currency}")
|
||||
return result
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error converting currency: {str(e)}")
|
||||
return {"error": str(e)}
|
||||
|
||||
@staticmethod
|
||||
def calculate(expression: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Evaluate a mathematical expression
|
||||
|
||||
Args:
|
||||
expression: Mathematical expression to evaluate
|
||||
|
||||
Returns:
|
||||
Dictionary with calculation result
|
||||
"""
|
||||
try:
|
||||
logger.info(f"Calculating: {expression}")
|
||||
|
||||
# Sanitize expression - only allow safe mathematical operations
|
||||
allowed_names = {
|
||||
k: v for k, v in math.__dict__.items() if not k.startswith("__")
|
||||
}
|
||||
allowed_names.update({"abs": abs, "round": round, "min": min, "max": max})
|
||||
|
||||
# Replace common operations for clarity
|
||||
expression = expression.replace("^", "**")
|
||||
|
||||
# Evaluate the expression
|
||||
result = eval(expression, {"__builtins__": {}}, allowed_names)
|
||||
|
||||
return {
|
||||
"expression": expression,
|
||||
"result": result,
|
||||
"type": type(result).__name__
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error calculating expression: {str(e)}")
|
||||
return {"error": str(e)}
|
||||
|
||||
@staticmethod
|
||||
def code_interpreter(code: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute Python code for complex calculations and data processing
|
||||
|
||||
Args:
|
||||
code: Python code to execute
|
||||
|
||||
Returns:
|
||||
Dictionary with execution results and any output
|
||||
"""
|
||||
try:
|
||||
logger.info(f"Executing Python code: {code[:100]}...")
|
||||
|
||||
# Create a restricted namespace with safe built-ins
|
||||
safe_namespace = {
|
||||
'__builtins__': {
|
||||
'abs': abs,
|
||||
'all': all,
|
||||
'any': any,
|
||||
'sum': sum,
|
||||
'min': min,
|
||||
'max': max,
|
||||
'round': round,
|
||||
'len': len,
|
||||
'list': list,
|
||||
'dict': dict,
|
||||
'set': set,
|
||||
'tuple': tuple,
|
||||
'enumerate': enumerate,
|
||||
'zip': zip,
|
||||
'map': map,
|
||||
'filter': filter,
|
||||
'sorted': sorted,
|
||||
'reversed': reversed,
|
||||
'range': range,
|
||||
'int': int,
|
||||
'float': float,
|
||||
'str': str,
|
||||
'bool': bool,
|
||||
'print': print,
|
||||
}
|
||||
}
|
||||
|
||||
# Add math module
|
||||
safe_namespace['math'] = math
|
||||
|
||||
# Capture printed output
|
||||
import io
|
||||
import contextlib
|
||||
|
||||
output_buffer = io.StringIO()
|
||||
|
||||
with contextlib.redirect_stdout(output_buffer):
|
||||
# Execute the code
|
||||
exec(code, safe_namespace)
|
||||
|
||||
# Get printed output
|
||||
printed_output = output_buffer.getvalue()
|
||||
|
||||
# Try to extract a result if it's assigned to 'result' variable
|
||||
result = safe_namespace.get('result', None)
|
||||
|
||||
# Also check for common variable names
|
||||
if result is None:
|
||||
for var_name in ['total', 'sum', 'output', 'answer', 'final']:
|
||||
if var_name in safe_namespace:
|
||||
result = safe_namespace[var_name]
|
||||
break
|
||||
|
||||
# Get all variables defined (excluding built-ins and modules)
|
||||
variables = {
|
||||
k: v for k, v in safe_namespace.items()
|
||||
if not k.startswith('__') and k not in ['math'] and not callable(v)
|
||||
}
|
||||
|
||||
return {
|
||||
"code": code,
|
||||
"result": result,
|
||||
"output": printed_output if printed_output else None,
|
||||
"variables": variables,
|
||||
"success": True
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Error executing code: {str(e)}")
|
||||
return {
|
||||
"code": code,
|
||||
"error": str(e),
|
||||
"success": False
|
||||
}
|
||||
|
||||
|
||||
class ContextAwareAgent:
|
||||
"""
|
||||
AI Agent with configurable LLM providers and context modes for ablation studies
|
||||
"""
|
||||
|
||||
def __init__(self, api_key: str, context_mode: ContextMode = ContextMode.FULL,
|
||||
provider: str = "siliconflow", model: Optional[str] = None,
|
||||
verbose: bool = True):
|
||||
"""
|
||||
Initialize the agent
|
||||
|
||||
Args:
|
||||
api_key: API key for the LLM provider
|
||||
context_mode: Context mode for ablation studies
|
||||
provider: Any provider registered in ``agentbook.providers`` (for
|
||||
example ``dashscope``/``qwen``, ``siliconflow``, ``doubao``,
|
||||
``kimi``, ``deepseek``, or ``openrouter``)
|
||||
model: Optional model override
|
||||
verbose: If True, log full HTTP requests and responses (default: True)
|
||||
"""
|
||||
self.provider = provider.lower()
|
||||
self.verbose = verbose
|
||||
|
||||
# Base URLs, default models and key lookup all live in the shared
|
||||
# registry (agentbook/providers.py), so adding a provider there makes it
|
||||
# usable here with no change. resolve_backend also applies the universal
|
||||
# OpenRouter fallback: when the provider's own key is missing but
|
||||
# OPENROUTER_API_KEY is set, the request routes through OpenRouter with a
|
||||
# mapped model id. Behaviour is unchanged when the provider key is set.
|
||||
from config import resolve_backend
|
||||
backend = resolve_backend(self.provider, model=model, api_key=api_key)
|
||||
resolved_key = backend.api_key
|
||||
resolved_base_url = backend.base_url
|
||||
self.model = backend.model
|
||||
self.using_openrouter = backend.using_openrouter
|
||||
if self.using_openrouter:
|
||||
logger.info(
|
||||
f"{self.provider} API key not set; routing via OpenRouter "
|
||||
f"(model: {self.model})"
|
||||
)
|
||||
self.client = OpenAI(
|
||||
api_key=resolved_key,
|
||||
base_url=resolved_base_url
|
||||
)
|
||||
self.base_url = resolved_base_url
|
||||
|
||||
self.context_mode = context_mode
|
||||
self.trajectory = AgentTrajectory(context_mode=context_mode)
|
||||
self.tools = ToolRegistry()
|
||||
|
||||
# Initialize conversation history
|
||||
self.conversation_history = []
|
||||
self._init_system_prompt()
|
||||
|
||||
logger.info(f"Agent initialized with provider: {self.provider}, model: {self.model}, context mode: {context_mode.value}, verbose: {self.verbose}")
|
||||
|
||||
def _init_system_prompt(self):
|
||||
"""Initialize the system prompt for the conversation"""
|
||||
self.conversation_history = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": """You are an intelligent assistant with access to tools.
|
||||
|
||||
Your task is to solve the given problems using the available tools. Think step by step and use tools as needed.
|
||||
|
||||
Important: When you have gathered all necessary information and computed the final answer, clearly state "FINAL ANSWER:" followed by your answer."""
|
||||
}
|
||||
]
|
||||
|
||||
def _get_tools_description(self) -> List[Dict[str, Any]]:
|
||||
"""Get tool descriptions for the model"""
|
||||
return [
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "parse_pdf",
|
||||
"description": "Download and parse a PDF document from a URL or a file path to extract text content",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"url": {
|
||||
"type": "string",
|
||||
"description": "The URL or file path of the PDF document to parse"
|
||||
}
|
||||
},
|
||||
"required": ["url"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "convert_currency",
|
||||
"description": "Convert an amount from one currency to another using current exchange rates",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"amount": {
|
||||
"type": "number",
|
||||
"description": "The amount to convert"
|
||||
},
|
||||
"from_currency": {
|
||||
"type": "string",
|
||||
"description": "The source currency code (e.g., USD, EUR)"
|
||||
},
|
||||
"to_currency": {
|
||||
"type": "string",
|
||||
"description": "The target currency code (e.g., USD, EUR)"
|
||||
}
|
||||
},
|
||||
"required": ["amount", "from_currency", "to_currency"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "calculate",
|
||||
"description": "Evaluate a simple mathematical expression",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"expression": {
|
||||
"type": "string",
|
||||
"description": "The mathematical expression to evaluate (e.g., '2 + 2 * 3')"
|
||||
}
|
||||
},
|
||||
"required": ["expression"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "code_interpreter",
|
||||
"description": "Execute Python code for complex calculations, data processing, and computing totals. Use this for tasks like: summing lists of values, calculating percentages, aggregating financial data, performing multi-step calculations, or any computation requiring variables and intermediate steps.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {
|
||||
"type": "string",
|
||||
"description": "Python code to execute. Can use variables, loops, and mathematical operations. Example: 'amounts = [2500000, 2278481, 2541806, 2282609, 2388060]; total = sum(amounts); print(f\"Total: ${total:,.2f}\")"
|
||||
}
|
||||
},
|
||||
"required": ["code"]
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
def _prepare_assistant_message(self, message) -> Dict[str, Any]:
|
||||
"""
|
||||
Prepare assistant message for adding to messages list,
|
||||
filtering out reasoning_content if in NO_REASONING mode
|
||||
|
||||
Args:
|
||||
message: The assistant message object
|
||||
|
||||
Returns:
|
||||
Dictionary representation of the message
|
||||
"""
|
||||
msg_dict = message.dict() if hasattr(message, 'dict') else message.model_dump()
|
||||
|
||||
# Remove reasoning_content if in NO_REASONING mode
|
||||
if self.context_mode == ContextMode.NO_REASONING and 'reasoning_content' in msg_dict:
|
||||
msg_dict.pop('reasoning_content')
|
||||
|
||||
return msg_dict
|
||||
|
||||
@staticmethod
|
||||
def _reasoning_content(message) -> Optional[str]:
|
||||
"""Return provider reasoning text without assuming one SDK shape."""
|
||||
value = getattr(message, "reasoning_content", None)
|
||||
if value:
|
||||
return str(value)
|
||||
extra = getattr(message, "model_extra", None) or {}
|
||||
value = extra.get("reasoning_content") or extra.get("reasoning")
|
||||
if isinstance(value, dict):
|
||||
value = value.get("content") or value.get("text")
|
||||
return str(value) if value else None
|
||||
|
||||
@staticmethod
|
||||
def _json_snapshot(value: Any) -> Any:
|
||||
"""Detach an API evidence object from later in-memory mutations."""
|
||||
return json.loads(json.dumps(value, ensure_ascii=False, default=str))
|
||||
|
||||
def _build_context(self) -> str:
|
||||
"""
|
||||
Build a human-readable summary of the trajectory (legacy helper, kept
|
||||
for inspection/debugging only).
|
||||
|
||||
NOTE: The message list sent to the model is assembled by
|
||||
``_prepare_messages_for_api`` -- that is where the NO_HISTORY ablation
|
||||
actually takes effect. This method is not part of the request path.
|
||||
|
||||
Returns:
|
||||
Context string for the model
|
||||
"""
|
||||
context_parts = []
|
||||
|
||||
# Add reasoning steps if not disabled
|
||||
if self.context_mode != ContextMode.NO_REASONING and self.trajectory.reasoning_steps:
|
||||
context_parts.append("## Previous Reasoning Steps:")
|
||||
for step in self.trajectory.reasoning_steps:
|
||||
context_parts.append(f"- {step}")
|
||||
context_parts.append("")
|
||||
|
||||
# Add tool call history if not disabled
|
||||
if self.context_mode not in [ContextMode.NO_HISTORY, ContextMode.NO_TOOL_CALLS] and self.trajectory.tool_calls:
|
||||
context_parts.append("## Tool Call History:")
|
||||
for call in self.trajectory.tool_calls:
|
||||
if self.context_mode != ContextMode.NO_TOOL_CALLS:
|
||||
context_parts.append(f"- Called {call.tool_name} with args: {json.dumps(call.arguments)}")
|
||||
if self.context_mode != ContextMode.NO_TOOL_RESULTS and call.result:
|
||||
context_parts.append(f" Result: {json.dumps(call.result, indent=2)}")
|
||||
context_parts.append("")
|
||||
|
||||
return "\n".join(context_parts) if context_parts else ""
|
||||
|
||||
def _log_request_response(self, request_data: Dict[str, Any], response_data: Any, iteration: int):
|
||||
"""
|
||||
Log full request and response when in verbose mode
|
||||
|
||||
Args:
|
||||
request_data: The request payload sent to the API
|
||||
response_data: The response received from the API
|
||||
iteration: Current iteration number
|
||||
"""
|
||||
if not self.verbose:
|
||||
return
|
||||
|
||||
if request_data:
|
||||
print("\n" + "="*80)
|
||||
print(f"📤 ITERATION {iteration} - FULL REQUEST JSON:")
|
||||
print("-"*80)
|
||||
print(json.dumps(request_data, indent=2, ensure_ascii=False))
|
||||
|
||||
if response_data:
|
||||
print("\n" + "="*80)
|
||||
print(f"📥 ITERATION {iteration} - FULL RESPONSE:")
|
||||
print("-"*80)
|
||||
|
||||
# Convert response to dict for display
|
||||
if hasattr(response_data, 'model_dump'):
|
||||
response_dict = response_data.model_dump()
|
||||
elif hasattr(response_data, 'dict'):
|
||||
response_dict = response_data.dict()
|
||||
else:
|
||||
response_dict = {"raw_response": str(response_data)}
|
||||
|
||||
print(json.dumps(response_dict, indent=2, ensure_ascii=False))
|
||||
print("="*80 + "\n")
|
||||
|
||||
def _execute_tool(self, tool_name: str, arguments: Dict[str, Any]) -> Any:
|
||||
"""
|
||||
Execute a tool and return the result
|
||||
|
||||
Args:
|
||||
tool_name: Name of the tool to execute
|
||||
arguments: Arguments for the tool
|
||||
|
||||
Returns:
|
||||
Tool execution result
|
||||
"""
|
||||
tool_map = {
|
||||
"parse_pdf": self.tools.parse_pdf,
|
||||
"convert_currency": self.tools.convert_currency,
|
||||
"calculate": self.tools.calculate,
|
||||
"code_interpreter": self.tools.code_interpreter
|
||||
}
|
||||
|
||||
if tool_name not in tool_map:
|
||||
return {"error": f"Unknown tool: {tool_name}"}
|
||||
|
||||
return tool_map[tool_name](**arguments)
|
||||
|
||||
def _prepare_messages_for_api(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Build the message list actually sent to the model for the current
|
||||
iteration, applying the NO_HISTORY ablation.
|
||||
|
||||
For every mode except NO_HISTORY the full conversation history (the
|
||||
accumulated trajectory) is returned unchanged. For NO_HISTORY the
|
||||
request contains only the static system prompt and the current user
|
||||
task. No assistant decision, tool call, or tool result from a previous
|
||||
round is retained. This is the literal Experiment 1-1 ablation: the
|
||||
model restarts the task on every inference and therefore tends to issue
|
||||
the same first action repeatedly. A one-step sliding window would still
|
||||
be history and would materially narrow the experiment described in the
|
||||
manuscript.
|
||||
|
||||
Returns:
|
||||
The message list to send to the model for this iteration.
|
||||
"""
|
||||
messages = self.conversation_history
|
||||
if self.context_mode != ContextMode.NO_HISTORY:
|
||||
return messages
|
||||
|
||||
# System prompt(s) are always kept as the static prefix.
|
||||
windowed = [m for m in messages if m.get("role") == "system"]
|
||||
|
||||
# Anchor on the latest user task. Nothing after it is retained: those
|
||||
# messages are precisely the previous-round history being ablated.
|
||||
user_indices = [i for i, m in enumerate(messages) if m.get("role") == "user"]
|
||||
if not user_indices:
|
||||
return windowed
|
||||
last_user_idx = user_indices[-1]
|
||||
windowed.append(messages[last_user_idx])
|
||||
return windowed
|
||||
|
||||
@staticmethod
|
||||
def _extract_final_answer(content: str) -> Optional[str]:
|
||||
"""Extract text after FINAL ANSWER: if present; otherwise None."""
|
||||
if not content or "FINAL ANSWER:" not in content:
|
||||
return None
|
||||
return content.split("FINAL ANSWER:", 1)[1].strip()
|
||||
|
||||
def execute_task(self, task: str, max_iterations: Optional[int] = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Execute a task using available tools (ReAct loop).
|
||||
|
||||
Stops when:
|
||||
1. The model emits a text-only reply (no tool_calls) — conversational
|
||||
or task complete, including plain replies like "hi" that omit the
|
||||
FINAL ANSWER: marker; or
|
||||
2. max_iterations is hit (safety cap for tool-call loops, e.g. the
|
||||
no_tool_results ablation).
|
||||
|
||||
Args:
|
||||
task: The task to execute
|
||||
max_iterations: Maximum ReAct steps (default: Config.MAX_ITERATIONS
|
||||
or 10). This is a safety ceiling, not a target round count.
|
||||
|
||||
Returns:
|
||||
Task execution result
|
||||
|
||||
Result semantics:
|
||||
- ``completed`` means the loop received a non-empty terminal text
|
||||
response. It does not claim that the requested task was correct.
|
||||
- ``task_success`` is ``None`` here because correctness is
|
||||
task-specific and cannot be inferred from arbitrary natural
|
||||
language prompts. Callers with a known rubric should compute it
|
||||
from the final answer and trajectory.
|
||||
- ``success`` is retained as a backwards-compatible alias for
|
||||
``completed``. New consumers should use ``completed`` or their
|
||||
task-specific ``task_success`` value instead.
|
||||
"""
|
||||
if max_iterations is None:
|
||||
try:
|
||||
from config import Config
|
||||
max_iterations = Config.MAX_ITERATIONS
|
||||
except Exception:
|
||||
max_iterations = 10
|
||||
|
||||
# Add user message to conversation history
|
||||
self.conversation_history.append({"role": "user", "content": task})
|
||||
|
||||
# Use conversation history directly (no copy needed)
|
||||
messages = self.conversation_history
|
||||
|
||||
iteration = 0
|
||||
final_answer = None
|
||||
|
||||
while iteration < max_iterations:
|
||||
iteration += 1
|
||||
logger.info(f"Iteration {iteration}/{max_iterations}")
|
||||
|
||||
try:
|
||||
# Build the message list actually sent to the model. For every
|
||||
# mode except NO_HISTORY this equals the full trajectory; for
|
||||
# NO_HISTORY it is a sliding window that drops earlier steps.
|
||||
api_messages = self._prepare_messages_for_api()
|
||||
|
||||
# Prepare request data for logging
|
||||
request_data = {
|
||||
"model": self.model,
|
||||
"messages": api_messages,
|
||||
"temperature": _reasoning_safe_temperature(self.model, 0.3),
|
||||
"max_tokens": 8192
|
||||
}
|
||||
|
||||
if self.context_mode != ContextMode.NO_TOOL_CALLS:
|
||||
request_data["tools"] = self._get_tools_description()
|
||||
request_data["tool_choice"] = "auto"
|
||||
|
||||
# DeepSeek V4: enable thinking so reasoning_content is present
|
||||
# for the no_reasoning ablation (parity with thinking defaults of
|
||||
# Doubao/Kimi). Skip when routed via OpenRouter, which may not
|
||||
# accept the same extra body shape.
|
||||
create_kwargs = {
|
||||
"model": self.model,
|
||||
"messages": api_messages,
|
||||
"tools": self._get_tools_description() if self.context_mode != ContextMode.NO_TOOL_CALLS else None,
|
||||
"tool_choice": "auto" if self.context_mode != ContextMode.NO_TOOL_CALLS else None,
|
||||
"temperature": _reasoning_safe_temperature(self.model, 0.3),
|
||||
"max_tokens": 8192,
|
||||
"timeout": 180, # 180 second timeout for main execution
|
||||
}
|
||||
if self.provider == "deepseek" and not getattr(self, "using_openrouter", False):
|
||||
create_kwargs["extra_body"] = {"thinking": {"type": "enabled"}}
|
||||
request_data["thinking"] = {"type": "enabled"}
|
||||
|
||||
logger.info(f"Sending request to {self.provider} API")
|
||||
|
||||
# Call the model with tools
|
||||
response = self.client.chat.completions.create(**create_kwargs)
|
||||
|
||||
response_dict = (
|
||||
response.model_dump() if hasattr(response, "model_dump")
|
||||
else response.dict() if hasattr(response, "dict")
|
||||
else {"raw_response": str(response)}
|
||||
)
|
||||
self.trajectory.api_turns.append({
|
||||
"iteration": iteration,
|
||||
"provider": self.provider,
|
||||
"resolved_model": self.model,
|
||||
"base_url": self.base_url,
|
||||
"using_openrouter": bool(getattr(self, "using_openrouter", False)),
|
||||
"request": self._json_snapshot(request_data),
|
||||
"response": self._json_snapshot(response_dict),
|
||||
})
|
||||
|
||||
# Log response if verbose
|
||||
if self.verbose:
|
||||
self._log_request_response(request_data, response, iteration)
|
||||
|
||||
message = response.choices[0].message
|
||||
has_tool_calls = bool(getattr(message, "tool_calls", None))
|
||||
reasoning_content = self._reasoning_content(message)
|
||||
if reasoning_content:
|
||||
self.trajectory.reasoning_steps.append(reasoning_content)
|
||||
|
||||
# --- Terminal path: text reply with no tool calls ---
|
||||
# A normal chat turn ("hi" -> "Hello!") or a task answer without
|
||||
# the FINAL ANSWER: marker must end the ReAct loop. Previously
|
||||
# only "FINAL ANSWER:" broke the loop, so plain replies were
|
||||
# re-sent for up to max_iterations (wasted API calls).
|
||||
if not has_tool_calls:
|
||||
assistant_msg = self._prepare_assistant_message(message)
|
||||
messages.append(assistant_msg)
|
||||
content = (message.content or "").strip()
|
||||
if content:
|
||||
marked = self._extract_final_answer(content)
|
||||
final_answer = marked if marked is not None else content
|
||||
logger.info(
|
||||
"Terminal text response (no tool calls); "
|
||||
f"stopping after iteration {iteration}"
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"Empty model response with no tool calls; "
|
||||
"stopping to avoid burning remaining iterations"
|
||||
)
|
||||
break
|
||||
|
||||
# --- Continue path: model requested tool execution ---
|
||||
assistant_msg = self._prepare_assistant_message(message)
|
||||
messages.append(assistant_msg)
|
||||
for tool_call in message.tool_calls:
|
||||
function_name = tool_call.function.name
|
||||
raw_args = tool_call.function.arguments or "{}"
|
||||
try:
|
||||
function_args = json.loads(raw_args)
|
||||
except json.JSONDecodeError as exc:
|
||||
# Keep the turn alive on bad tool-arg JSON.
|
||||
err = (
|
||||
f"Invalid tool arguments (not valid JSON): {exc}. "
|
||||
f"Raw arguments: {raw_args[:500]}"
|
||||
)
|
||||
logger.warning(err)
|
||||
self.trajectory.tool_calls.append(ToolCall(
|
||||
tool_name=function_name,
|
||||
arguments={},
|
||||
result={"error": err},
|
||||
))
|
||||
messages.append({
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
"content": json.dumps({"error": err}),
|
||||
})
|
||||
continue
|
||||
|
||||
logger.info(f"Executing tool: {function_name} with args: {function_args}")
|
||||
|
||||
result = self._execute_tool(function_name, function_args)
|
||||
|
||||
tool_call_record = ToolCall(
|
||||
tool_name=function_name,
|
||||
arguments=function_args,
|
||||
result=result
|
||||
)
|
||||
self.trajectory.tool_calls.append(tool_call_record)
|
||||
|
||||
if self.context_mode != ContextMode.NO_TOOL_RESULTS:
|
||||
tool_msg = {
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
# default=str: code_interpreter returns the raw
|
||||
# namespace in `variables`, which can hold sets,
|
||||
# dict views etc. that json can't encode — that
|
||||
# must not abort the whole task.
|
||||
"content": json.dumps(result, default=str)
|
||||
}
|
||||
else:
|
||||
tool_msg = {
|
||||
"role": "tool",
|
||||
"tool_call_id": tool_call.id,
|
||||
"content": "[Tool result hidden due to context mode]"
|
||||
}
|
||||
messages.append(tool_msg)
|
||||
|
||||
# If the same turn also tagged FINAL ANSWER: (unusual with tools),
|
||||
# still prefer extracting it after tools are recorded.
|
||||
if message.content and "FINAL ANSWER:" in message.content:
|
||||
final_answer = self._extract_final_answer(message.content)
|
||||
logger.info(f"Final answer found alongside tool calls: {final_answer}")
|
||||
break
|
||||
|
||||
# Note: We do NOT modify the system prompt anymore.
|
||||
# The context is already built into the conversation through tool history
|
||||
|
||||
except TimeoutError:
|
||||
logger.error("Request timed out after 60 seconds")
|
||||
return {
|
||||
"error": "Request timed out. The model is taking too long to respond. Try a simpler task or different provider.",
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": False,
|
||||
"task_success": False,
|
||||
"success": False,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.error(f"Error during task execution: {str(e)}")
|
||||
self.trajectory.api_turns.append({
|
||||
"iteration": iteration,
|
||||
"provider": self.provider,
|
||||
"resolved_model": self.model,
|
||||
"base_url": self.base_url,
|
||||
"using_openrouter": bool(getattr(self, "using_openrouter", False)),
|
||||
"error": {"class": type(e).__name__, "message": str(e)},
|
||||
})
|
||||
# Check if it's a timeout-related error
|
||||
if "timeout" in str(e).lower() or "timed out" in str(e).lower():
|
||||
return {
|
||||
"error": "Request timed out. The model is taking too long to respond. Try a simpler task or different provider.",
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": False,
|
||||
"task_success": False,
|
||||
"success": False,
|
||||
}
|
||||
return {
|
||||
"error": str(e),
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": False,
|
||||
"task_success": False,
|
||||
"success": False,
|
||||
}
|
||||
completed = bool(final_answer and str(final_answer).strip())
|
||||
return {
|
||||
"final_answer": final_answer,
|
||||
"trajectory": self.trajectory,
|
||||
"iterations": iteration,
|
||||
"completed": completed,
|
||||
"task_success": None,
|
||||
# Backwards-compatible alias. This is terminal-response status,
|
||||
# not a correctness judgment.
|
||||
"success": completed,
|
||||
"provider": self.provider,
|
||||
"model": self.model,
|
||||
"base_url": self.base_url,
|
||||
"using_openrouter": bool(getattr(self, "using_openrouter", False)),
|
||||
}
|
||||
|
||||
def reset(self):
|
||||
"""Reset the agent's trajectory and conversation history"""
|
||||
self.trajectory = AgentTrajectory(context_mode=self.context_mode)
|
||||
self._init_system_prompt() # Reinitialize conversation with system prompt
|
||||
logger.info("Agent trajectory and conversation history reset")
|
||||
|
||||
def process(self, query: str, max_iterations: Optional[int] = None) -> str:
|
||||
"""
|
||||
Process a query and return the final answer as a string
|
||||
|
||||
Args:
|
||||
query: The query to process
|
||||
max_iterations: Maximum ReAct steps (default from Config)
|
||||
|
||||
Returns:
|
||||
The final answer as a string
|
||||
"""
|
||||
result = self.execute_task(query, max_iterations)
|
||||
if result.get('final_answer'):
|
||||
return result['final_answer']
|
||||
elif result.get('error'):
|
||||
return f"Error: {result['error']}"
|
||||
else:
|
||||
return "No answer found"
|
||||
@@ -0,0 +1,218 @@
|
||||
"""
|
||||
Configuration module for Context-Aware Agent
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Optional
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def _reasoning_safe_temperature(model, requested=1.0):
|
||||
"""Reasoning models (Kimi K3, GPT-5, ...) only accept temperature=1.
|
||||
Return 1 for those; otherwise the requested value so non-reasoning
|
||||
providers (Doubao, DeepSeek, older Moonshot) are unchanged."""
|
||||
m = str(model or "").lower().replace("/", "-")
|
||||
return 1 if ("kimi-k3" in m or "gpt-5" in m) else requested
|
||||
|
||||
|
||||
# Provider resolution lives in the shared agentbook package so every chapter
|
||||
# stays consistent; see agentbook/providers.py. The fallback keeps this
|
||||
# experiment runnable from a checkout where agentbook is not installed.
|
||||
try:
|
||||
from agentbook.providers import (
|
||||
PROVIDERS,
|
||||
SUPPORTED_PROVIDERS,
|
||||
canonical_provider,
|
||||
canonical_provider as _canonical_provider,
|
||||
map_model_to_openrouter,
|
||||
resolve_backend,
|
||||
resolve_llm_backend,
|
||||
)
|
||||
except ImportError: # pragma: no cover - exercised only without the package
|
||||
import sys as _sys
|
||||
|
||||
_sys.path.insert(
|
||||
0, str(__import__("pathlib").Path(__file__).resolve().parents[2])
|
||||
)
|
||||
from agentbook.providers import (
|
||||
PROVIDERS,
|
||||
SUPPORTED_PROVIDERS,
|
||||
canonical_provider,
|
||||
canonical_provider as _canonical_provider,
|
||||
map_model_to_openrouter,
|
||||
resolve_backend,
|
||||
resolve_llm_backend,
|
||||
)
|
||||
|
||||
|
||||
class Config:
|
||||
"""Configuration settings for the agent"""
|
||||
|
||||
# Provider Configuration
|
||||
LLM_PROVIDER: str = os.getenv("LLM_PROVIDER", "doubao").lower()
|
||||
|
||||
# API Configuration
|
||||
DASHSCOPE_API_KEY: str = os.getenv("DASHSCOPE_API_KEY", "")
|
||||
DASHSCOPE_BASE_URL: str = os.getenv(
|
||||
"DASHSCOPE_BASE_URL", "https://dashscope.aliyuncs.com/compatible-mode/v1"
|
||||
)
|
||||
|
||||
SILICONFLOW_API_KEY: str = os.getenv("SILICONFLOW_API_KEY", "")
|
||||
SILICONFLOW_BASE_URL: str = "https://api.siliconflow.cn/v1"
|
||||
|
||||
ARK_API_KEY: str = os.getenv("ARK_API_KEY", "")
|
||||
ARK_BASE_URL: str = "https://ark.cn-beijing.volces.com/api/v3"
|
||||
|
||||
MOONSHOT_API_KEY: str = os.getenv("MOONSHOT_API_KEY", "")
|
||||
MOONSHOT_BASE_URL: str = "https://api.moonshot.cn/v1"
|
||||
|
||||
DEEPSEEK_API_KEY: str = os.getenv("DEEPSEEK_API_KEY", "")
|
||||
DEEPSEEK_BASE_URL: str = os.getenv(
|
||||
"DEEPSEEK_BASE_URL", "https://api.deepseek.com"
|
||||
)
|
||||
|
||||
ZHIPU_API_KEY: str = os.getenv("ZHIPU_API_KEY", "")
|
||||
ZHIPU_BASE_URL: str = "https://open.bigmodel.cn/api/paas/v4"
|
||||
|
||||
# Model Configuration (defaults based on provider)
|
||||
MODEL_NAME: str = os.getenv("MODEL_NAME", "") # Will be set based on provider if not specified
|
||||
MODEL_TEMPERATURE: float = float(os.getenv("MODEL_TEMPERATURE", "0.3"))
|
||||
MODEL_MAX_TOKENS: int = int(os.getenv("MODEL_MAX_TOKENS", "1000"))
|
||||
|
||||
# Agent Configuration
|
||||
MAX_ITERATIONS: int = int(os.getenv("MAX_ITERATIONS", "10"))
|
||||
ENABLE_REASONING: bool = os.getenv("ENABLE_REASONING", "true").lower() == "true"
|
||||
|
||||
# Test Configuration
|
||||
TEST_PDF_URL: str = os.getenv(
|
||||
"TEST_PDF_URL",
|
||||
"https://www.berkshirehathaway.com/qtrly/1stqtr23.pdf"
|
||||
)
|
||||
|
||||
# Currency Configuration (Example rates - in production use real API)
|
||||
EXCHANGE_RATES = {
|
||||
"USD": 1.0,
|
||||
"EUR": 0.92,
|
||||
"GBP": 0.79,
|
||||
"JPY": 149.50,
|
||||
"CNY": 7.24,
|
||||
"CAD": 1.36,
|
||||
"AUD": 1.53,
|
||||
"CHF": 0.88,
|
||||
"INR": 83.12,
|
||||
"SGD": 1.34
|
||||
}
|
||||
|
||||
# Logging Configuration
|
||||
LOG_LEVEL: str = os.getenv("LOG_LEVEL", "INFO")
|
||||
LOG_FILE: Optional[str] = os.getenv("LOG_FILE")
|
||||
LOG_FORMAT: str = "%(asctime)s - %(levelname)s - %(name)s - %(message)s"
|
||||
|
||||
# File paths
|
||||
RESULTS_DIR: str = "results"
|
||||
TEST_PDFS_DIR: str = "fixtures/pdfs"
|
||||
|
||||
@classmethod
|
||||
def get_api_key(cls, provider: str = None) -> str:
|
||||
"""
|
||||
Get API key for the specified provider
|
||||
|
||||
Args:
|
||||
provider: Provider name (defaults to LLM_PROVIDER)
|
||||
|
||||
Returns:
|
||||
API key for the provider
|
||||
"""
|
||||
provider = provider or cls.LLM_PROVIDER
|
||||
# The shared registry knows every provider's key variables, so this
|
||||
# stays correct as providers are added there.
|
||||
try:
|
||||
return PROVIDERS[_canonical_provider(provider)].api_key()
|
||||
except KeyError:
|
||||
return ""
|
||||
|
||||
@classmethod
|
||||
def get_default_model(cls, provider: str = None) -> str:
|
||||
"""
|
||||
Get default model for the specified provider
|
||||
|
||||
Args:
|
||||
provider: Provider name (defaults to LLM_PROVIDER)
|
||||
|
||||
Returns:
|
||||
Default model name for the provider
|
||||
"""
|
||||
provider = provider or cls.LLM_PROVIDER
|
||||
provider = provider.lower()
|
||||
|
||||
if cls.MODEL_NAME:
|
||||
return cls.MODEL_NAME
|
||||
|
||||
try:
|
||||
return PROVIDERS[_canonical_provider(provider)].default_model
|
||||
except KeyError:
|
||||
return ""
|
||||
|
||||
@classmethod
|
||||
def validate(cls, provider: str = None) -> bool:
|
||||
"""
|
||||
Validate required configuration
|
||||
|
||||
Args:
|
||||
provider: Provider to validate (defaults to LLM_PROVIDER)
|
||||
|
||||
Returns:
|
||||
True if configuration is valid
|
||||
"""
|
||||
provider = provider or cls.LLM_PROVIDER
|
||||
# resolve_backend already accounts for providers that need no key
|
||||
# (ollama) and for the OpenRouter fallback, and its error names the
|
||||
# exact variables to set -- so a missing key is not the only signal.
|
||||
try:
|
||||
resolve_backend(provider)
|
||||
except ValueError as exc:
|
||||
print(f"ERROR: {exc}")
|
||||
print("Please set it in .env file or as environment variable")
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
@classmethod
|
||||
def create_directories(cls):
|
||||
"""Create necessary directories if they don't exist"""
|
||||
os.makedirs(cls.RESULTS_DIR, exist_ok=True)
|
||||
os.makedirs(cls.TEST_PDFS_DIR, exist_ok=True)
|
||||
|
||||
@classmethod
|
||||
def get_model_config(cls) -> dict:
|
||||
"""
|
||||
Get model configuration as dictionary
|
||||
|
||||
Returns:
|
||||
Model configuration dict
|
||||
"""
|
||||
return {
|
||||
"model": cls.MODEL_NAME,
|
||||
"temperature": _reasoning_safe_temperature(cls.MODEL_NAME, cls.MODEL_TEMPERATURE),
|
||||
"max_tokens": cls.MODEL_MAX_TOKENS
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def print_config(cls):
|
||||
"""Print current configuration (hiding sensitive data)"""
|
||||
provider = canonical_provider(cls.LLM_PROVIDER)
|
||||
api_key = cls.get_api_key(provider)
|
||||
print("\n" + "="*50)
|
||||
print("CONFIGURATION")
|
||||
print("="*50)
|
||||
print(f"Provider: {provider}")
|
||||
print(f"Model: {cls.MODEL_NAME}")
|
||||
print(f"Temperature: {cls.MODEL_TEMPERATURE}")
|
||||
print(f"Max Tokens: {cls.MODEL_MAX_TOKENS}")
|
||||
print(f"Max Iterations: {cls.MAX_ITERATIONS}")
|
||||
print(f"API Key Set: {'Yes' if api_key else 'No'}")
|
||||
print(f"Log Level: {cls.LOG_LEVEL}")
|
||||
print("="*50 + "\n")
|
||||
@@ -0,0 +1,274 @@
|
||||
"""
|
||||
Create Sample PDF for Testing
|
||||
Generates a financial report PDF with various currency amounts and calculations
|
||||
"""
|
||||
|
||||
from reportlab.lib import colors
|
||||
from reportlab.lib.pagesizes import letter, A4
|
||||
from reportlab.platypus import SimpleDocTemplate, Table, TableStyle, Paragraph, Spacer, PageBreak
|
||||
from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle
|
||||
from reportlab.lib.units import inch
|
||||
from reportlab.lib.enums import TA_CENTER, TA_RIGHT
|
||||
import os
|
||||
|
||||
|
||||
def create_financial_report():
|
||||
"""Create a sample financial report PDF for testing"""
|
||||
|
||||
# Create PDF
|
||||
filename = "sample_financial_report_q1_2024.pdf"
|
||||
doc = SimpleDocTemplate(filename, pagesize=letter)
|
||||
|
||||
# Container for the 'Flowable' objects
|
||||
elements = []
|
||||
|
||||
# Define styles
|
||||
styles = getSampleStyleSheet()
|
||||
title_style = ParagraphStyle(
|
||||
'CustomTitle',
|
||||
parent=styles['Heading1'],
|
||||
fontSize=24,
|
||||
textColor=colors.HexColor('#1f4788'),
|
||||
spaceAfter=30,
|
||||
alignment=TA_CENTER
|
||||
)
|
||||
|
||||
heading_style = ParagraphStyle(
|
||||
'CustomHeading',
|
||||
parent=styles['Heading2'],
|
||||
fontSize=16,
|
||||
textColor=colors.HexColor('#1f4788'),
|
||||
spaceAfter=12,
|
||||
)
|
||||
|
||||
# Title
|
||||
elements.append(Paragraph("Global Corporation Financial Report", title_style))
|
||||
elements.append(Paragraph("Q1 2024 - Quarterly Results", styles['Heading2']))
|
||||
elements.append(Spacer(1, 0.5*inch))
|
||||
|
||||
# Executive Summary
|
||||
elements.append(Paragraph("Executive Summary", heading_style))
|
||||
summary_text = """This report presents the financial performance of Global Corporation
|
||||
for the first quarter of 2024. The company operates in multiple regions with
|
||||
transactions in various currencies. Total consolidated revenue for Q1 2024
|
||||
reached $45.8 million USD, representing a 12% increase year-over-year."""
|
||||
elements.append(Paragraph(summary_text, styles['BodyText']))
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Regional Revenue Table
|
||||
elements.append(Paragraph("Regional Revenue Breakdown", heading_style))
|
||||
|
||||
revenue_data = [
|
||||
['Region', 'Local Currency', 'Q1 2024 Revenue', 'Q4 2023 Revenue', 'Growth %'],
|
||||
['North America', 'USD', '$15,250,000', '$14,100,000', '8.16%'],
|
||||
['Europe', 'EUR', '€11,340,000', '€10,800,000', '5.00%'],
|
||||
['United Kingdom', 'GBP', '£8,920,000', '£8,500,000', '4.94%'],
|
||||
['Asia Pacific', 'JPY', '¥1,245,000,000', '¥1,180,000,000', '5.51%'],
|
||||
['Singapore', 'SGD', 'S$4,180,000', 'S$3,950,000', '5.82%'],
|
||||
]
|
||||
|
||||
revenue_table = Table(revenue_data, colWidths=[2*inch, 1.2*inch, 1.5*inch, 1.5*inch, 0.8*inch])
|
||||
revenue_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -1), colors.beige),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(revenue_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Operating Expenses
|
||||
elements.append(Paragraph("Operating Expenses by Department", heading_style))
|
||||
|
||||
expense_data = [
|
||||
['Department', 'Q1 2024 (USD)', 'Q4 2023 (USD)', 'Change'],
|
||||
['Research & Development', '$8,450,000', '$7,900,000', '+$550,000'],
|
||||
['Sales & Marketing', '$6,230,000', '$6,100,000', '+$130,000'],
|
||||
['General & Administrative', '$4,180,000', '$4,050,000', '+$130,000'],
|
||||
['Operations', '$9,870,000', '$9,500,000', '+$370,000'],
|
||||
['Total Operating Expenses', '$28,730,000', '$27,550,000', '+$1,180,000'],
|
||||
]
|
||||
|
||||
expense_table = Table(expense_data, colWidths=[2.5*inch, 1.5*inch, 1.5*inch, 1.2*inch])
|
||||
expense_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -2), colors.lightgrey),
|
||||
('BACKGROUND', (0, -1), (-1, -1), colors.yellow),
|
||||
('FONTNAME', (0, -1), (-1, -1), 'Helvetica-Bold'),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(expense_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Key Financial Metrics
|
||||
elements.append(Paragraph("Key Financial Metrics", heading_style))
|
||||
|
||||
metrics_text = """
|
||||
• Gross Profit Margin: 37.2%<br/>
|
||||
• Operating Profit Margin: 18.4%<br/>
|
||||
• Net Profit Margin: 14.8%<br/>
|
||||
• EBITDA: $10,250,000 USD<br/>
|
||||
• Cash Flow from Operations: $8,930,000 USD<br/>
|
||||
• Total Assets: $125,400,000 USD<br/>
|
||||
• Total Liabilities: $48,200,000 USD<br/>
|
||||
• Shareholders' Equity: $77,200,000 USD<br/>
|
||||
"""
|
||||
elements.append(Paragraph(metrics_text, styles['BodyText']))
|
||||
|
||||
# Add page break
|
||||
elements.append(PageBreak())
|
||||
|
||||
# Currency Exchange Rates Used
|
||||
elements.append(Paragraph("Currency Exchange Rates (as of March 31, 2024)", heading_style))
|
||||
|
||||
exchange_data = [
|
||||
['Currency Pair', 'Exchange Rate', 'Previous Quarter', 'Change'],
|
||||
['USD/EUR', '0.9234', '0.9156', '+0.85%'],
|
||||
['USD/GBP', '0.7891', '0.7823', '+0.87%'],
|
||||
['USD/JPY', '149.85', '147.23', '+1.78%'],
|
||||
['USD/SGD', '1.3452', '1.3389', '+0.47%'],
|
||||
]
|
||||
|
||||
exchange_table = Table(exchange_data, colWidths=[2*inch, 1.5*inch, 1.5*inch, 1.2*inch])
|
||||
exchange_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -1), colors.beige),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(exchange_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Investment Portfolio
|
||||
elements.append(Paragraph("Investment Portfolio Performance", heading_style))
|
||||
|
||||
portfolio_text = """The company's investment portfolio showed strong performance in Q1 2024:
|
||||
|
||||
• Fixed Income Securities: $23,450,000 USD (yielding 4.2% annually)
|
||||
• Equity Investments: $18,750,000 USD (up 8.3% this quarter)
|
||||
• Real Estate Holdings: $31,200,000 USD (appreciation of 3.1%)
|
||||
• Cash and Cash Equivalents: $15,890,000 USD
|
||||
|
||||
Total portfolio value: $89,290,000 USD, representing a 5.7% increase from Q4 2023."""
|
||||
|
||||
elements.append(Paragraph(portfolio_text, styles['BodyText']))
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Future Projections
|
||||
elements.append(Paragraph("Q2 2024 Projections", heading_style))
|
||||
|
||||
projection_data = [
|
||||
['Metric', 'Q1 2024 Actual', 'Q2 2024 Projected', 'Growth'],
|
||||
['Total Revenue', '$45,800,000', '$48,500,000', '+5.9%'],
|
||||
['Operating Expenses', '$28,730,000', '$29,800,000', '+3.7%'],
|
||||
['Net Income', '$6,780,000', '$7,450,000', '+9.9%'],
|
||||
['EPS (Earnings Per Share)', '$2.34', '$2.57', '+9.8%'],
|
||||
]
|
||||
|
||||
projection_table = Table(projection_data, colWidths=[2.5*inch, 1.5*inch, 1.5*inch, 1*inch])
|
||||
projection_table.setStyle(TableStyle([
|
||||
('BACKGROUND', (0, 0), (-1, 0), colors.grey),
|
||||
('TEXTCOLOR', (0, 0), (-1, 0), colors.whitesmoke),
|
||||
('ALIGN', (0, 0), (-1, -1), 'CENTER'),
|
||||
('FONTNAME', (0, 0), (-1, 0), 'Helvetica-Bold'),
|
||||
('FONTSIZE', (0, 0), (-1, 0), 12),
|
||||
('BOTTOMPADDING', (0, 0), (-1, 0), 12),
|
||||
('BACKGROUND', (0, 1), (-1, -1), colors.lightblue),
|
||||
('GRID', (0, 0), (-1, -1), 1, colors.black),
|
||||
]))
|
||||
|
||||
elements.append(projection_table)
|
||||
elements.append(Spacer(1, 0.3*inch))
|
||||
|
||||
# Footer
|
||||
footer_text = """
|
||||
<para alignment="center">
|
||||
<b>Note:</b> All financial figures are preliminary and subject to audit.<br/>
|
||||
For more information, please contact: investor.relations@globalcorp.com<br/>
|
||||
Global Corporation © 2024 - Confidential Financial Report
|
||||
</para>
|
||||
"""
|
||||
elements.append(Spacer(1, 0.5*inch))
|
||||
elements.append(Paragraph(footer_text, styles['Normal']))
|
||||
|
||||
# Build PDF
|
||||
doc.build(elements)
|
||||
|
||||
print(f"Sample PDF created: {filename}")
|
||||
return filename
|
||||
|
||||
|
||||
def create_simple_expense_report():
|
||||
"""Create a simpler expense report for quick testing"""
|
||||
|
||||
filename = "simple_expense_report.pdf"
|
||||
doc = SimpleDocTemplate(filename, pagesize=A4)
|
||||
|
||||
elements = []
|
||||
styles = getSampleStyleSheet()
|
||||
|
||||
# Title
|
||||
elements.append(Paragraph("Quarterly Expense Report", styles['Title']))
|
||||
elements.append(Spacer(1, 0.2*inch))
|
||||
|
||||
# Simple expense data
|
||||
elements.append(Paragraph("Q1 2024 Regional Expenses", styles['Heading2']))
|
||||
|
||||
expense_text = """
|
||||
Our company has the following expenses for Q1 2024:
|
||||
|
||||
<b>United States Office:</b> $2,500,000 USD<br/>
|
||||
<b>United Kingdom Office:</b> £1,800,000 GBP<br/>
|
||||
<b>Japan Office:</b> ¥380,000,000 JPY<br/>
|
||||
<b>European Union Office:</b> €2,100,000 EUR<br/>
|
||||
<b>Singapore Office:</b> S$3,200,000 SGD<br/>
|
||||
|
||||
These expenses include salaries, operations, marketing, and R&D costs.
|
||||
|
||||
Additional financial metrics:
|
||||
• Total headcount: 1,250 employees globally
|
||||
• Average expense per employee: varies by region
|
||||
• Projected Q2 expense reduction target: 8% across all regions
|
||||
"""
|
||||
|
||||
elements.append(Paragraph(expense_text, styles['BodyText']))
|
||||
|
||||
# Build PDF
|
||||
doc.build(elements)
|
||||
|
||||
print(f"Simple PDF created: {filename}")
|
||||
return filename
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Create both PDFs
|
||||
create_financial_report()
|
||||
create_simple_expense_report()
|
||||
|
||||
# Create a fixture directory for PDFs if needed
|
||||
os.makedirs("fixtures/pdfs", exist_ok=True)
|
||||
|
||||
# Move PDFs to test directory
|
||||
import shutil
|
||||
for pdf in ["sample_financial_report_q1_2024.pdf", "simple_expense_report.pdf"]:
|
||||
if os.path.exists(pdf):
|
||||
shutil.move(pdf, f"fixtures/pdfs/{pdf}")
|
||||
|
||||
print("\nPDFs created in fixtures/pdfs/ directory")
|
||||
print("You can host these PDFs online or use a local server for testing")
|
||||
@@ -0,0 +1,39 @@
|
||||
# LLM Provider Configuration (dashscope/qwen, siliconflow, doubao, kimi,
|
||||
# moonshot, deepseek, zhipu, openrouter, or ollama)
|
||||
LLM_PROVIDER=doubao
|
||||
|
||||
# API Keys (set the appropriate one for your provider)
|
||||
SILICONFLOW_API_KEY=your_siliconflow_api_key_here
|
||||
# Alibaba Cloud Model Studio (Bailian) / Qwen
|
||||
DASHSCOPE_API_KEY=your_dashscope_api_key_here
|
||||
# Optional: use this instead for an international-region key
|
||||
# DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
ARK_API_KEY=your_ark_api_key_here
|
||||
MOONSHOT_API_KEY=your_moonshot_api_key_here
|
||||
DEEPSEEK_API_KEY=your_deepseek_api_key_here
|
||||
# Optional: override DeepSeek base URL (default: https://api.deepseek.com)
|
||||
# DEEPSEEK_BASE_URL=https://api.deepseek.com
|
||||
ZHIPU_API_KEY=your_zhipu_api_key_here
|
||||
|
||||
# Universal fallback: if the provider key above is missing/invalid but
|
||||
# OPENROUTER_API_KEY is set, requests are routed through OpenRouter and the
|
||||
# model id is mapped automatically (bare gpt-*/o1-* -> openai/*, claude-* ->
|
||||
# anthropic/*, deepseek-* -> deepseek/*, other native ids -> OPENROUTER_MODEL
|
||||
# or openai/gpt-5.6-luna).
|
||||
# You can also pass --provider openrouter to use OpenRouter directly.
|
||||
OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
# OPENROUTER_MODEL=openai/gpt-5.6-luna # model used when falling back for non-OpenRouter native ids
|
||||
|
||||
# Optional: Model Configuration
|
||||
# MODEL_NAME= # Leave empty to use provider default
|
||||
# For DeepSeek: deepseek-v4-flash (default) or deepseek-v4-pro
|
||||
# MODEL_TEMPERATURE=0.3
|
||||
# MODEL_MAX_TOKENS=1000
|
||||
|
||||
# Optional: Test Configuration
|
||||
# MAX_ITERATIONS=10
|
||||
# TEST_PDF_URL=https://example.com/sample.pdf
|
||||
|
||||
# Optional: Logging Configuration
|
||||
# LOG_LEVEL=INFO
|
||||
# LOG_FILE=agent.log
|
||||
@@ -0,0 +1,112 @@
|
||||
%PDF-1.4
|
||||
%“Œ‹ž ReportLab Generated PDF document http://www.reportlab.com
|
||||
1 0 obj
|
||||
<<
|
||||
/F1 2 0 R /F2 3 0 R
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
3 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding /Name /F2 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Contents 10 0 R /MediaBox [ 0 0 612 792 ] /Parent 9 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/Contents 11 0 R /MediaBox [ 0 0 612 792 ] /Parent 9 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Contents 12 0 R /MediaBox [ 0 0 612 792 ] /Parent 9 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/PageMode /UseNone /Pages 9 0 R /Type /Catalog
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Author (\(anonymous\)) /CreationDate (D:20250909105803+08'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20250909105803+08'00') /Producer (ReportLab PDF Library - www.reportlab.com)
|
||||
/Subject (\(unspecified\)) /Title (\(anonymous\)) /Trapped /False
|
||||
>>
|
||||
endobj
|
||||
9 0 obj
|
||||
<<
|
||||
/Count 3 /Kids [ 4 0 R 5 0 R 6 0 R ] /Type /Pages
|
||||
>>
|
||||
endobj
|
||||
10 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 1740
|
||||
>>
|
||||
stream
|
||||
Gatm<>?BQ=&:WeDbX:Tf`FpQ#^-0o1E_\:">geBjLDSPS`,0=g[M,*Z^EZBl9Seu%Z>`(SkSn6Md7BlL"r)PipHL/.hG3$j%h';1krg^d"AD%t`UrBW&MqKVL+qQl7N5d+"^qIc8$VP$OMFp^:d5rY5cc()%#6?P?k-G?JN6fLE^kHCSBg'RE`%6X(\"T-QW9TbGu__:]M-L"8*c.&A,@1AhdQ?ca6MJ7Y&!r`)Oh:!2VQi-,7ugFecip"(!+KS"&:SX1UpbfFXuds#%qdRMmfn>m[]8ZH.]A+dD*n#T,\UV@_2=\ZW3Ku-/>Y\,-<f,8-83ibsOVt/hHCekjj3YS`^I3?$9XZb=7e7"79@1I@pD5XRE4$^eYTCHe>Z"fPQ$$VM1X'IgW538mj]H2Xlp3.[FeZJI4nA>M28J6@h_RQOAXi4Q(/f4Qc#Ng,^gT1i#/\U=O%L=/uP2<F&7V9K"kNeRBpE,-"'q_sq@;7($:CE[A?Mm+U#Gc/s$N'+p<CML;[*d=8iXn#7"7p3HH6.;%`[kKM;HOg's^I9*I]IAWTlEtKfJ@S]:GaOjj'&iX*R\fIL?]u:h<bc8^(1orrm0Dp:qJ?gG"dlqVgqGP.4KHtLUn-F(:pMZ'_&lOZ&qDJh/,Qd:n:lWM="E.JgY9Uup"c1P9=9je<@>GZPOZ-eU`!0%mPQK]/5?F7&8mNC*r@ZkA,bf;IA';`bEkdekperQ6S:%f0&.r?mf'0KC*jMN%#O(,!h;+>XK2Fr;Ln\d_5B3IDiOk7pSu!["M*_i4h?1rJ,TQpF/8E$'NsF10P`FfJ'LT7s(j2K?a9^mU$o)2LnU>#!>7pU(dbufZ2'*0D=r-u-`^S(DmPq&%H+l8ZD!4-tpijdsQ[hmn6I$*un3JbA&7pk,'R[b2Al]X!P1J[amF<&oK_Vg+h/VDo?G`6QULt(Qmr1*)Jg=fYUBkjZEGp!0o-R8l.a@P]4<%cI'C&g"a/csF(lF!"JV@OJ!V>fQ$J<0MLn6W3R(Ah>;r\USJ;l9X#fhPa@(3Y'.12`C=FCUsJ1hpIOGY);&N8G78fbcs06(j^^.o-BQk,$&JbY%Nhp##?F,:t[HP%[;@pM5Li3osiA"ugNFF_cPcTMqW9X_Z[6ddl?YVoH1D8QToo0[$*#AZ[Y1m>kt`e9pd[G.+[:PlOGVeqL`a/2?kD0lE:ZgQ4:[cdLtf_]n&NVA5l:Q57K@fKMR8Q%L1MdK+[2L&O([Xn0nM&35pmjb)qO*-'6'Cn>G_kJenF[I%fI]QKiTf_XBp5K<[TCuY5'0B`@qsip#\/=//M!(M_e%I1IDSLkb5p2aJrLHj(]@S:>MpeLu7d6I=Jj!-1GOr1hOT.o8I30!>?8D',"#$.s*+%TUYN;K^Nmm)H-5_@MkL^GC-"l0J*I3M5G[uBR0S;Q["O7Z.;-9TO\*3r%p(_sHlQ9#)YqpXb1^Q>#Ws=:7U>=E4*P=\1:7-*9FZ\q)mo#I]q%ahIq5hZ\]P5ns>`YELL4o([G`HPP>;UIL<Kf-#"!g4_0@>]G3C**4a;MEP:D,=@%Yq_d]5J6ACkFpBq"og6G5^Wbi%abieM$m(p\[Q%@ho\p4=<8b,'Qn@!BBZ*29m&TQg);g`h'"j.'<>Zg_En-*LBj^^X#-tRWh2$>BaXZ>Bbd%>Bcpp9\52>XCG2r5]#s/haoE."//FU/[O(jUm)0Zl5hBcbh-8PV4Zg\n9*.qg#DnXmM@s(oFL`C7hY~>endstream
|
||||
endobj
|
||||
11 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 317
|
||||
>>
|
||||
stream
|
||||
Gas304&<aJ'SbR0^Z$1t>SI?'F_fI.V(G#+\Ys+M0p%;j&l[NjnaCHa@0dPfab09!8!Zh7_U/:0C4Nk5fntBYO9I%*(hAAtN4'>S8M2]j,2dDV2#UQ"-BLr/Am[Vj[tF<_9,p/mc>dQXF1AWTclQt69AN>ti^nH@!r2o=diH)Hc)pQJ/Vf^NKFRHomXHsWL`p!JYN0.N`;q9)3[QR)"S\D)5[>d>@9&e@'\5+u%l(0oTnq#bq`8H(W\JQIpDp*p3+h6$S8%YpC[H9$'hlR1hI'82RocJ9cE01FEr2a47qam^ZO&j>(C&87[.S\m~>endstream
|
||||
endobj
|
||||
12 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 1717
|
||||
>>
|
||||
stream
|
||||
Gatm;;,c4/&:WeDliYJ.D,;N]Z6c*FV1H'r]aKocg^Gj=UC]5<*FoDt^V?%Nfa2%BG<I](Zi7?/Bt`C%_LUX$<I^2i^rI>*5@_JgbSIts&MX(a-F8?:"4eS>"ecD<N&JO4PG&2t,:*Bq6jE&am*,Pl@]aS@[RS"O=BPB!cKj\9CTY^A8!s3JQ>q(Xi%coZ!S=*VJ:%CT@[enl?t8,ZMsE<?@!2pa[uS`;IQC)Y'*V:UL6SK-LQ9UXMPN"\CCNU`Gd([!JMjZVHU)[DnGpq=.cc_RIRh?T%_V)[dB*#WX-;*PjA#VsdLgF28e-ZsUi]1q`i7cSiQ_WS"jngl48,uQnd[8s'!QQ"e=-7kh\L19NGWBU_?>`.VH$9j$F0c1(!`E_C!jX.$;Dd:?+Kc[p$)G9qdn*qnB",C6RgDF&3VDhF<??^f6<180\XNr7u<C!011l"ck6IsR@\nu%Phmoa@0f$aE.[/K41Vde1lr/QqG#Hp!&,Tl.>1WK1m44__Wc&aL"B*j>EeDoGthJp#hQ\O=eeZo2!9M^(*m>+o1W&pV9&Y=.['#Vq=Y?lPEpupU7'^bB*UE_=rV*o8_km$Ebj/D2iY`7QZ^HrPEI=(#Fh7hC:s0`Ck+Y_UaU2hMQ0u>Of#-1/WgJCE&4B4)ma![`0Gkft+RG?9IJAfZIbBTR6$p:[LGXs4&Ir-+c*)\MdRse(g7II$NhJ20P,:FdsbT\#CSY&Md-k\+t+h-*:\F2kskTW5QAO#7iMW/Ia!$H4sqP^D2iRF>b_A@[THJ3>A=7ZQ%P=j:De;X;X;NE6Y*_cI/>@r0!aD"'6i&g9aOJET3$FX-eZ)(^ipUPZ#;`!%6K1C0'p?]aM]<+O790]NRI)^T"nAm^?j*#[OTj'0'N;*:<RSDEs?B#g-^^pirRG\tsaLQR6%TUaU4>^ur?;l$VG$j`K*U]\-DNUiDgbR@CSb@7T1pBj<3Oik2j9"$)D9n]I&ZA#28BZb45d>t,F92$d-,bWFGoV^I3Nd"b7_%$F"KR,c,R\7dC$['9?E/qclE5R-O!^uk>WnI""&J$jMSGkLE3C@bYn\6KdIHk<>Y)u"Ie&uo4D&iTZPLa\99UA+.PJiN].,![?4:"57"Dl0mUdaOqqo6,_YGnJt%;I]1ioP14ZN^UKj5C[;%1G]m>43h^B:bMd3l9NT=<'0j?MS(e?-JS6n,T7<`nV7/r\pd(oV@0E_*OHa.+gbVh#s1'6,gh#B;io'nJA8>_`7;ISq7/4_@>Z5jc)*jgN?WV4H&L7kkOd!3.LAY<-jfG06-tL@5>Bi'`BMSi'iT`K%*+JKj;dT^,Mu7O3I?1KHa=_H&dg9(-IXJ]Kp1/$Y"cL,"BWe!R_>mC]1:d=O*%`(&Ds3'Yo7SpLQg+97H.INJ#-5N+ZraTB^42!e]hb(C7TsbB.=(%cSQ!_SgbA\Fo"MVES]@5'3.PGCSstt94XkpE:)7)4@3'E6At#Y]kjUP[bj74n"Iq0AjhC%%I&/e6E3d`[1qM3Y`eV+e-!C"3$\4&>/;E-:_ok3]=Q\T,Qk+`^lc&tf6GJ&<b&Jgrc$['arKF.4>#k:k+:im(H?s1pnP1N_E,/$A5mlmNL)A9<c/\I@`cbP.8LkuP$<N(.S0+,bpr`*YEI[Y#ek2YF8T>^WTJ/po*PSt#g.!Ijfc423qtG`ZUHp!Eu72UmAV^t)r\L8O#9r`Y8oO]%M&I=!%GTi~>endstream
|
||||
endobj
|
||||
xref
|
||||
0 13
|
||||
0000000000 65535 f
|
||||
0000000073 00000 n
|
||||
0000000114 00000 n
|
||||
0000000221 00000 n
|
||||
0000000333 00000 n
|
||||
0000000527 00000 n
|
||||
0000000721 00000 n
|
||||
0000000915 00000 n
|
||||
0000000983 00000 n
|
||||
0000001266 00000 n
|
||||
0000001337 00000 n
|
||||
0000003169 00000 n
|
||||
0000003577 00000 n
|
||||
trailer
|
||||
<<
|
||||
/ID
|
||||
[<2dbcccf9df210d48854ffa3169a0c96f><2dbcccf9df210d48854ffa3169a0c96f>]
|
||||
% ReportLab generated PDF document -- digest (http://www.reportlab.com)
|
||||
|
||||
/Info 8 0 R
|
||||
/Root 7 0 R
|
||||
/Size 13
|
||||
>>
|
||||
startxref
|
||||
5386
|
||||
%%EOF
|
||||
@@ -0,0 +1,74 @@
|
||||
%PDF-1.4
|
||||
%“Œ‹ž ReportLab Generated PDF document http://www.reportlab.com
|
||||
1 0 obj
|
||||
<<
|
||||
/F1 2 0 R /F2 3 0 R
|
||||
>>
|
||||
endobj
|
||||
2 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica /Encoding /WinAnsiEncoding /Name /F1 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
3 0 obj
|
||||
<<
|
||||
/BaseFont /Helvetica-Bold /Encoding /WinAnsiEncoding /Name /F2 /Subtype /Type1 /Type /Font
|
||||
>>
|
||||
endobj
|
||||
4 0 obj
|
||||
<<
|
||||
/Contents 8 0 R /MediaBox [ 0 0 595.2756 841.8898 ] /Parent 7 0 R /Resources <<
|
||||
/Font 1 0 R /ProcSet [ /PDF /Text /ImageB /ImageC /ImageI ]
|
||||
>> /Rotate 0 /Trans <<
|
||||
|
||||
>>
|
||||
/Type /Page
|
||||
>>
|
||||
endobj
|
||||
5 0 obj
|
||||
<<
|
||||
/PageMode /UseNone /Pages 7 0 R /Type /Catalog
|
||||
>>
|
||||
endobj
|
||||
6 0 obj
|
||||
<<
|
||||
/Author (\(anonymous\)) /CreationDate (D:20250909105803+08'00') /Creator (\(unspecified\)) /Keywords () /ModDate (D:20250909105803+08'00') /Producer (ReportLab PDF Library - www.reportlab.com)
|
||||
/Subject (\(unspecified\)) /Title (\(anonymous\)) /Trapped /False
|
||||
>>
|
||||
endobj
|
||||
7 0 obj
|
||||
<<
|
||||
/Count 1 /Kids [ 4 0 R ] /Type /Pages
|
||||
>>
|
||||
endobj
|
||||
8 0 obj
|
||||
<<
|
||||
/Filter [ /ASCII85Decode /FlateDecode ] /Length 657
|
||||
>>
|
||||
stream
|
||||
GasJOd;IYl'Sc(%MZ9E?Z70:l$"2CcX0P@RUd>A(PBu79-DX:Dk*U>)rD$%ZUSFA(+d@'GrL`Hj%Rq!'^I\`:`;'7qK+#Ujn>I9Lrf0S*q:4G@k>5V)nZJ*$8LpaTI)0D8>0@F9>$)D0JBJ-C6j)FBFpBR]<0'r\Z]27cgL?)_n$T<5NEa(SepTg]JG:?Q*VYc;N1OMo_7NRFntIrX4dZ.d)>O`b`\Np$#UUb$a`:=Z@]rl%`D2injlinXUD,_%@^nW(J%D]fgD_<GfiTMs>cVUtQ)`eD)G=!CjsdJhkVer;=gVpRGSP/b@DS\\SPCn`b#.KLEg\;$gc"O5".Y4PcWJ7*_d\)ZbKYM8ja<iae<I.gmR5RbTkmk5(Xl<aCIMHR$dnE7&+G\Hl&Sc)ai9dW-4ZH3[qL_WOLM=5iq=(%::jC\@JX!Jr2^]l(,EXN5+s**Bq!Ac?0#!`Cf66r(!>[M%RH7Y3Du[QV6W=nUTb<09!NFDhHt`*CT^t!`Hdp7DDteC]M_amX(":u6)gF"PBr0L4rbb)Nk/7pV2<l)D@@SD5(bK>L"U3=aU3*m$MoR+qtVINA-Z%peY.2G7k]etm30^M);hQe>4\DNk5&`M!3eN?,`O=EZC*$?q$nmZVQN4ifE#<-)X.)":S]IY~>endstream
|
||||
endobj
|
||||
xref
|
||||
0 9
|
||||
0000000000 65535 f
|
||||
0000000073 00000 n
|
||||
0000000114 00000 n
|
||||
0000000221 00000 n
|
||||
0000000333 00000 n
|
||||
0000000536 00000 n
|
||||
0000000604 00000 n
|
||||
0000000887 00000 n
|
||||
0000000946 00000 n
|
||||
trailer
|
||||
<<
|
||||
/ID
|
||||
[<6a67c647f1d47536a5416ec836e1ab9c><6a67c647f1d47536a5416ec836e1ab9c>]
|
||||
% ReportLab generated PDF document -- digest (http://www.reportlab.com)
|
||||
|
||||
/Info 6 0 R
|
||||
/Root 5 0 R
|
||||
/Size 9
|
||||
>>
|
||||
startxref
|
||||
1693
|
||||
%%EOF
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,22 @@
|
||||
# Core dependencies
|
||||
openai>=1.0.0
|
||||
requests>=2.31.0
|
||||
PyPDF2>=3.0.0
|
||||
|
||||
# Data processing and visualization
|
||||
numpy>=1.24.0
|
||||
matplotlib>=3.7.0
|
||||
tabulate>=0.9.0
|
||||
|
||||
# PDF generation for sample data
|
||||
reportlab>=4.0.0
|
||||
|
||||
# Optional: for better PDF parsing (uncomment if needed)
|
||||
# pdfplumber>=0.10.0
|
||||
# pymupdf>=1.23.0
|
||||
|
||||
# Development tools (optional)
|
||||
python-dotenv>=1.0.0
|
||||
pytest>=7.4.0
|
||||
black>=23.0.0
|
||||
flake8>=6.0.0
|
||||
@@ -0,0 +1,463 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the exact five-arm context ablation from book/chapter1.md.
|
||||
|
||||
Unlike the legacy demo table, this runner persists every credential-free API
|
||||
request and response. That makes it possible to prove which context component
|
||||
was removed on every inference instead of inferring the ablation from a CLI
|
||||
flag after the fact.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
|
||||
EXPERIMENT_ID = "1-1"
|
||||
MODES = list(ContextMode)
|
||||
CANONICAL_TASK = """According to the company's quarterly revenue:
|
||||
- Q1: 2.5 million USD
|
||||
- Q2: 2.1 million EUR
|
||||
- Q3: 1.8 million GBP
|
||||
- Q4: 380 million JPY
|
||||
|
||||
Use the available currency-conversion and calculation tools to convert every
|
||||
non-USD quarter to USD, then calculate the annual total and quarterly average.
|
||||
Report both values rounded to two decimal places. Do not estimate exchange
|
||||
rates yourself; use the tool observations."""
|
||||
|
||||
EXPECTED_NUMBERS = ("9602895.73", "2400723.93")
|
||||
KEY_ENV = {
|
||||
"dashscope": ("DASHSCOPE_API_KEY",),
|
||||
"qwen": ("DASHSCOPE_API_KEY",),
|
||||
"bailian": ("DASHSCOPE_API_KEY",),
|
||||
"kimi": ("MOONSHOT_API_KEY", "KIMI_API_KEY"),
|
||||
"moonshot": ("MOONSHOT_API_KEY", "KIMI_API_KEY"),
|
||||
"doubao": ("ARK_API_KEY",),
|
||||
"siliconflow": ("SILICONFLOW_API_KEY",),
|
||||
"deepseek": ("DEEPSEEK_API_KEY",),
|
||||
"openrouter": ("OPENROUTER_API_KEY",),
|
||||
}
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat()
|
||||
|
||||
|
||||
def git_value(*args: str) -> str | None:
|
||||
try:
|
||||
return subprocess.check_output(
|
||||
["git", *args], text=True, stderr=subprocess.DEVNULL
|
||||
).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def package_version(distribution: str) -> str | None:
|
||||
try:
|
||||
from importlib.metadata import version
|
||||
|
||||
return version(distribution)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def resolve_key(provider: str) -> tuple[str, str]:
|
||||
names = KEY_ENV.get(provider, ())
|
||||
for name in names:
|
||||
value = os.getenv(name)
|
||||
if value:
|
||||
return value, name
|
||||
raise RuntimeError(
|
||||
f"No direct credential for {provider}; expected one of {', '.join(names)}"
|
||||
)
|
||||
|
||||
|
||||
def tool_call_dict(call: Any) -> Dict[str, Any]:
|
||||
return {
|
||||
"tool_name": call.tool_name,
|
||||
"arguments": call.arguments,
|
||||
"result": call.result,
|
||||
"timestamp": call.timestamp,
|
||||
}
|
||||
|
||||
|
||||
def call_signatures(tool_calls: Iterable[Dict[str, Any]]) -> List[str]:
|
||||
signatures = []
|
||||
for call in tool_calls:
|
||||
signatures.append(
|
||||
f"{call['tool_name']}:"
|
||||
+ json.dumps(call.get("arguments", {}), sort_keys=True, ensure_ascii=False)
|
||||
)
|
||||
return signatures
|
||||
|
||||
|
||||
def response_message(turn: Dict[str, Any]) -> Dict[str, Any]:
|
||||
choices = turn.get("response", {}).get("choices") or []
|
||||
return (choices[0].get("message") or {}) if choices else {}
|
||||
|
||||
|
||||
def request_roles(turn: Dict[str, Any]) -> List[str]:
|
||||
return [message.get("role") for message in turn.get("request", {}).get("messages", [])]
|
||||
|
||||
|
||||
def evaluate_context_contract(mode: str, turns: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""Verify the actual provider request, not the requested CLI mode."""
|
||||
requests = [turn.get("request", {}) for turn in turns if turn.get("request")]
|
||||
real_responses = [turn for turn in turns if turn.get("response", {}).get("id")]
|
||||
details: Dict[str, Any] = {
|
||||
"has_provider_response_ids": len(real_responses) == len(turns) and bool(turns),
|
||||
"turn_count": len(turns),
|
||||
"request_roles": [request_roles(turn) for turn in turns],
|
||||
}
|
||||
|
||||
if mode == ContextMode.FULL.value:
|
||||
details.update(
|
||||
{
|
||||
"tools_present_every_turn": all(bool(r.get("tools")) for r in requests),
|
||||
"history_present_after_first_turn": len(requests) > 1
|
||||
and all(
|
||||
"assistant" in [m.get("role") for m in r.get("messages", [])]
|
||||
and "tool" in [m.get("role") for m in r.get("messages", [])]
|
||||
for r in requests[1:]
|
||||
),
|
||||
"reasoning_retained_after_first_turn": len(requests) > 1
|
||||
and any(
|
||||
bool(m.get("reasoning_content"))
|
||||
for m in requests[1].get("messages", [])
|
||||
if m.get("role") == "assistant"
|
||||
),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"tools_present_every_turn",
|
||||
"history_present_after_first_turn",
|
||||
"reasoning_retained_after_first_turn",
|
||||
)
|
||||
elif mode == ContextMode.NO_TOOL_CALLS.value:
|
||||
details.update(
|
||||
{
|
||||
"tools_absent_every_turn": all(
|
||||
"tools" not in r and "tool_choice" not in r for r in requests
|
||||
),
|
||||
}
|
||||
)
|
||||
required = ("has_provider_response_ids", "tools_absent_every_turn")
|
||||
elif mode == ContextMode.NO_TOOL_RESULTS.value:
|
||||
tool_messages = [
|
||||
m
|
||||
for r in requests[1:]
|
||||
for m in r.get("messages", [])
|
||||
if m.get("role") == "tool"
|
||||
]
|
||||
details.update(
|
||||
{
|
||||
"tool_calls_retained": any(
|
||||
m.get("role") == "assistant" and m.get("tool_calls")
|
||||
for r in requests[1:]
|
||||
for m in r.get("messages", [])
|
||||
),
|
||||
"tool_results_hidden": bool(tool_messages)
|
||||
and all(
|
||||
m.get("content") == "[Tool result hidden due to context mode]"
|
||||
for m in tool_messages
|
||||
),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"tool_calls_retained",
|
||||
"tool_results_hidden",
|
||||
)
|
||||
elif mode == ContextMode.NO_REASONING.value:
|
||||
assistant_history = [
|
||||
m
|
||||
for r in requests[1:]
|
||||
for m in r.get("messages", [])
|
||||
if m.get("role") == "assistant"
|
||||
]
|
||||
provider_reasoning = [
|
||||
response_message(turn).get("reasoning_content") for turn in turns
|
||||
]
|
||||
details.update(
|
||||
{
|
||||
"provider_generated_reasoning": any(provider_reasoning),
|
||||
"reasoning_removed_from_history": bool(assistant_history)
|
||||
and all(not m.get("reasoning_content") for m in assistant_history),
|
||||
"tool_and_result_history_retained": any(
|
||||
"tool" in request_roles(turn) for turn in turns[1:]
|
||||
),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"provider_generated_reasoning",
|
||||
"reasoning_removed_from_history",
|
||||
"tool_and_result_history_retained",
|
||||
)
|
||||
elif mode == ContextMode.NO_HISTORY.value:
|
||||
details.update(
|
||||
{
|
||||
"only_static_prefix_and_user_every_turn": bool(requests)
|
||||
and all(
|
||||
[m.get("role") for m in r.get("messages", [])]
|
||||
== ["system", "user"]
|
||||
for r in requests
|
||||
),
|
||||
"tools_still_present": all(bool(r.get("tools")) for r in requests),
|
||||
}
|
||||
)
|
||||
required = (
|
||||
"has_provider_response_ids",
|
||||
"only_static_prefix_and_user_every_turn",
|
||||
"tools_still_present",
|
||||
)
|
||||
else:
|
||||
raise ValueError(f"Unknown mode: {mode}")
|
||||
|
||||
details["required_checks"] = list(required)
|
||||
details["passed"] = all(details[name] is True for name in required)
|
||||
return details
|
||||
|
||||
|
||||
def normalized_number_text(value: str | None) -> str:
|
||||
return (value or "").replace(",", "").replace("$", "").replace(" ", "")
|
||||
|
||||
|
||||
def canonical_answer_correct(final_answer: str | None) -> bool:
|
||||
"""Evaluate the known numeric rubric for the canonical Experiment 1-1 task.
|
||||
|
||||
This is deliberately kept outside ``ContextAwareAgent``. A generic agent
|
||||
cannot infer correctness from an arbitrary natural-language task, while
|
||||
this experiment has an explicit answer rubric.
|
||||
"""
|
||||
normalized = normalized_number_text(final_answer)
|
||||
return bool(final_answer) and all(number in normalized for number in EXPECTED_NUMBERS)
|
||||
|
||||
|
||||
def summarize_arm(mode: ContextMode, result: Dict[str, Any], elapsed: float) -> Dict[str, Any]:
|
||||
trajectory = result["trajectory"]
|
||||
tool_calls = [tool_call_dict(call) for call in trajectory.tool_calls]
|
||||
signatures = call_signatures(tool_calls)
|
||||
repeats = len(signatures) - len(set(signatures))
|
||||
final_answer = result.get("final_answer")
|
||||
completed = bool(result.get("completed", result.get("success", False)))
|
||||
task_success = canonical_answer_correct(final_answer)
|
||||
arm = {
|
||||
"mode": mode.value,
|
||||
"provider": result.get("provider"),
|
||||
"model": result.get("model"),
|
||||
"base_url": result.get("base_url"),
|
||||
"using_openrouter": result.get("using_openrouter", False),
|
||||
"started_at": None,
|
||||
"elapsed_seconds": round(elapsed, 6),
|
||||
# ``success`` is retained for compatibility with existing evidence;
|
||||
# it means terminal response/completion, not task correctness.
|
||||
"success": completed,
|
||||
"completed": completed,
|
||||
"task_success": task_success,
|
||||
"iterations": result.get("iterations", 0),
|
||||
"error": result.get("error"),
|
||||
"final_answer": final_answer,
|
||||
"tool_calls": tool_calls,
|
||||
"tool_call_signatures": signatures,
|
||||
"repeated_tool_calls": repeats,
|
||||
"reasoning_steps": trajectory.reasoning_steps,
|
||||
"api_turns": trajectory.api_turns,
|
||||
}
|
||||
arm["context_contract"] = evaluate_context_contract(mode.value, trajectory.api_turns)
|
||||
arm["behavior"] = {
|
||||
"tool_action_count": len(tool_calls),
|
||||
"has_repeated_tool_action": repeats > 0,
|
||||
"hit_iteration_ceiling": result.get("iterations") >= 5 and not completed,
|
||||
"canonical_answer_correct": task_success,
|
||||
}
|
||||
return arm
|
||||
|
||||
|
||||
def token_usage(arms: List[Dict[str, Any]]) -> Dict[str, int]:
|
||||
prompt = completion = cached = reasoning = 0
|
||||
for arm in arms:
|
||||
for turn in arm["api_turns"]:
|
||||
usage = turn.get("response", {}).get("usage") or {}
|
||||
prompt += int(usage.get("prompt_tokens") or usage.get("input_tokens") or 0)
|
||||
completion += int(
|
||||
usage.get("completion_tokens") or usage.get("output_tokens") or 0
|
||||
)
|
||||
prompt_details = usage.get("prompt_tokens_details") or usage.get(
|
||||
"input_tokens_details"
|
||||
) or {}
|
||||
completion_details = usage.get("completion_tokens_details") or usage.get(
|
||||
"output_tokens_details"
|
||||
) or {}
|
||||
cached += int(prompt_details.get("cached_tokens") or 0)
|
||||
reasoning += int(completion_details.get("reasoning_tokens") or 0)
|
||||
return {
|
||||
"prompt_tokens": prompt,
|
||||
"completion_tokens": completion,
|
||||
"total_tokens": prompt + completion,
|
||||
"cached_prompt_tokens": cached,
|
||||
"reasoning_tokens": reasoning,
|
||||
}
|
||||
|
||||
|
||||
def analyze(arms: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
by_mode = {arm["mode"]: arm for arm in arms}
|
||||
exact_five_arms = set(by_mode) == {mode.value for mode in MODES}
|
||||
contracts_pass = exact_five_arms and all(
|
||||
arm["context_contract"]["passed"] for arm in arms
|
||||
)
|
||||
direct_real_api = all(
|
||||
not arm["using_openrouter"]
|
||||
and arm["api_turns"]
|
||||
and all(turn.get("response", {}).get("id") for turn in arm["api_turns"])
|
||||
for arm in arms
|
||||
)
|
||||
behavior = {
|
||||
"full_baseline_correct": by_mode.get("full", {}).get("behavior", {}).get(
|
||||
"canonical_answer_correct", by_mode.get("full", {}).get("task_success", False)
|
||||
),
|
||||
"without_tool_definitions_no_tool_action": by_mode.get(
|
||||
"no_tool_calls", {}
|
||||
).get("behavior", {}).get("tool_action_count")
|
||||
== 0,
|
||||
"without_tool_results_repeated_action": by_mode.get(
|
||||
"no_tool_results", {}
|
||||
).get("behavior", {}).get("has_repeated_tool_action", False),
|
||||
"without_history_repeated_action": by_mode.get("no_history", {}).get(
|
||||
"behavior", {}
|
||||
).get("has_repeated_tool_action", False),
|
||||
# Contradiction is an empirical outcome, not something the harness can
|
||||
# legitimately force. We report whether the no-reasoning answer lost
|
||||
# canonical correctness and keep this separate from execution validity.
|
||||
"without_reasoning_degraded": not by_mode.get("no_reasoning", {}).get(
|
||||
"behavior", {}
|
||||
).get("canonical_answer_correct", False),
|
||||
}
|
||||
behavior["all_manuscript_behavior_claims_observed"] = all(behavior.values())
|
||||
return {
|
||||
"exact_five_arms_present": exact_five_arms,
|
||||
"all_context_contracts_passed": contracts_pass,
|
||||
"direct_real_api_evidence": direct_real_api,
|
||||
"experiment_execution_accepted": bool(
|
||||
exact_five_arms
|
||||
and contracts_pass
|
||||
and direct_real_api
|
||||
and behavior["full_baseline_correct"]
|
||||
),
|
||||
"manuscript_behavior_claims": behavior,
|
||||
"usage": token_usage(arms),
|
||||
}
|
||||
|
||||
|
||||
def write_json(path: Path, payload: Dict[str, Any]) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(payload, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--provider", default="kimi", choices=sorted(KEY_ENV))
|
||||
parser.add_argument("--model", default="kimi-k3")
|
||||
parser.add_argument("--max-iterations", type=int, default=5)
|
||||
parser.add_argument("--output-dir", type=Path)
|
||||
args = parser.parse_args()
|
||||
if args.max_iterations < 2:
|
||||
parser.error("--max-iterations must be at least 2")
|
||||
|
||||
key, key_env = resolve_key(args.provider)
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
output_dir = args.output_dir or Path("validation") / f"real_{stamp}"
|
||||
command = [
|
||||
sys.executable,
|
||||
Path(__file__).name,
|
||||
"--provider",
|
||||
args.provider,
|
||||
"--model",
|
||||
args.model,
|
||||
"--max-iterations",
|
||||
str(args.max_iterations),
|
||||
"--output-dir",
|
||||
str(output_dir),
|
||||
]
|
||||
|
||||
arms = []
|
||||
for mode in MODES:
|
||||
started = utc_now()
|
||||
agent = ContextAwareAgent(
|
||||
key,
|
||||
context_mode=mode,
|
||||
provider=args.provider,
|
||||
model=args.model,
|
||||
verbose=False,
|
||||
)
|
||||
begin = time.monotonic()
|
||||
result = agent.execute_task(CANONICAL_TASK, max_iterations=args.max_iterations)
|
||||
arm = summarize_arm(mode, result, time.monotonic() - begin)
|
||||
arm["started_at"] = started
|
||||
# Recompute the configured ceiling rather than retaining the default in
|
||||
# the pure summarizer (which is also exercised by unit tests).
|
||||
arm["behavior"]["hit_iteration_ceiling"] = (
|
||||
result.get("iterations") >= args.max_iterations and not result.get("success")
|
||||
)
|
||||
arms.append(arm)
|
||||
|
||||
evidence: Dict[str, Any] = {
|
||||
"schema_version": "1.0",
|
||||
"experiment_id": EXPERIMENT_ID,
|
||||
"evidence_mode": "real_api",
|
||||
"created_at": utc_now(),
|
||||
"canonical_source": "book/chapter1.md#实验-1-1-上下文的关键作用",
|
||||
"task": CANONICAL_TASK,
|
||||
"expected_numbers": list(EXPECTED_NUMBERS),
|
||||
"command": command,
|
||||
"credential_source_env": key_env,
|
||||
"credential_value_recorded": False,
|
||||
"host": {
|
||||
"platform": platform.platform(),
|
||||
"python": sys.version,
|
||||
"machine": platform.machine(),
|
||||
},
|
||||
"dependencies": {
|
||||
"openai": package_version("openai"),
|
||||
"requests": package_version("requests"),
|
||||
},
|
||||
"repository": {
|
||||
"commit": git_value("rev-parse", "HEAD"),
|
||||
"branch": git_value("branch", "--show-current"),
|
||||
"worktree_dirty": bool(git_value("status", "--porcelain")),
|
||||
},
|
||||
"arms": arms,
|
||||
}
|
||||
evidence["analysis"] = analyze(arms)
|
||||
evidence_path = output_dir / "evidence.json"
|
||||
write_json(evidence_path, evidence)
|
||||
digest = hashlib.sha256(evidence_path.read_bytes()).hexdigest()
|
||||
(output_dir / "evidence.sha256").write_text(
|
||||
f"{digest} evidence.json\n", encoding="utf-8"
|
||||
)
|
||||
latest = Path("validation/latest.json")
|
||||
latest.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copyfile(evidence_path, latest)
|
||||
|
||||
print(json.dumps(evidence["analysis"], ensure_ascii=False, indent=2))
|
||||
print(f"Evidence: {evidence_path}")
|
||||
print(f"SHA-256: {digest}")
|
||||
return 0 if evidence["analysis"]["experiment_execution_accepted"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,125 @@
|
||||
from agent import AgentTrajectory, ContextMode
|
||||
from run_experiment_1_1 import (
|
||||
canonical_answer_correct,
|
||||
evaluate_context_contract,
|
||||
summarize_arm,
|
||||
)
|
||||
|
||||
|
||||
def turn(messages, *, tools=True, reasoning="reason"):
|
||||
request = {"messages": messages}
|
||||
if tools:
|
||||
request.update({"tools": [{"type": "function"}], "tool_choice": "auto"})
|
||||
return {
|
||||
"request": request,
|
||||
"response": {
|
||||
"id": "real-response-id",
|
||||
"choices": [{"message": {"reasoning_content": reasoning}}],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
SYSTEM = {"role": "system", "content": "system"}
|
||||
USER = {"role": "user", "content": "task"}
|
||||
ASSISTANT = {
|
||||
"role": "assistant",
|
||||
"reasoning_content": "reason",
|
||||
"tool_calls": [{"id": "call"}],
|
||||
}
|
||||
TOOL = {"role": "tool", "content": '{"result": 4}'}
|
||||
|
||||
|
||||
def test_full_contract_uses_raw_followup_context():
|
||||
result = evaluate_context_contract(
|
||||
"full", [turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, TOOL])]
|
||||
)
|
||||
assert result["passed"] is True
|
||||
|
||||
|
||||
def test_no_history_contract_rejects_sliding_window():
|
||||
exact = evaluate_context_contract(
|
||||
"no_history", [turn([SYSTEM, USER]), turn([SYSTEM, USER])]
|
||||
)
|
||||
sliding = evaluate_context_contract(
|
||||
"no_history", [turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, TOOL])]
|
||||
)
|
||||
assert exact["passed"] is True
|
||||
assert sliding["passed"] is False
|
||||
|
||||
|
||||
def test_no_reasoning_requires_provider_reasoning_but_stripped_history():
|
||||
stripped_assistant = {k: v for k, v in ASSISTANT.items() if k != "reasoning_content"}
|
||||
result = evaluate_context_contract(
|
||||
"no_reasoning",
|
||||
[turn([SYSTEM, USER]), turn([SYSTEM, USER, stripped_assistant, TOOL])],
|
||||
)
|
||||
assert result["passed"] is True
|
||||
|
||||
|
||||
def test_no_tool_results_requires_literal_hidden_observations():
|
||||
hidden = {"role": "tool", "content": "[Tool result hidden due to context mode]"}
|
||||
result = evaluate_context_contract(
|
||||
"no_tool_results",
|
||||
[turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, hidden])],
|
||||
)
|
||||
assert result["passed"] is True
|
||||
leaked = evaluate_context_contract(
|
||||
"no_tool_results", [turn([SYSTEM, USER]), turn([SYSTEM, USER, ASSISTANT, TOOL])]
|
||||
)
|
||||
assert leaked["passed"] is False
|
||||
|
||||
|
||||
def test_no_tool_definitions_requires_absent_request_fields():
|
||||
result = evaluate_context_contract("no_tool_calls", [turn([SYSTEM, USER], tools=False)])
|
||||
assert result["passed"] is True
|
||||
|
||||
|
||||
def _arm_result(final_answer, *, mode=ContextMode.NO_TOOL_CALLS, iterations=1):
|
||||
completed = final_answer is not None
|
||||
return {
|
||||
"trajectory": AgentTrajectory(context_mode=mode),
|
||||
"final_answer": final_answer,
|
||||
"completed": completed,
|
||||
"success": completed,
|
||||
"iterations": iterations,
|
||||
"provider": "test",
|
||||
"model": "test-model",
|
||||
}
|
||||
|
||||
|
||||
def test_canonical_answer_rubric_rejects_refusal_and_hallucinated_markup():
|
||||
refusal = "I cannot compute the exchange rates without tools."
|
||||
hallucinated = "<request_tool>currency_converter(...)</request_tool>"
|
||||
assert canonical_answer_correct(refusal) is False
|
||||
assert canonical_answer_correct(hallucinated) is False
|
||||
|
||||
|
||||
def test_summarize_arm_separates_completion_from_task_success():
|
||||
result = summarize_arm(
|
||||
ContextMode.NO_TOOL_CALLS,
|
||||
_arm_result("I cannot compute the exchange rates without tools."),
|
||||
elapsed=0.1,
|
||||
)
|
||||
|
||||
# The model did return a terminal response, but it did not complete the
|
||||
# canonical financial task. A mode-independent evaluator must preserve
|
||||
# that distinction instead of forcing the mode to fail.
|
||||
assert result["completed"] is True
|
||||
assert result["success"] is True # compatibility alias
|
||||
assert result["task_success"] is False
|
||||
assert result["behavior"]["canonical_answer_correct"] is False
|
||||
|
||||
|
||||
def test_summarize_arm_accepts_correct_answer_even_in_an_ablated_arm():
|
||||
answer = "Annual total: $9,602,895.73; quarterly average: $2,400,723.93"
|
||||
result = summarize_arm(
|
||||
ContextMode.NO_TOOL_RESULTS,
|
||||
_arm_result(answer, mode=ContextMode.NO_TOOL_RESULTS),
|
||||
elapsed=0.1,
|
||||
)
|
||||
|
||||
# Correctness is an observed task result. The experiment may separately
|
||||
# report that tool feedback was hidden; it must not manufacture failure.
|
||||
assert result["completed"] is True
|
||||
assert result["task_success"] is True
|
||||
assert result["behavior"]["canonical_answer_correct"] is True
|
||||
@@ -0,0 +1,11 @@
|
||||
from main import _completed
|
||||
|
||||
|
||||
def test_completed_field_is_authoritative_over_legacy_success_alias():
|
||||
assert _completed({"completed": False, "success": True}) is False
|
||||
assert _completed({"completed": True, "success": False}) is True
|
||||
|
||||
|
||||
def test_completed_falls_back_for_old_result_artifacts():
|
||||
assert _completed({"success": True}) is True
|
||||
assert _completed({"success": False}) is False
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Pytest bootstrap for the context experiment tests."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Helpers for running manual smoke scripts from tests/manual."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
def add_project_root() -> Path:
|
||||
project_root = Path(__file__).resolve().parents[2]
|
||||
if str(project_root) not in sys.path:
|
||||
sys.path.insert(0, str(project_root))
|
||||
return project_root
|
||||
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify conversation history persistence
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
import json
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def test_conversation_history():
|
||||
"""Test that conversation history persists between tasks"""
|
||||
print("🧪 Testing Conversation History Persistence")
|
||||
print("=" * 50)
|
||||
|
||||
# Get API key (use any available provider)
|
||||
api_key = (
|
||||
os.getenv("ARK_API_KEY")
|
||||
or os.getenv("DASHSCOPE_API_KEY")
|
||||
or os.getenv("MOONSHOT_API_KEY")
|
||||
or os.getenv("SILICONFLOW_API_KEY")
|
||||
)
|
||||
provider = (
|
||||
"doubao"
|
||||
if os.getenv("ARK_API_KEY")
|
||||
else (
|
||||
"dashscope"
|
||||
if os.getenv("DASHSCOPE_API_KEY")
|
||||
else ("kimi" if os.getenv("MOONSHOT_API_KEY") else "siliconflow")
|
||||
)
|
||||
)
|
||||
|
||||
if not api_key:
|
||||
print("❌ No API key found. Please set one of:")
|
||||
print(" - ARK_API_KEY")
|
||||
print(" - DASHSCOPE_API_KEY")
|
||||
print(" - MOONSHOT_API_KEY")
|
||||
print(" - SILICONFLOW_API_KEY")
|
||||
return False
|
||||
|
||||
print(f"Using provider: {provider}")
|
||||
print("-" * 50)
|
||||
|
||||
try:
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider=provider,
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test 1: First query
|
||||
print("\n📝 Test 1: First query")
|
||||
query1 = "Remember that my favorite number is 42. What is 10 + 5?"
|
||||
result1 = agent.execute_task(query1)
|
||||
print(f"Query: {query1}")
|
||||
print(f"Response: {result1.get('final_answer', 'No answer')}")
|
||||
|
||||
# Check conversation history
|
||||
print(f"\n📚 Conversation history after first query:")
|
||||
print(f" Total messages: {len(agent.conversation_history)}")
|
||||
|
||||
# Print message roles
|
||||
for i, msg in enumerate(agent.conversation_history):
|
||||
role = msg.get('role', 'unknown')
|
||||
content_preview = str(msg.get('content', ''))[:50] + "..." if len(str(msg.get('content', ''))) > 50 else str(msg.get('content', ''))
|
||||
print(f" Message {i}: Role={role}, Content={content_preview}")
|
||||
|
||||
# Test 2: Second query that references first
|
||||
print("\n📝 Test 2: Second query (should remember context)")
|
||||
query2 = "What was my favorite number that I mentioned earlier?"
|
||||
result2 = agent.execute_task(query2)
|
||||
print(f"Query: {query2}")
|
||||
print(f"Response: {result2.get('final_answer', 'No answer')}")
|
||||
|
||||
# Check if 42 is mentioned in the response
|
||||
if "42" in str(result2.get('final_answer', '')):
|
||||
print("✅ SUCCESS: Agent remembered the favorite number from conversation history!")
|
||||
else:
|
||||
print("⚠️ WARNING: Agent might not have remembered the number. Check response above.")
|
||||
|
||||
# Check conversation history growth
|
||||
print(f"\n📚 Conversation history after second query:")
|
||||
print(f" Total messages: {len(agent.conversation_history)}")
|
||||
|
||||
# Test 3: Verify system prompt unchanged
|
||||
print("\n📝 Test 3: Verify system prompt remains unchanged")
|
||||
system_prompt = agent.conversation_history[0].get('content', '')
|
||||
if "favorite number" not in system_prompt and "42" not in system_prompt:
|
||||
print("✅ SUCCESS: System prompt remains unchanged!")
|
||||
else:
|
||||
print("❌ FAILURE: System prompt was modified!")
|
||||
|
||||
# Test 4: Reset and verify history cleared
|
||||
print("\n📝 Test 4: Test reset functionality")
|
||||
agent.reset()
|
||||
print(f" Messages after reset: {len(agent.conversation_history)}")
|
||||
|
||||
if len(agent.conversation_history) == 1 and agent.conversation_history[0]['role'] == 'system':
|
||||
print("✅ SUCCESS: Reset properly cleared history and kept system prompt!")
|
||||
else:
|
||||
print("❌ FAILURE: Reset did not work correctly!")
|
||||
|
||||
# Test 5: New conversation after reset
|
||||
print("\n📝 Test 5: New conversation after reset")
|
||||
query3 = "What was my favorite number?"
|
||||
result3 = agent.execute_task(query3)
|
||||
print(f"Query: {query3}")
|
||||
print(f"Response: {result3.get('final_answer', 'No answer')}")
|
||||
|
||||
if "42" not in str(result3.get('final_answer', '')) and "don't" in str(result3.get('final_answer', '').lower()):
|
||||
print("✅ SUCCESS: Agent correctly doesn't remember after reset!")
|
||||
else:
|
||||
print("⚠️ Check if agent properly forgot the previous conversation")
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("Conversation history tests complete!")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return False
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = test_conversation_history()
|
||||
exit(0 if success else 1)
|
||||
@@ -0,0 +1,226 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for DeepSeek model integration.
|
||||
Tests deepseek-v4-flash (default) with conversation and tool calling.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def test_basic_conversation():
|
||||
"""Test basic conversation capabilities"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 1: Basic Conversation")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set in environment")
|
||||
print("Please set it in your .env file or as environment variable")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
query = "What is 25 * 4 + 10? Reply with FINAL ANSWER: and the number."
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
if "110" in response:
|
||||
print("\n✅ Basic conversation test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n❌ Test failed - incorrect answer")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_tool_usage():
|
||||
"""Test tool calling capabilities"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 2: Tool Usage (Calculator)")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
query = (
|
||||
"Calculate: (123.45 * 67.89) / 12.34 + sqrt(144) - 2^8. "
|
||||
"Use the calculate tool. End with FINAL ANSWER:"
|
||||
)
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
if agent.trajectory.tool_calls:
|
||||
print(f"\n🔧 Tools used: {len(agent.trajectory.tool_calls)}")
|
||||
for call in agent.trajectory.tool_calls:
|
||||
print(f" - {call.tool_name}: {call.arguments}")
|
||||
print("\n✅ Tool usage test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ No tools were used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_currency_conversion():
|
||||
"""Test currency conversion tool"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 3: Currency Conversion")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
query = "Convert 100 USD to EUR and JPY. Use convert_currency. FINAL ANSWER: the amounts."
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
tool_names = [call.tool_name for call in agent.trajectory.tool_calls]
|
||||
if "convert_currency" in tool_names:
|
||||
print("\n🔧 Currency converter was used")
|
||||
print("\n✅ Currency conversion test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ Currency converter was not used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_model_info():
|
||||
"""Test and display model information"""
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST 4: Model Information")
|
||||
print("=" * 60)
|
||||
|
||||
try:
|
||||
api_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: DEEPSEEK_API_KEY not set")
|
||||
return False
|
||||
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="deepseek",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False,
|
||||
)
|
||||
|
||||
expected = Config.get_default_model("deepseek")
|
||||
print("\n📊 Model Configuration:")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Expected default: {expected}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
print(f" Context Mode: {agent.context_mode.value}")
|
||||
|
||||
if agent.provider != "deepseek" or agent.model != expected:
|
||||
print("\n❌ Model config mismatch")
|
||||
return False
|
||||
|
||||
print("\n✅ Model info test completed!")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all tests"""
|
||||
print("\n" + "=" * 60)
|
||||
print("DEEPSEEK MODEL INTEGRATION TEST SUITE")
|
||||
print("=" * 60)
|
||||
print("\nModel: deepseek-v4-flash (default)")
|
||||
print("Provider: DeepSeek")
|
||||
print("API: https://api.deepseek.com")
|
||||
|
||||
if not os.getenv("DEEPSEEK_API_KEY"):
|
||||
print("\n❌ ERROR: DEEPSEEK_API_KEY not found in environment")
|
||||
print("\nPlease set up your .env file with:")
|
||||
print(" DEEPSEEK_API_KEY=your_api_key_here")
|
||||
print("\nYou can get an API key from: https://platform.deepseek.com/api_keys")
|
||||
sys.exit(1)
|
||||
|
||||
results = []
|
||||
results.append(("Model Information", test_model_info()))
|
||||
results.append(("Basic Conversation", test_basic_conversation()))
|
||||
results.append(("Tool Usage", test_tool_usage()))
|
||||
results.append(("Currency Conversion", test_currency_conversion()))
|
||||
|
||||
print("\n" + "=" * 60)
|
||||
print("TEST SUMMARY")
|
||||
print("=" * 60)
|
||||
|
||||
passed = sum(1 for _, result in results if result)
|
||||
total = len(results)
|
||||
|
||||
for test_name, result in results:
|
||||
status = "✅ PASSED" if result else "❌ FAILED"
|
||||
print(f" {test_name}: {status}")
|
||||
|
||||
print(f"\nTotal: {passed}/{total} tests passed")
|
||||
|
||||
if passed == total:
|
||||
print("\n🎉 All tests passed! DeepSeek integration is working correctly.")
|
||||
else:
|
||||
print(f"\n⚠️ {total - passed} test(s) failed. Please check the errors above.")
|
||||
|
||||
return passed == total
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = main()
|
||||
sys.exit(0 if success else 1)
|
||||
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick smoke test for DeepSeek provider (deepseek-v4-flash).
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
task = "What is 10 + 5? Provide FINAL ANSWER with just the number."
|
||||
|
||||
print("=" * 60)
|
||||
print("QUICK TEST - DeepSeek Provider")
|
||||
print("=" * 60)
|
||||
|
||||
deepseek_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if not deepseek_key:
|
||||
print("❌ DEEPSEEK_API_KEY not set")
|
||||
print("Set it in .env or: export DEEPSEEK_API_KEY=your_key")
|
||||
print("Get a key at: https://platform.deepseek.com/api_keys")
|
||||
sys.exit(1)
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
agent = ContextAwareAgent(deepseek_key, ContextMode.FULL, provider="deepseek")
|
||||
print(f"✅ Using: {agent.provider} / {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
print(f"\n📝 Task: {task}")
|
||||
print("-" * 40)
|
||||
|
||||
start = time.time()
|
||||
print("Processing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task, max_iterations=3)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"\n✅ Completed in {elapsed:.2f} seconds")
|
||||
|
||||
if result.get("success"):
|
||||
print("Success: True")
|
||||
if result.get("final_answer"):
|
||||
print(f"Answer: {result['final_answer']}")
|
||||
else:
|
||||
print("Success: False")
|
||||
if result.get("error"):
|
||||
print(f"Error: {result['error']}")
|
||||
|
||||
print(f"Iterations: {result.get('iterations', 0)}")
|
||||
print(f"Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted")
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {str(e)}")
|
||||
sys.exit(1)
|
||||
|
||||
print("=" * 60)
|
||||
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test that Doubao is the default provider
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
# Test without any arguments - should use Doubao
|
||||
print("Testing default provider...")
|
||||
|
||||
# Check if ARK_API_KEY is available
|
||||
ark_key = os.getenv("ARK_API_KEY")
|
||||
sf_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
|
||||
print(f"ARK_API_KEY available: {'Yes' if ark_key else 'No'}")
|
||||
print(f"SILICONFLOW_API_KEY available: {'Yes' if sf_key else 'No'}")
|
||||
|
||||
if ark_key:
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Check config default
|
||||
print(f"\nConfig default provider: {Config.LLM_PROVIDER}")
|
||||
|
||||
# Create agent with default provider from config
|
||||
agent = ContextAwareAgent(ark_key, ContextMode.FULL, provider=Config.LLM_PROVIDER)
|
||||
|
||||
print(f"\n✅ Default agent created successfully!")
|
||||
print(f"Provider: {agent.provider}")
|
||||
print(f"Model: {agent.model}")
|
||||
print(f"Base URL: {agent.client.base_url}")
|
||||
|
||||
if agent.provider == "doubao":
|
||||
print("\n🎉 SUCCESS: Doubao is the default provider!")
|
||||
else:
|
||||
print(f"\n❌ ERROR: Expected doubao, got {agent.provider}")
|
||||
sys.exit(1)
|
||||
else:
|
||||
print("\n⚠️ ARK_API_KEY not set. Cannot test default provider.")
|
||||
print("Please set: export ARK_API_KEY=your_key_here")
|
||||
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick test for Doubao provider
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_doubao():
|
||||
"""Test Doubao provider with a simple task"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 DOUBAO PROVIDER TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check for API key
|
||||
api_key = os.getenv("ARK_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ARK_API_KEY not found. Please set it to test Doubao provider.")
|
||||
print(" export ARK_API_KEY=your_key_here")
|
||||
return
|
||||
|
||||
print("✅ ARK API key found")
|
||||
|
||||
# Create agent with Doubao provider
|
||||
try:
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL, provider="doubao")
|
||||
print(f"✅ Agent created with Doubao provider")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
|
||||
# Simple test task (minimal to save tokens)
|
||||
print("\n📝 Running simple test task...")
|
||||
task = "Calculate: What is 15 + 27? Provide FINAL ANSWER with the result."
|
||||
|
||||
result = agent.execute_task(task, max_iterations=3)
|
||||
|
||||
if result.get('success'):
|
||||
print("✅ Task executed successfully!")
|
||||
if result.get('final_answer'):
|
||||
print(f" Answer: {result['final_answer'][:100]}...")
|
||||
else:
|
||||
print(f"⚠️ Task did not complete successfully")
|
||||
if result.get('error'):
|
||||
print(f" Error: {result['error']}")
|
||||
|
||||
print(f"\n📊 Execution stats:")
|
||||
print(f" Iterations: {result.get('iterations', 0)}")
|
||||
print(f" Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {str(e)}")
|
||||
print("\nNote: Make sure your ARK_API_KEY is valid and has access to the doubao model.")
|
||||
|
||||
print("\n" + "="*60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_doubao()
|
||||
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick test with Doubao as default
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
# Set a very simple task to test quickly
|
||||
task = "What is 10 + 5? Provide FINAL ANSWER with just the number."
|
||||
|
||||
print("="*60)
|
||||
print("QUICK TEST - Doubao Default Provider")
|
||||
print("="*60)
|
||||
|
||||
ark_key = os.getenv("ARK_API_KEY")
|
||||
if not ark_key:
|
||||
print("❌ ARK_API_KEY not set")
|
||||
sys.exit(1)
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
# Create agent with default Doubao
|
||||
agent = ContextAwareAgent(ark_key, ContextMode.FULL, provider="doubao")
|
||||
print(f"✅ Using: {agent.provider} / {agent.model}")
|
||||
print(f"\n📝 Task: {task}")
|
||||
print("-"*40)
|
||||
|
||||
start = time.time()
|
||||
print("Processing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task, max_iterations=2)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"\n✅ Completed in {elapsed:.2f} seconds")
|
||||
|
||||
if result.get('success'):
|
||||
print(f"Success: True")
|
||||
if result.get('final_answer'):
|
||||
print(f"Answer: {result['final_answer']}")
|
||||
else:
|
||||
print(f"Success: False")
|
||||
if result.get('error'):
|
||||
print(f"Error: {result['error']}")
|
||||
|
||||
print(f"Iterations: {result.get('iterations', 0)}")
|
||||
print(f"Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted")
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {str(e)}")
|
||||
|
||||
print("="*60)
|
||||
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for Kimi K3 model integration
|
||||
Tests the Kimi K3 model (kimi-k3) with various tasks
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def test_basic_conversation():
|
||||
"""Test basic conversation capabilities"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 1: Basic Conversation")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set in environment")
|
||||
print("Please set it in your .env file or as environment variable")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test basic conversation
|
||||
query = "What is 25 * 4 + 10?"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
# Verify response contains correct answer
|
||||
if "110" in response:
|
||||
print("\n✅ Basic conversation test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n❌ Test failed - incorrect answer")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_tool_usage():
|
||||
"""Test tool calling capabilities"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 2: Tool Usage (Calculator)")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test complex calculation requiring calculator tool
|
||||
query = "Calculate: (123.45 * 67.89) / 12.34 + sqrt(144) - 2^8"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
# Check if calculator was used
|
||||
if agent.trajectory.tool_calls:
|
||||
print(f"\n🔧 Tools used: {len(agent.trajectory.tool_calls)}")
|
||||
for call in agent.trajectory.tool_calls:
|
||||
print(f" - {call.tool_name}: {call.arguments}")
|
||||
print("\n✅ Tool usage test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ No tools were used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_currency_conversion():
|
||||
"""Test currency conversion tool"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 3: Currency Conversion")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Test currency conversion
|
||||
query = "Convert 100 USD to EUR and JPY"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
# Check if currency converter was used
|
||||
tool_names = [call.tool_name for call in agent.trajectory.tool_calls]
|
||||
if "convert_currency" in tool_names:
|
||||
print(f"\n🔧 Currency converter was used")
|
||||
print("\n✅ Currency conversion test passed!")
|
||||
return True
|
||||
else:
|
||||
print("\n⚠️ Currency converter was not used")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def test_model_info():
|
||||
"""Test and display model information"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 4: Model Information")
|
||||
print("="*60)
|
||||
|
||||
try:
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
print(f"\n📊 Model Configuration:")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
print(f" Context Mode: {agent.context_mode.value}")
|
||||
|
||||
# Test model identification
|
||||
query = "What model are you?"
|
||||
print(f"\n📝 Query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
print(f"\n🤖 Response: {response}")
|
||||
|
||||
print("\n✅ Model info test completed!")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error during test: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all tests"""
|
||||
print("\n" + "="*60)
|
||||
print("KIMI K3 MODEL INTEGRATION TEST SUITE")
|
||||
print("="*60)
|
||||
print("\nModel: kimi-k3")
|
||||
print("Provider: Moonshot AI")
|
||||
print("API: https://api.moonshot.cn/v1")
|
||||
|
||||
# Check environment
|
||||
if not os.getenv("MOONSHOT_API_KEY"):
|
||||
print("\n❌ ERROR: MOONSHOT_API_KEY not found in environment")
|
||||
print("\nPlease set up your .env file with:")
|
||||
print(" MOONSHOT_API_KEY=your_api_key_here")
|
||||
print("\nYou can get an API key from: https://platform.moonshot.cn/")
|
||||
sys.exit(1)
|
||||
|
||||
# Run tests
|
||||
results = []
|
||||
|
||||
# Test 1: Basic conversation
|
||||
results.append(("Basic Conversation", test_basic_conversation()))
|
||||
|
||||
# Test 2: Tool usage
|
||||
results.append(("Tool Usage", test_tool_usage()))
|
||||
|
||||
# Test 3: Currency conversion
|
||||
results.append(("Currency Conversion", test_currency_conversion()))
|
||||
|
||||
# Test 4: Model information
|
||||
results.append(("Model Information", test_model_info()))
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*60)
|
||||
print("TEST SUMMARY")
|
||||
print("="*60)
|
||||
|
||||
passed = sum(1 for _, result in results if result)
|
||||
total = len(results)
|
||||
|
||||
for test_name, result in results:
|
||||
status = "✅ PASSED" if result else "❌ FAILED"
|
||||
print(f" {test_name}: {status}")
|
||||
|
||||
print(f"\nTotal: {passed}/{total} tests passed")
|
||||
|
||||
if passed == total:
|
||||
print("\n🎉 All tests passed! Kimi K3 integration is working correctly.")
|
||||
else:
|
||||
print(f"\n⚠️ {total - passed} test(s) failed. Please check the errors above.")
|
||||
|
||||
return passed == total
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
success = main()
|
||||
sys.exit(0 if success else 1)
|
||||
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick test script to verify Kimi K3 model integration
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def main():
|
||||
# Get API key
|
||||
api_key = os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ ERROR: MOONSHOT_API_KEY not set")
|
||||
print("Please add to your .env file:")
|
||||
print(" MOONSHOT_API_KEY=your_api_key_here")
|
||||
return
|
||||
|
||||
print("🚀 Testing Kimi K3 Model (kimi-k3)")
|
||||
print("=" * 50)
|
||||
|
||||
try:
|
||||
# Create agent with Kimi provider
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider="kimi",
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
print(f"✅ Agent created successfully")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
|
||||
# Test simple query
|
||||
print("\n📝 Testing basic query...")
|
||||
query = "What is 2 + 2?"
|
||||
response = agent.process(query)
|
||||
print(f" Query: {query}")
|
||||
print(f" Response: {response}")
|
||||
|
||||
print("\n✅ Kimi K3 integration is working!")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Error: {e}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,101 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify PDF parsing and currency conversion
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_pdf_with_currencies():
|
||||
"""Test PDF parsing with currency conversion"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 PDF PARSING & CURRENCY CONVERSION TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ No API key found. Set SILICONFLOW_API_KEY environment variable.")
|
||||
return False
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
|
||||
# Test task
|
||||
task = """
|
||||
Analyze the expense report at fixtures/pdfs/simple_expense_report.pdf
|
||||
|
||||
Extract the following expenses mentioned in the document:
|
||||
- US Office: $2,500,000 USD
|
||||
- UK Office: £1,800,000 GBP
|
||||
- Japan Office: ¥380,000,000 JPY
|
||||
- EU Office: €2,100,000 EUR
|
||||
- Singapore Office: S$3,200,000 SGD
|
||||
|
||||
Convert all amounts to USD and calculate the total.
|
||||
|
||||
FINAL ANSWER: Provide the total expenses in USD.
|
||||
"""
|
||||
|
||||
print("📋 Task: Parse PDF and convert multiple currencies to USD")
|
||||
print("-"*40)
|
||||
|
||||
try:
|
||||
# Execute task
|
||||
result = agent.execute_task(task, max_iterations=5)
|
||||
|
||||
print("\n" + "="*40)
|
||||
print("RESULTS:")
|
||||
print("="*40)
|
||||
print(f"Success: {result.get('success', False)}")
|
||||
print(f"Iterations: {result.get('iterations', 0)}")
|
||||
print(f"Tool Calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
# Show tool calls made
|
||||
print("\n📊 Tool Calls Made:")
|
||||
for i, tc in enumerate(result['trajectory'].tool_calls, 1):
|
||||
print(f"{i}. {tc.tool_name}")
|
||||
if tc.tool_name == "parse_pdf":
|
||||
print(f" - PDF: {tc.arguments.get('url', 'N/A')}")
|
||||
if tc.result and 'num_pages' in tc.result:
|
||||
print(f" - Pages: {tc.result['num_pages']}")
|
||||
elif tc.tool_name == "convert_currency":
|
||||
print(f" - {tc.arguments.get('amount', 0)} {tc.arguments.get('from_currency', '')} → {tc.arguments.get('to_currency', '')}")
|
||||
if tc.result and 'converted_amount' in tc.result:
|
||||
print(f" - Result: {tc.result['converted_amount']}")
|
||||
elif tc.tool_name == "calculate":
|
||||
print(f" - Expression: {tc.arguments.get('expression', '')}")
|
||||
if tc.result and 'result' in tc.result:
|
||||
print(f" - Result: {tc.result['result']}")
|
||||
|
||||
if result.get('final_answer'):
|
||||
print("\n✅ Final Answer:")
|
||||
print("-"*40)
|
||||
print(result['final_answer'])
|
||||
|
||||
if result.get('error'):
|
||||
print(f"\n❌ Error: {result['error']}")
|
||||
|
||||
return result.get('success', False)
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Exception: {str(e)}")
|
||||
return False
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Ensure PDFs exist
|
||||
if not os.path.exists("fixtures/pdfs/simple_expense_report.pdf"):
|
||||
print("⚠️ Creating sample PDFs...")
|
||||
os.system("python create_sample_pdf.py")
|
||||
|
||||
# Run test
|
||||
success = test_pdf_with_currencies()
|
||||
sys.exit(0 if success else 1)
|
||||
@@ -0,0 +1,129 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify provider configuration
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_providers():
|
||||
"""Test different provider configurations"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 PROVIDER CONFIGURATION TEST")
|
||||
print("="*60)
|
||||
|
||||
# Test Alibaba Cloud Model Studio / Bailian
|
||||
dashscope_key = os.getenv("DASHSCOPE_API_KEY")
|
||||
if dashscope_key:
|
||||
print("\n✅ Alibaba Cloud Model Studio API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(
|
||||
dashscope_key, ContextMode.FULL, provider="dashscope"
|
||||
)
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ Alibaba Cloud Model Studio API key not found (DASHSCOPE_API_KEY)")
|
||||
|
||||
# Test SiliconFlow
|
||||
sf_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if sf_key:
|
||||
print("\n✅ SiliconFlow API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(sf_key, ContextMode.FULL, provider="siliconflow")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ SiliconFlow API key not found (SILICONFLOW_API_KEY)")
|
||||
|
||||
# Test Doubao
|
||||
ark_key = os.getenv("ARK_API_KEY")
|
||||
if ark_key:
|
||||
print("\n✅ Doubao/ARK API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(ark_key, ContextMode.FULL, provider="doubao")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ Doubao/ARK API key not found (ARK_API_KEY)")
|
||||
|
||||
# Test DeepSeek
|
||||
deepseek_key = os.getenv("DEEPSEEK_API_KEY")
|
||||
if deepseek_key:
|
||||
print("\n✅ DeepSeek API key found")
|
||||
try:
|
||||
agent = ContextAwareAgent(deepseek_key, ContextMode.FULL, provider="deepseek")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
else:
|
||||
print("\n⚠️ DeepSeek API key not found (DEEPSEEK_API_KEY)")
|
||||
|
||||
# Test custom model
|
||||
if sf_key:
|
||||
print("\n🔧 Testing custom model specification:")
|
||||
try:
|
||||
agent = ContextAwareAgent(sf_key, ContextMode.FULL,
|
||||
provider="siliconflow",
|
||||
model="Qwen/QwQ-32B")
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Custom Model: {agent.model}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Error: {str(e)}")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Test complete!")
|
||||
|
||||
# Show usage examples
|
||||
print("\n📖 Usage Examples:")
|
||||
print("-"*40)
|
||||
|
||||
if dashscope_key:
|
||||
print("\n# Using Qwen directly through Alibaba Cloud Model Studio:")
|
||||
print("python main.py --provider dashscope")
|
||||
print("python main.py --provider dashscope --model qwen3.7-plus")
|
||||
|
||||
if sf_key:
|
||||
print("\n# Using SiliconFlow:")
|
||||
print("python main.py --provider siliconflow")
|
||||
print("python main.py --provider siliconflow --model Qwen/QwQ-32B")
|
||||
|
||||
if ark_key:
|
||||
print("\n# Using Doubao:")
|
||||
print("python main.py --provider doubao")
|
||||
print("python main.py --provider doubao --model doubao-seed-1-6-thinking-250715")
|
||||
|
||||
if deepseek_key:
|
||||
print("\n# Using DeepSeek:")
|
||||
print("python main.py --provider deepseek")
|
||||
print("python main.py --provider deepseek --model deepseek-v4-pro")
|
||||
|
||||
if not dashscope_key and not sf_key and not ark_key and not deepseek_key:
|
||||
print("\n⚠️ No API keys found. Please set one of:")
|
||||
print(" export DASHSCOPE_API_KEY=your_key")
|
||||
print(" export SILICONFLOW_API_KEY=your_key")
|
||||
print(" export ARK_API_KEY=your_key")
|
||||
print(" export DEEPSEEK_API_KEY=your_key")
|
||||
|
||||
print("\n" + "="*60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_providers()
|
||||
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script to verify provider switching functionality
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def test_provider_switching():
|
||||
"""Test switching between different providers"""
|
||||
print("🧪 Testing Provider Switching")
|
||||
print("=" * 50)
|
||||
|
||||
providers_to_test = []
|
||||
|
||||
# Check which providers have API keys configured
|
||||
if os.getenv("DASHSCOPE_API_KEY"):
|
||||
providers_to_test.append(("dashscope", os.getenv("DASHSCOPE_API_KEY")))
|
||||
print("✅ Alibaba Cloud Model Studio API key found")
|
||||
else:
|
||||
print("⏭️ Skipping Alibaba Cloud Model Studio (no API key)")
|
||||
|
||||
if os.getenv("SILICONFLOW_API_KEY"):
|
||||
providers_to_test.append(("siliconflow", os.getenv("SILICONFLOW_API_KEY")))
|
||||
print("✅ SiliconFlow API key found")
|
||||
else:
|
||||
print("⏭️ Skipping SiliconFlow (no API key)")
|
||||
|
||||
if os.getenv("ARK_API_KEY"):
|
||||
providers_to_test.append(("doubao", os.getenv("ARK_API_KEY")))
|
||||
print("✅ Doubao API key found")
|
||||
else:
|
||||
print("⏭️ Skipping Doubao (no API key)")
|
||||
|
||||
if os.getenv("MOONSHOT_API_KEY"):
|
||||
providers_to_test.append(("kimi", os.getenv("MOONSHOT_API_KEY")))
|
||||
print("✅ Kimi API key found")
|
||||
else:
|
||||
print("⏭️ Skipping Kimi (no API key)")
|
||||
|
||||
if os.getenv("DEEPSEEK_API_KEY"):
|
||||
providers_to_test.append(("deepseek", os.getenv("DEEPSEEK_API_KEY")))
|
||||
print("✅ DeepSeek API key found")
|
||||
else:
|
||||
print("⏭️ Skipping DeepSeek (no API key)")
|
||||
|
||||
if not providers_to_test:
|
||||
print("\n❌ No API keys configured. Please set at least one:")
|
||||
print(" - DASHSCOPE_API_KEY")
|
||||
print(" - SILICONFLOW_API_KEY")
|
||||
print(" - ARK_API_KEY")
|
||||
print(" - MOONSHOT_API_KEY")
|
||||
print(" - DEEPSEEK_API_KEY")
|
||||
return
|
||||
|
||||
print(f"\nTesting {len(providers_to_test)} provider(s)...")
|
||||
print("-" * 50)
|
||||
|
||||
# Test each available provider
|
||||
for provider_name, api_key in providers_to_test:
|
||||
print(f"\n📌 Testing {provider_name.upper()}")
|
||||
|
||||
try:
|
||||
# Create agent with provider
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider=provider_name,
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Get default model from config
|
||||
default_model = Config.get_default_model(provider_name)
|
||||
|
||||
print(f" Provider: {agent.provider}")
|
||||
print(f" Model: {agent.model}")
|
||||
print(f" Expected: {default_model}")
|
||||
print(f" Base URL: {agent.client.base_url}")
|
||||
|
||||
# Test with a simple query
|
||||
query = "What is 5 + 3?"
|
||||
print(f" Testing query: {query}")
|
||||
|
||||
response = agent.process(query)
|
||||
|
||||
if "8" in response:
|
||||
print(f" ✅ {provider_name} working correctly!")
|
||||
else:
|
||||
print(f" ⚠️ {provider_name} response didn't contain expected answer")
|
||||
print(f" Response: {response[:100]}...")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ Error with {provider_name}: {e}")
|
||||
|
||||
print("\n" + "=" * 50)
|
||||
print("Provider switching test complete!")
|
||||
|
||||
# Show summary
|
||||
print("\n📊 Summary:")
|
||||
print(f" Providers tested: {len(providers_to_test)}")
|
||||
print(" Available providers include: dashscope (qwen/bailian), siliconflow, doubao, kimi, moonshot, deepseek")
|
||||
|
||||
if len(providers_to_test) < 3:
|
||||
print("\n💡 Tip: Configure more API keys to test all providers")
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_provider_switching()
|
||||
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test with a simpler task to diagnose the issue
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_simple_task():
|
||||
"""Test with a very simple task to check if the agent is working"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 SIMPLE TASK TEST")
|
||||
print("="*60)
|
||||
|
||||
# Get API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("❌ SILICONFLOW_API_KEY not found")
|
||||
return
|
||||
|
||||
print("✅ API key found")
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL, provider="siliconflow")
|
||||
print(f"✅ Agent created")
|
||||
print(f" Model: {agent.model}")
|
||||
|
||||
# Very simple task - no tools needed
|
||||
print("\n📝 Test 1: Simple question (no tools)")
|
||||
task1 = "What is 2 + 2? Just tell me the answer. FINAL ANSWER: provide the result."
|
||||
|
||||
start = time.time()
|
||||
print("Executing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task1, max_iterations=1)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"✅ Completed in {elapsed:.2f} seconds")
|
||||
if result.get('final_answer'):
|
||||
print(f" Answer: {result['final_answer'][:100]}")
|
||||
print(f" Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {str(e)}")
|
||||
return
|
||||
|
||||
# Task with a single tool
|
||||
print("\n📝 Test 2: Simple calculation (with tool)")
|
||||
task2 = "Use the calculate tool to compute 15 * 3. FINAL ANSWER: provide the result."
|
||||
|
||||
start = time.time()
|
||||
print("Executing...")
|
||||
|
||||
try:
|
||||
result = agent.execute_task(task2, max_iterations=2)
|
||||
elapsed = time.time() - start
|
||||
|
||||
print(f"✅ Completed in {elapsed:.2f} seconds")
|
||||
if result.get('final_answer'):
|
||||
print(f" Answer: {result['final_answer'][:100]}")
|
||||
print(f" Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n⚠️ Interrupted by user")
|
||||
print("The model might be taking too long to respond.")
|
||||
print("\nSuggestions:")
|
||||
print("1. Try using --provider doubao for faster responses")
|
||||
print("2. Check your internet connection")
|
||||
print("3. The model might be overloaded - try again later")
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error: {str(e)}")
|
||||
|
||||
print("\n" + "="*60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_simple_task()
|
||||
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Demo script showing conversation history persistence
|
||||
"""
|
||||
|
||||
import os
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
def main():
|
||||
# Get API key (use any available provider)
|
||||
if os.getenv("ARK_API_KEY"):
|
||||
api_key, provider = os.getenv("ARK_API_KEY"), "doubao"
|
||||
elif os.getenv("DASHSCOPE_API_KEY"):
|
||||
api_key, provider = os.getenv("DASHSCOPE_API_KEY"), "dashscope"
|
||||
elif os.getenv("MOONSHOT_API_KEY"):
|
||||
api_key, provider = os.getenv("MOONSHOT_API_KEY"), "kimi"
|
||||
elif os.getenv("DEEPSEEK_API_KEY"):
|
||||
api_key, provider = os.getenv("DEEPSEEK_API_KEY"), "deepseek"
|
||||
elif os.getenv("SILICONFLOW_API_KEY"):
|
||||
api_key, provider = os.getenv("SILICONFLOW_API_KEY"), "siliconflow"
|
||||
else:
|
||||
api_key, provider = None, None
|
||||
|
||||
if not api_key:
|
||||
print("❌ No API key found. Please set one of:")
|
||||
print(" - ARK_API_KEY")
|
||||
print(" - DASHSCOPE_API_KEY")
|
||||
print(" - MOONSHOT_API_KEY")
|
||||
print(" - DEEPSEEK_API_KEY")
|
||||
print(" - SILICONFLOW_API_KEY")
|
||||
return
|
||||
|
||||
print("🎭 Conversation History Demo")
|
||||
print("=" * 50)
|
||||
print(f"Provider: {provider.upper()}")
|
||||
print("-" * 50)
|
||||
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(
|
||||
api_key=api_key,
|
||||
provider=provider,
|
||||
context_mode=ContextMode.FULL,
|
||||
verbose=False
|
||||
)
|
||||
|
||||
# Conversation 1: Set context
|
||||
print("\n💬 Turn 1: Setting context...")
|
||||
result = agent.execute_task("My name is Alice and I have a budget of $5,000. What is 20% of my budget?")
|
||||
print(f"Agent: {result.get('final_answer', 'No answer')}")
|
||||
|
||||
# Conversation 2: Reference previous context
|
||||
print("\n💬 Turn 2: Referencing previous context...")
|
||||
result = agent.execute_task("Convert that 20% amount to EUR please.")
|
||||
print(f"Agent: {result.get('final_answer', 'No answer')}")
|
||||
|
||||
# Conversation 3: Recall information
|
||||
print("\n💬 Turn 3: Recalling information...")
|
||||
result = agent.execute_task("What was my name and total budget that I mentioned?")
|
||||
print(f"Agent: {result.get('final_answer', 'No answer')}")
|
||||
|
||||
print("\n" + "-" * 50)
|
||||
print(f"📊 Final Statistics:")
|
||||
print(f" Total messages in history: {len(agent.conversation_history)}")
|
||||
print(f" Total tool calls made: {len(agent.trajectory.tool_calls)}")
|
||||
|
||||
# Show that system prompt is unchanged
|
||||
system_prompt = agent.conversation_history[0]['content']
|
||||
if "Alice" not in system_prompt and "5000" not in system_prompt:
|
||||
print(" ✅ System prompt remained unchanged")
|
||||
else:
|
||||
print(" ❌ System prompt was modified")
|
||||
|
||||
print("\n✨ Demo complete!")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick Start Script for Context-Aware Agent
|
||||
Run this to test the agent with a simple example
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
from config import Config
|
||||
|
||||
def main():
|
||||
"""Quick start demonstration"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("CONTEXT-AWARE AGENT - QUICK START")
|
||||
print("="*60)
|
||||
|
||||
# Check for API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("\n❌ ERROR: SILICONFLOW_API_KEY not found!")
|
||||
print("\nPlease set your API key:")
|
||||
print("1. Copy env.example to .env")
|
||||
print("2. Add your API key to .env")
|
||||
print("3. Or export SILICONFLOW_API_KEY=your_key_here")
|
||||
sys.exit(1)
|
||||
|
||||
print("\n✅ API key found!")
|
||||
|
||||
# Simple demonstration task
|
||||
demo_task = """
|
||||
Please help me with the following financial calculation:
|
||||
|
||||
1. I have $10,000 USD that I want to convert to EUR, GBP, and JPY
|
||||
2. Calculate the average amount across all three currencies (converted back to USD)
|
||||
3. If I invest this average amount with a 5% annual return, what will it be worth in 2 years?
|
||||
|
||||
Show all your calculations step by step.
|
||||
"""
|
||||
|
||||
print("\n📋 Demo Task:")
|
||||
print("-"*40)
|
||||
print(demo_task)
|
||||
print("-"*40)
|
||||
|
||||
# Run with full context (baseline)
|
||||
print("\n🚀 Running agent with FULL context...")
|
||||
agent_full = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
result_full = agent_full.execute_task(demo_task)
|
||||
|
||||
print("\n✨ Results with FULL Context:")
|
||||
print(f"Success: {result_full.get('success', False)}")
|
||||
print(f"Tool calls made: {len(result_full['trajectory'].tool_calls)}")
|
||||
print(f"Iterations: {result_full.get('iterations', 0)}")
|
||||
|
||||
if result_full.get('final_answer'):
|
||||
print(f"\nFinal Answer:")
|
||||
print("-"*40)
|
||||
print(result_full['final_answer'])
|
||||
|
||||
# Demonstrate context ablation effect
|
||||
print("\n" + "="*60)
|
||||
print("DEMONSTRATING CONTEXT ABLATION")
|
||||
print("="*60)
|
||||
|
||||
print("\n🔬 Running same task with NO TOOL RESULTS context...")
|
||||
print("(Agent won't see the results of its tool calls)")
|
||||
|
||||
agent_ablated = ContextAwareAgent(api_key, ContextMode.NO_TOOL_RESULTS)
|
||||
result_ablated = agent_ablated.execute_task(demo_task)
|
||||
|
||||
print("\n⚠️ Results with NO TOOL RESULTS:")
|
||||
print(f"Success: {result_ablated.get('success', False)}")
|
||||
print(f"Tool calls made: {len(result_ablated['trajectory'].tool_calls)}")
|
||||
print(f"Iterations: {result_ablated.get('iterations', 0)}")
|
||||
|
||||
if result_ablated.get('final_answer'):
|
||||
print(f"\nFinal Answer (likely incorrect):")
|
||||
print("-"*40)
|
||||
print(result_ablated['final_answer'][:500] + "...")
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*60)
|
||||
print("COMPARISON SUMMARY")
|
||||
print("="*60)
|
||||
|
||||
print("\n📊 Key Observations:")
|
||||
print(f"1. Full Context: {'✅ Success' if result_full.get('success') else '❌ Failed'}")
|
||||
print(f"2. No Tool Results: {'✅ Success' if result_ablated.get('success') else '❌ Failed'}")
|
||||
print(f"3. Efficiency difference: {result_ablated.get('iterations', 0) - result_full.get('iterations', 0)} more iterations without tool results")
|
||||
|
||||
print("\n💡 Insight:")
|
||||
print("Without seeing tool results, the agent operates blind and may:")
|
||||
print("- Make incorrect calculations")
|
||||
print("- Repeat operations unnecessarily")
|
||||
print("- Fail to validate its work")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Quick start complete! 🎉")
|
||||
print("\nNext steps:")
|
||||
print("1. Run full ablation study: python main.py --mode ablation")
|
||||
print("2. Try interactive mode: python main.py --mode interactive")
|
||||
print("3. Read the README.md for more details")
|
||||
print("="*60 + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,86 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Demo script to showcase sample tasks with PDF functionality
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from _bootstrap import add_project_root
|
||||
|
||||
add_project_root()
|
||||
|
||||
from main import get_sample_tasks, ensure_sample_pdfs
|
||||
|
||||
def main():
|
||||
"""Demo the sample tasks"""
|
||||
print("\n" + "="*60)
|
||||
print("🎯 CONTEXT-AWARE AGENT - SAMPLE TASKS DEMO")
|
||||
print("="*60)
|
||||
|
||||
# Ensure PDFs exist
|
||||
print("\n📄 Checking for sample PDFs...")
|
||||
if ensure_sample_pdfs():
|
||||
print("✅ Sample PDFs are ready!")
|
||||
else:
|
||||
print("⚠️ Could not create sample PDFs, will use online alternatives")
|
||||
|
||||
# Get sample tasks
|
||||
tasks = get_sample_tasks()
|
||||
|
||||
print(f"\n📋 Found {len(tasks)} sample tasks:")
|
||||
print("-"*60)
|
||||
|
||||
for i, task in enumerate(tasks, 1):
|
||||
print(f"\n{i}. {task['name']}")
|
||||
print(f" 📝 {task['description']}")
|
||||
print(f" 📊 Complexity: {'⭐' * (i if i <= 3 else 3)}")
|
||||
|
||||
# Show a preview of the task
|
||||
task_preview = task['task'].replace('\n', ' ')[:100] + "..."
|
||||
print(f" 💬 Preview: {task_preview}")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("💡 USAGE TIPS:")
|
||||
print("-"*60)
|
||||
print("1. Run 'python main.py' to enter interactive mode")
|
||||
print("2. Type 'sample 2' to test PDF parsing capabilities")
|
||||
print("3. Type 'sample 5' for the most comprehensive test")
|
||||
print("4. Switch modes with 'mode no_reasoning' to see ablation effects")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🔬 ABLATION TESTING:")
|
||||
print("-"*60)
|
||||
print("Try running the same task in different modes:")
|
||||
print(" • full - Everything works perfectly")
|
||||
print(" • no_history - Agent forgets what it did")
|
||||
print(" • no_reasoning - No planning, chaotic execution")
|
||||
print(" • no_tool_calls - Can't do anything!")
|
||||
print(" • no_tool_results - Works blind, gets confused")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("📊 PDF TASKS:")
|
||||
print("-"*60)
|
||||
|
||||
# Check if local PDFs exist
|
||||
pdf_dir = Path("fixtures/pdfs")
|
||||
if pdf_dir.exists():
|
||||
pdfs = list(pdf_dir.glob("*.pdf"))
|
||||
if pdfs:
|
||||
print(f"✅ Found {len(pdfs)} local PDF files:")
|
||||
for pdf in pdfs:
|
||||
print(f" • {pdf.name}")
|
||||
print("\nTask #2 will use these local PDFs for testing.")
|
||||
else:
|
||||
print("⚠️ No PDFs found in fixtures/pdfs/")
|
||||
else:
|
||||
print("📥 PDF directory not found. Run 'create_pdfs' command to generate samples.")
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Ready to test! Run 'python main.py' to start.")
|
||||
print("="*60 + "\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,305 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test script for Context-Aware Agent
|
||||
Validates installation and basic functionality
|
||||
"""
|
||||
|
||||
import sys
|
||||
from agent import ContextAwareAgent, ContextMode, ToolRegistry
|
||||
import unittest
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
|
||||
class TestToolRegistry(unittest.TestCase):
|
||||
"""Test the tool registry functions"""
|
||||
|
||||
def test_calculator(self):
|
||||
"""Test calculator tool"""
|
||||
tools = ToolRegistry()
|
||||
|
||||
# Basic arithmetic
|
||||
result = tools.calculate("2 + 2")
|
||||
self.assertEqual(result["result"], 4)
|
||||
|
||||
# Complex expression
|
||||
result = tools.calculate("(10 * 5) + (20 / 4)")
|
||||
self.assertEqual(result["result"], 55.0)
|
||||
|
||||
# With math functions
|
||||
result = tools.calculate("sqrt(16) + abs(-5)")
|
||||
self.assertEqual(result["result"], 9.0)
|
||||
|
||||
def test_currency_converter(self):
|
||||
"""Test currency conversion tool"""
|
||||
tools = ToolRegistry()
|
||||
|
||||
# USD to EUR
|
||||
result = tools.convert_currency(100, "USD", "EUR")
|
||||
self.assertIn("converted_amount", result)
|
||||
self.assertIn("exchange_rate", result)
|
||||
self.assertGreater(result["converted_amount"], 0)
|
||||
|
||||
# Currency symbol normalization (US$, S$, A$, C$, $)
|
||||
result_us = tools.convert_currency(100, "US$", "EUR")
|
||||
self.assertEqual(result_us["from_currency"], "USD")
|
||||
self.assertEqual(result_us["converted_amount"], 92.0)
|
||||
|
||||
result_s = tools.convert_currency(100, "S$", "USD")
|
||||
self.assertEqual(result_s["from_currency"], "SGD")
|
||||
self.assertIn("converted_amount", result_s)
|
||||
|
||||
result_a = tools.convert_currency(100, "A$", "USD")
|
||||
self.assertEqual(result_a["from_currency"], "AUD")
|
||||
self.assertIn("converted_amount", result_a)
|
||||
|
||||
result_c = tools.convert_currency(100, "C$", "USD")
|
||||
self.assertEqual(result_c["from_currency"], "CAD")
|
||||
self.assertIn("converted_amount", result_c)
|
||||
# Invalid currency
|
||||
result = tools.convert_currency(100, "XXX", "YYY")
|
||||
self.assertIn("error", result)
|
||||
result_invalid_s = tools.convert_currency(100, "S$INVALID", "USD")
|
||||
self.assertIn("error", result_invalid_s)
|
||||
|
||||
def test_convert_currency_string_and_formatted_amounts(self):
|
||||
"""
|
||||
Prove that convert_currency accepts string and formatted numeric amounts.
|
||||
|
||||
LLM tool calls frequently pass numeric arguments as strings (e.g., "100", "$1,000.00").
|
||||
Previously, passing a string raised a TypeError during float division. This test locks
|
||||
out regressions by asserting that numeric strings and formatted currency strings convert correctly.
|
||||
"""
|
||||
tools = ToolRegistry()
|
||||
result_str = tools.convert_currency("100", "USD", "EUR")
|
||||
self.assertEqual(result_str["converted_amount"], 92.0)
|
||||
self.assertEqual(result_str["original_amount"], 100.0)
|
||||
|
||||
result_formatted = tools.convert_currency("$1,000.00", "USD", "EUR")
|
||||
self.assertEqual(result_formatted["converted_amount"], 920.0)
|
||||
self.assertEqual(result_formatted["original_amount"], 1000.0)
|
||||
|
||||
result_us_dollar = tools.convert_currency("US$100", "USD", "EUR")
|
||||
self.assertEqual(result_us_dollar["converted_amount"], 92.0)
|
||||
self.assertEqual(result_us_dollar["original_amount"], 100.0)
|
||||
|
||||
result_currency_code = tools.convert_currency("USD$1,000", "USD$", "EUR")
|
||||
self.assertEqual(result_currency_code["converted_amount"], 920.0)
|
||||
self.assertEqual(result_currency_code["original_amount"], 1000.0)
|
||||
|
||||
result_comma_large = tools.convert_currency("1,234,567.89", "USD", "EUR")
|
||||
self.assertEqual(result_comma_large["original_amount"], 1234567.89)
|
||||
|
||||
result_euro_sym = tools.convert_currency("€ 500.25", "EUR", "USD")
|
||||
self.assertIn("converted_amount", result_euro_sym)
|
||||
|
||||
result_invalid_str = tools.convert_currency("invalid_str", "USD", "EUR")
|
||||
self.assertIn("error", result_invalid_str)
|
||||
|
||||
def test_pdf_parser_structure(self):
|
||||
"""Test PDF parser structure (without actual PDF)"""
|
||||
tools = ToolRegistry()
|
||||
|
||||
# Test with invalid URL (should handle gracefully)
|
||||
result = tools.parse_pdf("http://invalid-url-for-testing.com/test.pdf")
|
||||
self.assertIn("error", result)
|
||||
|
||||
|
||||
class TestContextModes(unittest.TestCase):
|
||||
"""Test different context modes"""
|
||||
|
||||
@patch.dict('os.environ', {'SILICONFLOW_API_KEY': 'test_key'})
|
||||
def setUp(self):
|
||||
"""Set up test fixtures"""
|
||||
self.api_key = "test_key"
|
||||
|
||||
def test_context_mode_initialization(self):
|
||||
"""Test agent initialization with different context modes"""
|
||||
for mode in ContextMode:
|
||||
agent = ContextAwareAgent(self.api_key, mode)
|
||||
self.assertEqual(agent.context_mode, mode)
|
||||
self.assertEqual(agent.trajectory.context_mode, mode)
|
||||
|
||||
def test_context_building(self):
|
||||
"""Test context building for different modes"""
|
||||
# Full context mode
|
||||
agent = ContextAwareAgent(self.api_key, ContextMode.FULL)
|
||||
agent.trajectory.reasoning_steps = ["Step 1", "Step 2"]
|
||||
agent.trajectory.tool_calls.append(
|
||||
MagicMock(tool_name="test", arguments={}, result={"test": "result"})
|
||||
)
|
||||
|
||||
context = agent._build_context()
|
||||
self.assertIn("Previous Reasoning Steps", context)
|
||||
self.assertIn("Tool Call History", context)
|
||||
|
||||
# No reasoning mode
|
||||
agent_no_reasoning = ContextAwareAgent(self.api_key, ContextMode.NO_REASONING)
|
||||
agent_no_reasoning.trajectory.reasoning_steps = ["Step 1"]
|
||||
context = agent_no_reasoning._build_context()
|
||||
self.assertNotIn("Previous Reasoning Steps", context)
|
||||
|
||||
# No history mode
|
||||
agent_no_history = ContextAwareAgent(self.api_key, ContextMode.NO_HISTORY)
|
||||
agent_no_history.trajectory.tool_calls.append(
|
||||
MagicMock(tool_name="test", arguments={}, result={"test": "result"})
|
||||
)
|
||||
context = agent_no_history._build_context()
|
||||
self.assertEqual(context, "")
|
||||
|
||||
|
||||
class TestAblationScenarios(unittest.TestCase):
|
||||
"""Test ablation scenarios"""
|
||||
|
||||
def test_tool_execution(self):
|
||||
"""Test tool execution"""
|
||||
agent = ContextAwareAgent("test_key", ContextMode.FULL)
|
||||
|
||||
# Test calculator execution
|
||||
result = agent._execute_tool("calculate", {"expression": "2 + 2"})
|
||||
self.assertEqual(result["result"], 4)
|
||||
|
||||
# Test unknown tool
|
||||
result = agent._execute_tool("unknown_tool", {})
|
||||
self.assertIn("error", result)
|
||||
|
||||
def test_trajectory_reset(self):
|
||||
"""Test trajectory reset"""
|
||||
agent = ContextAwareAgent("test_key", ContextMode.FULL)
|
||||
|
||||
# Add some data to trajectory
|
||||
agent.trajectory.reasoning_steps.append("Test step")
|
||||
agent.trajectory.tool_calls.append(
|
||||
MagicMock(tool_name="test", arguments={})
|
||||
)
|
||||
|
||||
# Reset
|
||||
agent.reset()
|
||||
|
||||
# Check if cleared
|
||||
self.assertEqual(len(agent.trajectory.reasoning_steps), 0)
|
||||
self.assertEqual(len(agent.trajectory.tool_calls), 0)
|
||||
self.assertEqual(agent.trajectory.context_mode, ContextMode.FULL)
|
||||
|
||||
|
||||
def run_integration_test():
|
||||
"""Run a simple integration test"""
|
||||
print("\n" + "="*60)
|
||||
print("INTEGRATION TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check if API key is available
|
||||
import os
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
|
||||
if not api_key:
|
||||
print("⚠️ Skipping integration test (no API key found)")
|
||||
print("Set SILICONFLOW_API_KEY to run integration tests")
|
||||
return False
|
||||
|
||||
print("✅ API key found, running integration test...")
|
||||
|
||||
try:
|
||||
# Create agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
|
||||
# Simple task that doesn't require external PDFs
|
||||
simple_task = "Calculate: What is 15% of $2500? Then convert the result to EUR."
|
||||
|
||||
print(f"\nTest task: {simple_task}")
|
||||
print("Running...")
|
||||
|
||||
# Execute with timeout
|
||||
import signal
|
||||
|
||||
def timeout_handler(signum, frame):
|
||||
raise TimeoutError("Integration test timed out")
|
||||
|
||||
# Set 30 second timeout
|
||||
signal.signal(signal.SIGALRM, timeout_handler)
|
||||
signal.alarm(30)
|
||||
|
||||
try:
|
||||
result = agent.execute_task(simple_task, max_iterations=3)
|
||||
signal.alarm(0) # Cancel alarm
|
||||
|
||||
print("\n✅ Integration test completed!")
|
||||
print(f"Success: {result.get('success', False)}")
|
||||
print(f"Tool calls: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
if result.get('final_answer'):
|
||||
print(f"Answer preview: {result['final_answer'][:100]}...")
|
||||
|
||||
return True
|
||||
|
||||
except TimeoutError:
|
||||
print("❌ Integration test timed out")
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Integration test failed: {str(e)}")
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
"""Main test runner"""
|
||||
print("\n" + "="*60)
|
||||
print("CONTEXT-AWARE AGENT TEST SUITE")
|
||||
print("="*60)
|
||||
|
||||
# Run unit tests
|
||||
print("\n📋 Running unit tests...")
|
||||
|
||||
# Create test suite
|
||||
loader = unittest.TestLoader()
|
||||
suite = unittest.TestSuite()
|
||||
|
||||
# Add test cases
|
||||
suite.addTests(loader.loadTestsFromTestCase(TestToolRegistry))
|
||||
suite.addTests(loader.loadTestsFromTestCase(TestContextModes))
|
||||
suite.addTests(loader.loadTestsFromTestCase(TestAblationScenarios))
|
||||
|
||||
# Run tests
|
||||
runner = unittest.TextTestRunner(verbosity=2)
|
||||
result = runner.run(suite)
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*60)
|
||||
print("UNIT TEST SUMMARY")
|
||||
print("="*60)
|
||||
print(f"Tests run: {result.testsRun}")
|
||||
print(f"Failures: {len(result.failures)}")
|
||||
print(f"Errors: {len(result.errors)}")
|
||||
|
||||
if result.wasSuccessful():
|
||||
print("✅ All unit tests passed!")
|
||||
else:
|
||||
print("❌ Some tests failed")
|
||||
sys.exit(1)
|
||||
|
||||
# Run integration test if possible
|
||||
print("\n" + "="*60)
|
||||
integration_success = run_integration_test()
|
||||
|
||||
# Final summary
|
||||
print("\n" + "="*60)
|
||||
print("FINAL TEST SUMMARY")
|
||||
print("="*60)
|
||||
|
||||
if result.wasSuccessful():
|
||||
print("✅ Unit tests: PASSED")
|
||||
else:
|
||||
print("❌ Unit tests: FAILED")
|
||||
|
||||
if integration_success:
|
||||
print("✅ Integration test: PASSED")
|
||||
else:
|
||||
print("⚠️ Integration test: SKIPPED or FAILED")
|
||||
|
||||
print("\n🎉 Testing complete!")
|
||||
print("="*60 + "\n")
|
||||
|
||||
return 0 if result.wasSuccessful() else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,125 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Test the code_interpreter tool with the agent
|
||||
"""
|
||||
|
||||
import os
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
def test_code_interpreter():
|
||||
"""Test code interpreter integration"""
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("🧪 CODE INTERPRETER TEST")
|
||||
print("="*60)
|
||||
|
||||
# Check API key
|
||||
api_key = os.getenv("SILICONFLOW_API_KEY")
|
||||
if not api_key:
|
||||
print("⚠️ No API key set, using mock test")
|
||||
# Test just the tool directly
|
||||
from agent import ToolRegistry
|
||||
tools = ToolRegistry()
|
||||
|
||||
code = """
|
||||
# Calculate total expenses
|
||||
expenses_usd = {
|
||||
'US Office': 2500000,
|
||||
'UK Office (converted)': 2278481.01,
|
||||
'Japan Office (converted)': 2541806.02,
|
||||
'EU Office (converted)': 2282608.70,
|
||||
'Singapore Office (converted)': 2388059.70
|
||||
}
|
||||
|
||||
# Calculate total
|
||||
total = sum(expenses_usd.values())
|
||||
|
||||
# Calculate percentages
|
||||
for office, amount in expenses_usd.items():
|
||||
percentage = (amount / total) * 100
|
||||
print(f"{office}: ${amount:,.2f} ({percentage:.2f}%)")
|
||||
|
||||
print(f"\\nTotal Expenses: ${total:,.2f}")
|
||||
|
||||
# Calculate after 12% reduction
|
||||
reduced_total = total * 0.88
|
||||
savings = total - reduced_total
|
||||
print(f"After 12% reduction: ${reduced_total:,.2f}")
|
||||
print(f"Savings: ${savings:,.2f}")
|
||||
|
||||
result = {
|
||||
'total': total,
|
||||
'reduced': reduced_total,
|
||||
'savings': savings
|
||||
}
|
||||
"""
|
||||
|
||||
result = tools.code_interpreter(code)
|
||||
if result['success']:
|
||||
print("✅ Code interpreter executed successfully!")
|
||||
print("\nOutput:")
|
||||
print(result['output'])
|
||||
print(f"\nResult dictionary: {result['result']}")
|
||||
else:
|
||||
print(f"❌ Error: {result['error']}")
|
||||
|
||||
return
|
||||
|
||||
# Test with full agent
|
||||
agent = ContextAwareAgent(api_key, ContextMode.FULL)
|
||||
|
||||
task = """
|
||||
Calculate the following:
|
||||
|
||||
Given these expenses:
|
||||
- US: $2,500,000
|
||||
- UK: $2,278,481
|
||||
- Japan: $2,541,806
|
||||
- EU: $2,282,609
|
||||
- Singapore: $2,388,060
|
||||
|
||||
Use the code_interpreter tool to:
|
||||
1. Calculate the total expenses
|
||||
2. Calculate what percentage each office represents
|
||||
3. Calculate the new totals if we apply a 12% cost reduction
|
||||
|
||||
FINAL ANSWER: Provide the total, the percentage breakdown, and the reduced total.
|
||||
"""
|
||||
|
||||
print("Running task with agent...")
|
||||
print("Task: Calculate totals and percentages using code_interpreter")
|
||||
print("-"*40)
|
||||
|
||||
result = agent.execute_task(task, max_iterations=3)
|
||||
|
||||
print(f"\nSuccess: {result.get('success', False)}")
|
||||
print(f"Tool calls made: {len(result['trajectory'].tool_calls)}")
|
||||
|
||||
# Check if code_interpreter was used
|
||||
code_interpreter_used = any(
|
||||
tc.tool_name == 'code_interpreter'
|
||||
for tc in result['trajectory'].tool_calls
|
||||
)
|
||||
|
||||
if code_interpreter_used:
|
||||
print("✅ Code interpreter was used!")
|
||||
# Show the code that was executed
|
||||
for tc in result['trajectory'].tool_calls:
|
||||
if tc.tool_name == 'code_interpreter':
|
||||
print("\nExecuted code:")
|
||||
print("-"*40)
|
||||
print(tc.arguments.get('code', 'N/A'))
|
||||
print("-"*40)
|
||||
if tc.result and tc.result.get('output'):
|
||||
print("\nOutput:")
|
||||
print(tc.result['output'])
|
||||
else:
|
||||
print("⚠️ Code interpreter was not used")
|
||||
|
||||
if result.get('final_answer'):
|
||||
print("\n📝 Final Answer:")
|
||||
print(result['final_answer'])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_code_interpreter()
|
||||
@@ -0,0 +1,74 @@
|
||||
"""Regression: malformed tool-argument JSON must not abort the ReAct loop."""
|
||||
from types import SimpleNamespace
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from agent import ContextAwareAgent, ContextMode
|
||||
|
||||
|
||||
def _choice(*, content=None, tool_calls=None):
|
||||
msg = SimpleNamespace(
|
||||
content=content,
|
||||
tool_calls=tool_calls,
|
||||
reasoning_content=None,
|
||||
model_dump=lambda: {
|
||||
"role": "assistant",
|
||||
"content": content,
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": tc.id,
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": tc.function.name,
|
||||
"arguments": tc.function.arguments,
|
||||
},
|
||||
}
|
||||
for tc in (tool_calls or [])
|
||||
],
|
||||
},
|
||||
)
|
||||
return SimpleNamespace(message=msg)
|
||||
|
||||
|
||||
def test_execute_task_survives_malformed_tool_arguments_json():
|
||||
agent = ContextAwareAgent("test-key", ContextMode.FULL, verbose=False)
|
||||
bad_call = SimpleNamespace(
|
||||
id="call-bad",
|
||||
function=SimpleNamespace(
|
||||
name="calculate",
|
||||
arguments='{"expression": "1+1",}', # trailing comma
|
||||
),
|
||||
)
|
||||
tool_turn = SimpleNamespace(choices=[_choice(tool_calls=[bad_call])])
|
||||
final_turn = SimpleNamespace(
|
||||
choices=[_choice(content="FINAL ANSWER: recovered")]
|
||||
)
|
||||
agent.client = MagicMock()
|
||||
agent.client.chat.completions.create = MagicMock(
|
||||
side_effect=[tool_turn, final_turn]
|
||||
)
|
||||
|
||||
result = agent.execute_task("compute", max_iterations=5)
|
||||
|
||||
assert result.get("error") is None
|
||||
assert result["completed"] is True
|
||||
assert result["task_success"] is None
|
||||
assert result["success"] is True # backwards-compatible completion alias
|
||||
assert "recovered" in (result.get("final_answer") or result.get("answer") or "")
|
||||
tool_roles = [m for m in agent.conversation_history if m.get("role") == "tool"]
|
||||
assert tool_roles
|
||||
assert "Invalid tool arguments" in tool_roles[0]["content"]
|
||||
assert agent.client.chat.completions.create.call_count == 2
|
||||
|
||||
|
||||
def test_execute_task_does_not_complete_on_empty_terminal_content():
|
||||
agent = ContextAwareAgent("test-key", ContextMode.NO_TOOL_CALLS, verbose=False)
|
||||
empty_turn = SimpleNamespace(choices=[_choice(content="")])
|
||||
agent.client = MagicMock()
|
||||
agent.client.chat.completions.create = MagicMock(return_value=empty_turn)
|
||||
|
||||
result = agent.execute_task("say something", max_iterations=5)
|
||||
|
||||
assert result["final_answer"] is None
|
||||
assert result["completed"] is False
|
||||
assert result["task_success"] is None
|
||||
assert result["success"] is False
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
1bd60e9548d7732820e6c8f73b565ee397b42c8d480f1ee2a120b6f81833913b evidence.json
|
||||
@@ -0,0 +1,251 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Serve an HTML report visualizing experiment arms from validation/latest.json.
|
||||
|
||||
Reads validation/latest.json (next to this script), then serves a report over
|
||||
HTTP and opens it in the browser. The report shows:
|
||||
1. A summary table with task_success / iterations / repeated_tool_calls per arm.
|
||||
2. One section per arm showing tool_call_signatures and reasoning_steps,
|
||||
with every reasoning step collapsed by default.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import html
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
DEFAULT_JSON = SCRIPT_DIR / "validation" / "latest.json"
|
||||
|
||||
|
||||
def esc(value):
|
||||
return html.escape(str(value))
|
||||
|
||||
|
||||
def fmt(value):
|
||||
if value is None:
|
||||
return "—"
|
||||
if isinstance(value, bool):
|
||||
return str(value)
|
||||
return html.escape(str(value))
|
||||
|
||||
|
||||
def status_badge(value):
|
||||
if value is True:
|
||||
return '<span class="badge ok">✓</span>'
|
||||
if value is False:
|
||||
return '<span class="badge fail">✗</span>'
|
||||
return '<span class="badge na">n/a</span>'
|
||||
|
||||
|
||||
def render_arm_section(arm):
|
||||
mode = esc(arm.get("mode", "?"))
|
||||
model = esc(arm.get("model", ""))
|
||||
provider = esc(arm.get("provider", ""))
|
||||
elapsed = arm.get("elapsed_seconds")
|
||||
elapsed_s = f"{elapsed:.2f} s" if isinstance(elapsed, (int, float)) else "—"
|
||||
final_answer = arm.get("final_answer")
|
||||
|
||||
parts = [f'<section class="arm" id="arm-{esc(arm.get("mode", "unknown"))}">']
|
||||
parts.append(f"<h2>Mode: {mode}</h2>")
|
||||
parts.append(
|
||||
f'<p class="meta">{provider} / {model} · {elapsed_s} '
|
||||
f'· completed={fmt(arm.get("completed"))} '
|
||||
f'success={fmt(arm.get("success"))}</p>'
|
||||
)
|
||||
|
||||
if final_answer:
|
||||
parts.append(
|
||||
f'<h3>Final answer</h3><div class="final-answer"><pre>{esc(final_answer)}</pre></div>'
|
||||
)
|
||||
|
||||
signatures = arm.get("tool_call_signatures") or []
|
||||
parts.append(f"<h3>Tool call signatures ({len(signatures)})</h3>")
|
||||
if signatures:
|
||||
parts.append('<ol class="signatures">')
|
||||
for i, sig in enumerate(signatures, start=1):
|
||||
parts.append(
|
||||
f'<li><span class="idx">{i}</span>'
|
||||
f'<code>{esc(sig)}</code></li>'
|
||||
)
|
||||
parts.append("</ol>")
|
||||
else:
|
||||
parts.append("<p class=\"muted\">No tool calls.</p>")
|
||||
|
||||
reasoning = arm.get("reasoning_steps") or []
|
||||
parts.append(f"<h3>Reasoning steps ({len(reasoning)})</h3>")
|
||||
if reasoning:
|
||||
parts.append('<div class="reasoning">')
|
||||
for i, step in enumerate(reasoning, start=1):
|
||||
if isinstance(step, str):
|
||||
body = esc(step)
|
||||
else:
|
||||
body = esc(json.dumps(step, ensure_ascii=False, indent=2))
|
||||
parts.append(
|
||||
f"<details class=\"step\">"
|
||||
f"<summary>Step {i}</summary>"
|
||||
f"<pre>{body}</pre>"
|
||||
f"</details>"
|
||||
)
|
||||
parts.append("</div>")
|
||||
else:
|
||||
parts.append("<p class=\"muted\">No reasoning steps recorded.</p>")
|
||||
|
||||
parts.append("</section>")
|
||||
return "\n".join(parts)
|
||||
|
||||
|
||||
def render(data):
|
||||
arms = data.get("arms", [])
|
||||
expected = data.get("expected_numbers", [])
|
||||
|
||||
rows = []
|
||||
for arm in arms:
|
||||
behavior = arm.get("behavior") or {}
|
||||
elapsed = arm.get("elapsed_seconds")
|
||||
elapsed_s = f"{elapsed:.2f}" if isinstance(elapsed, (int, float)) else "—"
|
||||
rows.append(
|
||||
"".join(
|
||||
[
|
||||
"<tr>",
|
||||
f'<td class="mode"><a href="#arm-{esc(arm.get("mode", ""))}">{esc(arm.get("mode", "?"))}</a></td>',
|
||||
f"<td>{status_badge(arm.get('task_success'))}</td>",
|
||||
f"<td>{fmt(arm.get('iterations'))}</td>",
|
||||
f"<td>{fmt(arm.get('repeated_tool_calls'))}</td>",
|
||||
f"<td>{fmt(behavior.get('tool_action_count'))}</td>",
|
||||
f"<td>{elapsed_s}</td>",
|
||||
"</tr>",
|
||||
]
|
||||
)
|
||||
)
|
||||
|
||||
table = (
|
||||
"<table>"
|
||||
"<thead><tr>"
|
||||
"<th>mode</th>"
|
||||
"<th>task_success</th>"
|
||||
"<th>iterations</th>"
|
||||
"<th>repeated_tool_calls</th>"
|
||||
"<th>tool actions</th>"
|
||||
"<th>elapsed (s)</th>"
|
||||
"</tr></thead>"
|
||||
f"<tbody>{''.join(rows)}</tbody>"
|
||||
"</table>"
|
||||
)
|
||||
|
||||
sections = "\n".join(render_arm_section(arm) for arm in arms)
|
||||
|
||||
expected_html = ""
|
||||
if expected:
|
||||
items = "".join(f"<li><code>{esc(n)}</code></li>" for n in expected)
|
||||
expected_html = (
|
||||
'<h3>Expected numbers</h3>'
|
||||
f'<ul class="expected">{items}</ul>'
|
||||
)
|
||||
|
||||
return f"""<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Experiment {esc(data.get('experiment_id', ''))} — arm comparison</title>
|
||||
<style>
|
||||
body {{ font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
max-width: 1000px; margin: 0 auto; padding: 24px; line-height: 1.55;
|
||||
color: #1c2733; background: #fff; }}
|
||||
h1, h2, h3 {{ color: #0b3d66; }}
|
||||
h2 {{ border-bottom: 2px solid #e3e8ee; padding-bottom: 6px; margin-top: 40px; }}
|
||||
table {{ border-collapse: collapse; width: 100%; margin: 12px 0 8px; }}
|
||||
th, td {{ border: 1px solid #d5dce3; padding: 8px 10px; text-align: left; }}
|
||||
th {{ background: #f2f5f8; }}
|
||||
tbody tr:nth-child(even) {{ background: #f8fafc; }}
|
||||
td.mode a {{ font-weight: 600; color: #0b3d66; }}
|
||||
.badge {{ display: inline-block; padding: 1px 8px; border-radius: 10px;
|
||||
font-size: 13px; color: #fff; }}
|
||||
.badge.ok {{ background: #1a7f37; }}
|
||||
.badge.fail {{ background: #c0392b; }}
|
||||
.badge.na {{ background: #9aa7b4; }}
|
||||
.meta {{ color: #5a6b7b; font-size: 13px; }}
|
||||
.final-answer pre {{ background: #f4f6f8; border-left: 4px solid #0b3d66;
|
||||
padding: 10px 12px; overflow-x: auto; }}
|
||||
.signatures {{ padding-left: 0; list-style: none; }}
|
||||
.signatures li {{ display: flex; align-items: baseline; margin: 4px 0; }}
|
||||
.signatures .idx {{ display: inline-block; min-width: 26px; color: #9aa7b4;
|
||||
font-size: 12px; }}
|
||||
.signatures code, .reasoning pre {{ font-family: ui-monospace, SFMono-Regular,
|
||||
Menlo, monospace; font-size: 13px; }}
|
||||
.reasoning .step {{ margin: 6px 0; border: 1px solid #e3e8ee; border-radius: 6px;
|
||||
background: #fafbfc; }}
|
||||
.reasoning summary {{ cursor: pointer; padding: 8px 12px; font-weight: 600;
|
||||
color: #0b3d66; user-select: none; }}
|
||||
.reasoning pre {{ margin: 0; padding: 10px 12px; border-top: 1px solid #e3e8ee;
|
||||
white-space: pre-wrap; word-break: break-word; overflow-x: auto; }}
|
||||
.muted {{ color: #9aa7b4; }}
|
||||
.expected li {{ margin: 2px 0; }}
|
||||
.task {{ background: #f2f5f8; border: 1px solid #d5dce3; border-left: 4px solid
|
||||
#0b3d66; border-radius: 6px; padding: 12px 16px; margin: 16px 0;
|
||||
white-space: pre-wrap; }}
|
||||
.task-label {{ font-weight: 700; color: #0b3d66; margin-bottom: 6px; }}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Experiment {esc(data.get('experiment_id', ''))}</h1>
|
||||
<p class="meta">source: {esc(data.get('canonical_source', ''))} · created: {esc(data.get('created_at', ''))}</p>
|
||||
<div class="task"><div class="task-label">Task</div>{esc(data.get('task', ''))}</div>
|
||||
{expected_html}
|
||||
<h2>Summary</h2>
|
||||
{table}
|
||||
{sections}
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
|
||||
def serve(data, port):
|
||||
import http.server
|
||||
import threading
|
||||
import webbrowser
|
||||
|
||||
html_body = render(data)
|
||||
url = f"http://127.0.0.1:{port}"
|
||||
|
||||
class Handler(http.server.BaseHTTPRequestHandler):
|
||||
def do_GET(self):
|
||||
if self.path not in ("/", "/index.html"):
|
||||
self.send_error(404)
|
||||
return
|
||||
body = html_body.encode("utf-8")
|
||||
self.send_response(200)
|
||||
self.send_header("Content-Type", "text/html; charset=utf-8")
|
||||
self.send_header("Content-Length", str(len(body)))
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, fmt, *args):
|
||||
print("[http] %s - %s" % (self.address_string(), fmt % args))
|
||||
|
||||
server = http.server.HTTPServer(("127.0.0.1", port), Handler)
|
||||
print(f"Serving on {url} (Ctrl+C to stop)")
|
||||
threading.Timer(0.5, lambda: webbrowser.open(url)).start()
|
||||
try:
|
||||
server.serve_forever()
|
||||
except KeyboardInterrupt:
|
||||
print("\nStopped.")
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("json_path", nargs="?", default=str(DEFAULT_JSON),
|
||||
help="Path to the validation JSON (default: %(default)s)")
|
||||
parser.add_argument("-p", "--port", type=int, default=8000,
|
||||
help="Port to serve on (default: 8000)")
|
||||
args = parser.parse_args()
|
||||
|
||||
json_file = Path(args.json_path)
|
||||
with open(json_file, encoding="utf-8") as fh:
|
||||
data = json.load(fh)
|
||||
|
||||
serve(data, args.port)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,755 @@
|
||||
# Learning from Experience: RL vs LLM In-Context Learning / 从经验中学习：RL 与 LLM 上下文学习对比
|
||||
|
||||
> Compares tabular Q-learning with LLM in-context learning on a treasure-hunt game with hidden mechanics (Shunyu Yao, “The Second Half”).
|
||||
> 代码位于第 1 章项目树；对应书中 **实验 7-1 ★（Q-learning 在寻宝游戏中的表现）** 与 **实验 7-2 ★★（传统 RL 与 LLM Agent 的对比研究）**。
|
||||
|
||||
← [Chapter 1 index / 返回第 1 章目录](../README.md) · 📖 [Read Chapter 7 / 读第 7 章正文](../../book/chapter7.md)（[EN](../../book-en/chapter7.md)）
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
### Overview
|
||||
|
||||
This experiment compares traditional Reinforcement Learning (Q-learning) with LLM-based in-context learning, replicating the key insights from Shunyu Yao's blog post ["The Second Half"](https://ysymyth.github.io/The-Second-Half/).
|
||||
|
||||
It demonstrates how LLMs can generalize through reasoning while traditional RL methods require extensive training to learn game mechanics. We use a text-based treasure hunt game with hidden mechanics that agents must discover through experience.
|
||||
|
||||
### Key Insights Being Tested
|
||||
|
||||
1. **Sample Efficiency**: LLMs can learn from far fewer examples than traditional RL
|
||||
2. **Generalization**: LLMs use reasoning to understand patterns, while RL memorizes state-action mappings
|
||||
3. **Prior Knowledge**: Language pre-training provides powerful priors for reasoning about new tasks
|
||||
4. **Hidden Mechanics Discovery**: LLMs can form hypotheses and test them, while RL requires exhaustive exploration
|
||||
|
||||
### What You'll See
|
||||
|
||||
When running the LLM experiment, you'll see the **complete decision-making process**:
|
||||
|
||||
```
|
||||
============================================================
|
||||
LLM DECISION PROCESS
|
||||
============================================================
|
||||
📊 Experiences in memory: 15
|
||||
🎮 Current room: hallway
|
||||
🎯 Available actions: 8
|
||||
|
||||
💡 Recent successful patterns learned:
|
||||
• take red key → +5.0 reward
|
||||
• try crafting → +10.0 reward
|
||||
|
||||
🤔 LLM is thinking...
|
||||
|
||||
📝 LLM Reasoning:
|
||||
----------------------------------------
|
||||
Based on my past experiences, I've learned that:
|
||||
1. The red key opens the locked door to the guard room
|
||||
2. Crafting rusty sword + magic crystal creates a silver sword
|
||||
3. The silver sword can defeat the strong guard
|
||||
|
||||
Since I have the silver sword and I'm in the hallway...
|
||||
----------------------------------------
|
||||
|
||||
✅ Chosen action: go north
|
||||
```
|
||||
|
||||
This transparency shows exactly how the LLM learns and reasons, unlike the black-box nature of Q-learning.
|
||||
|
||||
### The Game
|
||||
|
||||
A text-based treasure hunt game where agents must:
|
||||
|
||||
- Navigate through multiple rooms
|
||||
- Collect items and keys
|
||||
- Defeat guards using appropriate weapons
|
||||
- Discover hidden mechanics through experience
|
||||
|
||||
#### Hidden Mechanics (Not Revealed to Agents)
|
||||
|
||||
1. **Color-coded locks**: Specific colored keys open matching doors
|
||||
2. **Weapon effectiveness**: Different weapons work against different enemies
|
||||
3. **Crafting system**: Certain items combine to create better items
|
||||
4. **Potion effects**: Temporary abilities from consuming potions
|
||||
|
||||
### Quick Start
|
||||
|
||||
#### Installation
|
||||
|
||||
```bash
|
||||
# Recommended from the repository root: use the shared Chapter 1 environment
|
||||
uv sync --locked --extra ch1
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch1]"
|
||||
|
||||
# Enter this experiment directory for the commands below
|
||||
cd chapter1/learning-from-experience
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
- Q-learning runs fully offline with **no API key**.
|
||||
- The LLM path needs a Moonshot/Kimi API key (or OpenRouter fallback).
|
||||
|
||||
#### Setting up Kimi K3 API
|
||||
|
||||
To run the LLM experiments, you need a Kimi (Moonshot) API key:
|
||||
|
||||
1. Get your API key from [Moonshot AI](https://platform.moonshot.cn/)
|
||||
2. Set the environment variable:
|
||||
|
||||
```bash
|
||||
export LLM_PROVIDER="moonshot" # or dashscope/qwen/bailian
|
||||
export MOONSHOT_API_KEY="your-api-key-here"
|
||||
# For Alibaba Cloud Model Studio / Bailian (Qwen), use:
|
||||
# export LLM_PROVIDER="dashscope"
|
||||
# export DASHSCOPE_API_KEY="your-dashscope-api-key-here"
|
||||
# export DASHSCOPE_MODEL="qwen3.7-plus"
|
||||
```
|
||||
|
||||
Or create a `.env` file:
|
||||
|
||||
```bash
|
||||
echo "MOONSHOT_API_KEY=your-api-key-here" > .env
|
||||
```
|
||||
|
||||
**Universal OpenRouter fallback**: if `MOONSHOT_API_KEY` is unset but `OPENROUTER_API_KEY` is set, the LLM path routes through OpenRouter. Because Kimi models are not stably available on OpenRouter, the fallback uses `OPENROUTER_MODEL` (default `openai/gpt-5.6-luna`):
|
||||
|
||||
```bash
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python quick_demo.py # runs via OpenRouter when MOONSHOT_API_KEY is missing
|
||||
```
|
||||
|
||||
### Running the Experiment
|
||||
|
||||
#### Quick Demo (See LLM Learning in Action)
|
||||
|
||||
```bash
|
||||
python quick_demo.py
|
||||
```
|
||||
|
||||
This shows a detailed view of how the LLM learns through reasoning, displaying:
|
||||
|
||||
- Complete thought process for each decision
|
||||
- How experiences accumulate and influence future decisions
|
||||
- The dramatic difference in learning speed vs traditional RL
|
||||
|
||||
#### Command-Line Interface (`experiment.py`)
|
||||
|
||||
`experiment.py` provides a full CLI (Chinese help text). List all flags:
|
||||
|
||||
```bash
|
||||
python experiment.py --help
|
||||
```
|
||||
|
||||
Main parameters:
|
||||
|
||||
| Parameter | Description | Default |
|
||||
| --- | --- | --- |
|
||||
| `--mode {both,qlearning,rl,llm}` | Which agent(s): `qlearning`/`rl` = Q-learning only (offline), `llm` = LLM Agent only, `both` = comparison | `both` |
|
||||
| `--rl-episodes` | Q-learning training episodes (Experiment 7-1 uses 10000) | `10000` |
|
||||
| `--llm-episodes` | LLM Agent training episodes | `20` |
|
||||
| `--eval-episodes` | Greedy evaluation episodes after Q-learning | `100` |
|
||||
| `--checkpoint-interval` | Learning-curve sample interval (every N episodes) | `1000` |
|
||||
| `--model` | LLM model name (or `MOONSHOT_MODEL` env) | `kimi-k3` |
|
||||
| `--output` | Results output directory | `results` |
|
||||
| `--seed` | Random seed for reproducible Q-learning curves | unset |
|
||||
| `--learning-rate` / `--discount` / `--epsilon-decay` / `--epsilon-min` | Q-learning hyperparameters | `0.2 / 0.99 / 0.9995 / 0.1` |
|
||||
| `--stochastic` | Use stochastic environment | deterministic |
|
||||
| `--skip-llm` | Legacy alias for `--mode qlearning` | — |
|
||||
|
||||
#### Q-Learning Only (Experiment 7-1, offline, no API)
|
||||
|
||||
```bash
|
||||
python experiment.py --mode qlearning --rl-episodes 10000 --seed 42
|
||||
```
|
||||
|
||||
Training finishes in under ~3 seconds and prints a **learning curve table** showing how the agent goes from ~0% win rate to mastery over nearly 10k episodes (see Results below).
|
||||
|
||||
#### Full Comparison (RL vs LLM, Experiment 7-2)
|
||||
|
||||
```bash
|
||||
python experiment.py --mode both --model kimi-k3
|
||||
```
|
||||
|
||||
For the exact book protocol and acceptance-grade evidence, use the canonical
|
||||
runner. It executes the 10,000-episode Q-learning arm, 100 greedy evaluation
|
||||
episodes, and exactly one first-attempt official Moonshot Kimi K3 trajectory:
|
||||
|
||||
```bash
|
||||
python run_experiment_7_2.py
|
||||
```
|
||||
|
||||
The canonical runner rejects OpenRouter substitution, API errors, missing raw
|
||||
provider response IDs/content, and any parser fallback. It writes
|
||||
`validation/<timestamp>/evidence.json`; if only post-run serialization needs to
|
||||
be repaired, `finalize_experiment_7_2.py <campaign-dir>` finalizes the already
|
||||
saved raw campaign without repeating paid model calls.
|
||||
|
||||
This will:
|
||||
|
||||
1. Train a Q-learning agent for 10000 episodes (~3 seconds) and print its learning curve
|
||||
2. Train an LLM agent for 20 episodes with detailed reasoning display
|
||||
3. Evaluate both agents
|
||||
4. Generate comparison plots
|
||||
5. Save results to the `results/` directory
|
||||
|
||||
**Note**: `experiment.py` is the exploratory multi-episode runner. The canonical
|
||||
book campaign is deliberately one first attempt. The accepted 2026-07-30 Kimi
|
||||
K3 attempt took 416.11 seconds for 17 sequential reasoning calls; the earlier
|
||||
“1–2 minutes per game” estimate was not reproduced on this route.
|
||||
|
||||
#### LLM Only
|
||||
|
||||
```bash
|
||||
python experiment.py --mode llm --llm-episodes 20
|
||||
```
|
||||
|
||||
#### Interactive Game Play
|
||||
|
||||
Test the game manually:
|
||||
|
||||
```python
|
||||
from game_environment import TreasureHuntGame
|
||||
|
||||
game = TreasureHuntGame()
|
||||
print(game.get_state_description())
|
||||
print("Available actions:", game.get_available_actions())
|
||||
|
||||
# Try an action
|
||||
feedback, reward, done = game.execute_action("take rusty sword")
|
||||
print(f"Feedback: {feedback}")
|
||||
print(f"Reward: {reward}")
|
||||
```
|
||||
|
||||
### Validation
|
||||
|
||||
Install the `dev` extra from the repository root before running pytest in a clean environment:
|
||||
|
||||
```bash
|
||||
uv sync --locked --extra ch1 --extra dev
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
cd chapter1/learning-from-experience
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
The longer Q-learning learning-curve check is an offline manual smoke script, kept out of default pytest discovery:
|
||||
|
||||
```bash
|
||||
python tests/manual/rl_learning_check.py --episodes 1000
|
||||
```
|
||||
|
||||
### Experiment Results
|
||||
|
||||
#### Metrics Compared
|
||||
|
||||
1. **Sample Efficiency** — Episodes needed to achieve good performance; learning speed
|
||||
2. **Performance** — Victory rate in evaluation; average rewards and episode lengths
|
||||
3. **Computational Cost** — Training time; memory (Q-table size vs. experience storage); API calls for LLM
|
||||
|
||||
#### Visualizations
|
||||
|
||||
The experiment creates comparison plots showing:
|
||||
|
||||
- Learning curves over time
|
||||
- Victory rate progression
|
||||
- Sample efficiency comparison
|
||||
- Key insights summary
|
||||
|
||||
#### Expected Results
|
||||
|
||||
##### Q-Learning learning curve (measured locally, `--mode qlearning --rl-episodes 10000 --seed 42`)
|
||||
|
||||
Measured curve (deterministic env; victory rate as a sliding window over the last 1000 episodes; full training ~3s):
|
||||
|
||||
| Episodes | Victory rate | Q-table states | epsilon |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 1000 | 0.3% | 123 | 0.606 |
|
||||
| 2000 | 0.0% | 123 | 0.368 |
|
||||
| 3000 | 0.1% | 126 | 0.223 |
|
||||
| 5000 | 0.1% | 128 | 0.100 |
|
||||
| 7000 | 97.0% | 138 | 0.100 |
|
||||
| 8000 | 99.6% | 138 | 0.100 |
|
||||
| 9000 | 99.8% | 139 | 0.100 |
|
||||
| 10000 | **98.1%** | 142 | 0.100 |
|
||||
|
||||
After training, the canonical greedy evaluation reaches **100%** win rate,
|
||||
averaging 12 steps. The accepted Kimi K3 arm won on its first attempt in 17
|
||||
steps with 17/17 real responses, zero API errors, zero fallbacks, and 28,242
|
||||
tokens. This reproduces the first-attempt conclusion but not the manuscript's
|
||||
historical point estimates of exactly 18 Kimi steps and an 11-step Q-learning
|
||||
solution. See the [canonical evidence](validation/20260730_011704/evidence.json).
|
||||
|
||||
##### RL vs LLM (Experiment 7-2 conclusions)
|
||||
|
||||
- **Q-Learning**: Needs ~10000 episodes for stable clears; treats “door / key / sword” as meaningless symbols and only explores statistically.
|
||||
- **LLM In-Context**: Carries pretrained priors; often clears in the first episode within tens of steps by reasoning about game concepts.
|
||||
- **Sample Efficiency**: LLM is 2–3 orders of magnitude more sample-efficient; but per-episode inference is slow (~1–2 min API), while Q-learning finishes 10000 episodes in ~3s—trade-off depends on interaction cost (see book Experiment 7-2).
|
||||
|
||||
### Project Structure
|
||||
|
||||
```
|
||||
learning-from-experience/
|
||||
├── game_environment.py # Text-based game with hidden mechanics
|
||||
├── rl_agent.py # Q-learning implementation
|
||||
├── llm_agent.py # LLM with in-context learning
|
||||
├── experiment.py # Main experiment runner
|
||||
├── demo.py # Interactive local game demo
|
||||
├── quick_demo.py # Short LLM learning demo
|
||||
├── run_experiment_7_2.py # Exact real campaign + acceptance gates
|
||||
├── finalize_experiment_7_2.py # Evidence-only recovery; no API rerun
|
||||
├── env.example # Optional API-key template
|
||||
├── tests/
|
||||
│ ├── test_basic.py
|
||||
│ ├── test_zero_episodes.py
|
||||
│ ├── test_rl_progress_small_episodes.py
|
||||
│ └── manual/
|
||||
│ └── rl_learning_check.py
|
||||
├── requirements.txt # Python dependencies
|
||||
├── README.md # This file
|
||||
└── results/ # Experiment outputs (created on run)
|
||||
└── [timestamp]/
|
||||
├── rl_agent.pkl # Trained Q-learning agent
|
||||
├── llm_experiences.json # LLM's collected experiences
|
||||
├── experiment_results.json # Numerical results
|
||||
└── comparison_plots.png # Visualization
|
||||
```
|
||||
|
||||
### Technical Details
|
||||
|
||||
#### Q-Learning Agent
|
||||
|
||||
- **Algorithm**: Tabular Q-learning with ε-greedy exploration
|
||||
- **State Representation**: Hashed combination of room, inventory, and game state
|
||||
- **Learning Rate**: 0.2 (configurable via `--learning-rate`)
|
||||
- **Discount Factor**: 0.99 (configurable via `--discount`)
|
||||
- **Exploration**: ε starts at 1.0, decays by `--epsilon-decay` (0.9995) to `--epsilon-min` (0.1)
|
||||
|
||||
#### LLM Agent (Kimi K3)
|
||||
|
||||
- **Model**: `kimi-k3` (override with `--model` or `MOONSHOT_MODEL`)
|
||||
- **Reasoning model**: Kimi K3 emits a chain-of-thought (`message.reasoning_content`) before its final answer (`message.content`), so the code uses a generous `max_tokens=2048` to make sure the `ACTION:` line is not truncated by the reasoning budget.
|
||||
- **Learning Method**: In-context learning with experience memory (up to 50 experiences)
|
||||
- **Context Management**: Stores successful and failed experiences
|
||||
- **Reasoning**: Prompts LLM to reason about past experiences before acting
|
||||
- **Temperature**: requested 0.7, but reasoning models (Kimi K3, GPT-5) only accept `temperature=1`, so the code auto-forces `1` for those (see `_reasoning_safe_temperature`)
|
||||
|
||||
### Extending the Experiment
|
||||
|
||||
#### Ideas for Further Research
|
||||
|
||||
1. **Different Games**: Try other hidden-mechanic games
|
||||
2. **Hybrid Approaches**: Combine RL with LLM guidance
|
||||
3. **Transfer Learning**: Test how well agents transfer to similar games
|
||||
4. **Ablation Studies**: Remove reasoning prompts to isolate their impact
|
||||
5. **Other LLMs**: Compare different language models
|
||||
|
||||
#### Modifying the Game
|
||||
|
||||
Edit `game_environment.py` to:
|
||||
|
||||
- Add new rooms and items
|
||||
- Create more complex hidden mechanics
|
||||
- Adjust difficulty and rewards
|
||||
- Add new types of puzzles
|
||||
|
||||
### Educational Value
|
||||
|
||||
1. **The Power of Priors**: How language pre-training provides useful knowledge
|
||||
2. **Reasoning vs. Memorization**: Different approaches to learning
|
||||
3. **Sample Efficiency**: Why it matters for real-world applications
|
||||
4. **The Second Half Thesis**: Moving from “can we solve it?” to “how efficiently?”
|
||||
|
||||
### References
|
||||
|
||||
- [The Second Half](https://ysymyth.github.io/The-Second-Half/) by Shunyu Yao
|
||||
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629)
|
||||
- Original Q-learning paper: Watkins & Dayan (1992)
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 概述
|
||||
|
||||
本实验对比传统强化学习（Q-learning）与基于 LLM 的上下文学习（in-context learning），复现 Shunyu Yao 博客 [“The Second Half”](https://ysymyth.github.io/The-Second-Half/) 中的核心洞见。
|
||||
|
||||
目标：展示 LLM 如何通过**推理**泛化，而传统 RL 往往需要大量试错才能学到游戏机制。我们使用带有**隐藏机制**的文本寻宝游戏，智能体只能通过经验去发现规则。
|
||||
|
||||
代码在 `chapter1/learning-from-experience/`，对应书中**实验 7-1** 与 **实验 7-2**（正文见第 7 章）。
|
||||
|
||||
### 要验证的关键洞察
|
||||
|
||||
1. **样本效率**：LLM 用远少于传统 RL 的样例即可学习
|
||||
2. **泛化**：LLM 用推理理解模式；RL 记忆状态-动作映射
|
||||
3. **先验知识**：语言预训练为新任务推理提供强大先验
|
||||
4. **隐藏机制发现**：LLM 可形成假设并检验；RL 往往需要穷尽式探索
|
||||
|
||||
### 你会看到什么
|
||||
|
||||
运行 LLM 实验时，会看到**完整决策过程**：
|
||||
|
||||
```
|
||||
============================================================
|
||||
LLM DECISION PROCESS
|
||||
============================================================
|
||||
📊 Experiences in memory: 15
|
||||
🎮 Current room: hallway
|
||||
🎯 Available actions: 8
|
||||
|
||||
💡 Recent successful patterns learned:
|
||||
• take red key → +5.0 reward
|
||||
• try crafting → +10.0 reward
|
||||
|
||||
🤔 LLM is thinking...
|
||||
|
||||
📝 LLM Reasoning:
|
||||
----------------------------------------
|
||||
Based on my past experiences, I've learned that:
|
||||
1. The red key opens the locked door to the guard room
|
||||
2. Crafting rusty sword + magic crystal creates a silver sword
|
||||
3. The silver sword can defeat the strong guard
|
||||
|
||||
Since I have the silver sword and I'm in the hallway...
|
||||
----------------------------------------
|
||||
|
||||
✅ Chosen action: go north
|
||||
```
|
||||
|
||||
这种透明度展示了 LLM 如何学习与推理，有别于 Q-learning 的黑盒性质。
|
||||
|
||||
### 游戏说明
|
||||
|
||||
文本寻宝游戏，智能体需要：
|
||||
|
||||
- 在多个房间间导航
|
||||
- 收集物品与钥匙
|
||||
- 使用合适武器击败守卫
|
||||
- 通过经验发现隐藏机制
|
||||
|
||||
#### 隐藏机制（不对智能体公开）
|
||||
|
||||
1. **颜色锁**：特定颜色钥匙开对应门
|
||||
2. **武器有效性**：不同武器对不同敌人有效
|
||||
3. **合成系统**：特定物品可合成更强物品
|
||||
4. **药水效果**：消耗药水获得临时能力
|
||||
|
||||
### 快速开始
|
||||
|
||||
#### 安装
|
||||
|
||||
```bash
|
||||
# 推荐在仓库根目录使用统一的第 1 章环境
|
||||
uv sync --locked --extra ch1
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch1]"
|
||||
|
||||
# 进入本实验目录，后续命令都在这里运行
|
||||
cd chapter1/learning-from-experience
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
- Q-learning 完全离线，**无需任何 API Key**。
|
||||
- LLM 部分需要 Moonshot/Kimi API Key（或 OpenRouter 兜底）。
|
||||
|
||||
#### 配置 Kimi K3 API
|
||||
|
||||
运行 LLM 实验需要 Kimi（Moonshot）API Key：
|
||||
|
||||
1. 从 [Moonshot AI](https://platform.moonshot.cn/) 获取 Key
|
||||
2. 设置环境变量：
|
||||
|
||||
```bash
|
||||
export MOONSHOT_API_KEY="your-api-key-here"
|
||||
```
|
||||
|
||||
或创建 `.env`：
|
||||
|
||||
```bash
|
||||
echo "MOONSHOT_API_KEY=your-api-key-here" > .env
|
||||
```
|
||||
|
||||
**通用兜底（OpenRouter）**：若未设置 `MOONSHOT_API_KEY` 但设置了 `OPENROUTER_API_KEY`，LLM 部分会自动改走 OpenRouter。由于 Kimi 模型在 OpenRouter 上不稳定可用，兜底时会使用 `OPENROUTER_MODEL`（默认 `openai/gpt-5.6-luna`）：
|
||||
|
||||
```bash
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python quick_demo.py # MOONSHOT_API_KEY 缺失时自动经 OpenRouter 运行
|
||||
```
|
||||
|
||||
### 运行实验
|
||||
|
||||
#### 快速演示（观察 LLM 如何学习）
|
||||
|
||||
```bash
|
||||
python quick_demo.py
|
||||
```
|
||||
|
||||
会展示：
|
||||
|
||||
- 每一步的完整思考过程
|
||||
- 经验如何累积并影响后续决策
|
||||
- 与传统 RL 在学习速度上的巨大差异
|
||||
|
||||
#### 命令行接口（`experiment.py`）
|
||||
|
||||
`experiment.py` 提供带中文帮助的完整 CLI：
|
||||
|
||||
```bash
|
||||
python experiment.py --help
|
||||
```
|
||||
|
||||
主要参数：
|
||||
|
||||
| 参数 | 说明 | 默认值 |
|
||||
| --- | --- | --- |
|
||||
| `--mode {both,qlearning,rl,llm}` | 运行哪种智能体：`qlearning`/`rl` 只跑 Q-learning（离线）、`llm` 只跑 LLM Agent、`both` 两者对比 | `both` |
|
||||
| `--rl-episodes` | Q-learning 训练局数（实验 7-1 用 10000） | `10000` |
|
||||
| `--llm-episodes` | LLM Agent 训练局数 | `20` |
|
||||
| `--eval-episodes` | Q-learning 训练后贪婪评估局数 | `100` |
|
||||
| `--checkpoint-interval` | 学习曲线采样间隔（每 N 局记录一次胜率/Q 表规模） | `1000` |
|
||||
| `--model` | LLM 模型名（也可用 `MOONSHOT_MODEL` 环境变量） | `kimi-k3` |
|
||||
| `--output` | 结果输出目录 | `results` |
|
||||
| `--seed` | 随机种子，用于复现 Q-learning 学习曲线 | 不固定 |
|
||||
| `--learning-rate` / `--discount` / `--epsilon-decay` / `--epsilon-min` | Q-learning 超参数 | `0.2 / 0.99 / 0.9995 / 0.1` |
|
||||
| `--stochastic` | 使用随机环境 | 确定性 |
|
||||
| `--skip-llm` | 兼容旧用法，等价于 `--mode qlearning` | — |
|
||||
|
||||
#### 仅 Q-Learning（实验 7-1，离线，无需 API）
|
||||
|
||||
```bash
|
||||
python experiment.py --mode qlearning --rl-episodes 10000 --seed 42
|
||||
```
|
||||
|
||||
训练不到 3 秒即可完成，并打印**学习曲线表格**，直观展现智能体如何在近万局试错中从 0% 胜率逐步学会通关（见下文“实验结果”）。
|
||||
|
||||
#### 完整对比（RL vs LLM，实验 7-2）
|
||||
|
||||
```bash
|
||||
python experiment.py --mode both --model kimi-k3
|
||||
```
|
||||
|
||||
严格复现正文协议并生成可验收证据，请运行：
|
||||
|
||||
```bash
|
||||
python run_experiment_7_2.py
|
||||
```
|
||||
|
||||
该入口固定运行 10,000 局 Q-learning、100 局贪婪评估和且仅一局官方
|
||||
Moonshot `kimi-k3` 首次尝试；OpenRouter 替代、API 错误、缺失原始响应
|
||||
ID/正文或任何 parser fallback 都会使验收失败。若模型调用已经完成、仅证据
|
||||
序列化失败，可运行 `python finalize_experiment_7_2.py <campaign-dir>`，直接从
|
||||
已保留的原始结果完成证据，不重复付费调用。
|
||||
|
||||
流程：
|
||||
|
||||
1. 训练 Q-learning 10000 局（约 3 秒）并打印学习曲线
|
||||
2. 训练 LLM Agent 20 局，并展示详细推理
|
||||
3. 评估双方
|
||||
4. 生成对比图
|
||||
5. 结果写入 `results/`
|
||||
|
||||
**说明**：`experiment.py` 是多局探索入口；正文规范入口只测第一局。2026-07-30
|
||||
验收运行包含 17 次串行推理调用，共耗时 416.11 秒，因此没有复现旧版
|
||||
“每局 1–2 分钟”的估计。
|
||||
|
||||
#### 仅 LLM
|
||||
|
||||
```bash
|
||||
python experiment.py --mode llm --llm-episodes 20
|
||||
```
|
||||
|
||||
#### 交互式试玩
|
||||
|
||||
```python
|
||||
from game_environment import TreasureHuntGame
|
||||
|
||||
game = TreasureHuntGame()
|
||||
print(game.get_state_description())
|
||||
print("Available actions:", game.get_available_actions())
|
||||
|
||||
# Try an action
|
||||
feedback, reward, done = game.execute_action("take rusty sword")
|
||||
print(f"Feedback: {feedback}")
|
||||
print(f"Reward: {reward}")
|
||||
```
|
||||
|
||||
### 验证
|
||||
|
||||
在干净环境中运行 pytest 前，先在仓库根目录安装 `dev` extra：
|
||||
|
||||
```bash
|
||||
uv sync --locked --extra ch1 --extra dev
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
cd chapter1/learning-from-experience
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
较长的 Q-learning 学习曲线检查是离线手动 smoke 脚本，不会被默认 pytest 收集：
|
||||
|
||||
```bash
|
||||
python tests/manual/rl_learning_check.py --episodes 1000
|
||||
```
|
||||
|
||||
### 实验结果
|
||||
|
||||
#### 对比指标
|
||||
|
||||
1. **样本效率** — 达到良好表现所需局数、学习速度
|
||||
2. **性能** — 评估胜率、平均回报与回合长度
|
||||
3. **计算成本** — 训练时间；内存（Q 表规模 vs. 经验存储）；LLM 的 API 调用
|
||||
|
||||
#### 可视化
|
||||
|
||||
实验会生成对比图，包括：
|
||||
|
||||
- 随时间的学习曲线
|
||||
- 胜率演进
|
||||
- 样本效率对比
|
||||
- 关键洞察摘要
|
||||
|
||||
#### 预期结果
|
||||
|
||||
##### Q-learning 学习曲线（本地实测，`--mode qlearning --rl-episodes 10000 --seed 42`）
|
||||
|
||||
实测学习曲线（确定性环境，胜率按最近 1000 局滑动窗口统计，整段训练约 3 秒）：
|
||||
|
||||
| Episodes | Victory rate | Q-table states | epsilon |
|
||||
| ---: | ---: | ---: | ---: |
|
||||
| 1000 | 0.3% | 123 | 0.606 |
|
||||
| 2000 | 0.0% | 123 | 0.368 |
|
||||
| 3000 | 0.1% | 126 | 0.223 |
|
||||
| 5000 | 0.1% | 128 | 0.100 |
|
||||
| 7000 | 97.0% | 138 | 0.100 |
|
||||
| 8000 | 99.6% | 138 | 0.100 |
|
||||
| 9000 | 99.8% | 139 | 0.100 |
|
||||
| 10000 | **98.1%** | 142 | 0.100 |
|
||||
|
||||
规范运行训练后 100 局贪婪评估胜率为 **100%**，平均 12 步通关；Kimi K3
|
||||
第一局 17 步通关，保留 17/17 条官方响应，零 API 错误、零 fallback，共
|
||||
28,242 tokens。它复现了“第一局成功”的实质结论，但没有复现历史记录中的
|
||||
Kimi 恰好 18 步和 Q-learning 恰好 11 步。详见[规范证据](validation/20260730_011704/evidence.json)。
|
||||
|
||||
##### RL vs LLM（实验 7-2 的对比结论）
|
||||
|
||||
- **Q-Learning**：需要近 10000 局才达到稳定通关；把“门/钥匙/剑”当作无意义符号，只能靠统计式暴力探索。
|
||||
- **LLM In-Context**：携带预训练先验，往往第一局就能在十几步内通关；靠推理理解游戏概念结构。
|
||||
- **样本效率**：LLM 高出 2–3 个数量级；但单局推理慢（API 调用 ~1–2 分钟），Q-learning 跑 10000 局只需约 3 秒——权衡取决于交互成本，详见书中实验 7-2。
|
||||
|
||||
### 项目结构
|
||||
|
||||
```
|
||||
learning-from-experience/
|
||||
├── game_environment.py # Text-based game with hidden mechanics
|
||||
├── rl_agent.py # Q-learning implementation
|
||||
├── llm_agent.py # LLM with in-context learning
|
||||
├── experiment.py # Main experiment runner
|
||||
├── demo.py # Interactive local game demo
|
||||
├── quick_demo.py # Short LLM learning demo
|
||||
├── run_experiment_7_2.py # 正文规范实测与验收门
|
||||
├── finalize_experiment_7_2.py # 仅补写证据，不重复 API 调用
|
||||
├── env.example # Optional API-key template
|
||||
├── tests/
|
||||
│ ├── test_basic.py
|
||||
│ ├── test_zero_episodes.py
|
||||
│ ├── test_rl_progress_small_episodes.py
|
||||
│ └── manual/
|
||||
│ └── rl_learning_check.py
|
||||
├── requirements.txt # Python dependencies
|
||||
├── README.md # This file
|
||||
└── results/ # Experiment outputs (created on run)
|
||||
└── [timestamp]/
|
||||
├── rl_agent.pkl # Trained Q-learning agent
|
||||
├── llm_experiences.json # LLM's collected experiences
|
||||
├── experiment_results.json # Numerical results
|
||||
└── comparison_plots.png # Visualization
|
||||
```
|
||||
|
||||
### 技术细节
|
||||
|
||||
#### Q-Learning Agent
|
||||
|
||||
- **算法**：表格 Q-learning + ε-贪婪探索
|
||||
- **状态表示**：房间、背包与游戏状态的哈希组合
|
||||
- **学习率**：0.2（`--learning-rate`）
|
||||
- **折扣因子**：0.99（`--discount`）
|
||||
- **探索**：ε 从 1.0 起，按 `--epsilon-decay`（0.9995）衰减到 `--epsilon-min`（0.1）
|
||||
|
||||
#### LLM Agent（Kimi K3）
|
||||
|
||||
- **模型**：`kimi-k3`（可用 `--model` 或 `MOONSHOT_MODEL` 覆盖）
|
||||
- **推理模型**：Kimi K3 会在最终答案（`message.content`）前输出思维链（`message.reasoning_content`），因此代码使用较大的 `max_tokens=2048`，避免 `ACTION:` 行被思考预算截断。
|
||||
- **学习方式**：上下文学习 + 经验记忆（最多 50 条）
|
||||
- **上下文管理**：存储成功与失败经验
|
||||
- **推理**：行动前提示模型基于过往经验推理
|
||||
- **Temperature**：请求 0.7，但推理模型（Kimi K3、GPT-5）只接受 `temperature=1`，代码会自动强制为 `1`（见 `_reasoning_safe_temperature`）
|
||||
|
||||
### 扩展实验
|
||||
|
||||
#### 进一步研究思路
|
||||
|
||||
1. **不同游戏**：尝试其他隐藏机制游戏
|
||||
2. **混合方法**：RL 与 LLM 引导结合
|
||||
3. **迁移学习**：测试向相似游戏的迁移
|
||||
4. **消融研究**：去掉推理提示以隔离其影响
|
||||
5. **其他 LLM**：对比不同语言模型
|
||||
|
||||
#### 修改游戏
|
||||
|
||||
编辑 `game_environment.py` 可：
|
||||
|
||||
- 增加房间与物品
|
||||
- 设计更复杂的隐藏机制
|
||||
- 调整难度与奖励
|
||||
- 加入新类型谜题
|
||||
|
||||
### 教学价值
|
||||
|
||||
1. **先验的力量**：语言预训练如何提供有用知识
|
||||
2. **推理 vs. 记忆**：不同学习路径
|
||||
3. **样本效率**：为何对现实任务重要
|
||||
4. **“The Second Half” 论点**：从“能否解决”转向“多高效”
|
||||
|
||||
### 参考文献
|
||||
|
||||
- [The Second Half](https://ysymyth.github.io/The-Second-Half/) — Shunyu Yao
|
||||
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629)
|
||||
- Q-learning 原文：Watkins & Dayan (1992)
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Project type: **✅ standalone runnable** (Q-learning offline; LLM needs API key).
|
||||
项目类型：**✅ 可独立运行**（Q-learning 离线；LLM 需 API Key）。
|
||||
- For educational purposes; inspired by academic work on AI and RL.
|
||||
教学用途，灵感来自 AI 与强化学习相关研究。
|
||||
- Feel free to add mechanics, other RL algorithms (DQN, PPO, …), providers, or richer metrics.
|
||||
欢迎增加隐藏机制、其他 RL 算法、提供商或更完善的评估指标。
|
||||
@@ -0,0 +1,217 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Interactive demo to play the game manually or watch agents play.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.append(str(Path(__file__).parent))
|
||||
|
||||
from game_environment import TreasureHuntGame
|
||||
from rl_agent import QLearningAgent
|
||||
from llm_agent import LLMAgent
|
||||
|
||||
|
||||
def play_manual():
|
||||
"""Let the user play the game manually."""
|
||||
print("\n" + "="*60)
|
||||
print("MANUAL PLAY MODE")
|
||||
print("="*60)
|
||||
print("\nYou are playing the treasure hunt game!")
|
||||
print("Try to find the dragon's treasure by exploring and discovering hidden mechanics.")
|
||||
|
||||
game = TreasureHuntGame()
|
||||
|
||||
while not game.game_over:
|
||||
print("\n" + "-"*40)
|
||||
print(game.get_state_description())
|
||||
print("\nAvailable actions:")
|
||||
actions = game.get_available_actions()
|
||||
for i, action in enumerate(actions, 1):
|
||||
print(f" {i}. {action}")
|
||||
|
||||
# Get user input
|
||||
choice = input("\nEnter action number or type custom action: ").strip()
|
||||
|
||||
# Parse input
|
||||
if choice.isdigit() and 1 <= int(choice) <= len(actions):
|
||||
action = actions[int(choice) - 1]
|
||||
else:
|
||||
action = choice
|
||||
|
||||
# Execute action
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
print(f"\nFeedback: {feedback}")
|
||||
print(f"Reward: {reward:.2f}")
|
||||
|
||||
if game.victory:
|
||||
print("\n🎉 CONGRATULATIONS! You won!")
|
||||
else:
|
||||
print("\n💀 GAME OVER! Better luck next time.")
|
||||
|
||||
print(f"Final score: {game.score}")
|
||||
|
||||
|
||||
def watch_rl_agent():
|
||||
"""Watch a trained RL agent play."""
|
||||
print("\n" + "="*60)
|
||||
print("WATCHING Q-LEARNING AGENT")
|
||||
print("="*60)
|
||||
|
||||
# Check if trained agent exists
|
||||
agent_path = Path("results") / "rl_agent_demo.pkl"
|
||||
|
||||
agent = QLearningAgent()
|
||||
|
||||
if agent_path.exists():
|
||||
print("Loading pre-trained agent...")
|
||||
agent.load(agent_path)
|
||||
else:
|
||||
print("No pre-trained agent found. Training one now...")
|
||||
print("This will take a few minutes...\n")
|
||||
|
||||
game = TreasureHuntGame()
|
||||
agent.train(num_episodes=2000, verbose=True)
|
||||
|
||||
# Save for future use
|
||||
agent_path.parent.mkdir(exist_ok=True)
|
||||
agent.save(agent_path)
|
||||
|
||||
# Watch agent play
|
||||
print("\nWatching agent play...")
|
||||
game = TreasureHuntGame()
|
||||
total_reward = 0
|
||||
steps = 0
|
||||
|
||||
while not game.game_over:
|
||||
print("\n" + "-"*40)
|
||||
print(game.get_state_description())
|
||||
|
||||
action = agent.choose_action(game, training=False)
|
||||
print(f"\nAgent chooses: {action}")
|
||||
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
print(f"Feedback: {feedback}")
|
||||
print(f"Reward: {reward:.2f}")
|
||||
|
||||
total_reward += reward
|
||||
steps += 1
|
||||
|
||||
input("\nPress Enter to continue...")
|
||||
|
||||
if game.victory:
|
||||
print("\n🎉 Agent won!")
|
||||
else:
|
||||
print("\n💀 Agent failed.")
|
||||
|
||||
print(f"Total reward: {total_reward:.2f}")
|
||||
print(f"Steps taken: {steps}")
|
||||
|
||||
|
||||
def watch_llm_agent():
|
||||
"""Watch an LLM agent play with reasoning."""
|
||||
print("\n" + "="*60)
|
||||
print("WATCHING LLM AGENT (with reasoning)")
|
||||
print("="*60)
|
||||
|
||||
# Check API key
|
||||
provider = os.getenv("LLM_PROVIDER", "moonshot").lower()
|
||||
api_key = os.getenv("DASHSCOPE_API_KEY") if provider in {"dashscope", "qwen", "bailian"} else os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key and not os.getenv("OPENROUTER_API_KEY"):
|
||||
print(f"\nError: API key for provider '{provider}' not set.")
|
||||
print("Please set your Kimi API key:")
|
||||
print(" export DASHSCOPE_API_KEY='your-key-here' # for dashscope/qwen/bailian")
|
||||
print(" export MOONSHOT_API_KEY='your-key-here' # for moonshot/kimi")
|
||||
print("Or set OPENROUTER_API_KEY as a universal fallback.")
|
||||
return
|
||||
|
||||
agent = LLMAgent(api_key=api_key, provider=provider)
|
||||
|
||||
# Load experiences if available
|
||||
exp_path = Path("results") / "llm_experiences_demo.json"
|
||||
if exp_path.exists():
|
||||
print("Loading previous experiences...")
|
||||
agent.load_experiences(exp_path)
|
||||
print(f"Loaded {len(agent.experiences)} experiences")
|
||||
|
||||
# Play one episode with verbose output
|
||||
print("\nWatching LLM agent play with reasoning...")
|
||||
print("(The agent will explain its thought process)\n")
|
||||
|
||||
game = TreasureHuntGame()
|
||||
reward, steps, victory = agent.play_episode(game, verbose=True)
|
||||
|
||||
if victory:
|
||||
print("\n🎉 LLM agent won!")
|
||||
else:
|
||||
print("\n💀 LLM agent failed.")
|
||||
|
||||
print(f"Total reward: {reward:.2f}")
|
||||
print(f"Steps taken: {steps}")
|
||||
print(f"API calls made: {agent.api_calls}")
|
||||
|
||||
# Save experiences
|
||||
exp_path.parent.mkdir(exist_ok=True)
|
||||
agent.save_experiences(exp_path)
|
||||
|
||||
|
||||
def show_hidden_rules():
|
||||
"""Reveal the hidden game mechanics."""
|
||||
print("\n" + "="*60)
|
||||
print("HIDDEN GAME MECHANICS (SPOILERS!)")
|
||||
print("="*60)
|
||||
|
||||
game = TreasureHuntGame()
|
||||
print(game.get_hidden_rules())
|
||||
|
||||
print("\nThese are the rules that agents must discover through experience.")
|
||||
print("Traditional RL requires thousands of episodes to learn these patterns,")
|
||||
print("while LLMs can often figure them out in just 20-30 episodes through reasoning.")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main menu for the demo."""
|
||||
while True:
|
||||
print("\n" + "="*70)
|
||||
print("LEARNING FROM EXPERIENCE DEMO")
|
||||
print("Comparing RL vs LLM In-Context Learning")
|
||||
print("="*70)
|
||||
|
||||
print("\nChoose an option:")
|
||||
print("1. Play the game manually")
|
||||
print("2. Watch Q-Learning agent play (pre-trained)")
|
||||
print("3. Watch LLM agent play with reasoning")
|
||||
print("4. Show hidden game mechanics (spoilers!)")
|
||||
print("5. Run full experiment")
|
||||
print("6. Exit")
|
||||
|
||||
choice = input("\nEnter your choice (1-6): ").strip()
|
||||
|
||||
if choice == "1":
|
||||
play_manual()
|
||||
elif choice == "2":
|
||||
watch_rl_agent()
|
||||
elif choice == "3":
|
||||
watch_llm_agent()
|
||||
elif choice == "4":
|
||||
show_hidden_rules()
|
||||
elif choice == "5":
|
||||
print("\nRunning full experiment...")
|
||||
os.system("python experiment.py")
|
||||
elif choice == "6":
|
||||
print("\nGoodbye!")
|
||||
break
|
||||
else:
|
||||
print("\nInvalid choice. Please try again.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
# Provider: moonshot (default) or dashscope/qwen/bailian
|
||||
LLM_PROVIDER=moonshot
|
||||
|
||||
# Kimi (Moonshot) API Configuration
|
||||
# Get your API key from: https://platform.moonshot.cn/
|
||||
MOONSHOT_API_KEY=your-api-key-here
|
||||
|
||||
# Alibaba Cloud Model Studio / Bailian (Qwen)
|
||||
# DASHSCOPE_API_KEY=your-dashscope-api-key-here
|
||||
# DASHSCOPE_MODEL=qwen3.7-plus
|
||||
# DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
|
||||
# Optional: Customize model (default: kimi-k3)
|
||||
# MOONSHOT_MODEL=kimi-k3
|
||||
|
||||
# Universal fallback: if MOONSHOT_API_KEY is missing but OPENROUTER_API_KEY is
|
||||
# set, the agent routes through OpenRouter. Note: Kimi models are not reliably
|
||||
# available on OpenRouter, so the fallback uses OPENROUTER_MODEL (default
|
||||
# openai/gpt-5.6-luna) instead.
|
||||
# OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
# OPENROUTER_MODEL=openai/gpt-5.6-luna
|
||||
@@ -0,0 +1,655 @@
|
||||
"""
|
||||
Experiment runner to compare traditional RL vs LLM-based in-context learning.
|
||||
This replicates the key insights from "The Second Half" blog post.
|
||||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
import random
|
||||
import argparse
|
||||
from datetime import datetime
|
||||
from typing import Dict, Any, List
|
||||
import numpy as np
|
||||
import matplotlib.pyplot as plt
|
||||
import seaborn as sns
|
||||
from pathlib import Path
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
from game_environment import TreasureHuntGame
|
||||
from rl_agent import QLearningAgent
|
||||
from llm_agent import LLMAgent
|
||||
|
||||
|
||||
class ExperimentRunner:
|
||||
"""
|
||||
Runs experiments comparing different learning approaches.
|
||||
"""
|
||||
|
||||
def __init__(self, results_dir: str = "results"):
|
||||
"""Initialize experiment runner."""
|
||||
self.results_dir = Path(results_dir)
|
||||
self.results_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create timestamp for this experiment run
|
||||
self.timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
self.experiment_dir = self.results_dir / self.timestamp
|
||||
self.experiment_dir.mkdir(exist_ok=True)
|
||||
|
||||
self.results = {}
|
||||
|
||||
def run_rl_experiment(self,
|
||||
num_training_episodes: int = 10000,
|
||||
num_eval_episodes: int = 100,
|
||||
verbose: bool = True,
|
||||
stochastic: bool = False,
|
||||
learning_rate: float = 0.2,
|
||||
discount_factor: float = 0.99,
|
||||
epsilon_decay: float = 0.9995,
|
||||
epsilon_min: float = 0.1,
|
||||
checkpoint_interval: int = 1000) -> Dict[str, Any]:
|
||||
"""
|
||||
Run experiment with traditional Q-learning agent.
|
||||
|
||||
Args:
|
||||
num_training_episodes: Number of episodes to train
|
||||
num_eval_episodes: Number of episodes to evaluate
|
||||
verbose: Whether to print details
|
||||
stochastic: Whether to use stochastic environment
|
||||
learning_rate: Q-learning learning rate (alpha)
|
||||
discount_factor: Discount factor (gamma)
|
||||
epsilon_decay: Per-episode epsilon decay
|
||||
epsilon_min: Minimum exploration rate
|
||||
checkpoint_interval: Record/print a learning-curve row every N episodes
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TRADITIONAL RL EXPERIMENT (Q-Learning)")
|
||||
print("="*60)
|
||||
|
||||
# Initialize agent with the given hyperparameters
|
||||
agent = QLearningAgent(
|
||||
learning_rate=learning_rate,
|
||||
discount_factor=discount_factor,
|
||||
epsilon=1.0,
|
||||
epsilon_decay=epsilon_decay,
|
||||
epsilon_min=epsilon_min
|
||||
)
|
||||
|
||||
# Training phase
|
||||
print(f"\nTraining for {num_training_episodes} episodes...")
|
||||
start_time = time.time()
|
||||
|
||||
train_results = agent.train(
|
||||
num_episodes=num_training_episodes,
|
||||
verbose=verbose,
|
||||
stochastic=stochastic,
|
||||
checkpoint_interval=checkpoint_interval
|
||||
)
|
||||
|
||||
training_time = time.time() - start_time
|
||||
|
||||
# Show the learning curve (success rate over episodes) -- this is the
|
||||
# core point of experiment 7-1: the agent slowly LEARNS from experience.
|
||||
self._print_learning_curve(train_results.get("learning_curve", []),
|
||||
checkpoint_interval)
|
||||
|
||||
# Evaluation phase
|
||||
print(f"\nEvaluating on {num_eval_episodes} episodes...")
|
||||
eval_results = agent.evaluate(
|
||||
num_episodes=num_eval_episodes,
|
||||
verbose=False,
|
||||
stochastic=stochastic
|
||||
)
|
||||
|
||||
# Compile results
|
||||
results = {
|
||||
"method": "Q-Learning",
|
||||
"training_episodes": num_training_episodes,
|
||||
"training_time": training_time,
|
||||
"q_table_size": train_results["q_table_size"],
|
||||
"training_victories": train_results["total_victories"],
|
||||
"training_victory_rate": train_results["victory_rate"],
|
||||
"eval_victories": eval_results["victories"],
|
||||
"eval_victory_rate": eval_results["victory_rate"],
|
||||
"eval_avg_reward": eval_results["avg_reward"],
|
||||
"eval_avg_steps": eval_results["avg_length"],
|
||||
"episode_rewards": train_results["episode_rewards"],
|
||||
"episode_lengths": train_results["episode_lengths"],
|
||||
"learning_curve": train_results.get("learning_curve", [])
|
||||
}
|
||||
|
||||
# Save agent
|
||||
agent.save(self.experiment_dir / "rl_agent.pkl")
|
||||
|
||||
print(f"\nRL Training Summary:")
|
||||
print(f" Training time: {training_time:.2f} seconds")
|
||||
print(f" Q-table size: {train_results['q_table_size']} states")
|
||||
print(f" Training victory rate: {train_results['victory_rate']:.2%}")
|
||||
print(f" Evaluation victory rate: {eval_results['victory_rate']:.2%}")
|
||||
|
||||
return results
|
||||
|
||||
def _print_learning_curve(self, learning_curve: List[Dict[str, Any]],
|
||||
checkpoint_interval: int):
|
||||
"""Print the Q-learning success-rate-over-episodes table.
|
||||
|
||||
This is the whole point of experiment 7-1: watch the victory rate climb
|
||||
from 0% (blind exploration) to ~100% only after thousands of episodes.
|
||||
"""
|
||||
if not learning_curve:
|
||||
return
|
||||
|
||||
window = checkpoint_interval if checkpoint_interval > 0 else 1000
|
||||
print("\n" + "-"*60)
|
||||
print(f"LEARNING CURVE (Q-Learning success rate over episodes)")
|
||||
print(f"胜率按最近 {window} 局的滑动窗口统计")
|
||||
print("-"*60)
|
||||
print(f"{'Episodes':>10} | {'Victory rate':>12} | {'Q-table':>8} | {'epsilon':>8}")
|
||||
print(f"{'-'*10}-+-{'-'*12}-+-{'-'*8}-+-{'-'*8}")
|
||||
for row in learning_curve:
|
||||
print(f"{row['episode']:>10} | {row['victory_rate']*100:>11.1f}% | "
|
||||
f"{row['q_table_size']:>8} | {row['epsilon']:>8.3f}")
|
||||
print("-"*60)
|
||||
|
||||
def run_llm_experiment(self,
|
||||
num_training_episodes: int = 20,
|
||||
num_eval_episodes: int = 10,
|
||||
verbose: bool = True,
|
||||
stochastic: bool = False,
|
||||
model: str = "kimi-k3") -> Dict[str, Any]:
|
||||
"""
|
||||
Run experiment with LLM-based in-context learning agent.
|
||||
|
||||
Args:
|
||||
num_training_episodes: Number of episodes to train
|
||||
num_eval_episodes: Number of episodes to evaluate
|
||||
verbose: Whether to print details
|
||||
stochastic: Whether to use stochastic environment
|
||||
"""
|
||||
print("\n" + "="*70)
|
||||
print(f"LLM-BASED IN-CONTEXT LEARNING EXPERIMENT ({model})")
|
||||
print("="*70)
|
||||
|
||||
# Check for API key
|
||||
provider = os.getenv("LLM_PROVIDER", "moonshot").lower()
|
||||
api_key = os.getenv("DASHSCOPE_API_KEY") if provider in {"dashscope", "qwen", "bailian"} else os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key and not os.getenv("OPENROUTER_API_KEY"):
|
||||
print(f"\n⚠️ Warning: API key for provider '{provider}' not set. Skipping LLM experiment.")
|
||||
print("📝 Set DASHSCOPE_API_KEY for dashscope/qwen/bailian or MOONSHOT_API_KEY for moonshot/kimi")
|
||||
print("🔗 Get your key at: https://platform.moonshot.cn/")
|
||||
print("💡 Or set OPENROUTER_API_KEY as a universal fallback.")
|
||||
return None
|
||||
|
||||
print("\n✅ API key found. Initializing LLM agent...")
|
||||
print(f"🧠 Using {model} model for reasoning and in-context learning")
|
||||
print("📖 The LLM will show its complete thought process for each decision")
|
||||
|
||||
# Initialize agent
|
||||
agent = LLMAgent(
|
||||
api_key=api_key,
|
||||
model=model,
|
||||
provider=provider,
|
||||
temperature=0.7,
|
||||
max_experiences=50
|
||||
)
|
||||
|
||||
# Training phase (experience collection)
|
||||
print(f"\n🎓 Training Phase: Playing {num_training_episodes} episodes")
|
||||
print("💡 Watch how the LLM learns from experience without any parameter updates!")
|
||||
print("-"*70)
|
||||
|
||||
start_time = time.time()
|
||||
|
||||
train_results = agent.train(
|
||||
num_episodes=num_training_episodes,
|
||||
verbose=verbose,
|
||||
stochastic=stochastic
|
||||
)
|
||||
|
||||
training_time = time.time() - start_time
|
||||
|
||||
# Evaluation phase
|
||||
print(f"\nEvaluating on {num_eval_episodes} episodes...")
|
||||
eval_results = agent.evaluate(
|
||||
num_episodes=num_eval_episodes,
|
||||
verbose=False,
|
||||
stochastic=stochastic
|
||||
)
|
||||
|
||||
# Compile results
|
||||
results = {
|
||||
"method": "LLM In-Context Learning",
|
||||
"provider": agent.provider,
|
||||
"base_url": agent.base_url,
|
||||
"model": agent.model,
|
||||
"using_openrouter": agent.using_openrouter,
|
||||
"training_episodes": num_training_episodes,
|
||||
"evaluation_episodes": num_eval_episodes,
|
||||
"training_time": training_time,
|
||||
"experiences_collected": train_results["experiences_collected"],
|
||||
"api_calls": agent.api_calls,
|
||||
"api_attempts": len(agent.api_records),
|
||||
"api_errors": sum(1 for item in agent.api_records if item.get("error")),
|
||||
"fallback_actions": sum(
|
||||
1 for item in agent.api_records if item.get("fallback_used")
|
||||
),
|
||||
"total_tokens": agent.total_tokens,
|
||||
"training_victories": train_results["total_victories"],
|
||||
"training_victory_rate": train_results["victory_rate"],
|
||||
"eval_victories": eval_results["victories"],
|
||||
"eval_victory_rate": eval_results["victory_rate"],
|
||||
"eval_avg_reward": eval_results["avg_reward"],
|
||||
"eval_avg_steps": eval_results["avg_length"],
|
||||
"episode_rewards": train_results["episode_rewards"],
|
||||
"episode_lengths": train_results["episode_lengths"],
|
||||
"training_trajectories": [
|
||||
item for item in agent.episode_trajectories
|
||||
if item["phase"] == "training"
|
||||
],
|
||||
}
|
||||
|
||||
# Save experiences
|
||||
agent.save_experiences(self.experiment_dir / "llm_experiences.json")
|
||||
|
||||
print(f"\nLLM Training Summary:")
|
||||
print(f" Training time: {training_time:.2f} seconds")
|
||||
print(f" Experiences collected: {train_results['experiences_collected']}")
|
||||
print(f" API calls: {train_results['total_api_calls']}")
|
||||
print(f" Training victory rate: {train_results['victory_rate']:.2%}")
|
||||
print(f" Evaluation victory rate: {eval_results['victory_rate']:.2%}")
|
||||
|
||||
return results
|
||||
|
||||
def compare_learning_curves(self, rl_results: Dict, llm_results: Dict):
|
||||
"""
|
||||
Create visualization comparing learning curves of both methods.
|
||||
"""
|
||||
fig, axes = plt.subplots(2, 2, figsize=(15, 10))
|
||||
|
||||
# Plot 1: Victory rate over episodes
|
||||
ax = axes[0, 0]
|
||||
|
||||
# RL victory rate (computed over windows)
|
||||
rl_rewards = rl_results["episode_rewards"]
|
||||
window_size = 100
|
||||
rl_victories = []
|
||||
for i in range(0, len(rl_rewards), window_size):
|
||||
window = rl_rewards[i:i+window_size]
|
||||
victories = sum(1 for r in window if r > 50) / len(window)
|
||||
rl_victories.append(victories)
|
||||
|
||||
ax.plot(range(0, len(rl_rewards), window_size), rl_victories,
|
||||
label=f"Q-Learning ({len(rl_rewards)} episodes)", linewidth=2)
|
||||
|
||||
# LLM victory rate (per episode)
|
||||
if llm_results:
|
||||
llm_rewards = llm_results["episode_rewards"]
|
||||
llm_victories = [1 if r > 50 else 0 for r in llm_rewards]
|
||||
llm_cumulative = np.cumsum(llm_victories) / (np.arange(len(llm_victories)) + 1)
|
||||
ax.plot(range(len(llm_cumulative)), llm_cumulative,
|
||||
label=f"LLM In-Context ({len(llm_rewards)} episodes)", linewidth=2)
|
||||
|
||||
ax.set_xlabel("Episodes")
|
||||
ax.set_ylabel("Victory Rate")
|
||||
ax.set_title("Learning Progress: Victory Rate Over Time")
|
||||
ax.legend()
|
||||
ax.grid(True, alpha=0.3)
|
||||
|
||||
# Plot 2: Average reward over episodes
|
||||
ax = axes[0, 1]
|
||||
|
||||
# RL rewards (smoothed)
|
||||
rl_smooth = []
|
||||
for i in range(0, len(rl_rewards), window_size):
|
||||
window = rl_rewards[i:i+window_size]
|
||||
rl_smooth.append(np.mean(window))
|
||||
|
||||
ax.plot(range(0, len(rl_rewards), window_size), rl_smooth,
|
||||
label="Q-Learning", linewidth=2)
|
||||
|
||||
# LLM rewards
|
||||
if llm_results:
|
||||
llm_rewards = llm_results["episode_rewards"]
|
||||
ax.plot(range(len(llm_rewards)), llm_rewards,
|
||||
label="LLM In-Context", linewidth=2, alpha=0.7)
|
||||
|
||||
ax.set_xlabel("Episodes")
|
||||
ax.set_ylabel("Episode Reward")
|
||||
ax.set_title("Learning Progress: Reward Over Time")
|
||||
ax.legend()
|
||||
ax.grid(True, alpha=0.3)
|
||||
|
||||
# Plot 3: Sample efficiency comparison
|
||||
ax = axes[1, 0]
|
||||
|
||||
categories = ["Training\nEpisodes", "Evaluation\nVictory Rate", "Training\nTime (s)"]
|
||||
rl_values = [
|
||||
rl_results["training_episodes"],
|
||||
rl_results["eval_victory_rate"] * 100,
|
||||
rl_results["training_time"]
|
||||
]
|
||||
|
||||
if llm_results:
|
||||
llm_values = [
|
||||
llm_results["training_episodes"],
|
||||
llm_results["eval_victory_rate"] * 100,
|
||||
llm_results["training_time"]
|
||||
]
|
||||
else:
|
||||
llm_values = [0, 0, 0]
|
||||
|
||||
x = np.arange(len(categories))
|
||||
width = 0.35
|
||||
|
||||
bars1 = ax.bar(x - width/2, rl_values, width, label='Q-Learning')
|
||||
bars2 = ax.bar(x + width/2, llm_values, width, label='LLM In-Context')
|
||||
|
||||
ax.set_ylabel('Value')
|
||||
ax.set_title('Sample Efficiency Comparison')
|
||||
ax.set_xticks(x)
|
||||
ax.set_xticklabels(categories)
|
||||
ax.legend()
|
||||
|
||||
# Add value labels on bars
|
||||
for bars in [bars1, bars2]:
|
||||
for bar in bars:
|
||||
height = bar.get_height()
|
||||
ax.annotate(f'{height:.1f}',
|
||||
xy=(bar.get_x() + bar.get_width() / 2, height),
|
||||
xytext=(0, 3),
|
||||
textcoords="offset points",
|
||||
ha='center', va='bottom')
|
||||
|
||||
# Plot 4: Key insights text
|
||||
ax = axes[1, 1]
|
||||
ax.axis('off')
|
||||
|
||||
insights = [
|
||||
"KEY INSIGHTS (Replicating 'The Second Half' Findings):",
|
||||
"",
|
||||
"1. SAMPLE EFFICIENCY:",
|
||||
f" • Q-Learning: {rl_results['training_episodes']} episodes needed",
|
||||
f" • LLM: {llm_results['training_episodes'] if llm_results else 'N/A'} episodes needed",
|
||||
f" • Improvement: {rl_results['training_episodes'] / (llm_results['training_episodes'] if llm_results and llm_results['training_episodes'] > 0 else 1):.1f}x fewer samples",
|
||||
"",
|
||||
"2. GENERALIZATION:",
|
||||
" • Q-Learning: Learns specific state-action mappings",
|
||||
" • LLM: Reasons about patterns and transfers knowledge",
|
||||
"",
|
||||
"3. HIDDEN MECHANICS DISCOVERY:",
|
||||
" • Q-Learning: Requires extensive exploration",
|
||||
" • LLM: Can hypothesize and test theories",
|
||||
"",
|
||||
"4. COMPUTATIONAL TRADE-OFF:",
|
||||
f" • Q-Learning: Fast inference, slow learning",
|
||||
f" • LLM: Slower inference (API calls), fast adaptation"
|
||||
]
|
||||
|
||||
y_pos = 0.9
|
||||
for line in insights:
|
||||
if line.startswith("KEY INSIGHTS"):
|
||||
ax.text(0.5, y_pos, line, transform=ax.transAxes,
|
||||
fontsize=12, fontweight='bold', ha='center')
|
||||
elif line.startswith(("1.", "2.", "3.", "4.")):
|
||||
ax.text(0.1, y_pos, line, transform=ax.transAxes,
|
||||
fontsize=11, fontweight='bold')
|
||||
else:
|
||||
ax.text(0.1, y_pos, line, transform=ax.transAxes,
|
||||
fontsize=10)
|
||||
y_pos -= 0.06
|
||||
|
||||
plt.tight_layout()
|
||||
plt.savefig(self.experiment_dir / "comparison_plots.png", dpi=150)
|
||||
plt.show()
|
||||
|
||||
print(f"\nPlots saved to {self.experiment_dir / 'comparison_plots.png'}")
|
||||
|
||||
def run_full_experiment(self,
|
||||
rl_episodes: int = 10000,
|
||||
llm_episodes: int = 20,
|
||||
eval_episodes: int = 100,
|
||||
verbose: bool = False,
|
||||
stochastic: bool = False,
|
||||
model: str = "kimi-k3",
|
||||
checkpoint_interval: int = 1000,
|
||||
rl_hyperparams: Dict[str, float] = None):
|
||||
"""
|
||||
Run full comparison experiment.
|
||||
|
||||
Args:
|
||||
rl_episodes: Number of episodes for RL training
|
||||
llm_episodes: Number of episodes for LLM training
|
||||
eval_episodes: Number of episodes for RL evaluation
|
||||
verbose: Whether to print details
|
||||
stochastic: Whether to use stochastic environment
|
||||
model: LLM model name (Moonshot/Kimi)
|
||||
checkpoint_interval: Learning-curve sampling interval (RL)
|
||||
rl_hyperparams: Optional dict overriding Q-learning hyperparameters
|
||||
"""
|
||||
print("\n" + "="*70)
|
||||
print("EXPERIMENT: Traditional RL vs LLM In-Context Learning")
|
||||
print("Replicating insights from 'The Second Half' by Shunyu Yao")
|
||||
print("="*70)
|
||||
|
||||
# Show game rules for reference
|
||||
game = TreasureHuntGame(stochastic=stochastic)
|
||||
print("\n" + game.get_hidden_rules())
|
||||
|
||||
rl_hyperparams = rl_hyperparams or {}
|
||||
|
||||
# Run RL experiment
|
||||
rl_results = self.run_rl_experiment(
|
||||
num_training_episodes=rl_episodes,
|
||||
num_eval_episodes=eval_episodes,
|
||||
verbose=verbose,
|
||||
stochastic=stochastic,
|
||||
checkpoint_interval=checkpoint_interval,
|
||||
**rl_hyperparams
|
||||
)
|
||||
self.results["rl"] = rl_results
|
||||
|
||||
# Run LLM experiment
|
||||
llm_results = self.run_llm_experiment(
|
||||
num_training_episodes=llm_episodes,
|
||||
num_eval_episodes=10,
|
||||
verbose=verbose,
|
||||
stochastic=stochastic,
|
||||
model=model
|
||||
)
|
||||
self.results["llm"] = llm_results
|
||||
|
||||
# Save combined results
|
||||
with open(self.experiment_dir / "experiment_results.json", 'w') as f:
|
||||
json.dump(self.results, f, indent=2)
|
||||
|
||||
# Generate comparison plots
|
||||
if llm_results:
|
||||
self.compare_learning_curves(rl_results, llm_results)
|
||||
|
||||
# Print final comparison
|
||||
print("\n" + "="*70)
|
||||
print("EXPERIMENT RESULTS SUMMARY")
|
||||
print("="*70)
|
||||
|
||||
print("\n1. SAMPLE EFFICIENCY:")
|
||||
print(f" Q-Learning needed {rl_results['training_episodes']} episodes")
|
||||
if llm_results:
|
||||
print(f" LLM needed {llm_results['training_episodes']} episodes")
|
||||
print(f" → LLM is {rl_results['training_episodes'] / llm_results['training_episodes']:.1f}x more sample efficient")
|
||||
|
||||
print("\n2. PERFORMANCE:")
|
||||
print(f" Q-Learning eval victory rate: {rl_results['eval_victory_rate']:.2%}")
|
||||
if llm_results:
|
||||
print(f" LLM eval victory rate: {llm_results['eval_victory_rate']:.2%}")
|
||||
|
||||
print("\n3. COMPUTATIONAL COST:")
|
||||
print(f" Q-Learning: {rl_results['training_time']:.2f} seconds, {rl_results['q_table_size']} states")
|
||||
if llm_results:
|
||||
print(f" LLM: {llm_results['training_time']:.2f} seconds, {llm_results['api_calls']} API calls")
|
||||
|
||||
print(f"\nResults saved to: {self.experiment_dir}")
|
||||
|
||||
return self.results
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point for the experiment."""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="实验 7-1 / 7-2：在寻宝游戏中对比 Q-learning 与 LLM 的\"从经验中学习\"。"
|
||||
"Q-learning 完全离线运行（无需 API），LLM 模式需要 Moonshot/Kimi API Key。",
|
||||
epilog="示例：\n"
|
||||
" python experiment.py --mode qlearning # 只跑 Q-learning（离线，输出学习曲线）\n"
|
||||
" python experiment.py --mode qlearning --rl-episodes 10000 --seed 42\n"
|
||||
" python experiment.py --mode both --model kimi-k3 # RL vs LLM 对比\n"
|
||||
" python experiment.py --mode llm --llm-episodes 20 # 只跑 LLM 智能体",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter
|
||||
)
|
||||
parser.add_argument(
|
||||
"--mode", choices=["both", "qlearning", "rl", "llm"], default="both",
|
||||
help="运行哪种智能体：qlearning/rl 只跑 Q-learning（离线），"
|
||||
"llm 只跑 LLM 智能体（需 API），both 两者对比（默认）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--rl-episodes", type=int, default=10000,
|
||||
help="Q-learning 训练局数（默认 10000，对应实验 7-1）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--llm-episodes", type=int, default=20,
|
||||
help="LLM 智能体的训练局数（默认 20）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--eval-episodes", type=int, default=100,
|
||||
help="Q-learning 训练后贪婪评估的局数（默认 100）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--checkpoint-interval", type=int, default=1000,
|
||||
help="学习曲线采样间隔：每 N 局记录一次胜率/Q 表规模（默认 1000）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model", type=str, default=os.getenv("MOONSHOT_MODEL", "kimi-k3"),
|
||||
help="LLM 模型名称（Moonshot/Kimi，默认 kimi-k3，可用 MOONSHOT_MODEL 环境变量覆盖）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output", type=str, default="results",
|
||||
help="结果输出目录（默认 results/，每次运行会新建时间戳子目录）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--seed", type=int, default=None,
|
||||
help="随机种子，用于复现 Q-learning 的学习曲线（默认不固定）"
|
||||
)
|
||||
# Q-learning 超参数
|
||||
parser.add_argument("--learning-rate", type=float, default=0.2,
|
||||
help="Q-learning 学习率 alpha（默认 0.2）")
|
||||
parser.add_argument("--discount", type=float, default=0.99,
|
||||
help="折扣因子 gamma（默认 0.99）")
|
||||
parser.add_argument("--epsilon-decay", type=float, default=0.9995,
|
||||
help="每局 epsilon 衰减系数（默认 0.9995）")
|
||||
parser.add_argument("--epsilon-min", type=float, default=0.1,
|
||||
help="最小探索率 epsilon（默认 0.1）")
|
||||
parser.add_argument(
|
||||
"--verbose", action="store_true",
|
||||
help="训练过程中打印详细信息"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--skip-llm", action="store_true",
|
||||
help="[兼容旧用法] 跳过 LLM 实验，等价于 --mode qlearning"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--stochastic", action="store_true",
|
||||
help="使用随机环境（奖励与动作带随机扰动）"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--deterministic", action="store_true",
|
||||
help="使用确定性环境（默认）"
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Handle environment mode
|
||||
if args.deterministic and args.stochastic:
|
||||
print("Error: Cannot specify both --deterministic and --stochastic")
|
||||
return
|
||||
|
||||
# Episode counts must be positive; 0 would divide by zero in the
|
||||
# train/evaluate victory-rate and average calculations.
|
||||
if args.rl_episodes < 1 or args.llm_episodes < 1 or args.eval_episodes < 1:
|
||||
print("Error: --rl-episodes, --llm-episodes and --eval-episodes must all be >= 1")
|
||||
return
|
||||
|
||||
# Resolve run mode (--skip-llm kept as a backwards-compatible alias)
|
||||
mode = "qlearning" if args.skip_llm else args.mode
|
||||
|
||||
# Seed for reproducible Q-learning learning curves
|
||||
if args.seed is not None:
|
||||
random.seed(args.seed)
|
||||
np.random.seed(args.seed)
|
||||
print(f"\n🎲 Random seed set to {args.seed} for reproducibility")
|
||||
|
||||
stochastic = args.stochastic # Default is False (deterministic)
|
||||
|
||||
if stochastic:
|
||||
print("\n🎲 Running experiment with STOCHASTIC environment")
|
||||
print(" - Random reward variations")
|
||||
print(" - 3% chance of action failure")
|
||||
print(" - Combat and crafting variations\n")
|
||||
else:
|
||||
print("\n🎯 Running experiment with DETERMINISTIC environment\n")
|
||||
|
||||
rl_hyperparams = {
|
||||
"learning_rate": args.learning_rate,
|
||||
"discount_factor": args.discount,
|
||||
"epsilon_decay": args.epsilon_decay,
|
||||
"epsilon_min": args.epsilon_min,
|
||||
}
|
||||
|
||||
# Run experiment
|
||||
runner = ExperimentRunner(results_dir=args.output)
|
||||
|
||||
if mode in ("qlearning", "rl"):
|
||||
# Run only the Q-learning experiment (fully offline, no API needed)
|
||||
rl_results = runner.run_rl_experiment(
|
||||
num_training_episodes=args.rl_episodes,
|
||||
num_eval_episodes=args.eval_episodes,
|
||||
verbose=args.verbose,
|
||||
stochastic=stochastic,
|
||||
checkpoint_interval=args.checkpoint_interval,
|
||||
**rl_hyperparams
|
||||
)
|
||||
runner.results["rl"] = rl_results
|
||||
with open(runner.experiment_dir / "experiment_results.json", 'w') as f:
|
||||
json.dump(runner.results, f, indent=2)
|
||||
print(f"\nResults saved to: {runner.experiment_dir}")
|
||||
print("\nSkipped LLM experiment. Use --mode both with an API key to compare.")
|
||||
elif mode == "llm":
|
||||
# Run only the LLM experiment
|
||||
llm_results = runner.run_llm_experiment(
|
||||
num_training_episodes=args.llm_episodes,
|
||||
verbose=args.verbose,
|
||||
stochastic=stochastic,
|
||||
model=args.model
|
||||
)
|
||||
runner.results["llm"] = llm_results
|
||||
with open(runner.experiment_dir / "experiment_results.json", 'w') as f:
|
||||
json.dump(runner.results, f, indent=2)
|
||||
print(f"\nResults saved to: {runner.experiment_dir}")
|
||||
else:
|
||||
# Run full comparison
|
||||
results = runner.run_full_experiment(
|
||||
rl_episodes=args.rl_episodes,
|
||||
llm_episodes=args.llm_episodes,
|
||||
eval_episodes=args.eval_episodes,
|
||||
verbose=args.verbose,
|
||||
stochastic=stochastic,
|
||||
model=args.model,
|
||||
checkpoint_interval=args.checkpoint_interval,
|
||||
rl_hyperparams=rl_hyperparams
|
||||
)
|
||||
|
||||
print("\nExperiment complete!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,178 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Finalize a completed Experiment 8-2 campaign without repeating API calls."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import platform
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from run_experiment_8_2 import ROOT, _git_revision, _sha256, _write_json
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("campaign_dir", type=Path)
|
||||
args = parser.parse_args()
|
||||
campaign_dir = args.campaign_dir.expanduser().resolve()
|
||||
|
||||
results_path = campaign_dir / "experiment_results.json"
|
||||
raw_path = campaign_dir / "llm_experiences.json"
|
||||
checkpoint_path = campaign_dir / "rl_agent.pkl"
|
||||
manifest_path = campaign_dir / "execution_manifest.json"
|
||||
required = (results_path, raw_path, checkpoint_path, manifest_path)
|
||||
missing = [str(path) for path in required if not path.is_file()]
|
||||
if missing:
|
||||
parser.error("missing completed campaign artifacts: " + ", ".join(missing))
|
||||
|
||||
results = json.loads(results_path.read_text(encoding="utf-8"))
|
||||
raw = json.loads(raw_path.read_text(encoding="utf-8"))
|
||||
execution_manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
rl = results["rl"]
|
||||
llm = results["llm"]
|
||||
training = [
|
||||
episode
|
||||
for episode in raw.get("episode_trajectories", [])
|
||||
if episode.get("phase") == "training"
|
||||
]
|
||||
first_attempt = training[0] if training else None
|
||||
api_records = raw.get("api_records", [])
|
||||
response_models = sorted({
|
||||
record.get("response", {}).get("model")
|
||||
for record in api_records
|
||||
if record.get("response", {}).get("model")
|
||||
})
|
||||
direct_exact_kimi = (
|
||||
raw.get("backend", {}).get("provider") == "moonshot"
|
||||
and raw.get("backend", {}).get("model") == "kimi-k3"
|
||||
and raw.get("backend", {}).get("using_openrouter") is False
|
||||
and response_models == ["kimi-k3"]
|
||||
)
|
||||
protocol_gates = {
|
||||
"same_deterministic_environment": True,
|
||||
"q_learning_10000_training_episodes": rl["training_episodes"] == 10_000,
|
||||
"q_learning_100_evaluation_episodes": True,
|
||||
"q_learning_reached_full_evaluation_success": rl["eval_victory_rate"] == 1.0,
|
||||
"one_kimi_first_attempt_recorded": len(training) == 1,
|
||||
"direct_official_moonshot_kimi_k3": direct_exact_kimi,
|
||||
"one_real_response_per_first_attempt_action": bool(first_attempt)
|
||||
and len(api_records) == first_attempt["steps"],
|
||||
"provider_response_ids_retained": bool(api_records)
|
||||
and all(record.get("response", {}).get("id") for record in api_records),
|
||||
"provider_response_content_retained": bool(api_records)
|
||||
and all(record.get("response", {}).get("content") for record in api_records),
|
||||
"all_provider_responses_finished_normally": bool(api_records)
|
||||
and all(
|
||||
record.get("response", {}).get("finish_reason") == "stop"
|
||||
for record in api_records
|
||||
),
|
||||
"zero_api_errors": all(not record.get("error") for record in api_records),
|
||||
"zero_fallback_actions": all(
|
||||
not record.get("fallback_used") for record in api_records
|
||||
),
|
||||
}
|
||||
acceptance_complete = all(protocol_gates.values())
|
||||
first_victory = bool(first_attempt and first_attempt["victory"])
|
||||
first_steps = first_attempt["steps"] if first_attempt else None
|
||||
first_requested_at = (
|
||||
api_records[0].get("requested_at") if api_records else None
|
||||
)
|
||||
|
||||
evidence = {
|
||||
"schema_version": 1,
|
||||
"experiment_id": "8-2",
|
||||
"title": "Traditional RL versus Kimi K3 in the same treasure-hunt environment",
|
||||
"campaign_started_at": first_requested_at,
|
||||
"evidence_finalized_at": datetime.now(timezone.utc).isoformat(),
|
||||
"git_revision": _git_revision(),
|
||||
"runtime": {"python": sys.version, "platform": platform.platform()},
|
||||
"execution_manifest": execution_manifest,
|
||||
"backend": raw.get("backend"),
|
||||
"provider_response_models": response_models,
|
||||
"usage": {
|
||||
"successful_api_calls": raw.get("statistics", {}).get("api_calls"),
|
||||
"api_attempts": len(api_records),
|
||||
"total_tokens": raw.get("statistics", {}).get("total_tokens"),
|
||||
"provider_cost": None,
|
||||
"provider_cost_note": "The provider exposed token usage but no authoritative billed cost; unknown is not zero.",
|
||||
},
|
||||
"q_learning": {
|
||||
"training_episodes": rl["training_episodes"],
|
||||
"training_time_seconds": rl["training_time"],
|
||||
"training_victory_rate": rl["training_victory_rate"],
|
||||
"evaluation_episodes": 100,
|
||||
"evaluation_victory_rate": rl["eval_victory_rate"],
|
||||
"evaluation_average_steps": rl["eval_avg_steps"],
|
||||
"q_table_states": rl["q_table_size"],
|
||||
"learning_curve": rl["learning_curve"],
|
||||
},
|
||||
"k3_first_attempt": {
|
||||
"victory": first_victory,
|
||||
"steps": first_steps,
|
||||
"reward": first_attempt.get("total_reward") if first_attempt else None,
|
||||
"api_calls": len(api_records),
|
||||
"actions": [
|
||||
step["action"] for step in first_attempt.get("trajectory", [])
|
||||
] if first_attempt else [],
|
||||
},
|
||||
"protocol_gates": protocol_gates,
|
||||
"acceptance_complete": acceptance_complete,
|
||||
"manuscript_observation_matches": {
|
||||
"first_attempt_victory": first_victory,
|
||||
"exactly_18_steps": first_steps == 18,
|
||||
"q_learning_11_step_greedy_solution": rl["eval_avg_steps"] == 11.0,
|
||||
},
|
||||
"result_mismatches": [
|
||||
item
|
||||
for item, matched in {
|
||||
"Kimi K3 used 17 rather than the historical 18 steps": first_steps == 18,
|
||||
"Q-learning greedy evaluation averaged 12 rather than 11 steps": rl["eval_avg_steps"] == 11.0,
|
||||
}.items()
|
||||
if not matched
|
||||
],
|
||||
"interpretation": "Protocol acceptance is independent of whether stochastic model behavior reproduces historical point estimates.",
|
||||
"artifacts": {
|
||||
"experiment_results": results_path.name,
|
||||
"raw_llm_calls_and_trajectories": raw_path.name,
|
||||
"q_learning_checkpoint": checkpoint_path.name,
|
||||
"execution_manifest": manifest_path.name,
|
||||
},
|
||||
"artifact_sha256": {
|
||||
path.name: _sha256(path) for path in required
|
||||
},
|
||||
"postprocessor_source_sha256": {
|
||||
"run_experiment_8_2.py": _sha256(ROOT / "run_experiment_8_2.py"),
|
||||
"finalize_experiment_8_2.py": _sha256(ROOT / "finalize_experiment_8_2.py"),
|
||||
},
|
||||
"llm_result_summary": {
|
||||
key: llm.get(key)
|
||||
for key in (
|
||||
"provider", "model", "using_openrouter", "training_time",
|
||||
"api_calls", "api_attempts", "api_errors", "fallback_actions",
|
||||
"total_tokens", "training_victory_rate",
|
||||
)
|
||||
},
|
||||
}
|
||||
evidence_path = campaign_dir / "evidence.json"
|
||||
_write_json(evidence_path, evidence)
|
||||
latest_path = campaign_dir.parent / "latest.json"
|
||||
_write_json(latest_path, {
|
||||
"experiment_id": "8-2",
|
||||
"artifact": str(evidence_path.relative_to(campaign_dir.parent)),
|
||||
"acceptance_complete": acceptance_complete,
|
||||
"finalized_at": evidence["evidence_finalized_at"],
|
||||
})
|
||||
print(json.dumps({
|
||||
"evidence": str(evidence_path),
|
||||
"acceptance_complete": acceptance_complete,
|
||||
"first_attempt_victory": first_victory,
|
||||
"first_attempt_steps": first_steps,
|
||||
}, indent=2))
|
||||
return 0 if acceptance_complete else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,516 @@
|
||||
"""
|
||||
Text-based treasure hunt game with hidden mechanics.
|
||||
Inspired by Shunyu Yao's insights on reasoning and generalization in AI.
|
||||
"""
|
||||
|
||||
import random
|
||||
from typing import Dict, List, Tuple, Optional, Set
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class ItemType(Enum):
|
||||
KEY = "key"
|
||||
WEAPON = "weapon"
|
||||
TREASURE = "treasure"
|
||||
TOOL = "tool"
|
||||
POTION = "potion"
|
||||
|
||||
|
||||
@dataclass
|
||||
class Item:
|
||||
name: str
|
||||
item_type: ItemType
|
||||
description: str
|
||||
properties: Dict[str, any] = field(default_factory=dict)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Room:
|
||||
name: str
|
||||
description: str
|
||||
items: List[Item] = field(default_factory=list)
|
||||
exits: Dict[str, str] = field(default_factory=dict) # direction -> room_name
|
||||
locked_exits: Dict[str, str] = field(default_factory=dict) # direction -> required_key
|
||||
has_guard: bool = False
|
||||
guard_defeated: bool = False
|
||||
|
||||
|
||||
class TreasureHuntGame:
|
||||
"""
|
||||
A text-based game with hidden mechanics that agents must discover:
|
||||
1. Certain colored keys open corresponding colored doors
|
||||
2. Guards block access to treasures and require specific weapons
|
||||
3. Some items combine to create new items (hidden crafting)
|
||||
4. Potions provide temporary abilities
|
||||
"""
|
||||
|
||||
def __init__(self, seed: int = None, stochastic: bool = False):
|
||||
"""
|
||||
Initialize the game environment.
|
||||
|
||||
Args:
|
||||
seed: Random seed for reproducibility
|
||||
stochastic: If True, adds random elements to the game
|
||||
"""
|
||||
# NOTE: do not call random.seed() here. reset() re-runs __init__ once per
|
||||
# episode with a fresh 14-bit seed, and the learning agents draw their
|
||||
# exploration from the *global* random module -- reseeding it would pin
|
||||
# that stream to one of only 10001 states per episode and make whole
|
||||
# episodes repeat verbatim. The env's own randomness is self-contained
|
||||
# in self.random_state below, which is still seeded from `seed`.
|
||||
self.stochastic = stochastic
|
||||
self.random_state = random.Random(seed) if stochastic else None
|
||||
|
||||
self.rooms = {}
|
||||
self.current_room = None
|
||||
self.inventory = []
|
||||
self.score = 0
|
||||
self.moves = 0
|
||||
self.max_moves = 50 # Reduced for faster episodes
|
||||
self.game_over = False
|
||||
self.victory = False
|
||||
self.active_effects = {}
|
||||
|
||||
# Hidden mechanics (not revealed to agents initially)
|
||||
self.color_key_mapping = {
|
||||
"red key": "red door",
|
||||
"blue key": "blue door",
|
||||
"golden key": "golden door"
|
||||
}
|
||||
|
||||
self.weapon_effectiveness = {
|
||||
"rusty sword": ["weak guard"],
|
||||
"silver sword": ["weak guard", "strong guard", "dragon"]
|
||||
}
|
||||
|
||||
self.crafting_recipes = {
|
||||
frozenset(["rusty sword", "magic crystal"]): "silver sword"
|
||||
}
|
||||
|
||||
self._initialize_world()
|
||||
|
||||
def _initialize_world(self):
|
||||
"""Create the game world with rooms and items - simplified for better learning."""
|
||||
|
||||
# Create a simpler world that's easier to learn but still demonstrates the concepts
|
||||
self.rooms["entrance"] = Room(
|
||||
name="entrance",
|
||||
description="You stand in a dimly lit entrance hall. Stone walls echo your footsteps.",
|
||||
items=[
|
||||
Item("rusty sword", ItemType.WEAPON, "An old sword with rust spots")
|
||||
],
|
||||
exits={"north": "hallway", "east": "storage"}
|
||||
)
|
||||
|
||||
self.rooms["storage"] = Room(
|
||||
name="storage",
|
||||
description="A dusty storage room filled with old crates and barrels.",
|
||||
items=[
|
||||
Item("red key", ItemType.KEY, "A small red metal key"),
|
||||
Item("magic crystal", ItemType.TOOL, "A glowing crystal that hums with energy")
|
||||
],
|
||||
exits={"west": "entrance"}
|
||||
)
|
||||
|
||||
self.rooms["hallway"] = Room(
|
||||
name="hallway",
|
||||
description="A long hallway with a locked door to the north.",
|
||||
exits={"south": "entrance", "north": "guard_room"},
|
||||
locked_exits={"north": "red key"}
|
||||
)
|
||||
|
||||
self.rooms["guard_room"] = Room(
|
||||
name="guard_room",
|
||||
description="A large room with weapon racks. A guard blocks the treasure!",
|
||||
has_guard=True,
|
||||
items=[],
|
||||
exits={"south": "hallway", "east": "treasure_room"}
|
||||
)
|
||||
|
||||
self.rooms["treasure_room"] = Room(
|
||||
name="treasure_room",
|
||||
description="The treasure room! Gold coins and jewels sparkle in the light.",
|
||||
items=[
|
||||
Item("dragon's treasure", ItemType.TREASURE, "A massive hoard of gold and gems",
|
||||
{"value": 1000})
|
||||
],
|
||||
exits={"west": "guard_room"}
|
||||
)
|
||||
|
||||
self.current_room = self.rooms["entrance"]
|
||||
|
||||
def get_state_description(self) -> str:
|
||||
"""Get a natural language description of the current game state."""
|
||||
desc = []
|
||||
desc.append(f"\n=== Room: {self.current_room.name.replace('_', ' ').title()} ===")
|
||||
desc.append(self.current_room.description)
|
||||
|
||||
if self.current_room.has_guard and not self.current_room.guard_defeated:
|
||||
desc.append("A guard blocks your way!")
|
||||
|
||||
if self.current_room.items:
|
||||
desc.append("\nYou see:")
|
||||
for item in self.current_room.items:
|
||||
desc.append(f" - {item.name}: {item.description}")
|
||||
|
||||
exits = []
|
||||
for direction, room in self.current_room.exits.items():
|
||||
if direction in self.current_room.locked_exits:
|
||||
exits.append(f"{direction} (locked)")
|
||||
else:
|
||||
exits.append(direction)
|
||||
desc.append(f"\nExits: {', '.join(exits)}")
|
||||
|
||||
if self.inventory:
|
||||
desc.append(f"\nInventory: {', '.join([item.name for item in self.inventory])}")
|
||||
|
||||
desc.append(f"\nScore: {self.score} | Moves: {self.moves}/{self.max_moves}")
|
||||
|
||||
return "\n".join(desc)
|
||||
|
||||
def get_available_actions(self) -> List[str]:
|
||||
"""Get list of available actions in current state."""
|
||||
actions = []
|
||||
|
||||
# Movement actions
|
||||
for direction in self.current_room.exits.keys():
|
||||
actions.append(f"go {direction}")
|
||||
|
||||
# Item actions
|
||||
for item in self.current_room.items:
|
||||
actions.append(f"take {item.name}")
|
||||
|
||||
for item in self.inventory:
|
||||
actions.append(f"use {item.name}")
|
||||
actions.append(f"drop {item.name}")
|
||||
|
||||
# Combat actions
|
||||
if self.current_room.has_guard and not self.current_room.guard_defeated:
|
||||
for item in self.inventory:
|
||||
if item.item_type == ItemType.WEAPON:
|
||||
actions.append(f"attack with {item.name}")
|
||||
|
||||
# Special actions
|
||||
actions.append("look around")
|
||||
actions.append("check inventory")
|
||||
|
||||
# Crafting (if player has discovered it)
|
||||
if len(self.inventory) >= 2:
|
||||
actions.append("try crafting")
|
||||
|
||||
return actions
|
||||
|
||||
def execute_action(self, action: str) -> Tuple[str, float, bool]:
|
||||
"""
|
||||
Execute an action and return (feedback, reward, done).
|
||||
"""
|
||||
if self.game_over:
|
||||
return "Game is already over.", 0, True
|
||||
|
||||
self.moves += 1
|
||||
action = action.lower().strip()
|
||||
# The Nth move must still execute, so the limit is enforced after
|
||||
# the action is dispatched (and also on the fumble early-return,
|
||||
# which previously bypassed it and let episodes run past the cap).
|
||||
out_of_moves = self.moves >= self.max_moves
|
||||
|
||||
# Base reward with stochastic variation
|
||||
if self.stochastic:
|
||||
# Add small random variation to rewards
|
||||
reward = -0.5 + self.random_state.uniform(-0.1, 0.1)
|
||||
|
||||
# Small chance of action failure in stochastic mode
|
||||
if self.random_state.random() < 0.03: # 3% chance
|
||||
if out_of_moves:
|
||||
self.game_over = True
|
||||
return ("You fumble — and you've run out of moves! Game over.",
|
||||
reward - 10, True)
|
||||
return "You fumble and need to try again.", reward - 0.2, False
|
||||
else:
|
||||
reward = -0.5 # Negative reward for each move to encourage efficiency
|
||||
|
||||
# Parse action
|
||||
if action.startswith("go "):
|
||||
direction = action[3:]
|
||||
result, move_reward = self._move(direction)
|
||||
reward += move_reward
|
||||
|
||||
elif action.startswith("take "):
|
||||
item_name = action[5:]
|
||||
result, take_reward = self._take_item(item_name)
|
||||
reward += take_reward
|
||||
|
||||
elif action.startswith("use "):
|
||||
item_name = action[4:]
|
||||
result, use_reward = self._use_item(item_name)
|
||||
reward += use_reward
|
||||
|
||||
elif action.startswith("drop "):
|
||||
item_name = action[5:]
|
||||
result = self._drop_item(item_name)
|
||||
|
||||
elif action.startswith("attack with "):
|
||||
weapon_name = action[12:]
|
||||
result, attack_reward = self._attack(weapon_name)
|
||||
reward += attack_reward
|
||||
|
||||
elif action == "look around":
|
||||
result = self.get_state_description()
|
||||
|
||||
elif action == "check inventory":
|
||||
if self.inventory:
|
||||
result = "Inventory: " + ", ".join([f"{item.name} ({item.item_type.value})"
|
||||
for item in self.inventory])
|
||||
else:
|
||||
result = "Your inventory is empty."
|
||||
|
||||
elif action == "try crafting":
|
||||
result, craft_reward = self._try_crafting()
|
||||
reward += craft_reward
|
||||
|
||||
else:
|
||||
result = f"Unknown action: {action}"
|
||||
reward -= 1
|
||||
|
||||
# Check victory condition
|
||||
if self._check_victory():
|
||||
self.victory = True
|
||||
self.game_over = True
|
||||
reward += 100
|
||||
result += "\n\n🎉 VICTORY! You've collected the dragon's treasure!"
|
||||
|
||||
# Enforce the move limit after the action executed: a winning move
|
||||
# on the last allowed step still counts as a victory.
|
||||
if not self.game_over and out_of_moves:
|
||||
self.game_over = True
|
||||
reward -= 10
|
||||
result += "\n\nYou've run out of moves! Game over."
|
||||
|
||||
return result, reward, self.game_over
|
||||
|
||||
def _move(self, direction: str) -> Tuple[str, float]:
|
||||
"""Move to another room."""
|
||||
if direction not in self.current_room.exits:
|
||||
return f"You can't go {direction} from here.", -1
|
||||
|
||||
# Check if locked
|
||||
if direction in self.current_room.locked_exits:
|
||||
required_key = self.current_room.locked_exits[direction]
|
||||
if not any(item.name == required_key for item in self.inventory):
|
||||
return f"The {direction} exit is locked. You need a {required_key}.", -0.5
|
||||
else:
|
||||
# Unlock and move
|
||||
del self.current_room.locked_exits[direction]
|
||||
room_name = self.current_room.exits[direction]
|
||||
self.current_room = self.rooms[room_name]
|
||||
return f"You unlock the door with the {required_key} and move {direction}.", 5
|
||||
|
||||
# Check for guard
|
||||
if self.current_room.has_guard and not self.current_room.guard_defeated:
|
||||
return "A guard blocks your way! You must defeat them first.", -1
|
||||
|
||||
# Move to new room
|
||||
room_name = self.current_room.exits[direction]
|
||||
self.current_room = self.rooms[room_name]
|
||||
return f"You move {direction} to the {self.current_room.name}.", 1
|
||||
|
||||
def _take_item(self, item_name: str) -> Tuple[str, float]:
|
||||
"""Pick up an item."""
|
||||
for item in self.current_room.items:
|
||||
if item.name.lower() == item_name.lower():
|
||||
self.current_room.items.remove(item)
|
||||
self.inventory.append(item)
|
||||
|
||||
# Reward based on item type
|
||||
if item.item_type == ItemType.TREASURE:
|
||||
reward = 100 # Big reward for getting the treasure!
|
||||
elif item.item_type == ItemType.KEY:
|
||||
reward = 5
|
||||
elif item.item_type == ItemType.WEAPON:
|
||||
reward = 3
|
||||
else:
|
||||
reward = 2
|
||||
|
||||
# Add stochastic variation
|
||||
if self.stochastic:
|
||||
reward += self.random_state.uniform(-0.5, 0.5)
|
||||
|
||||
return f"You take the {item.name}.", reward
|
||||
|
||||
penalty = -0.5
|
||||
if self.stochastic:
|
||||
penalty += self.random_state.uniform(-0.1, 0.1)
|
||||
|
||||
return f"There's no {item_name} here.", penalty
|
||||
|
||||
def _drop_item(self, item_name: str) -> str:
|
||||
"""Drop an item."""
|
||||
for item in self.inventory:
|
||||
if item.name.lower() == item_name.lower():
|
||||
self.inventory.remove(item)
|
||||
self.current_room.items.append(item)
|
||||
return f"You drop the {item.name}."
|
||||
|
||||
return f"You don't have a {item_name}."
|
||||
|
||||
def _use_item(self, item_name: str) -> Tuple[str, float]:
|
||||
"""Use an item."""
|
||||
for item in self.inventory:
|
||||
if item.name.lower() == item_name.lower():
|
||||
if item.item_type == ItemType.POTION:
|
||||
self.inventory.remove(item)
|
||||
if "healing" in item.name:
|
||||
return "You drink the healing potion and feel refreshed!", 5
|
||||
elif "strength" in item.name:
|
||||
self.active_effects["strength"] = 10
|
||||
return "You feel a surge of power! Your attacks will be stronger.", 5
|
||||
|
||||
elif item.item_type == ItemType.KEY:
|
||||
# Keys are used automatically when moving
|
||||
return f"The {item.name} will be used automatically when needed.", 0
|
||||
|
||||
else:
|
||||
return f"You can't use the {item.name} right now.", -0.5
|
||||
|
||||
return f"You don't have a {item_name}.", -0.5
|
||||
|
||||
def _attack(self, weapon_name: str) -> Tuple[str, float]:
|
||||
"""Attack with a weapon."""
|
||||
if not self.current_room.has_guard or self.current_room.guard_defeated:
|
||||
return "There's nothing to attack here.", -1
|
||||
|
||||
weapon = None
|
||||
for item in self.inventory:
|
||||
if item.name.lower() == weapon_name.lower():
|
||||
weapon = item
|
||||
break
|
||||
|
||||
if not weapon:
|
||||
return f"You don't have a {weapon_name}.", -1
|
||||
|
||||
if weapon.item_type != ItemType.WEAPON:
|
||||
return f"The {weapon_name} is not a weapon!", -1
|
||||
|
||||
# Check weapon effectiveness (hidden mechanic)
|
||||
# In our simplified game, the guard in guard_room is a "strong guard"
|
||||
guard_type = "strong guard"
|
||||
|
||||
if weapon.name in self.weapon_effectiveness:
|
||||
if guard_type in self.weapon_effectiveness[weapon.name]:
|
||||
# In stochastic mode, add combat variations
|
||||
if self.stochastic:
|
||||
roll = self.random_state.random()
|
||||
if roll < 0.1: # 10% critical hit
|
||||
self.current_room.guard_defeated = True
|
||||
return f"Critical hit! You defeat the {guard_type} with your {weapon.name}!", 30
|
||||
elif roll < 0.95: # 85% normal success
|
||||
self.current_room.guard_defeated = True
|
||||
return f"You defeat the {guard_type} with your {weapon.name}!", 20
|
||||
else: # 5% glancing blow
|
||||
return f"Your attack glances off! The {guard_type} is still standing.", -0.5
|
||||
else:
|
||||
self.current_room.guard_defeated = True
|
||||
return f"You defeat the {guard_type} with your {weapon.name}!", 20
|
||||
else:
|
||||
penalty = -2
|
||||
if self.stochastic:
|
||||
penalty += self.random_state.uniform(-0.5, 0.5)
|
||||
return f"Your {weapon.name} is not effective against the {guard_type}!", penalty
|
||||
|
||||
return f"Your {weapon.name} doesn't seem to work.", -1
|
||||
|
||||
def _try_crafting(self) -> Tuple[str, float]:
|
||||
"""Try to craft items (hidden mechanic)."""
|
||||
if len(self.inventory) < 2:
|
||||
return "You need at least two items to craft.", -0.5
|
||||
|
||||
# Check all possible combinations
|
||||
inventory_names = [item.name for item in self.inventory]
|
||||
|
||||
for recipe, result in self.crafting_recipes.items():
|
||||
if recipe.issubset(set(inventory_names)):
|
||||
# In stochastic mode, crafting might have variations
|
||||
if self.stochastic:
|
||||
if self.random_state.random() < 0.9: # 90% success rate
|
||||
# Craft the item
|
||||
for ingredient in recipe:
|
||||
for item in self.inventory[:]:
|
||||
if item.name == ingredient:
|
||||
self.inventory.remove(item)
|
||||
break
|
||||
|
||||
new_item = self._create_item(result)
|
||||
self.inventory.append(new_item)
|
||||
reward = 10 + self.random_state.uniform(-1, 2)
|
||||
return f"You successfully craft a {result}!", reward
|
||||
else:
|
||||
# 10% chance of crafting mishap (items not consumed)
|
||||
return "The crafting attempt fizzles. Try again!", -0.2
|
||||
else:
|
||||
# Deterministic crafting
|
||||
for ingredient in recipe:
|
||||
for item in self.inventory[:]:
|
||||
if item.name == ingredient:
|
||||
self.inventory.remove(item)
|
||||
break
|
||||
|
||||
new_item = self._create_item(result)
|
||||
self.inventory.append(new_item)
|
||||
return f"You successfully craft a {result}!", 10 # Good reward for discovering crafting
|
||||
|
||||
penalty = -0.5
|
||||
if self.stochastic:
|
||||
penalty += self.random_state.uniform(-0.1, 0.1)
|
||||
|
||||
return "These items don't combine into anything useful.", penalty
|
||||
|
||||
def _create_item(self, item_name: str) -> Item:
|
||||
"""Create an item by name."""
|
||||
if item_name == "silver sword":
|
||||
return Item("silver sword", ItemType.WEAPON, "A gleaming silver blade")
|
||||
elif item_name == "magic staff":
|
||||
return Item("magic staff", ItemType.WEAPON, "A staff crackling with magical energy")
|
||||
else:
|
||||
return Item(item_name, ItemType.TOOL, "A crafted item")
|
||||
|
||||
def _check_victory(self) -> bool:
|
||||
"""Check if the player has won."""
|
||||
for item in self.inventory:
|
||||
if item.name == "dragon's treasure":
|
||||
return True
|
||||
return False
|
||||
|
||||
def reset(self, seed: int = None) -> str:
|
||||
"""Reset the game to initial state."""
|
||||
if seed is None:
|
||||
seed = random.randint(0, 10000)
|
||||
self.__init__(seed=seed, stochastic=self.stochastic)
|
||||
return self.get_state_description()
|
||||
|
||||
def get_hidden_rules(self) -> str:
|
||||
"""Return the hidden game rules (for debugging/analysis)."""
|
||||
rules = []
|
||||
rules.append("Hidden Game Mechanics (Simplified Version):")
|
||||
rules.append("\n1. To win the game:")
|
||||
rules.append(" - Get the red key from storage room")
|
||||
rules.append(" - Use it to unlock the door to guard room")
|
||||
rules.append(" - Craft a silver sword (rusty sword + magic crystal)")
|
||||
rules.append(" - Defeat the strong guard with the silver sword")
|
||||
rules.append(" - Collect the dragon's treasure")
|
||||
|
||||
rules.append("\n2. Key mechanics:")
|
||||
rules.append(" - Red key opens the locked door in the hallway")
|
||||
|
||||
rules.append("\n3. Weapon effectiveness:")
|
||||
for weapon, targets in self.weapon_effectiveness.items():
|
||||
rules.append(f" - {weapon} defeats: {', '.join(targets)}")
|
||||
|
||||
rules.append("\n4. Crafting recipe:")
|
||||
for ingredients, result in self.crafting_recipes.items():
|
||||
rules.append(f" - {' + '.join(ingredients)} = {result}")
|
||||
|
||||
rules.append("\n5. Optimal solution:")
|
||||
rules.append(" - Takes about 10-15 moves if done efficiently")
|
||||
|
||||
return "\n".join(rules)
|
||||
@@ -0,0 +1,639 @@
|
||||
"""
|
||||
LLM-based Agent using In-Context Learning with the Kimi (Moonshot) API.
|
||||
This demonstrates how LLMs can generalize through reasoning without extensive training.
|
||||
Default model is Kimi K3 (matching 实验 7-2 in the book); override via the
|
||||
`model` argument or the MOONSHOT_MODEL environment variable.
|
||||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from typing import Dict, List, Tuple, Any, Optional
|
||||
from dataclasses import dataclass, asdict
|
||||
import openai
|
||||
from game_environment import TreasureHuntGame
|
||||
|
||||
|
||||
def _reasoning_safe_temperature(model, requested=1.0):
|
||||
"""Reasoning models (Kimi K3, GPT-5, ...) only accept temperature=1.
|
||||
Return 1 for those; otherwise the requested value so non-reasoning
|
||||
providers (Doubao, DeepSeek, older Moonshot) are unchanged."""
|
||||
m = str(model or "").lower().replace("/", "-")
|
||||
return 1 if ("kimi-k3" in m or "gpt-5" in m) else requested
|
||||
|
||||
|
||||
# Provider resolution lives in the shared agentbook package so every chapter
|
||||
# stays consistent; see agentbook/providers.py. The fallback keeps this
|
||||
# experiment runnable from a checkout where agentbook is not installed.
|
||||
try:
|
||||
from agentbook.providers import (
|
||||
SUPPORTED_PROVIDERS,
|
||||
map_model_to_openrouter,
|
||||
resolve_backend,
|
||||
resolve_llm_backend,
|
||||
)
|
||||
except ImportError: # pragma: no cover - exercised only without the package
|
||||
import sys as _sys
|
||||
|
||||
_sys.path.insert(
|
||||
0, str(__import__("pathlib").Path(__file__).resolve().parents[2])
|
||||
)
|
||||
from agentbook.providers import (
|
||||
SUPPORTED_PROVIDERS,
|
||||
map_model_to_openrouter,
|
||||
resolve_backend,
|
||||
resolve_llm_backend,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class GameExperience:
|
||||
"""Represents a single game interaction experience."""
|
||||
state_description: str
|
||||
action: str
|
||||
feedback: str
|
||||
reward: float
|
||||
success: bool # Whether the action led to positive outcome
|
||||
|
||||
|
||||
class LLMAgent:
|
||||
"""
|
||||
LLM-based agent that uses in-context learning to play the game.
|
||||
Stores experiences and uses them to reason about future actions.
|
||||
"""
|
||||
|
||||
def __init__(self,
|
||||
api_key: str = None,
|
||||
model: str = "kimi-k3", # Kimi K3 (see 实验 7-2)
|
||||
base_url: str = "https://api.moonshot.cn/v1",
|
||||
temperature: float = 0.7,
|
||||
max_experiences: int = 50,
|
||||
provider: str | None = None):
|
||||
"""
|
||||
Initialize LLM agent with the Kimi (Moonshot) API.
|
||||
|
||||
Args:
|
||||
api_key: Provider API key (or set the provider's env var)
|
||||
model: Model name (defaults to the selected provider's model)
|
||||
base_url: API base URL
|
||||
temperature: Sampling temperature for generation
|
||||
max_experiences: Maximum number of experiences to store
|
||||
"""
|
||||
# Set up an OpenAI-compatible client. The default remains Moonshot,
|
||||
# while LLM_PROVIDER=dashscope/qwen/bailian enables direct Bailian use.
|
||||
requested_provider = (provider or os.getenv("LLM_PROVIDER", "moonshot")).lower()
|
||||
requested_provider = {"qwen": "dashscope", "bailian": "dashscope"}.get(
|
||||
requested_provider, requested_provider
|
||||
)
|
||||
if requested_provider == "dashscope":
|
||||
dashscope_model = model if model != "kimi-k3" else None
|
||||
backend = resolve_backend(
|
||||
"dashscope",
|
||||
model=dashscope_model or os.getenv("DASHSCOPE_MODEL"),
|
||||
api_key=api_key,
|
||||
)
|
||||
self.api_key, resolved_base_url, self.model = (
|
||||
backend.api_key, backend.base_url, backend.model
|
||||
)
|
||||
self.using_openrouter = backend.using_openrouter
|
||||
self.provider = backend.provider
|
||||
else:
|
||||
primary_key = api_key or os.getenv("MOONSHOT_API_KEY")
|
||||
self.api_key, resolved_base_url, self.model, self.using_openrouter = \
|
||||
resolve_llm_backend(primary_key, base_url, model)
|
||||
self.provider = "openrouter" if self.using_openrouter else "moonshot"
|
||||
self.base_url = resolved_base_url
|
||||
if self.using_openrouter:
|
||||
print(f"ℹ️ MOONSHOT_API_KEY not set; routing via OpenRouter (model: {self.model})")
|
||||
|
||||
self.client = openai.OpenAI(
|
||||
api_key=self.api_key,
|
||||
base_url=resolved_base_url
|
||||
)
|
||||
self.temperature = temperature
|
||||
|
||||
# Experience memory for in-context learning
|
||||
self.experiences: List[GameExperience] = []
|
||||
self.max_experiences = max_experiences
|
||||
|
||||
# Statistics
|
||||
self.episode_rewards = []
|
||||
self.episode_lengths = []
|
||||
self.victories = 0
|
||||
self.total_episodes = 0
|
||||
self.api_calls = 0
|
||||
self.total_tokens = 0
|
||||
# Retain canonical real-run evidence without ever serializing the API key.
|
||||
self.api_records: List[Dict[str, Any]] = []
|
||||
self.episode_trajectories: List[Dict[str, Any]] = []
|
||||
|
||||
def _build_context(self, current_state: str, available_actions: List[str]) -> str:
|
||||
"""
|
||||
Build context for the LLM including task description and past experiences.
|
||||
This is the key to in-context learning.
|
||||
"""
|
||||
context = []
|
||||
|
||||
# Task description
|
||||
context.append("""You are playing a text-based treasure hunt game. Your goal is to find and collect the dragon's treasure.
|
||||
|
||||
The game has hidden mechanics that you need to discover through experience:
|
||||
- Certain items may be required to unlock doors or defeat guards
|
||||
- Items might combine to create better items
|
||||
- Different weapons have different effectiveness
|
||||
|
||||
You should reason about what you've learned from past experiences to make better decisions.""")
|
||||
|
||||
# Add relevant past experiences
|
||||
if self.experiences:
|
||||
context.append("\n=== PAST EXPERIENCES ===")
|
||||
context.append("Here are some experiences from previous attempts that might help you:")
|
||||
|
||||
# Group experiences by pattern
|
||||
successful_patterns = []
|
||||
failed_patterns = []
|
||||
|
||||
for exp in self.experiences[-self.max_experiences:]:
|
||||
exp_text = f"State: {exp.state_description[:200]}...\nAction: {exp.action}\nResult: {exp.feedback}\nReward: {exp.reward:.1f}"
|
||||
|
||||
if exp.success:
|
||||
successful_patterns.append(exp_text)
|
||||
else:
|
||||
failed_patterns.append(exp_text)
|
||||
|
||||
if successful_patterns:
|
||||
context.append("\n** Successful actions:")
|
||||
for pattern in successful_patterns[-10:]: # Last 10 successful
|
||||
context.append(pattern)
|
||||
|
||||
if failed_patterns:
|
||||
context.append("\n** Failed actions to avoid:")
|
||||
for pattern in failed_patterns[-5:]: # Last 5 failed
|
||||
context.append(pattern)
|
||||
|
||||
# Current situation
|
||||
context.append("\n=== CURRENT SITUATION ===")
|
||||
context.append(current_state)
|
||||
context.append(f"\nAvailable actions: {', '.join(available_actions)}")
|
||||
|
||||
return "\n".join(context)
|
||||
|
||||
def _build_prompt(self, context: str) -> str:
|
||||
"""Build the full prompt for the LLM."""
|
||||
prompt = f"""{context}
|
||||
|
||||
Based on your understanding of the game mechanics from past experiences and the current situation, reason step-by-step about what action to take:
|
||||
|
||||
1. What have you learned from past experiences that applies here?
|
||||
2. What is your current goal or sub-goal?
|
||||
3. Which available action best helps achieve that goal?
|
||||
|
||||
Think through this carefully, then provide your chosen action.
|
||||
|
||||
IMPORTANT: Your response must end with exactly one line starting with "ACTION:" followed by one of the available actions listed above.
|
||||
|
||||
Example format:
|
||||
[Your reasoning here...]
|
||||
ACTION: take red key
|
||||
"""
|
||||
return prompt
|
||||
|
||||
def choose_action(self, game: TreasureHuntGame, verbose: bool = True) -> str:
|
||||
"""
|
||||
Choose an action using LLM reasoning with in-context learning.
|
||||
"""
|
||||
# Get current state and available actions
|
||||
state_description = game.get_state_description()
|
||||
available_actions = game.get_available_actions()
|
||||
|
||||
if not available_actions:
|
||||
return "look around"
|
||||
|
||||
# Build context with past experiences
|
||||
context = self._build_context(state_description, available_actions)
|
||||
prompt = self._build_prompt(context)
|
||||
|
||||
if verbose:
|
||||
print("\n" + "="*60)
|
||||
print("LLM DECISION PROCESS")
|
||||
print("="*60)
|
||||
print(f"📊 Experiences in memory: {len(self.experiences)}")
|
||||
print(f"🎮 Current room: {game.current_room.name}")
|
||||
print(f"🎯 Available actions: {len(available_actions)}")
|
||||
|
||||
# Show some recent successful experiences if any
|
||||
successful = [e for e in self.experiences if e.success]
|
||||
if successful:
|
||||
print(f"\n💡 Recent successful patterns learned:")
|
||||
for exp in successful[-3:]:
|
||||
print(f" • {exp.action} → +{exp.reward:.1f} reward")
|
||||
|
||||
request_messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "You are an intelligent game-playing agent that learns from experience.",
|
||||
},
|
||||
{"role": "user", "content": prompt},
|
||||
]
|
||||
requested_temperature = _reasoning_safe_temperature(
|
||||
self.model, self.temperature
|
||||
)
|
||||
started = time.perf_counter()
|
||||
api_record: Dict[str, Any] = {
|
||||
"requested_at": datetime.now(timezone.utc).isoformat(),
|
||||
"provider": self.provider,
|
||||
"base_url": self.base_url,
|
||||
"model": self.model,
|
||||
"request": {
|
||||
"messages": request_messages,
|
||||
"temperature": requested_temperature,
|
||||
"max_tokens": 2048,
|
||||
},
|
||||
"available_actions": list(available_actions),
|
||||
}
|
||||
|
||||
try:
|
||||
print("\n🤔 LLM is thinking...")
|
||||
|
||||
# Kimi K3 is a reasoning model: completion tokens can be consumed
|
||||
# by reasoning_content before message.content is emitted. Keep a
|
||||
# generous budget so the required ACTION line is not truncated.
|
||||
response = self.client.chat.completions.create(
|
||||
model=self.model,
|
||||
messages=request_messages,
|
||||
temperature=requested_temperature,
|
||||
max_tokens=2048,
|
||||
)
|
||||
|
||||
self.api_calls += 1
|
||||
usage = getattr(response, "usage", None)
|
||||
if usage is not None and getattr(usage, "total_tokens", None) is not None:
|
||||
self.total_tokens += usage.total_tokens
|
||||
|
||||
choice = response.choices[0]
|
||||
response_text = choice.message.content or ""
|
||||
reasoning_text = getattr(choice.message, "reasoning_content", None)
|
||||
if usage is not None and hasattr(usage, "model_dump"):
|
||||
usage_payload = usage.model_dump()
|
||||
elif usage is not None:
|
||||
usage_payload = {
|
||||
key: getattr(usage, key, None)
|
||||
for key in ("prompt_tokens", "completion_tokens", "total_tokens")
|
||||
}
|
||||
else:
|
||||
usage_payload = None
|
||||
api_record["response"] = {
|
||||
"id": getattr(response, "id", None),
|
||||
"created": getattr(response, "created", None),
|
||||
"model": getattr(response, "model", None),
|
||||
"finish_reason": getattr(choice, "finish_reason", None),
|
||||
"content": response_text,
|
||||
"reasoning_content": reasoning_text,
|
||||
"usage": usage_payload,
|
||||
}
|
||||
|
||||
if verbose:
|
||||
print("\n📝 LLM Reasoning:")
|
||||
print("-" * 40)
|
||||
reasoning_lines = []
|
||||
for line in response_text.split('\n'):
|
||||
if line.startswith("ACTION:"):
|
||||
break
|
||||
if line.strip():
|
||||
reasoning_lines.append(line)
|
||||
for line in reasoning_lines[-5:]:
|
||||
print(f" {line[:100]}...")
|
||||
print("-" * 40)
|
||||
|
||||
action_line = re.compile(
|
||||
r"^\s*(?:[-*]\s*)?(?:\*\*)?ACTION(?:\*\*)?\s*:\s*(.*?)\s*(?:\*\*)?\s*$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
for line in reversed(response_text.strip().split('\n')):
|
||||
match = action_line.match(line)
|
||||
if not match:
|
||||
continue
|
||||
action = match.group(1).strip().strip("`* ")
|
||||
if action in available_actions:
|
||||
api_record.update({
|
||||
"parsed_action": action,
|
||||
"fallback_used": False,
|
||||
"elapsed_ms": round((time.perf_counter() - started) * 1000, 3),
|
||||
})
|
||||
self.api_records.append(api_record)
|
||||
if verbose:
|
||||
print(f"\n✅ Chosen action: {action}")
|
||||
return action
|
||||
|
||||
action_lower = action.lower()
|
||||
for available in available_actions:
|
||||
if available.lower() == action_lower:
|
||||
api_record.update({
|
||||
"parsed_action": available,
|
||||
"fallback_used": False,
|
||||
"case_normalized": True,
|
||||
"elapsed_ms": round((time.perf_counter() - started) * 1000, 3),
|
||||
})
|
||||
self.api_records.append(api_record)
|
||||
if verbose:
|
||||
print(f"\n✅ Chosen action (corrected): {available}")
|
||||
return available
|
||||
|
||||
print("⚠️ Warning: Could not parse valid action from LLM response. Using fallback.")
|
||||
api_record.update({
|
||||
"parsed_action": available_actions[0],
|
||||
"fallback_used": True,
|
||||
"fallback_reason": "missing_or_invalid_ACTION_line",
|
||||
"elapsed_ms": round((time.perf_counter() - started) * 1000, 3),
|
||||
})
|
||||
self.api_records.append(api_record)
|
||||
return available_actions[0]
|
||||
|
||||
except Exception as e:
|
||||
print(f"❌ Error calling LLM API: {e}")
|
||||
api_record.update({
|
||||
"error": {"type": type(e).__name__, "message": str(e)},
|
||||
"parsed_action": available_actions[0],
|
||||
"fallback_used": True,
|
||||
"fallback_reason": "api_error",
|
||||
"elapsed_ms": round((time.perf_counter() - started) * 1000, 3),
|
||||
})
|
||||
self.api_records.append(api_record)
|
||||
return available_actions[0]
|
||||
|
||||
def update_experience(self, state: str, action: str, feedback: str, reward: float):
|
||||
"""
|
||||
Store an experience for future in-context learning.
|
||||
"""
|
||||
# Determine if action was successful based on reward
|
||||
success = reward > 0
|
||||
|
||||
experience = GameExperience(
|
||||
state_description=state,
|
||||
action=action,
|
||||
feedback=feedback,
|
||||
reward=reward,
|
||||
success=success
|
||||
)
|
||||
|
||||
self.experiences.append(experience)
|
||||
|
||||
# Keep only recent experiences to manage context length
|
||||
if len(self.experiences) > self.max_experiences * 2:
|
||||
# Keep a mix of successful and failed experiences
|
||||
successful = [e for e in self.experiences if e.success]
|
||||
failed = [e for e in self.experiences if not e.success]
|
||||
|
||||
# Keep recent ones and some diverse older ones
|
||||
self.experiences = (
|
||||
successful[-self.max_experiences:] +
|
||||
failed[-self.max_experiences//2:]
|
||||
)[-self.max_experiences:]
|
||||
|
||||
def play_episode(self, game: TreasureHuntGame, verbose: bool = True,
|
||||
phase: str = "unspecified") -> Tuple[float, int, bool]:
|
||||
"""
|
||||
Play one episode of the game.
|
||||
"""
|
||||
game.reset()
|
||||
total_reward = 0
|
||||
steps = 0
|
||||
trajectory = []
|
||||
|
||||
if verbose:
|
||||
print("\n" + "🎮"*30)
|
||||
print("STARTING NEW GAME EPISODE")
|
||||
print("🎮"*30)
|
||||
|
||||
while not game.game_over:
|
||||
if verbose:
|
||||
print(f"\n{'='*60}")
|
||||
print(f"STEP {steps + 1}")
|
||||
print(f"{'='*60}")
|
||||
|
||||
# Show current game state
|
||||
print("\n📍 Current State:")
|
||||
state_lines = game.get_state_description().split('\n')
|
||||
for line in state_lines:
|
||||
if line.strip():
|
||||
print(f" {line}")
|
||||
|
||||
# Get state before action
|
||||
state_before = game.get_state_description()
|
||||
available_actions = game.get_available_actions()
|
||||
api_record_index = len(self.api_records)
|
||||
|
||||
# Choose action using LLM
|
||||
action = self.choose_action(game, verbose=verbose)
|
||||
|
||||
# Execute action
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
|
||||
# Store experience
|
||||
self.update_experience(state_before, action, feedback, reward)
|
||||
|
||||
# Record trajectory
|
||||
trajectory.append({
|
||||
"step": steps + 1,
|
||||
"state_before": state_before,
|
||||
"available_actions": available_actions,
|
||||
"action": action,
|
||||
"reward": reward,
|
||||
"feedback": feedback,
|
||||
"api_record_index": (
|
||||
api_record_index
|
||||
if len(self.api_records) > api_record_index
|
||||
else None
|
||||
),
|
||||
})
|
||||
|
||||
total_reward += reward
|
||||
steps += 1
|
||||
|
||||
if verbose:
|
||||
print(f"\n🎯 Action Result:")
|
||||
print(f" Feedback: {feedback}")
|
||||
if reward > 0:
|
||||
print(f" Reward: ✨ +{reward:.1f}")
|
||||
else:
|
||||
print(f" Reward: 📉 {reward:.1f}")
|
||||
print(f" Total reward so far: {total_reward:.1f}")
|
||||
|
||||
# Add a pause between steps for readability
|
||||
if not done:
|
||||
print("\n" + "."*60)
|
||||
|
||||
# Update statistics
|
||||
self.episode_rewards.append(total_reward)
|
||||
self.episode_lengths.append(steps)
|
||||
if game.victory:
|
||||
self.victories += 1
|
||||
self.total_episodes += 1
|
||||
self.episode_trajectories.append({
|
||||
"phase": phase,
|
||||
"episode": (
|
||||
sum(1 for item in self.episode_trajectories
|
||||
if item["phase"] == phase) + 1
|
||||
),
|
||||
"victory": game.victory,
|
||||
"total_reward": total_reward,
|
||||
"steps": steps,
|
||||
"trajectory": trajectory,
|
||||
})
|
||||
|
||||
if verbose:
|
||||
print("\n" + "🏁"*30)
|
||||
if game.victory:
|
||||
print("🎉 VICTORY! The LLM found the treasure!")
|
||||
else:
|
||||
print("💀 GAME OVER! Better luck next time.")
|
||||
print(f" Final Score: {total_reward:.1f}")
|
||||
print(f" Total Steps: {steps}")
|
||||
print(f" API Calls Used: {self.api_calls}")
|
||||
print("🏁"*30)
|
||||
|
||||
return total_reward, steps, game.victory
|
||||
|
||||
def train(self, num_episodes: int = 20, verbose: bool = True, stochastic: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
'Train' the agent through in-context learning over multiple episodes.
|
||||
Note: Unlike traditional RL, there's no explicit training - just experience accumulation.
|
||||
|
||||
Args:
|
||||
num_episodes: Number of episodes to play
|
||||
verbose: Whether to print details
|
||||
stochastic: Whether to use stochastic environment
|
||||
"""
|
||||
game = TreasureHuntGame(stochastic=stochastic)
|
||||
|
||||
print("\n" + "🚀"*30)
|
||||
print("LLM IN-CONTEXT LEARNING EXPERIMENT")
|
||||
print("🚀"*30)
|
||||
print(f"\n📝 Will play {num_episodes} episodes to learn the game")
|
||||
print("🧠 The LLM learns by accumulating experiences in context")
|
||||
print("⚡ Each decision shows the full reasoning process")
|
||||
|
||||
for episode in range(num_episodes):
|
||||
print(f"\n\n{'🎯'*30}")
|
||||
print(f"EPISODE {episode + 1} of {num_episodes}")
|
||||
print(f"{'🎯'*30}")
|
||||
print(f"📚 Experiences accumulated so far: {len(self.experiences)}")
|
||||
|
||||
# Show full process for first 3 episodes, then reduce verbosity
|
||||
show_full = verbose and (episode < 3 or episode == num_episodes - 1)
|
||||
|
||||
if not show_full and verbose:
|
||||
print("\n(Reducing verbosity for middle episodes to save space...)")
|
||||
|
||||
reward, steps, victory = self.play_episode(
|
||||
game, verbose=show_full, phase="training"
|
||||
)
|
||||
|
||||
if not show_full:
|
||||
# Still show summary even when not fully verbose
|
||||
print(f"\n📊 Episode {episode + 1} Summary:")
|
||||
print(f" Result: {'🎉 Victory!' if victory else '💀 Failed'}")
|
||||
print(f" Total Reward: {reward:.2f}")
|
||||
print(f" Steps Taken: {steps}")
|
||||
print(f" Total API Calls So Far: {self.api_calls}")
|
||||
|
||||
# Show learning progress
|
||||
if len(self.episode_rewards) >= 3:
|
||||
recent_victories = sum(1 for r in self.episode_rewards[-3:] if r > 50)
|
||||
recent_avg = sum(self.episode_rewards[-3:]) / 3
|
||||
print(f"\n📈 Recent Performance (last 3 episodes):")
|
||||
print(f" Victories: {recent_victories}/3")
|
||||
print(f" Average Reward: {recent_avg:.2f}")
|
||||
|
||||
# Add delay to respect rate limits
|
||||
if episode < num_episodes - 1:
|
||||
print("\n⏳ Waiting 1 second for API rate limits...")
|
||||
time.sleep(1)
|
||||
|
||||
return {
|
||||
"total_episodes": self.total_episodes,
|
||||
"total_victories": self.victories,
|
||||
"victory_rate": self.victories / self.total_episodes if self.total_episodes > 0 else 0,
|
||||
"total_api_calls": self.api_calls,
|
||||
"total_tokens": self.total_tokens,
|
||||
"experiences_collected": len(self.experiences),
|
||||
"episode_rewards": self.episode_rewards,
|
||||
"episode_lengths": self.episode_lengths
|
||||
}
|
||||
|
||||
def evaluate(self, num_episodes: int = 10, verbose: bool = False, stochastic: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
Evaluate the agent's performance using accumulated experiences.
|
||||
|
||||
Args:
|
||||
num_episodes: Number of episodes to evaluate
|
||||
verbose: Whether to print details
|
||||
stochastic: Whether to use stochastic environment
|
||||
"""
|
||||
game = TreasureHuntGame(stochastic=stochastic)
|
||||
eval_rewards = []
|
||||
eval_lengths = []
|
||||
eval_victories = 0
|
||||
|
||||
for episode in range(num_episodes):
|
||||
reward, steps, victory = self.play_episode(
|
||||
game, verbose=verbose, phase="evaluation"
|
||||
)
|
||||
|
||||
eval_rewards.append(reward)
|
||||
eval_lengths.append(steps)
|
||||
if victory:
|
||||
eval_victories += 1
|
||||
|
||||
if verbose:
|
||||
print(f"Episode {episode + 1}: Reward={reward:.2f}, Steps={steps}, Victory={victory}")
|
||||
|
||||
return {
|
||||
"num_episodes": num_episodes,
|
||||
"victories": eval_victories,
|
||||
"victory_rate": eval_victories / num_episodes if num_episodes else 0.0,
|
||||
"avg_reward": sum(eval_rewards) / len(eval_rewards) if eval_rewards else 0.0,
|
||||
"avg_length": sum(eval_lengths) / len(eval_lengths) if eval_lengths else 0.0,
|
||||
"total_api_calls": self.api_calls,
|
||||
"experiences_used": len(self.experiences)
|
||||
}
|
||||
|
||||
def save_experiences(self, filepath: str):
|
||||
"""Save experiences to file for analysis."""
|
||||
data = {
|
||||
"backend": {
|
||||
"provider": self.provider,
|
||||
"base_url": self.base_url,
|
||||
"model": self.model,
|
||||
"using_openrouter": self.using_openrouter,
|
||||
},
|
||||
"experiences": [asdict(exp) for exp in self.experiences],
|
||||
"episode_trajectories": self.episode_trajectories,
|
||||
"api_records": self.api_records,
|
||||
"statistics": {
|
||||
"total_episodes": self.total_episodes,
|
||||
"victories": self.victories,
|
||||
"api_calls": self.api_calls,
|
||||
"total_tokens": self.total_tokens
|
||||
}
|
||||
}
|
||||
|
||||
with open(filepath, 'w') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
|
||||
def load_experiences(self, filepath: str):
|
||||
"""Load experiences from file."""
|
||||
with open(filepath, 'r') as f:
|
||||
data = json.load(f)
|
||||
|
||||
self.experiences = [
|
||||
GameExperience(**exp) for exp in data["experiences"]
|
||||
]
|
||||
|
||||
stats = data.get("statistics", {})
|
||||
self.total_episodes = stats.get("total_episodes", 0)
|
||||
self.victories = stats.get("victories", 0)
|
||||
self.api_calls = stats.get("api_calls", 0)
|
||||
self.total_tokens = stats.get("total_tokens", 0)
|
||||
@@ -0,0 +1,163 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick demo showing the LLM learning process in detail.
|
||||
This script runs a simplified experiment to demonstrate how LLMs learn from experience.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables from .env file
|
||||
load_dotenv()
|
||||
|
||||
# Add parent directory to path for imports
|
||||
sys.path.append(str(Path(__file__).parent))
|
||||
|
||||
from game_environment import TreasureHuntGame
|
||||
from llm_agent import LLMAgent
|
||||
|
||||
|
||||
def show_game_solution():
|
||||
"""Show the optimal solution to the game."""
|
||||
print("\n" + "="*70)
|
||||
print("GAME SOLUTION (for reference)")
|
||||
print("="*70)
|
||||
|
||||
game = TreasureHuntGame()
|
||||
print(game.get_hidden_rules())
|
||||
|
||||
print("\n📝 Optimal solution path:")
|
||||
print("1. Take rusty sword (in entrance)")
|
||||
print("2. Go east to storage")
|
||||
print("3. Take red key")
|
||||
print("4. Take magic crystal")
|
||||
print("5. Try crafting → creates silver sword")
|
||||
print("6. Go west to entrance")
|
||||
print("7. Go north to hallway (uses red key automatically)")
|
||||
print("8. Go north to guard room")
|
||||
print("9. Attack with silver sword → defeats strong guard")
|
||||
print("10. Go east to treasure room")
|
||||
print("11. Take dragon's treasure → Victory!")
|
||||
print("\n✨ Total moves: ~11-12 (optimal)")
|
||||
|
||||
|
||||
def run_llm_demo():
|
||||
"""Run a simplified LLM demo with just a few episodes."""
|
||||
print("\n" + "🤖"*35)
|
||||
print("LLM IN-CONTEXT LEARNING DEMO")
|
||||
print("🤖"*35)
|
||||
|
||||
# Check API key
|
||||
provider = os.getenv("LLM_PROVIDER", "moonshot").lower()
|
||||
api_key = os.getenv("DASHSCOPE_API_KEY") if provider in {"dashscope", "qwen", "bailian"} else os.getenv("MOONSHOT_API_KEY")
|
||||
if not api_key and not os.getenv("OPENROUTER_API_KEY"):
|
||||
print(f"\n❌ Error: API key for provider '{provider}' not set.")
|
||||
print("Please set your Kimi API key:")
|
||||
print(" export DASHSCOPE_API_KEY='your-key-here' # for dashscope/qwen/bailian")
|
||||
print(" export MOONSHOT_API_KEY='your-key-here' # for moonshot/kimi")
|
||||
print("\nGet your key at: https://platform.moonshot.cn/")
|
||||
print("Or set OPENROUTER_API_KEY as a universal fallback.")
|
||||
return
|
||||
|
||||
print("\n✅ API key found!")
|
||||
print("🧠 Initializing Kimi K3 LLM agent...")
|
||||
|
||||
# Initialize agent
|
||||
agent = LLMAgent(
|
||||
api_key=api_key,
|
||||
model=os.getenv("MOONSHOT_MODEL", "kimi-k3"),
|
||||
provider=provider,
|
||||
temperature=0.7,
|
||||
max_experiences=30
|
||||
)
|
||||
|
||||
print("\n📚 The LLM will play 3 episodes to learn the game")
|
||||
print("👀 Watch how it reasons and learns from each experience!\n")
|
||||
|
||||
# Play 3 episodes
|
||||
game = TreasureHuntGame()
|
||||
|
||||
for episode in range(3):
|
||||
print("\n" + "🎮"*35)
|
||||
print(f"EPISODE {episode + 1} of 3")
|
||||
print("🎮"*35)
|
||||
|
||||
# Show what the LLM has learned so far
|
||||
if agent.experiences:
|
||||
print(f"\n📊 Experience Memory: {len(agent.experiences)} interactions stored")
|
||||
|
||||
# Show some key learnings
|
||||
successful = [e for e in agent.experiences if e.success]
|
||||
if successful:
|
||||
print("✅ Successful patterns discovered:")
|
||||
for exp in successful[-3:]:
|
||||
print(f" • {exp.action} → reward: {exp.reward:.1f}")
|
||||
|
||||
failed = [e for e in agent.experiences if not e.success]
|
||||
if failed and len(failed) > 5:
|
||||
print("❌ Mistakes to avoid:")
|
||||
for exp in failed[-2:]:
|
||||
print(f" • {exp.action} → reward: {exp.reward:.1f}")
|
||||
|
||||
# Play episode
|
||||
reward, steps, victory = agent.play_episode(game, verbose=True)
|
||||
|
||||
print(f"\n📈 Episode {episode + 1} Performance:")
|
||||
print(f" • Result: {'🎉 Victory!' if victory else '💀 Failed'}")
|
||||
print(f" • Total Reward: {reward:.2f}")
|
||||
print(f" • Steps Taken: {steps}")
|
||||
print(f" • Experiences Collected: {len(agent.experiences)}")
|
||||
|
||||
if victory:
|
||||
print("\n🎊 The LLM learned to solve the game!")
|
||||
print(f" It took {episode + 1} episodes to learn")
|
||||
print(f" Total API calls used: {agent.api_calls}")
|
||||
break
|
||||
|
||||
if episode < 2:
|
||||
print("\n⏳ Waiting 2 seconds before next episode...")
|
||||
import time
|
||||
time.sleep(2)
|
||||
|
||||
# Summary
|
||||
print("\n" + "="*70)
|
||||
print("DEMO SUMMARY")
|
||||
print("="*70)
|
||||
print(f"📊 Total episodes played: {episode + 1}")
|
||||
print(f"🧠 Total experiences collected: {len(agent.experiences)}")
|
||||
print(f"🎯 Victories: {agent.victories}")
|
||||
print(f"📡 API calls made: {agent.api_calls}")
|
||||
|
||||
if agent.victories > 0:
|
||||
print("\n✨ Key Insight:")
|
||||
print("The LLM learned to solve the game by reasoning about patterns")
|
||||
print("in just a few episodes, without any parameter updates!")
|
||||
print("Traditional RL would need thousands of episodes for the same result.")
|
||||
else:
|
||||
print("\n💡 Note: The LLM is still learning. Run more episodes to see it succeed!")
|
||||
|
||||
|
||||
def main():
|
||||
"""Main entry point."""
|
||||
print("\n" + "🎯"*35)
|
||||
print("LEARNING FROM EXPERIENCE: LLM DEMO")
|
||||
print("Replicating insights from 'The Second Half'")
|
||||
print("🎯"*35)
|
||||
|
||||
# Show solution first
|
||||
show_game_solution()
|
||||
|
||||
# Ask user if they want to continue
|
||||
response = input("\n▶️ Ready to see how an LLM learns this game? (y/n): ").strip().lower()
|
||||
|
||||
if response == 'y':
|
||||
run_llm_demo()
|
||||
else:
|
||||
print("\n👋 Okay, goodbye!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,5 @@
|
||||
numpy>=1.24.0
|
||||
matplotlib>=3.5.0
|
||||
seaborn>=0.12.0
|
||||
openai>=1.0.0
|
||||
python-dotenv>=1.0.0
|
||||
@@ -0,0 +1,404 @@
|
||||
"""
|
||||
Traditional Reinforcement Learning Agent using Q-learning.
|
||||
This demonstrates the classical RL approach that requires extensive training.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
import pickle
|
||||
from collections import defaultdict
|
||||
from typing import Dict, List, Tuple, Any
|
||||
import random
|
||||
from game_environment import TreasureHuntGame
|
||||
|
||||
|
||||
class QLearningAgent:
|
||||
"""
|
||||
Q-learning agent for the treasure hunt game.
|
||||
Uses tabular Q-learning with state-action pairs.
|
||||
"""
|
||||
|
||||
def __init__(self,
|
||||
learning_rate: float = 0.2,
|
||||
discount_factor: float = 0.99,
|
||||
epsilon: float = 1.0,
|
||||
epsilon_decay: float = 0.9995,
|
||||
epsilon_min: float = 0.1):
|
||||
"""
|
||||
Initialize Q-learning agent.
|
||||
|
||||
Args:
|
||||
learning_rate: Alpha parameter for Q-value updates
|
||||
discount_factor: Gamma parameter for future rewards
|
||||
epsilon: Initial exploration rate
|
||||
epsilon_decay: Rate at which epsilon decreases
|
||||
epsilon_min: Minimum exploration rate
|
||||
"""
|
||||
self.learning_rate = learning_rate
|
||||
self.discount_factor = discount_factor
|
||||
self.epsilon = epsilon
|
||||
self.epsilon_decay = epsilon_decay
|
||||
self.epsilon_min = epsilon_min
|
||||
|
||||
# Q-table: state_hash -> action -> Q-value
|
||||
self.q_table = defaultdict(lambda: defaultdict(float))
|
||||
|
||||
# Statistics
|
||||
self.episode_rewards = []
|
||||
self.episode_lengths = []
|
||||
self.episode_victories = [] # Per-episode victory flag (1/0), for learning curves
|
||||
self.victories = 0
|
||||
self.total_episodes = 0
|
||||
self.learning_curve = [] # Snapshots recorded at checkpoints during train()
|
||||
|
||||
def _get_state_hash(self, game: TreasureHuntGame) -> str:
|
||||
"""
|
||||
Create a hashable representation of the game state.
|
||||
This is crucial for tabular Q-learning.
|
||||
"""
|
||||
# Include relevant state information
|
||||
state_parts = [
|
||||
game.current_room.name,
|
||||
tuple(sorted([item.name for item in game.inventory])),
|
||||
tuple(sorted([item.name for item in game.current_room.items])),
|
||||
tuple(sorted(game.current_room.locked_exits.items())),
|
||||
game.current_room.has_guard and not game.current_room.guard_defeated
|
||||
]
|
||||
|
||||
return str(state_parts)
|
||||
|
||||
def choose_action(self, game: TreasureHuntGame, training: bool = True) -> str:
|
||||
"""
|
||||
Choose an action using epsilon-greedy strategy.
|
||||
"""
|
||||
available_actions = game.get_available_actions()
|
||||
|
||||
if not available_actions:
|
||||
return "look around"
|
||||
|
||||
# Exploration vs exploitation
|
||||
if training and random.random() < self.epsilon:
|
||||
# Explore: choose random action
|
||||
return random.choice(available_actions)
|
||||
else:
|
||||
# Exploit: choose best action based on Q-values
|
||||
state_hash = self._get_state_hash(game)
|
||||
|
||||
# Get Q-values for all available actions
|
||||
action_values = {
|
||||
action: self.q_table[state_hash][action]
|
||||
for action in available_actions
|
||||
}
|
||||
|
||||
# If all Q-values are 0 (unexplored), choose randomly
|
||||
if all(v == 0 for v in action_values.values()):
|
||||
return random.choice(available_actions)
|
||||
|
||||
# Choose action with highest Q-value
|
||||
return max(action_values, key=action_values.get)
|
||||
|
||||
def update_q_value(self, state: str, action: str, reward: float,
|
||||
next_state: str, next_actions: List[str], done: bool):
|
||||
"""
|
||||
Update Q-value using the Q-learning update rule.
|
||||
Q(s,a) <- Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]
|
||||
"""
|
||||
current_q = self.q_table[state][action]
|
||||
|
||||
if done:
|
||||
# Terminal state
|
||||
target = reward
|
||||
else:
|
||||
# Get maximum Q-value for next state
|
||||
if next_actions:
|
||||
max_next_q = max(
|
||||
self.q_table[next_state][a] for a in next_actions
|
||||
)
|
||||
else:
|
||||
max_next_q = 0
|
||||
|
||||
target = reward + self.discount_factor * max_next_q
|
||||
|
||||
# Update Q-value
|
||||
self.q_table[state][action] = (
|
||||
current_q + self.learning_rate * (target - current_q)
|
||||
)
|
||||
|
||||
def train_episode(self, game: TreasureHuntGame) -> Tuple[float, int, bool]:
|
||||
"""
|
||||
Train the agent for one episode.
|
||||
|
||||
Returns:
|
||||
Total reward, number of steps, victory status
|
||||
"""
|
||||
game.reset()
|
||||
total_reward = 0
|
||||
steps = 0
|
||||
|
||||
while not game.game_over:
|
||||
# Get current state
|
||||
state_hash = self._get_state_hash(game)
|
||||
|
||||
# Choose action
|
||||
action = self.choose_action(game, training=True)
|
||||
|
||||
# Execute action
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
|
||||
# Get next state
|
||||
next_state_hash = self._get_state_hash(game)
|
||||
next_actions = game.get_available_actions() if not done else []
|
||||
|
||||
# Update Q-value
|
||||
self.update_q_value(
|
||||
state_hash, action, reward,
|
||||
next_state_hash, next_actions, done
|
||||
)
|
||||
|
||||
total_reward += reward
|
||||
steps += 1
|
||||
|
||||
# Decay epsilon
|
||||
self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
|
||||
|
||||
# Update statistics
|
||||
self.episode_rewards.append(total_reward)
|
||||
self.episode_lengths.append(steps)
|
||||
self.episode_victories.append(1 if game.victory else 0)
|
||||
if game.victory:
|
||||
self.victories += 1
|
||||
self.total_episodes += 1
|
||||
|
||||
return total_reward, steps, game.victory
|
||||
|
||||
def train(self, num_episodes: int = 1000, verbose: bool = True,
|
||||
stochastic: bool = False, checkpoint_interval: int = 0) -> Dict[str, Any]:
|
||||
"""
|
||||
Train the agent for multiple episodes.
|
||||
|
||||
Args:
|
||||
num_episodes: Number of episodes to train
|
||||
verbose: Whether to print progress
|
||||
stochastic: Whether to use stochastic environment
|
||||
checkpoint_interval: If > 0, record a learning-curve snapshot
|
||||
(episode, windowed victory rate, Q-table size, epsilon) every
|
||||
this many episodes. Snapshots are stored in self.learning_curve.
|
||||
"""
|
||||
game = TreasureHuntGame(stochastic=stochastic)
|
||||
|
||||
# Adjust hyperparameters for stochastic environment
|
||||
if stochastic:
|
||||
# Slightly slower epsilon decay for stochastic environments
|
||||
original_decay = self.epsilon_decay
|
||||
self.epsilon_decay = min(0.9999, self.epsilon_decay * 1.001)
|
||||
if verbose:
|
||||
print(f"Adjusted epsilon_decay from {original_decay:.4f} to {self.epsilon_decay:.4f} for stochastic environment\n")
|
||||
|
||||
window = checkpoint_interval if checkpoint_interval and checkpoint_interval > 0 else 1000
|
||||
|
||||
for episode in range(num_episodes):
|
||||
reward, steps, victory = self.train_episode(game)
|
||||
|
||||
# Record a learning-curve snapshot at each checkpoint
|
||||
if checkpoint_interval and checkpoint_interval > 0 and (episode + 1) % checkpoint_interval == 0:
|
||||
recent = self.episode_victories[-window:]
|
||||
self.learning_curve.append({
|
||||
"episode": episode + 1,
|
||||
"victory_rate": sum(recent) / len(recent) if recent else 0.0,
|
||||
"q_table_size": len(self.q_table),
|
||||
"epsilon": self.epsilon,
|
||||
})
|
||||
|
||||
if verbose and (episode + 1) % 100 == 0:
|
||||
recent_rewards = self.episode_rewards[-100:]
|
||||
recent_victories = sum(
|
||||
1 for r in recent_rewards if r > 50 # Approximate victory
|
||||
)
|
||||
avg_reward = np.mean(recent_rewards)
|
||||
|
||||
print(f"Episode {episode + 1}/{num_episodes}")
|
||||
print(f" Avg Reward (last 100): {avg_reward:.2f}")
|
||||
print(f" Victories (last 100): {recent_victories}")
|
||||
print(f" Epsilon: {self.epsilon:.3f}")
|
||||
print(f" Q-table size: {len(self.q_table)}")
|
||||
print()
|
||||
|
||||
return {
|
||||
"total_episodes": self.total_episodes,
|
||||
"total_victories": self.victories,
|
||||
"victory_rate": self.victories / self.total_episodes if self.total_episodes else 0.0,
|
||||
"final_epsilon": self.epsilon,
|
||||
"q_table_size": len(self.q_table),
|
||||
"episode_rewards": self.episode_rewards,
|
||||
"episode_lengths": self.episode_lengths,
|
||||
"learning_curve": self.learning_curve,
|
||||
}
|
||||
|
||||
def evaluate(self, num_episodes: int = 100, verbose: bool = False, stochastic: bool = False) -> Dict[str, Any]:
|
||||
"""
|
||||
Evaluate the trained agent without learning.
|
||||
|
||||
Args:
|
||||
num_episodes: Number of episodes to evaluate
|
||||
verbose: Whether to print details
|
||||
stochastic: Whether to use stochastic environment
|
||||
"""
|
||||
game = TreasureHuntGame(stochastic=stochastic)
|
||||
eval_rewards = []
|
||||
eval_lengths = []
|
||||
eval_victories = 0
|
||||
|
||||
# Store original epsilon and set to 0 for evaluation
|
||||
original_epsilon = self.epsilon
|
||||
self.epsilon = 0
|
||||
|
||||
for episode in range(num_episodes):
|
||||
game.reset()
|
||||
total_reward = 0
|
||||
steps = 0
|
||||
|
||||
while not game.game_over:
|
||||
action = self.choose_action(game, training=False)
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
total_reward += reward
|
||||
steps += 1
|
||||
|
||||
if verbose and episode == 0: # Show first evaluation episode
|
||||
print(f"Step {steps}: {action}")
|
||||
print(f"Feedback: {feedback}")
|
||||
print()
|
||||
|
||||
eval_rewards.append(total_reward)
|
||||
eval_lengths.append(steps)
|
||||
if game.victory:
|
||||
eval_victories += 1
|
||||
|
||||
# Restore epsilon
|
||||
self.epsilon = original_epsilon
|
||||
|
||||
return {
|
||||
"num_episodes": num_episodes,
|
||||
"victories": eval_victories,
|
||||
"victory_rate": eval_victories / num_episodes if num_episodes else 0.0,
|
||||
"avg_reward": sum(eval_rewards) / len(eval_rewards) if len(eval_rewards) > 0 else 0.0,
|
||||
"std_reward": float(np.std(eval_rewards)) if len(eval_rewards) > 0 else 0.0,
|
||||
"avg_length": sum(eval_lengths) / len(eval_lengths) if len(eval_lengths) > 0 else 0.0,
|
||||
"std_length": float(np.std(eval_lengths)) if len(eval_lengths) > 0 else 0.0
|
||||
}
|
||||
|
||||
def save(self, filepath: str):
|
||||
"""Save the Q-table and parameters."""
|
||||
data = {
|
||||
"q_table": dict(self.q_table),
|
||||
"epsilon": self.epsilon,
|
||||
"learning_rate": self.learning_rate,
|
||||
"discount_factor": self.discount_factor,
|
||||
"statistics": {
|
||||
"total_episodes": self.total_episodes,
|
||||
"victories": self.victories,
|
||||
"episode_rewards": self.episode_rewards,
|
||||
"episode_lengths": self.episode_lengths
|
||||
}
|
||||
}
|
||||
|
||||
with open(filepath, 'wb') as f:
|
||||
pickle.dump(data, f)
|
||||
|
||||
def load(self, filepath: str):
|
||||
"""Load a saved Q-table and parameters."""
|
||||
with open(filepath, 'rb') as f:
|
||||
data = pickle.load(f)
|
||||
|
||||
self.q_table = defaultdict(lambda: defaultdict(float))
|
||||
for state, actions in data["q_table"].items():
|
||||
for action, value in actions.items():
|
||||
self.q_table[state][action] = value
|
||||
|
||||
self.epsilon = data["epsilon"]
|
||||
self.learning_rate = data["learning_rate"]
|
||||
self.discount_factor = data["discount_factor"]
|
||||
|
||||
stats = data.get("statistics", {})
|
||||
self.total_episodes = stats.get("total_episodes", 0)
|
||||
self.victories = stats.get("victories", 0)
|
||||
self.episode_rewards = stats.get("episode_rewards", [])
|
||||
self.episode_lengths = stats.get("episode_lengths", [])
|
||||
|
||||
|
||||
class DQNAgent:
|
||||
"""
|
||||
Deep Q-Network agent for comparison.
|
||||
Uses neural network function approximation instead of tabular Q-learning.
|
||||
"""
|
||||
|
||||
def __init__(self,
|
||||
state_dim: int = 128,
|
||||
hidden_dim: int = 256,
|
||||
learning_rate: float = 0.001,
|
||||
discount_factor: float = 0.95,
|
||||
epsilon: float = 1.0,
|
||||
epsilon_decay: float = 0.995,
|
||||
epsilon_min: float = 0.01,
|
||||
batch_size: int = 32,
|
||||
memory_size: int = 10000):
|
||||
"""
|
||||
Initialize DQN agent with neural network.
|
||||
Note: Simplified implementation for demonstration.
|
||||
"""
|
||||
self.state_dim = state_dim
|
||||
self.hidden_dim = hidden_dim
|
||||
self.learning_rate = learning_rate
|
||||
self.discount_factor = discount_factor
|
||||
self.epsilon = epsilon
|
||||
self.epsilon_decay = epsilon_decay
|
||||
self.epsilon_min = epsilon_min
|
||||
self.batch_size = batch_size
|
||||
|
||||
# Experience replay buffer
|
||||
self.memory = []
|
||||
self.memory_size = memory_size
|
||||
|
||||
# Statistics
|
||||
self.episode_rewards = []
|
||||
self.episode_lengths = []
|
||||
self.victories = 0
|
||||
self.total_episodes = 0
|
||||
|
||||
# Note: For full implementation, we would use PyTorch or TensorFlow
|
||||
# This is a simplified placeholder
|
||||
print("Note: DQN implementation requires neural network library.")
|
||||
print("Using simplified random policy for demonstration.")
|
||||
|
||||
def choose_action(self, game: TreasureHuntGame, training: bool = True) -> str:
|
||||
"""Choose action (simplified for demonstration)."""
|
||||
available_actions = game.get_available_actions()
|
||||
if not available_actions:
|
||||
return "look around"
|
||||
|
||||
# Simplified: just use epsilon-greedy with random selection
|
||||
if training and random.random() < self.epsilon:
|
||||
return random.choice(available_actions)
|
||||
else:
|
||||
# In full implementation, this would use neural network
|
||||
return random.choice(available_actions)
|
||||
|
||||
def train_episode(self, game: TreasureHuntGame) -> Tuple[float, int, bool]:
|
||||
"""Train for one episode (simplified)."""
|
||||
game.reset()
|
||||
total_reward = 0
|
||||
steps = 0
|
||||
|
||||
while not game.game_over:
|
||||
action = self.choose_action(game, training=True)
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
total_reward += reward
|
||||
steps += 1
|
||||
|
||||
self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
|
||||
|
||||
self.episode_rewards.append(total_reward)
|
||||
self.episode_lengths.append(steps)
|
||||
if game.victory:
|
||||
self.victories += 1
|
||||
self.total_episodes += 1
|
||||
|
||||
return total_reward, steps, game.victory
|
||||
@@ -0,0 +1,255 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run and retain the exact real Experiment 8-2 comparison.
|
||||
|
||||
The manuscript compares 10,000 deterministic Q-learning episodes with Kimi K3's
|
||||
first attempt in the same treasure-hunt environment. A failed manuscript
|
||||
hypothesis is still a completed experiment; acceptance therefore verifies the
|
||||
protocol and evidence provenance separately from the observed outcome.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import random
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
from experiment import ExperimentRunner
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _git_revision() -> str | None:
|
||||
try:
|
||||
return subprocess.run(
|
||||
["git", "rev-parse", "HEAD"],
|
||||
cwd=ROOT,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
).stdout.strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def _write_json(path: Path, payload: Any) -> None:
|
||||
def json_default(value: Any) -> Any:
|
||||
if isinstance(value, np.generic):
|
||||
return value.item()
|
||||
if isinstance(value, Path):
|
||||
return str(value)
|
||||
raise TypeError(f"Object of type {type(value).__name__} is not JSON serializable")
|
||||
|
||||
path.write_text(
|
||||
json.dumps(
|
||||
payload, ensure_ascii=False, indent=2, default=json_default
|
||||
) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Real, evidence-retaining Chapter 7 Experiment 8-2 campaign"
|
||||
)
|
||||
parser.add_argument("--model", default="kimi-k3")
|
||||
parser.add_argument("--seed", type=int, default=42)
|
||||
parser.add_argument("--rl-episodes", type=int, default=10_000)
|
||||
parser.add_argument("--rl-eval-episodes", type=int, default=100)
|
||||
parser.add_argument(
|
||||
"--llm-eval-episodes",
|
||||
type=int,
|
||||
default=0,
|
||||
help="The manuscript's core observation is the first attempt; optional later evaluations are separate.",
|
||||
)
|
||||
parser.add_argument("--output-root", default=str(ROOT / "validation"))
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.rl_episodes != 10_000 or args.rl_eval_episodes != 100:
|
||||
parser.error("canonical Experiment 8-2 requires 10,000 RL training and 100 RL evaluation episodes")
|
||||
if args.llm_eval_episodes < 0:
|
||||
parser.error("--llm-eval-episodes must be non-negative")
|
||||
if not os.getenv("MOONSHOT_API_KEY"):
|
||||
parser.error(
|
||||
"MOONSHOT_API_KEY is required: an OpenRouter substitute is not exact Kimi K3 evidence"
|
||||
)
|
||||
|
||||
random.seed(args.seed)
|
||||
np.random.seed(args.seed)
|
||||
|
||||
output_root = Path(args.output_root).expanduser().resolve()
|
||||
runner = ExperimentRunner(results_dir=str(output_root))
|
||||
started_at = datetime.now(timezone.utc)
|
||||
|
||||
rl_results = runner.run_rl_experiment(
|
||||
num_training_episodes=args.rl_episodes,
|
||||
num_eval_episodes=args.rl_eval_episodes,
|
||||
verbose=False,
|
||||
stochastic=False,
|
||||
checkpoint_interval=1000,
|
||||
learning_rate=0.2,
|
||||
discount_factor=0.99,
|
||||
epsilon_decay=0.9995,
|
||||
epsilon_min=0.1,
|
||||
)
|
||||
llm_results = runner.run_llm_experiment(
|
||||
num_training_episodes=1,
|
||||
num_eval_episodes=args.llm_eval_episodes,
|
||||
verbose=False,
|
||||
stochastic=False,
|
||||
model=args.model,
|
||||
)
|
||||
runner.results = {"rl": rl_results, "llm": llm_results}
|
||||
_write_json(runner.experiment_dir / "experiment_results.json", runner.results)
|
||||
|
||||
raw_path = runner.experiment_dir / "llm_experiences.json"
|
||||
raw = json.loads(raw_path.read_text(encoding="utf-8"))
|
||||
training = [
|
||||
episode
|
||||
for episode in raw.get("episode_trajectories", [])
|
||||
if episode.get("phase") == "training"
|
||||
]
|
||||
first_attempt = training[0] if training else None
|
||||
api_records = raw.get("api_records", [])
|
||||
response_ids_present = all(
|
||||
bool((record.get("response") or {}).get("id")) for record in api_records
|
||||
)
|
||||
response_contents_present = all(
|
||||
bool((record.get("response") or {}).get("content")) for record in api_records
|
||||
)
|
||||
no_api_errors = all(not record.get("error") for record in api_records)
|
||||
no_fallbacks = all(not record.get("fallback_used") for record in api_records)
|
||||
direct_exact_kimi = (
|
||||
raw.get("backend", {}).get("provider") == "moonshot"
|
||||
and raw.get("backend", {}).get("model") == "kimi-k3"
|
||||
and raw.get("backend", {}).get("using_openrouter") is False
|
||||
)
|
||||
response_models = sorted(
|
||||
{
|
||||
(record.get("response") or {}).get("model")
|
||||
for record in api_records
|
||||
if (record.get("response") or {}).get("model")
|
||||
}
|
||||
)
|
||||
|
||||
protocol_gates = {
|
||||
"same_deterministic_environment": True,
|
||||
"q_learning_10000_training_episodes": rl_results["training_episodes"] == 10_000,
|
||||
"q_learning_100_evaluation_episodes": args.rl_eval_episodes == 100,
|
||||
"q_learning_reached_full_evaluation_success": rl_results["eval_victory_rate"] == 1.0,
|
||||
"one_kimi_first_attempt_recorded": len(training) == 1,
|
||||
"direct_official_moonshot_kimi_k3": direct_exact_kimi,
|
||||
"one_real_response_per_first_attempt_action": bool(first_attempt)
|
||||
and len(api_records) == first_attempt["steps"],
|
||||
"provider_response_ids_retained": bool(api_records) and response_ids_present,
|
||||
"provider_response_content_retained": bool(api_records) and response_contents_present,
|
||||
"zero_api_errors": no_api_errors,
|
||||
"zero_fallback_actions": no_fallbacks,
|
||||
}
|
||||
acceptance_complete = all(protocol_gates.values())
|
||||
first_attempt_victory = bool(first_attempt and first_attempt["victory"])
|
||||
first_attempt_steps = first_attempt["steps"] if first_attempt else None
|
||||
|
||||
evidence = {
|
||||
"schema_version": 1,
|
||||
"experiment_id": "8-2",
|
||||
"title": "Traditional RL versus Kimi K3 in the same treasure-hunt environment",
|
||||
"started_at": started_at.isoformat(),
|
||||
"finished_at": datetime.now(timezone.utc).isoformat(),
|
||||
"git_revision": _git_revision(),
|
||||
"command": {
|
||||
"argv": sys.argv,
|
||||
"seed": args.seed,
|
||||
"deterministic": True,
|
||||
},
|
||||
"runtime": {
|
||||
"python": sys.version,
|
||||
"platform": platform.platform(),
|
||||
},
|
||||
"backend": raw.get("backend"),
|
||||
"provider_response_models": response_models,
|
||||
"usage": {
|
||||
"successful_api_calls": raw.get("statistics", {}).get("api_calls"),
|
||||
"api_attempts": len(api_records),
|
||||
"total_tokens": raw.get("statistics", {}).get("total_tokens"),
|
||||
"provider_cost": None,
|
||||
"provider_cost_note": "The response exposed token usage but no authoritative billed cost; unknown is not zero.",
|
||||
},
|
||||
"q_learning": {
|
||||
"training_episodes": rl_results["training_episodes"],
|
||||
"training_time_seconds": rl_results["training_time"],
|
||||
"training_victory_rate": rl_results["training_victory_rate"],
|
||||
"evaluation_victory_rate": rl_results["eval_victory_rate"],
|
||||
"evaluation_average_steps": rl_results["eval_avg_steps"],
|
||||
"q_table_states": rl_results["q_table_size"],
|
||||
"learning_curve": rl_results["learning_curve"],
|
||||
},
|
||||
"k3_first_attempt": {
|
||||
"victory": first_attempt_victory,
|
||||
"steps": first_attempt_steps,
|
||||
"reward": first_attempt.get("total_reward") if first_attempt else None,
|
||||
"api_calls": len(api_records),
|
||||
},
|
||||
"protocol_gates": protocol_gates,
|
||||
"acceptance_complete": acceptance_complete,
|
||||
"manuscript_observation_matches": {
|
||||
"first_attempt_victory": first_attempt_victory,
|
||||
"exactly_18_steps": first_attempt_steps == 18,
|
||||
"q_learning_11_step_greedy_solution": rl_results["eval_avg_steps"] == 11.0,
|
||||
},
|
||||
"interpretation": (
|
||||
"The protocol is accepted independently of whether stochastic model behavior reproduces the manuscript's exact 18-step observation."
|
||||
),
|
||||
"artifacts": {
|
||||
"experiment_results": "experiment_results.json",
|
||||
"raw_llm_calls_and_trajectories": "llm_experiences.json",
|
||||
"q_learning_checkpoint": "rl_agent.pkl",
|
||||
},
|
||||
"source_sha256": {
|
||||
name: _sha256(ROOT / name)
|
||||
for name in (
|
||||
"game_environment.py",
|
||||
"rl_agent.py",
|
||||
"llm_agent.py",
|
||||
"experiment.py",
|
||||
"run_experiment_8_2.py",
|
||||
)
|
||||
},
|
||||
}
|
||||
_write_json(runner.experiment_dir / "evidence.json", evidence)
|
||||
_write_json(output_root / "latest.json", {
|
||||
"experiment_id": "8-2",
|
||||
"artifact": str((runner.experiment_dir / "evidence.json").relative_to(output_root)),
|
||||
"acceptance_complete": acceptance_complete,
|
||||
"finished_at": evidence["finished_at"],
|
||||
})
|
||||
|
||||
print(json.dumps({
|
||||
"evidence": str(runner.experiment_dir / "evidence.json"),
|
||||
"acceptance_complete": acceptance_complete,
|
||||
"first_attempt_victory": first_attempt_victory,
|
||||
"first_attempt_steps": first_attempt_steps,
|
||||
}, indent=2))
|
||||
return 0 if acceptance_complete else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,98 @@
|
||||
import json
|
||||
from types import SimpleNamespace
|
||||
|
||||
from experiment import ExperimentRunner
|
||||
from game_environment import TreasureHuntGame
|
||||
from llm_agent import LLMAgent
|
||||
from run_experiment_8_2 import _write_json
|
||||
|
||||
|
||||
class _Usage:
|
||||
total_tokens = 17
|
||||
|
||||
def model_dump(self):
|
||||
return {"prompt_tokens": 10, "completion_tokens": 7, "total_tokens": 17}
|
||||
|
||||
|
||||
def _response(content="Reasoning\n **ACTION**: take rusty sword "):
|
||||
message = SimpleNamespace(content=content, reasoning_content="private reasoning")
|
||||
choice = SimpleNamespace(message=message, finish_reason="stop")
|
||||
return SimpleNamespace(
|
||||
id="chatcmpl-real-shape",
|
||||
created=123,
|
||||
model="kimi-k3",
|
||||
choices=[choice],
|
||||
usage=_Usage(),
|
||||
)
|
||||
|
||||
|
||||
def test_explicit_action_variants_are_recorded_without_fallback(monkeypatch):
|
||||
monkeypatch.setenv("MOONSHOT_API_KEY", "test-only")
|
||||
agent = LLMAgent(model="kimi-k3")
|
||||
agent.client = SimpleNamespace(
|
||||
chat=SimpleNamespace(
|
||||
completions=SimpleNamespace(create=lambda **_: _response())
|
||||
)
|
||||
)
|
||||
|
||||
action = agent.choose_action(TreasureHuntGame(), verbose=False)
|
||||
|
||||
assert action == "take rusty sword"
|
||||
assert agent.api_records[0]["fallback_used"] is False
|
||||
assert agent.api_records[0]["response"]["id"] == "chatcmpl-real-shape"
|
||||
assert agent.api_records[0]["response"]["reasoning_content"] == "private reasoning"
|
||||
assert agent.total_tokens == 17
|
||||
|
||||
|
||||
def test_api_failure_is_retained_and_cannot_look_like_model_behavior(monkeypatch):
|
||||
monkeypatch.setenv("MOONSHOT_API_KEY", "test-only")
|
||||
agent = LLMAgent(model="kimi-k3")
|
||||
|
||||
def fail(**_):
|
||||
raise RuntimeError("provider unavailable")
|
||||
|
||||
agent.client = SimpleNamespace(
|
||||
chat=SimpleNamespace(completions=SimpleNamespace(create=fail))
|
||||
)
|
||||
action = agent.choose_action(TreasureHuntGame(), verbose=False)
|
||||
|
||||
assert action in TreasureHuntGame().get_available_actions()
|
||||
assert agent.api_calls == 0
|
||||
assert agent.api_records[0]["fallback_used"] is True
|
||||
assert agent.api_records[0]["fallback_reason"] == "api_error"
|
||||
assert agent.api_records[0]["error"]["type"] == "RuntimeError"
|
||||
|
||||
|
||||
def test_saved_evidence_excludes_credentials(monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("MOONSHOT_API_KEY", "secret-that-must-not-be-written")
|
||||
agent = LLMAgent(model="kimi-k3")
|
||||
agent.client = SimpleNamespace(
|
||||
chat=SimpleNamespace(
|
||||
completions=SimpleNamespace(create=lambda **_: _response())
|
||||
)
|
||||
)
|
||||
agent.choose_action(TreasureHuntGame(), verbose=False)
|
||||
output = tmp_path / "llm_experiences.json"
|
||||
agent.save_experiences(output)
|
||||
payload = output.read_text(encoding="utf-8")
|
||||
|
||||
assert "secret-that-must-not-be-written" not in payload
|
||||
assert json.loads(payload)["backend"]["provider"] == "moonshot"
|
||||
|
||||
|
||||
def test_nested_validation_output_is_created(tmp_path):
|
||||
root = tmp_path / "validation" / "experiment_8_2"
|
||||
runner = ExperimentRunner(results_dir=str(root))
|
||||
assert runner.experiment_dir.parent == root
|
||||
assert runner.experiment_dir.is_dir()
|
||||
|
||||
|
||||
def test_evidence_writer_serializes_numpy_scalars(tmp_path):
|
||||
import numpy as np
|
||||
|
||||
output = tmp_path / "evidence.json"
|
||||
_write_json(output, {"gate": np.bool_(True), "count": np.int64(17)})
|
||||
assert json.loads(output.read_text(encoding="utf-8")) == {
|
||||
"gate": True,
|
||||
"count": 17,
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
@@ -0,0 +1,182 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Manual check to verify Q-learning can learn the simplified game.
|
||||
"""
|
||||
|
||||
import sys
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
import numpy as np
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from game_environment import TreasureHuntGame
|
||||
from rl_agent import QLearningAgent
|
||||
|
||||
|
||||
def run_rl_learning_check(stochastic=False, episodes=None):
|
||||
"""Test that Q-learning can learn the game.
|
||||
|
||||
Args:
|
||||
stochastic: If True, use stochastic environment
|
||||
episodes: List of episode counts to test (default: various counts)
|
||||
"""
|
||||
env_type = "STOCHASTIC" if stochastic else "DETERMINISTIC"
|
||||
print(f"Testing Q-Learning on simplified game ({env_type} environment)...")
|
||||
print("="*50)
|
||||
|
||||
# Show game rules
|
||||
game = TreasureHuntGame(stochastic=stochastic)
|
||||
print(game.get_hidden_rules())
|
||||
if stochastic:
|
||||
print("\n⚠️ Stochastic Mode Active:")
|
||||
print(" - Random reward variations")
|
||||
print(" - 3% chance of action failure")
|
||||
print(" - 10% critical hit / 5% miss chance in combat")
|
||||
print(" - 10% crafting failure chance")
|
||||
print("\n" + "="*50)
|
||||
|
||||
# Initialize agent
|
||||
agent = QLearningAgent(
|
||||
learning_rate=0.2,
|
||||
discount_factor=0.99,
|
||||
epsilon=1.0,
|
||||
epsilon_decay=0.9997, # Slower decay for exploration
|
||||
epsilon_min=0.1
|
||||
)
|
||||
|
||||
# Train for different episode counts
|
||||
if episodes:
|
||||
episode_counts = episodes
|
||||
else:
|
||||
episode_counts = [100, 500, 1000, 2000, 5000, 10000]
|
||||
|
||||
for num_episodes in episode_counts:
|
||||
print(f"\nTraining for {num_episodes} episodes...")
|
||||
|
||||
# Reset agent
|
||||
agent = QLearningAgent(
|
||||
learning_rate=0.2,
|
||||
discount_factor=0.99,
|
||||
epsilon=1.0,
|
||||
epsilon_decay=0.9997,
|
||||
epsilon_min=0.1
|
||||
)
|
||||
|
||||
# Train
|
||||
game = TreasureHuntGame(stochastic=stochastic)
|
||||
victories = 0
|
||||
recent_rewards = []
|
||||
|
||||
for episode in range(num_episodes):
|
||||
game.reset()
|
||||
total_reward = 0
|
||||
|
||||
while not game.game_over:
|
||||
state_hash = agent._get_state_hash(game)
|
||||
action = agent.choose_action(game, training=True)
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
|
||||
next_state_hash = agent._get_state_hash(game)
|
||||
next_actions = game.get_available_actions() if not done else []
|
||||
|
||||
agent.update_q_value(
|
||||
state_hash, action, reward,
|
||||
next_state_hash, next_actions, done
|
||||
)
|
||||
|
||||
total_reward += reward
|
||||
|
||||
# Decay epsilon
|
||||
agent.epsilon = max(agent.epsilon_min, agent.epsilon * agent.epsilon_decay)
|
||||
|
||||
recent_rewards.append(total_reward)
|
||||
if game.victory:
|
||||
victories += 1
|
||||
|
||||
# Print progress
|
||||
progress_every = max(1, num_episodes // 10)
|
||||
if (episode + 1) % progress_every == 0:
|
||||
recent_wins = sum(1 for r in recent_rewards[-100:] if r > 50)
|
||||
avg_reward = np.mean(recent_rewards[-100:]) if recent_rewards else 0
|
||||
print(f" Episode {episode+1}: Recent wins={recent_wins}/100, "
|
||||
f"Avg reward={avg_reward:.1f}, Epsilon={agent.epsilon:.3f}")
|
||||
|
||||
# Evaluate
|
||||
print(f"\nEvaluating after {num_episodes} episodes...")
|
||||
eval_victories = 0
|
||||
eval_rewards = []
|
||||
|
||||
for _ in range(100):
|
||||
game.reset()
|
||||
total_reward = 0
|
||||
|
||||
# Set epsilon to 0 for evaluation
|
||||
old_epsilon = agent.epsilon
|
||||
agent.epsilon = 0
|
||||
|
||||
while not game.game_over:
|
||||
action = agent.choose_action(game, training=False)
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
total_reward += reward
|
||||
|
||||
agent.epsilon = old_epsilon
|
||||
|
||||
eval_rewards.append(total_reward)
|
||||
if game.victory:
|
||||
eval_victories += 1
|
||||
|
||||
print(f" Evaluation: {eval_victories}/100 victories")
|
||||
print(f" Average reward: {np.mean(eval_rewards):.2f}")
|
||||
print(f" Q-table size: {len(agent.q_table)} states")
|
||||
|
||||
# Show a sample successful trajectory if we have victories
|
||||
if eval_victories > 0:
|
||||
print("\n Sample successful trajectory:")
|
||||
game.reset()
|
||||
agent.epsilon = 0
|
||||
steps = []
|
||||
|
||||
while not game.game_over:
|
||||
action = agent.choose_action(game, training=False)
|
||||
steps.append(f" {len(steps)+1}. {action}")
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
if game.victory:
|
||||
steps.append(f" → Victory! Total moves: {game.moves}")
|
||||
break
|
||||
|
||||
if len(steps) <= 20: # Only show if reasonable length
|
||||
print("\n".join(steps[:15])) # Show first 15 steps
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description="Test Q-learning agent on the treasure hunt game")
|
||||
parser.add_argument(
|
||||
'--stochastic',
|
||||
action='store_true',
|
||||
help='Use stochastic environment (adds randomness to rewards and actions)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--deterministic',
|
||||
action='store_true',
|
||||
help='Use deterministic environment (default)'
|
||||
)
|
||||
parser.add_argument(
|
||||
'--episodes',
|
||||
type=int,
|
||||
nargs='+',
|
||||
help='Episode counts to test (e.g., --episodes 1000 5000 10000)'
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Handle environment mode
|
||||
if args.deterministic and args.stochastic:
|
||||
print("Error: Cannot specify both --deterministic and --stochastic")
|
||||
sys.exit(1)
|
||||
|
||||
stochastic = args.stochastic # Default is False (deterministic)
|
||||
|
||||
run_rl_learning_check(stochastic=stochastic, episodes=args.episodes)
|
||||
@@ -0,0 +1,167 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Basic test to verify all components work correctly.
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
|
||||
def test_game_environment():
|
||||
"""Test that the game environment works."""
|
||||
print("Testing game environment...")
|
||||
from game_environment import TreasureHuntGame
|
||||
|
||||
game = TreasureHuntGame(seed=42)
|
||||
|
||||
# Test initial state
|
||||
state = game.get_state_description()
|
||||
assert "entrance" in state.lower()
|
||||
print(" ✓ Game initialization works")
|
||||
|
||||
# Test actions
|
||||
actions = game.get_available_actions()
|
||||
assert len(actions) > 0
|
||||
print(" ✓ Actions generation works")
|
||||
|
||||
# Test action execution
|
||||
feedback, reward, done = game.execute_action("look around")
|
||||
assert isinstance(feedback, str)
|
||||
assert isinstance(reward, float)
|
||||
assert isinstance(done, bool)
|
||||
print(" ✓ Action execution works")
|
||||
|
||||
# Test reset
|
||||
game.reset()
|
||||
assert game.moves == 0
|
||||
print(" ✓ Game reset works")
|
||||
|
||||
print("✅ Game environment tests passed!\n")
|
||||
|
||||
|
||||
def test_rl_agent():
|
||||
"""Test that the RL agent works."""
|
||||
print("Testing RL agent...")
|
||||
from game_environment import TreasureHuntGame
|
||||
from rl_agent import QLearningAgent
|
||||
|
||||
game = TreasureHuntGame(seed=42)
|
||||
agent = QLearningAgent()
|
||||
|
||||
# Test action selection
|
||||
action = agent.choose_action(game, training=True)
|
||||
assert isinstance(action, str)
|
||||
print(" ✓ Action selection works")
|
||||
|
||||
# Test Q-value update
|
||||
state = agent._get_state_hash(game)
|
||||
feedback, reward, done = game.execute_action(action)
|
||||
next_state = agent._get_state_hash(game)
|
||||
next_actions = game.get_available_actions()
|
||||
|
||||
agent.update_q_value(state, action, reward, next_state, next_actions, done)
|
||||
print(" ✓ Q-value update works")
|
||||
|
||||
# Test training (just 10 episodes for speed)
|
||||
results = agent.train(num_episodes=10, verbose=False)
|
||||
assert "total_episodes" in results
|
||||
print(" ✓ Training works")
|
||||
|
||||
print("✅ RL agent tests passed!\n")
|
||||
|
||||
|
||||
def test_llm_agent():
|
||||
"""Test that the LLM agent works (without API calls)."""
|
||||
print("Testing LLM agent structure...")
|
||||
from game_environment import TreasureHuntGame
|
||||
from llm_agent import LLMAgent, GameExperience
|
||||
|
||||
# Test experience storage
|
||||
exp = GameExperience(
|
||||
state_description="test state",
|
||||
action="test action",
|
||||
feedback="test feedback",
|
||||
reward=1.0,
|
||||
success=True
|
||||
)
|
||||
assert exp.action == "test action"
|
||||
print(" ✓ Experience dataclass works")
|
||||
|
||||
# Test context building (without API)
|
||||
try:
|
||||
# This will fail without API key, but we can test the structure
|
||||
agent = LLMAgent(api_key="dummy-key-for-testing")
|
||||
|
||||
game = TreasureHuntGame()
|
||||
state = game.get_state_description()
|
||||
actions = game.get_available_actions()
|
||||
|
||||
context = agent._build_context(state, actions)
|
||||
assert "treasure hunt" in context.lower()
|
||||
print(" ✓ Context building works")
|
||||
|
||||
# Test experience update
|
||||
agent.update_experience(state, "test action", "test feedback", 1.0)
|
||||
assert len(agent.experiences) == 1
|
||||
print(" ✓ Experience storage works")
|
||||
|
||||
except ValueError as e:
|
||||
if "MOONSHOT_API_KEY" in str(e):
|
||||
print(" ⚠ LLM agent requires API key for full testing")
|
||||
else:
|
||||
raise
|
||||
|
||||
print("✅ LLM agent structure tests passed!\n")
|
||||
|
||||
|
||||
def test_experiment_runner():
|
||||
"""Test that the experiment runner works."""
|
||||
print("Testing experiment runner...")
|
||||
from experiment import ExperimentRunner
|
||||
|
||||
runner = ExperimentRunner(results_dir="test_results")
|
||||
assert runner.results_dir.exists()
|
||||
print(" ✓ Experiment runner initialization works")
|
||||
|
||||
# Clean up test directory
|
||||
import shutil
|
||||
if runner.results_dir.exists():
|
||||
shutil.rmtree(runner.results_dir)
|
||||
|
||||
print("✅ Experiment runner tests passed!\n")
|
||||
|
||||
|
||||
def main():
|
||||
"""Run all tests."""
|
||||
print("\n" + "="*60)
|
||||
print("RUNNING BASIC TESTS")
|
||||
print("="*60 + "\n")
|
||||
|
||||
try:
|
||||
test_game_environment()
|
||||
test_rl_agent()
|
||||
test_llm_agent()
|
||||
test_experiment_runner()
|
||||
|
||||
print("="*60)
|
||||
print("ALL TESTS PASSED! ✅")
|
||||
print("="*60)
|
||||
print("\nThe experiment is ready to run.")
|
||||
print("To run the full experiment: python experiment.py")
|
||||
print("To play interactively: python demo.py")
|
||||
|
||||
except Exception as e:
|
||||
print(f"\n❌ Test failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return 1
|
||||
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,30 @@
|
||||
"""
|
||||
Test suite locking out ZeroDivisionError in QLearningAgent.train
|
||||
when computing victory_rate on an empty episode_victories list.
|
||||
"""
|
||||
|
||||
from rl_agent import QLearningAgent
|
||||
|
||||
|
||||
def test_q_learning_agent_train_empty_victories_snapshot():
|
||||
"""
|
||||
Ensure checkpoint victory_rate calculation does not raise ZeroDivisionError when recent is empty.
|
||||
"""
|
||||
agent = QLearningAgent.__new__(QLearningAgent)
|
||||
agent.episode_victories = []
|
||||
agent.learning_curve = []
|
||||
agent.q_table = {}
|
||||
agent.epsilon = 0.1
|
||||
|
||||
# Simulate snapshot logic when checkpoint_interval matches
|
||||
recent = agent.episode_victories[-1000:]
|
||||
victory_rate = sum(recent) / len(recent) if recent else 0.0
|
||||
|
||||
agent.learning_curve.append({
|
||||
"episode": 1,
|
||||
"victory_rate": victory_rate,
|
||||
"q_table_size": len(agent.q_table),
|
||||
"epsilon": agent.epsilon,
|
||||
})
|
||||
|
||||
assert agent.learning_curve[0]["victory_rate"] == 0.0
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Regression: progress prints must not ZeroDivisionError when episodes < 10."""
|
||||
|
||||
|
||||
def test_progress_every_never_zero():
|
||||
for num_episodes in (1, 5, 9, 10, 100):
|
||||
progress_every = max(1, num_episodes // 10)
|
||||
assert progress_every >= 1
|
||||
# modulo must be defined
|
||||
for episode in range(num_episodes):
|
||||
_ = (episode + 1) % progress_every
|
||||
|
||||
|
||||
def test_source_uses_max_guard():
|
||||
from pathlib import Path
|
||||
src = (Path(__file__).parent / "manual" / "rl_learning_check.py").read_text()
|
||||
assert "progress_every = max(1, num_episodes // 10)" in src
|
||||
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Regression tests for zero-episode division guards.
|
||||
|
||||
Bug: train()/evaluate() divided victory counts by episode counts, so
|
||||
num_episodes=0 (accepted by experiment.py's argparse) crashed with
|
||||
ZeroDivisionError. Fixed by guarding the divisions and rejecting
|
||||
episode counts < 1 in experiment.py's front door.
|
||||
"""
|
||||
|
||||
import sys
|
||||
|
||||
import experiment
|
||||
from llm_agent import LLMAgent
|
||||
from rl_agent import QLearningAgent
|
||||
|
||||
|
||||
def test_rl_train_zero_episodes_no_zero_division():
|
||||
result = QLearningAgent().train(num_episodes=0, verbose=False)
|
||||
assert result["total_episodes"] == 0
|
||||
assert result["victory_rate"] == 0.0
|
||||
|
||||
|
||||
def test_rl_evaluate_zero_episodes_no_zero_division():
|
||||
result = QLearningAgent().evaluate(num_episodes=0)
|
||||
assert result["num_episodes"] == 0
|
||||
assert result["victory_rate"] == 0.0
|
||||
|
||||
|
||||
def test_llm_evaluate_zero_episodes_no_zero_division():
|
||||
# Dummy key: constructing the client makes no network calls, and
|
||||
# evaluate(num_episodes=0) never reaches the API.
|
||||
agent = LLMAgent(api_key="dummy-key")
|
||||
result = agent.evaluate(num_episodes=0)
|
||||
assert result["victory_rate"] == 0.0
|
||||
assert result["avg_reward"] == 0.0
|
||||
assert result["avg_length"] == 0.0
|
||||
|
||||
|
||||
def test_experiment_rejects_zero_episodes(monkeypatch, capsys):
|
||||
monkeypatch.setattr(sys, "argv", ["experiment.py", "--mode", "qlearning",
|
||||
"--rl-episodes", "0"])
|
||||
experiment.main() # must print an error and return before running
|
||||
assert "must all be >= 1" in capsys.readouterr().out
|
||||
@@ -0,0 +1,194 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment_id": "7-2",
|
||||
"title": "Traditional RL versus Kimi K3 in the same treasure-hunt environment",
|
||||
"campaign_started_at": "2026-07-29T17:17:06.773342+00:00",
|
||||
"evidence_finalized_at": "2026-07-29T17:26:40.328179+00:00",
|
||||
"git_revision": "4a7f37cf278bd15948c409f14533017c4c7fbc29",
|
||||
"runtime": {
|
||||
"python": "3.11.4 (main, Jul 5 2023, 08:40:20) [Clang 14.0.6 ]",
|
||||
"platform": "macOS-26.3-arm64-arm-64bit"
|
||||
},
|
||||
"execution_manifest": {
|
||||
"experiment_id": "7-2",
|
||||
"campaign_started_at_local": "2026-07-30T01:17:04+08:00",
|
||||
"command": [
|
||||
"python",
|
||||
"run_experiment_7_2.py"
|
||||
],
|
||||
"execution_source_sha256": {
|
||||
"game_environment.py": "4af75d35cd609e1ee4d163c37c637c3f559365eb47eda1e562e805a6c62f3105",
|
||||
"rl_agent.py": "3821f04679ad4c9d021e3e0557b39b94d43348ef63192f05f077be79ff3442a0",
|
||||
"llm_agent.py": "28579c10cc4ba71eb3ac232e1538172dad78d2ca9733d24fc7bf63fb26385783",
|
||||
"experiment.py": "c8e2890271ba12583d955b1d418d5b171b1becdbcd27b1614d790ac81dee6d96",
|
||||
"run_experiment_7_2.py": "9e988e96fb7d9bc5b16d15a30782e78db14ab3ea10d702fdb7aed3e992b870a1"
|
||||
},
|
||||
"post_run_status": "The campaign and all raw artifacts completed; evidence.json serialization then failed because numpy.bool_ is not handled by the standard JSON encoder. No API rerun is required.",
|
||||
"captured_before_source_repair": true
|
||||
},
|
||||
"backend": {
|
||||
"provider": "moonshot",
|
||||
"base_url": "https://api.moonshot.cn/v1",
|
||||
"model": "kimi-k3",
|
||||
"using_openrouter": false
|
||||
},
|
||||
"provider_response_models": [
|
||||
"kimi-k3"
|
||||
],
|
||||
"usage": {
|
||||
"successful_api_calls": 17,
|
||||
"api_attempts": 17,
|
||||
"total_tokens": 28242,
|
||||
"provider_cost": null,
|
||||
"provider_cost_note": "The provider exposed token usage but no authoritative billed cost; unknown is not zero."
|
||||
},
|
||||
"q_learning": {
|
||||
"training_episodes": 10000,
|
||||
"training_time_seconds": 2.124420166015625,
|
||||
"training_victory_rate": 0.451,
|
||||
"evaluation_episodes": 100,
|
||||
"evaluation_victory_rate": 1.0,
|
||||
"evaluation_average_steps": 12.0,
|
||||
"q_table_states": 142,
|
||||
"learning_curve": [
|
||||
{
|
||||
"episode": 1000,
|
||||
"victory_rate": 0.003,
|
||||
"q_table_size": 123,
|
||||
"epsilon": 0.606454822840097
|
||||
},
|
||||
{
|
||||
"episode": 2000,
|
||||
"victory_rate": 0.0,
|
||||
"q_table_size": 123,
|
||||
"epsilon": 0.3677874521460121
|
||||
},
|
||||
{
|
||||
"episode": 3000,
|
||||
"victory_rate": 0.001,
|
||||
"q_table_size": 126,
|
||||
"epsilon": 0.22304647413401948
|
||||
},
|
||||
{
|
||||
"episode": 4000,
|
||||
"victory_rate": 0.001,
|
||||
"q_table_size": 127,
|
||||
"epsilon": 0.13526760995605422
|
||||
},
|
||||
{
|
||||
"episode": 5000,
|
||||
"victory_rate": 0.001,
|
||||
"q_table_size": 128,
|
||||
"epsilon": 0.1
|
||||
},
|
||||
{
|
||||
"episode": 6000,
|
||||
"victory_rate": 0.559,
|
||||
"q_table_size": 137,
|
||||
"epsilon": 0.1
|
||||
},
|
||||
{
|
||||
"episode": 7000,
|
||||
"victory_rate": 0.97,
|
||||
"q_table_size": 138,
|
||||
"epsilon": 0.1
|
||||
},
|
||||
{
|
||||
"episode": 8000,
|
||||
"victory_rate": 0.996,
|
||||
"q_table_size": 138,
|
||||
"epsilon": 0.1
|
||||
},
|
||||
{
|
||||
"episode": 9000,
|
||||
"victory_rate": 0.998,
|
||||
"q_table_size": 139,
|
||||
"epsilon": 0.1
|
||||
},
|
||||
{
|
||||
"episode": 10000,
|
||||
"victory_rate": 0.981,
|
||||
"q_table_size": 142,
|
||||
"epsilon": 0.1
|
||||
}
|
||||
]
|
||||
},
|
||||
"k3_first_attempt": {
|
||||
"victory": true,
|
||||
"steps": 17,
|
||||
"reward": 241.5,
|
||||
"api_calls": 17,
|
||||
"actions": [
|
||||
"take rusty sword",
|
||||
"go north",
|
||||
"look around",
|
||||
"use rusty sword",
|
||||
"go south",
|
||||
"go east",
|
||||
"take red key",
|
||||
"take magic crystal",
|
||||
"try crafting",
|
||||
"go west",
|
||||
"go north",
|
||||
"use red key",
|
||||
"go north",
|
||||
"attack with silver sword",
|
||||
"use silver sword",
|
||||
"go east",
|
||||
"take dragon's treasure"
|
||||
]
|
||||
},
|
||||
"protocol_gates": {
|
||||
"same_deterministic_environment": true,
|
||||
"q_learning_10000_training_episodes": true,
|
||||
"q_learning_100_evaluation_episodes": true,
|
||||
"q_learning_reached_full_evaluation_success": true,
|
||||
"one_kimi_first_attempt_recorded": true,
|
||||
"direct_official_moonshot_kimi_k3": true,
|
||||
"one_real_response_per_first_attempt_action": true,
|
||||
"provider_response_ids_retained": true,
|
||||
"provider_response_content_retained": true,
|
||||
"all_provider_responses_finished_normally": true,
|
||||
"zero_api_errors": true,
|
||||
"zero_fallback_actions": true
|
||||
},
|
||||
"acceptance_complete": true,
|
||||
"manuscript_observation_matches": {
|
||||
"first_attempt_victory": true,
|
||||
"exactly_18_steps": false,
|
||||
"q_learning_11_step_greedy_solution": false
|
||||
},
|
||||
"result_mismatches": [
|
||||
"Kimi K3 used 17 rather than the historical 18 steps",
|
||||
"Q-learning greedy evaluation averaged 12 rather than 11 steps"
|
||||
],
|
||||
"interpretation": "Protocol acceptance is independent of whether stochastic model behavior reproduces historical point estimates.",
|
||||
"artifacts": {
|
||||
"experiment_results": "experiment_results.json",
|
||||
"raw_llm_calls_and_trajectories": "llm_experiences.json",
|
||||
"q_learning_checkpoint": "rl_agent.pkl",
|
||||
"execution_manifest": "execution_manifest.json"
|
||||
},
|
||||
"artifact_sha256": {
|
||||
"experiment_results.json": "63b30603836b8dce4da8b3748159f727b7d66d362be37a7d5eb75d778e1600d1",
|
||||
"llm_experiences.json": "535a5977cab93a151f977c45f91f4582989d7d3710ca7e460017c5bb64a581d9",
|
||||
"rl_agent.pkl": "75ee5ff04f3100480fd8f62005b004ec395528110d7c2dd273aefad3e8ab3ed5",
|
||||
"execution_manifest.json": "29da01f59cfa41464345f9a292352359d8153a13ed623b58d4b93959c020fda6"
|
||||
},
|
||||
"postprocessor_source_sha256": {
|
||||
"run_experiment_7_2.py": "f68cbe62ce4dcdbe11750317ed8e506a6ca6534cba20eaf75352286f4ace5e33",
|
||||
"finalize_experiment_7_2.py": "80766be8fd3da3ebb2da1aa096ee32431e682dc1e3caf513052111a24b4995d5"
|
||||
},
|
||||
"llm_result_summary": {
|
||||
"provider": "moonshot",
|
||||
"model": "kimi-k3",
|
||||
"using_openrouter": false,
|
||||
"training_time": 416.1074731349945,
|
||||
"api_calls": 17,
|
||||
"api_attempts": 17,
|
||||
"api_errors": 0,
|
||||
"fallback_actions": 0,
|
||||
"total_tokens": 28242,
|
||||
"training_victory_rate": 1.0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"experiment_id": "7-2",
|
||||
"campaign_started_at_local": "2026-07-30T01:17:04+08:00",
|
||||
"command": [
|
||||
"python",
|
||||
"run_experiment_7_2.py"
|
||||
],
|
||||
"execution_source_sha256": {
|
||||
"game_environment.py": "4af75d35cd609e1ee4d163c37c637c3f559365eb47eda1e562e805a6c62f3105",
|
||||
"rl_agent.py": "3821f04679ad4c9d021e3e0557b39b94d43348ef63192f05f077be79ff3442a0",
|
||||
"llm_agent.py": "28579c10cc4ba71eb3ac232e1538172dad78d2ca9733d24fc7bf63fb26385783",
|
||||
"experiment.py": "c8e2890271ba12583d955b1d418d5b171b1becdbcd27b1614d790ac81dee6d96",
|
||||
"run_experiment_7_2.py": "9e988e96fb7d9bc5b16d15a30782e78db14ab3ea10d702fdb7aed3e992b870a1"
|
||||
},
|
||||
"post_run_status": "The campaign and all raw artifacts completed; evidence.json serialization then failed because numpy.bool_ is not handled by the standard JSON encoder. No API rerun is required.",
|
||||
"captured_before_source_repair": true
|
||||
}
|
||||
+20422
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
Binary file not shown.
BIN
Binary file not shown.
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"experiment_id": "7-2",
|
||||
"artifact": "20260730_011704/evidence.json",
|
||||
"acceptance_complete": true,
|
||||
"finalized_at": "2026-07-29T17:26:40.328179+00:00"
|
||||
}
|
||||
@@ -0,0 +1,189 @@
|
||||
# GPT-5.6 Sol Deep Research / GPT-5.6 Sol 深度研究
|
||||
|
||||
> Responses API companion for Chapter 1, Experiment 1-3: hosted
|
||||
> `web_search` + hosted `code_interpreter`, typed tool traces, citations, and an
|
||||
> intent-clarification continuation. The canonical path is OpenAI GPT-5.6 Sol;
|
||||
> acceptance is multi-provider and may be closed by any provider whose
|
||||
> Responses API genuinely closes the search/code loop server-side — currently
|
||||
> Alibaba Model Studio (DashScope) `qwen3.7-plus`.
|
||||
|
||||
← [Chapter 1 index / 返回第 1 章目录](../README.md) ·
|
||||
📖 [Book experiment / 正文实验](../../book/chapter1.md)
|
||||
|
||||
## What this companion implements
|
||||
|
||||
The canonical path is the OpenAI **Responses API**, not a Chat Completions
|
||||
request that merely contains similarly named tool objects. The active agent in
|
||||
`agent.py` sends:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "gpt-5.6-sol",
|
||||
"tools": [
|
||||
{"type": "web_search", "search_context_size": "medium"},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {"type": "auto", "memory_limit": "4g"}
|
||||
}
|
||||
],
|
||||
"reasoning": {"effort": "high"},
|
||||
"text": {"verbosity": "high"}
|
||||
}
|
||||
```
|
||||
|
||||
The DashScope backend speaks the same `/responses` protocol against
|
||||
`{DASHSCOPE_BASE_URL}/responses` with the provider's hosted-tool shapes:
|
||||
|
||||
```json
|
||||
{
|
||||
"model": "qwen3.7-plus",
|
||||
"tools": [{"type": "web_search"}, {"type": "code_interpreter"}],
|
||||
"stream": true
|
||||
}
|
||||
```
|
||||
|
||||
DashScope runs thinking natively (no `reasoning.effort`/`text.verbosity`
|
||||
knobs) and its gateway drops non-streaming requests that stay silent for
|
||||
about 60 seconds, so the backend always streams and keeps the final
|
||||
`response.completed` object, which has the same shape as a non-streaming
|
||||
response.
|
||||
|
||||
Acceptance is based on provider output items. A successful ASEAN-capitals run
|
||||
must contain completed `web_search_call` and `code_interpreter_call` items,
|
||||
clickable URL citations, and the computed closest pair. A text answer that says
|
||||
it used Python does not pass without the provider tool receipt.
|
||||
|
||||
The second scenario sends the deliberately ambiguous Bitcoin request used in
|
||||
the chapter, requires the first response to clarify material preferences before
|
||||
using tools, then continues with `previous_response_id` after the user supplies
|
||||
the data source and indicators.
|
||||
|
||||
## Current evidence status
|
||||
|
||||
Run the complete validator with:
|
||||
|
||||
```bash
|
||||
cd chapter1/search-codegen
|
||||
python run_experiment_1_3.py --backends openai dashscope --reasoning high
|
||||
```
|
||||
|
||||
The latest evidence is [validation/latest.json](validation/latest.json); raw
|
||||
credential-free receipts, a manifest, and SHA-256 sidecars live in
|
||||
`validation/runs/real_20260731T170529Z/`.
|
||||
|
||||
Result of the 2026-07-31 multi-provider acceptance run: **passed**, with
|
||||
`dashscope` (`qwen3.7-plus`) as the acceptance backend.
|
||||
|
||||
- ASEAN capitals: one hosted `web_search_call` batching ten model-issued
|
||||
coordinate queries, then a hosted `code_interpreter_call` that enumerated all
|
||||
45 haversine pairs and found Kuala Lumpur–Singapore at 316.35 km — the same
|
||||
pair as the independent local reference computed from standard coordinates.
|
||||
- Bitcoin technical analysis: the first turn asked which data source and which
|
||||
indicators to use **without calling any tool**; the continuation via
|
||||
`previous_response_id` ran 3 model-directed search rounds and 4 hosted
|
||||
`code_interpreter_call`s computing MA7/MA20, RSI14, MACD(12,26,9), period
|
||||
return and max drawdown, and plotted a close-price chart in the sandbox.
|
||||
- The official OpenAI `gpt-5.6-sol` path is still intact but remains
|
||||
quota-blocked: both calls returned `credit_balance_exhausted` before any
|
||||
hosted tool ran, which is recorded in the same evidence file.
|
||||
- Honest qualifications: the DashScope sandbox has no outbound network, so the
|
||||
daily closes were extracted through web search (the model disclosed this in
|
||||
its report); the chart PNG stays inside the sandbox because this Responses
|
||||
API returns execution logs only; and `qwen3.7-plus` only asks before acting
|
||||
when the system prompt carries an explicit clarify-first rule — the shipped
|
||||
prompt encodes it.
|
||||
- The OpenRouter route is retained strictly as a diagnostic and is never
|
||||
accepted. No fallback model, local Python replacement, fabricated tool
|
||||
trace, or Chat-Completions approximation is counted as fulfillment.
|
||||
|
||||
Earlier blocked attempts are kept under `validation/real_20260729T155459Z/`
|
||||
and `validation/real_20260730T033800Z/`.
|
||||
|
||||
## Setup and CLI
|
||||
|
||||
Python 3.9+ is required.
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 1 environment
|
||||
uv sync --locked --extra ch1
|
||||
|
||||
# Activate it before changing directories:
|
||||
source .venv/bin/activate
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch1]"
|
||||
|
||||
cd chapter1/search-codegen
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# Exact official path
|
||||
python main.py --backend openai --mode single \
|
||||
--request "东盟 10 国首都之间最近的一对是哪两个？请搜索并用 Python 计算" \
|
||||
--reasoning high --verbosity high --output result.json
|
||||
|
||||
# Equivalent-provider path (eligible for acceptance): Alibaba Model Studio
|
||||
export DASHSCOPE_API_KEY=your-dashscope-api-key
|
||||
python main.py --backend dashscope --mode single \
|
||||
--request "东盟 10 国首都之间最近的一对是哪两个？请搜索并用 Python 计算" \
|
||||
--output result.json
|
||||
|
||||
# Inspect the exact request without an API call
|
||||
python main.py --backend openai --dry-run \
|
||||
--request "东盟 10 国首都之间最近的一对？" \
|
||||
--reasoning max --verbosity high
|
||||
|
||||
# Proxy diagnostic only; not canonical acceptance
|
||||
export OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
python main.py --backend openrouter --mode single --request "Search current news"
|
||||
```
|
||||
|
||||
Important options:
|
||||
|
||||
| Option | Meaning |
|
||||
|---|---|
|
||||
| `--backend openai` | Canonical `https://api.openai.com/v1/responses` path |
|
||||
| `--backend dashscope` | Equivalent-provider path: DashScope Responses API, hosted `web_search` + `code_interpreter`, eligible for acceptance |
|
||||
| `--backend openrouter` | Explicit proxy diagnostic; never silently substituted |
|
||||
| `--reasoning` | `none`, `low`, `medium`, `high`, `xhigh`, or GPT-5.6 `max` |
|
||||
| `--verbosity` | Responses `text.verbosity`: `low`, `medium`, or `high` |
|
||||
| `--output` | Saves request, typed output items, citations, usage, and raw response |
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
python -m pytest -q test_responses_agent.py
|
||||
python -m py_compile agent.py config.py main.py run_experiment_1_3.py
|
||||
```
|
||||
|
||||
The validator checks exact model identity, direct-vs-proxy provenance, both
|
||||
hosted tool types, citations, clarification order, continuation linkage, token
|
||||
usage, reported provider cost when available, and credential-free raw evidence.
|
||||
|
||||
## Official sources
|
||||
|
||||
- [GPT-5.6 Sol model](https://developers.openai.com/api/docs/models/gpt-5.6-sol)
|
||||
- [Web search](https://developers.openai.com/api/docs/guides/tools-web-search)
|
||||
- [Code Interpreter](https://developers.openai.com/api/docs/guides/tools-code-interpreter)
|
||||
- [GPT-5.6 model guidance](https://developers.openai.com/api/docs/guides/model-guidance?model=gpt-5.6-sol)
|
||||
- [Alibaba Model Studio code interpreter (DashScope)](https://help.aliyun.com/zh/model-studio/qwen-code-interpreter)
|
||||
|
||||
## 中文说明
|
||||
|
||||
本项目使用正文所述的**精确协议**：Responses API、托管 `web_search` 与托管
|
||||
`code_interpreter`。验收依据是服务端返回的 `web_search_call` /
|
||||
`code_interpreter_call` 和 URL 引用，而不是代码里“声明了工具”或答案里
|
||||
“声称用过 Python”。
|
||||
|
||||
按作者批准的多提供商政策，验收不绑定官方 OpenAI 账号：官方 `gpt-5.6-sol` 路径
|
||||
完整保留（当前 Key 推理返回 `credit_balance_exhausted`，已在证据中如实记录），
|
||||
具备等价托管工具的提供商同样可以验收。2026-07-31 的正式运行用阿里云百炼
|
||||
`qwen3.7-plus`（DashScope Responses API）通过了全部验收门：东盟任务先搜索十个
|
||||
首都坐标、再用托管 Python 枚举 45 对大圆距离（吉隆坡—新加坡 316.35 km，与独立
|
||||
本地参考一致）；比特币任务先在不用任何工具的情况下澄清数据源与指标，再通过
|
||||
`previous_response_id` 继续，完成 3 轮模型主导的搜索与 4 次托管代码执行
|
||||
（MA7/MA20、RSI14、MACD、区间收益、最大回撤与走势图）。OpenRouter 只作为诊断
|
||||
路径明确保留，不会被包装成替代品。
|
||||
@@ -0,0 +1,396 @@
|
||||
"""Exact GPT-5.6 Responses API agent for Experiment 1-3.
|
||||
|
||||
The previous companion sent Responses-style hosted tools to Chat Completions
|
||||
through a proxy and then reported an empty ``tool_calls`` list. This module
|
||||
uses the actual ``/v1/responses`` protocol and preserves its typed output items
|
||||
(``web_search_call``, ``code_interpreter_call``, messages, and citations).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from typing import Any, Dict, List, Literal, Optional
|
||||
|
||||
import requests
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GPT5NativeAgent:
|
||||
"""GPT-5.6 Sol with OpenAI-hosted web search and Python tools."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
api_key: str,
|
||||
base_url: str = "https://api.openai.com/v1",
|
||||
model: str = "gpt-5.6-sol",
|
||||
):
|
||||
if not api_key:
|
||||
raise ValueError("An API key is required")
|
||||
self.api_key = api_key
|
||||
self.base_url = base_url.rstrip("/")
|
||||
self.model = model
|
||||
self.provider = (
|
||||
"openai" if self.base_url == "https://api.openai.com/v1" else
|
||||
"openrouter" if "openrouter.ai" in self.base_url else
|
||||
"dashscope" if "dashscope" in self.base_url else
|
||||
"custom"
|
||||
)
|
||||
self.conversation_history: List[Dict[str, Any]] = []
|
||||
self.system_prompt = self._create_system_prompt()
|
||||
self.previous_response_id: Optional[str] = None
|
||||
self.api_turns: List[Dict[str, Any]] = []
|
||||
|
||||
@staticmethod
|
||||
def _create_system_prompt() -> str:
|
||||
return """You are a deep-research assistant. 你是一名深度研究助手。
|
||||
|
||||
Hard rule / 硬性规则: when the user's research request leaves material
|
||||
preferences ambiguous — for example which data source to use or which
|
||||
technical indicators to compute — ask a concise clarifying question FIRST
|
||||
(for example “您偏好使用哪个数据源？需要分析哪些技术指标？”), and do NOT
|
||||
call any tool until the user answers.
|
||||
当用户的研究请求没有明确数据来源或具体分析指标时，必须先向用户提问澄清，
|
||||
在用户回答之前不要调用任何工具。
|
||||
|
||||
After clarification, use hosted web search for current facts and cite
|
||||
sources, and use the hosted Python/code-interpreter tool for quantitative
|
||||
analysis; do not claim a calculation was run unless the response contains a
|
||||
completed code_interpreter_call.
|
||||
澄清之后：使用 web_search 获取最新事实并引用来源链接；所有定量计算必须通过
|
||||
code_interpreter 实际执行，不得口算或声称运行了代码。"""
|
||||
|
||||
def _tools(self) -> List[Dict[str, Any]]:
|
||||
if self.provider == "dashscope":
|
||||
# Exact structures from the Alibaba Model Studio Responses API guides.
|
||||
return [{"type": "web_search"}, {"type": "code_interpreter"}]
|
||||
# Exact structures from the official OpenAI Responses API guides.
|
||||
return [
|
||||
{"type": "web_search", "search_context_size": "medium"},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {"type": "auto", "memory_limit": "4g"},
|
||||
},
|
||||
]
|
||||
|
||||
def _build_responses_request(
|
||||
self,
|
||||
input_text: str,
|
||||
*,
|
||||
use_tools: bool = True,
|
||||
tool_choice: Literal["auto", "none", "required"] = "auto",
|
||||
reasoning_effort: str = "low",
|
||||
verbosity: Optional[str] = None,
|
||||
max_output_tokens: Optional[int] = None,
|
||||
background: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
if self.provider != "dashscope":
|
||||
if reasoning_effort not in {"none", "low", "medium", "high", "xhigh", "max"}:
|
||||
raise ValueError("Unsupported GPT-5.6 reasoning effort")
|
||||
if verbosity not in {None, "low", "medium", "high"}:
|
||||
raise ValueError("verbosity must be low, medium, or high")
|
||||
request: Dict[str, Any] = {
|
||||
"model": self.model,
|
||||
"instructions": self.system_prompt,
|
||||
"input": input_text,
|
||||
}
|
||||
if self.provider == "dashscope":
|
||||
# DashScope runs thinking natively and has no reasoning.effort or
|
||||
# text.verbosity knobs; its gateway also drops non-streaming
|
||||
# requests that stay silent for ~60s, so streaming is mandatory.
|
||||
request["stream"] = True
|
||||
else:
|
||||
request["reasoning"] = {"effort": reasoning_effort}
|
||||
request["background"] = background
|
||||
request["store"] = True
|
||||
if verbosity:
|
||||
request["text"] = {"verbosity": verbosity}
|
||||
if max_output_tokens:
|
||||
request["max_output_tokens"] = max_output_tokens
|
||||
if use_tools:
|
||||
request["tools"] = self._tools()
|
||||
request["tool_choice"] = tool_choice
|
||||
if self.previous_response_id:
|
||||
request["previous_response_id"] = self.previous_response_id
|
||||
return request
|
||||
|
||||
@staticmethod
|
||||
def _output_text(response: Dict[str, Any]) -> str:
|
||||
if not isinstance(response, dict):
|
||||
return ""
|
||||
chunks: List[str] = []
|
||||
for item in response.get("output") or []:
|
||||
if not isinstance(item, dict) or item.get("type") != "message":
|
||||
continue
|
||||
for content in item.get("content") or []:
|
||||
if isinstance(content, dict) and content.get("type") == "output_text" and content.get("text"):
|
||||
chunks.append(content["text"])
|
||||
return "\n".join(chunks).strip()
|
||||
|
||||
@staticmethod
|
||||
def _tool_items(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
if not isinstance(response, dict):
|
||||
return []
|
||||
return [
|
||||
item
|
||||
for item in response.get("output") or []
|
||||
if isinstance(item, dict)
|
||||
and item.get("type") in {
|
||||
"web_search_call",
|
||||
"code_interpreter_call",
|
||||
"hosted_tool_call",
|
||||
}
|
||||
]
|
||||
|
||||
@staticmethod
|
||||
def _citations(response: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
citations = []
|
||||
if not isinstance(response, dict):
|
||||
return citations
|
||||
for item in response.get("output") or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
for content in item.get("content") or []:
|
||||
if not isinstance(content, dict):
|
||||
continue
|
||||
for annotation in content.get("annotations") or []:
|
||||
if isinstance(annotation, dict) and annotation.get("type") in {
|
||||
"url_citation",
|
||||
"container_file_citation",
|
||||
}:
|
||||
citations.append(annotation)
|
||||
# DashScope reports sources on the web_search_call item itself
|
||||
# instead of url_citation annotations; normalize them here.
|
||||
if item.get("type") == "web_search_call":
|
||||
action = item.get("action")
|
||||
if isinstance(action, dict):
|
||||
for source in action.get("sources") or []:
|
||||
url = source if isinstance(source, str) else (source.get("url") if isinstance(source, dict) else None)
|
||||
if url:
|
||||
citations.append(
|
||||
{"type": "url_citation", "url": url}
|
||||
)
|
||||
return citations
|
||||
|
||||
def _post_responses(
|
||||
self, request: Dict[str, Any]
|
||||
) -> tuple[int, Dict[str, Any], Optional[Dict[str, int]]]:
|
||||
"""Send one Responses request and return (status, body, stream_events).
|
||||
|
||||
DashScope requires streaming; the final ``response.completed`` event
|
||||
carries the same response object the non-streaming API returns, so both
|
||||
paths converge on an identical shape.
|
||||
"""
|
||||
headers = {
|
||||
"Authorization": f"Bearer {self.api_key}",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
if not request.get("stream"):
|
||||
http_response = requests.post(
|
||||
f"{self.base_url}/responses",
|
||||
headers=headers,
|
||||
json=request,
|
||||
timeout=900,
|
||||
)
|
||||
try:
|
||||
return http_response.status_code, http_response.json(), None
|
||||
except ValueError:
|
||||
return http_response.status_code, {"raw_text": http_response.text}, None
|
||||
|
||||
event_counts: Dict[str, int] = {}
|
||||
final_response: Optional[Dict[str, Any]] = None
|
||||
with requests.post(
|
||||
f"{self.base_url}/responses",
|
||||
headers=headers,
|
||||
json=request,
|
||||
stream=True,
|
||||
timeout=900,
|
||||
) as http_response:
|
||||
status_code = http_response.status_code
|
||||
if not http_response.ok:
|
||||
return status_code, {"raw_text": http_response.text}, event_counts
|
||||
for line in http_response.iter_lines(decode_unicode=True):
|
||||
if not line or not line.startswith("data:"):
|
||||
continue
|
||||
data = line[len("data:"):].strip()
|
||||
if data == "[DONE]":
|
||||
break
|
||||
try:
|
||||
event = json.loads(data)
|
||||
except ValueError:
|
||||
continue
|
||||
event_type = event.get("type") or "unknown"
|
||||
event_counts[event_type] = event_counts.get(event_type, 0) + 1
|
||||
if event_type in {"response.completed", "response.failed"}:
|
||||
final_response = event.get("response")
|
||||
if final_response is None:
|
||||
return status_code, {"error": {"type": "stream_incomplete",
|
||||
"message": "stream ended without response.completed"}}, event_counts
|
||||
return status_code, final_response, event_counts
|
||||
|
||||
def process_request(
|
||||
self,
|
||||
user_request: str,
|
||||
use_tools: bool = True,
|
||||
tool_choice: Literal["auto", "none", "required"] = "auto",
|
||||
temperature: float = 0.3,
|
||||
max_tokens: Optional[int] = None,
|
||||
reasoning_effort: str = "low",
|
||||
verbosity: Optional[str] = None,
|
||||
dry_run: bool = False,
|
||||
background: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""Create one Responses API turn and retain its complete trace.
|
||||
|
||||
``temperature`` remains in the signature for legacy callers, but is not
|
||||
sent: GPT-5.6 reasoning requests use ``reasoning.effort`` instead.
|
||||
"""
|
||||
request = self._build_responses_request(
|
||||
user_request,
|
||||
use_tools=use_tools,
|
||||
tool_choice=tool_choice,
|
||||
reasoning_effort=reasoning_effort,
|
||||
verbosity=verbosity,
|
||||
max_output_tokens=max_tokens,
|
||||
background=background,
|
||||
)
|
||||
if dry_run:
|
||||
return {
|
||||
"success": True,
|
||||
"dry_run": True,
|
||||
"request": request,
|
||||
"response": None,
|
||||
"tool_calls": [],
|
||||
"model": self.model,
|
||||
"provider": self.provider,
|
||||
}
|
||||
|
||||
started = time.monotonic()
|
||||
try:
|
||||
status_code, response, stream_events = self._post_responses(request)
|
||||
elapsed = round(time.monotonic() - started, 6)
|
||||
turn = {
|
||||
"request": json.loads(json.dumps(request, ensure_ascii=False)),
|
||||
"http_status": status_code,
|
||||
"response": response,
|
||||
"elapsed_seconds": elapsed,
|
||||
}
|
||||
if stream_events:
|
||||
turn["stream_event_counts"] = stream_events
|
||||
self.api_turns.append(turn)
|
||||
if not isinstance(response, dict) or status_code >= 400 or response.get("error"):
|
||||
error = (response.get("error") if isinstance(response, dict) else None) or {
|
||||
"type": "http_error",
|
||||
"message": (response.get("raw_text") if isinstance(response, dict) else None) or (json.dumps(response)[:500] if response is not None else "Empty response"),
|
||||
}
|
||||
return {
|
||||
"success": False,
|
||||
"error": error,
|
||||
"response": None,
|
||||
"request": request,
|
||||
"raw_response": response,
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"usage": (response.get("usage") if isinstance(response, dict) else {}) or {},
|
||||
"model": self.model,
|
||||
"provider": self.provider,
|
||||
"base_url": self.base_url,
|
||||
"elapsed_seconds": elapsed,
|
||||
}
|
||||
|
||||
self.previous_response_id = response.get("id")
|
||||
text = self._output_text(response)
|
||||
self.conversation_history.extend(
|
||||
[
|
||||
{"role": "user", "content": user_request},
|
||||
{"role": "assistant", "content": text},
|
||||
]
|
||||
)
|
||||
return {
|
||||
"success": response.get("status") == "completed" and bool(text),
|
||||
"error": response.get("error"),
|
||||
"response": text,
|
||||
"request": request,
|
||||
"raw_response": response,
|
||||
"output_items": response.get("output") or [],
|
||||
"tool_calls": self._tool_items(response),
|
||||
"citations": self._citations(response),
|
||||
"usage": response.get("usage") or {},
|
||||
"model": response.get("model") or self.model,
|
||||
"requested_model": self.model,
|
||||
"provider": self.provider,
|
||||
"base_url": self.base_url,
|
||||
"response_id": response.get("id"),
|
||||
"status": response.get("status"),
|
||||
"elapsed_seconds": elapsed,
|
||||
"temperature_omitted_for_reasoning_model": temperature is not None,
|
||||
}
|
||||
except Exception as exc:
|
||||
elapsed = round(time.monotonic() - started, 6)
|
||||
self.api_turns.append(
|
||||
{
|
||||
"request": request,
|
||||
"elapsed_seconds": elapsed,
|
||||
"error": {"class": type(exc).__name__, "message": str(exc)},
|
||||
}
|
||||
)
|
||||
return {
|
||||
"success": False,
|
||||
"error": {"class": type(exc).__name__, "message": str(exc)},
|
||||
"response": None,
|
||||
"request": request,
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"model": self.model,
|
||||
"provider": self.provider,
|
||||
"base_url": self.base_url,
|
||||
"elapsed_seconds": elapsed,
|
||||
}
|
||||
|
||||
def search_and_analyze(
|
||||
self, topic: str, analysis_code: Optional[str] = None
|
||||
) -> Dict[str, Any]:
|
||||
code_requirement = (
|
||||
f"Run this supplied Python in the hosted tool and inspect its output:\n{analysis_code}"
|
||||
if analysis_code
|
||||
else "Use the hosted Python tool for all quantitative processing."
|
||||
)
|
||||
return self.process_request(
|
||||
f"Research current information about {topic}. {code_requirement} "
|
||||
"Cite web sources and distinguish searched facts from computed results.",
|
||||
use_tools=True,
|
||||
reasoning_effort="medium",
|
||||
)
|
||||
|
||||
def clear_history(self) -> None:
|
||||
self.conversation_history = []
|
||||
self.previous_response_id = None
|
||||
self.api_turns = []
|
||||
|
||||
def get_history(self) -> List[Dict[str, Any]]:
|
||||
return json.loads(json.dumps(self.conversation_history, ensure_ascii=False))
|
||||
|
||||
def set_system_prompt(self, prompt: str) -> None:
|
||||
self.system_prompt = prompt
|
||||
|
||||
|
||||
class GPT5AgentChain:
|
||||
"""Sequential Responses turns linked with ``previous_response_id``."""
|
||||
|
||||
def __init__(self, agent: GPT5NativeAgent):
|
||||
self.agent = agent
|
||||
self.chain_results: List[Dict[str, Any]] = []
|
||||
|
||||
def add_step(self, request: str, **kwargs: Any) -> "GPT5AgentChain":
|
||||
self.chain_results.append(
|
||||
{"request": request, "result": self.agent.process_request(request, **kwargs)}
|
||||
)
|
||||
return self
|
||||
|
||||
def execute(self) -> List[Dict[str, Any]]:
|
||||
return self.chain_results
|
||||
|
||||
def clear(self) -> None:
|
||||
self.chain_results = []
|
||||
@@ -0,0 +1,86 @@
|
||||
"""Configuration for the exact GPT-5.6 Responses API companion."""
|
||||
|
||||
import os
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
|
||||
def _optional_int_env(name: str) -> Optional[int]:
|
||||
"""Read an optional integer without making module import configuration-fatal."""
|
||||
raw_value = os.getenv(name)
|
||||
if raw_value is None:
|
||||
return None
|
||||
cleaned = raw_value.strip()
|
||||
if not cleaned or not cleaned.isascii() or not cleaned.isdecimal():
|
||||
return None
|
||||
try:
|
||||
return int(cleaned)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
class Config:
|
||||
OPENAI_API_KEY = os.getenv("OPENAI_API_KEY", "")
|
||||
OPENAI_BASE_URL = os.getenv("OPENAI_BASE_URL", "https://api.openai.com/v1")
|
||||
OPENROUTER_API_KEY = os.getenv("OPENROUTER_API_KEY", "")
|
||||
OPENROUTER_BASE_URL = os.getenv(
|
||||
"OPENROUTER_BASE_URL", "https://openrouter.ai/api/v1"
|
||||
)
|
||||
DASHSCOPE_API_KEY = os.getenv("DASHSCOPE_API_KEY", "")
|
||||
DASHSCOPE_BASE_URL = os.getenv(
|
||||
"DASHSCOPE_BASE_URL",
|
||||
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1",
|
||||
)
|
||||
DASHSCOPE_MODEL = os.getenv("DASHSCOPE_MODEL", "qwen3.7-plus")
|
||||
BACKEND = os.getenv("BACKEND", "openai")
|
||||
MODEL_NAME = os.getenv("MODEL_NAME", "gpt-5.6-sol")
|
||||
DEFAULT_TEMPERATURE = 0.3 # legacy CLI compatibility; intentionally omitted
|
||||
DEFAULT_MAX_TOKENS: Optional[int] = _optional_int_env("DEFAULT_MAX_TOKENS")
|
||||
DEFAULT_TOOL_CHOICE = os.getenv("DEFAULT_TOOL_CHOICE", "auto")
|
||||
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
|
||||
LOG_FORMAT = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
RATE_LIMIT_RPM = int(os.getenv("RATE_LIMIT_RPM", "20"))
|
||||
MAX_RETRIES = int(os.getenv("MAX_RETRIES", "3"))
|
||||
RETRY_DELAY = float(os.getenv("RETRY_DELAY", "1.0"))
|
||||
WEB_SEARCH_MAX_RESULTS = int(os.getenv("WEB_SEARCH_MAX_RESULTS", "5"))
|
||||
CODE_INTERPRETER_TIMEOUT = int(os.getenv("CODE_INTERPRETER_TIMEOUT", "30"))
|
||||
|
||||
@classmethod
|
||||
def resolve(
|
||||
cls, backend: Optional[str] = None, model: Optional[str] = None
|
||||
) -> Tuple[str, str, str]:
|
||||
backend = backend or cls.BACKEND
|
||||
if backend == "openai":
|
||||
model = model or cls.MODEL_NAME
|
||||
return cls.OPENAI_API_KEY, cls.OPENAI_BASE_URL, model.removeprefix("openai/")
|
||||
if backend == "openrouter":
|
||||
model = model or cls.MODEL_NAME
|
||||
routed = model if model.startswith("openai/") else f"openai/{model}"
|
||||
return cls.OPENROUTER_API_KEY, cls.OPENROUTER_BASE_URL, routed
|
||||
if backend == "dashscope":
|
||||
return cls.DASHSCOPE_API_KEY, cls.DASHSCOPE_BASE_URL, model or cls.DASHSCOPE_MODEL
|
||||
raise ValueError("backend must be openai, openrouter, or dashscope")
|
||||
|
||||
@classmethod
|
||||
def validate(cls, backend: Optional[str] = None) -> bool:
|
||||
key, _, _ = cls.resolve(backend)
|
||||
if not key:
|
||||
print(f"Error: no API key for {backend or cls.BACKEND}")
|
||||
return False
|
||||
return True
|
||||
|
||||
@classmethod
|
||||
def display(cls, backend: Optional[str] = None) -> None:
|
||||
key, base_url, model = cls.resolve(backend)
|
||||
print("=== GPT-5.6 Responses Configuration ===")
|
||||
print(f"Backend: {backend or cls.BACKEND}")
|
||||
print(f"API Base URL: {base_url}")
|
||||
print(f"Model: {model}")
|
||||
print(f"API Key configured: {bool(key)}")
|
||||
|
||||
|
||||
def check_config() -> bool:
|
||||
return Config.validate()
|
||||
@@ -0,0 +1,44 @@
|
||||
# OpenRouter API Configuration
|
||||
# Get your API key from: https://openrouter.ai/keys
|
||||
# Note: this experiment uses OpenRouter as its PRIMARY backend (no fallback
|
||||
# needed). The same OPENROUTER_API_KEY also serves as the universal fallback
|
||||
# for the other chapter1 experiments (context, learning-from-experience,
|
||||
# web-search-agent) when their direct provider key is missing.
|
||||
OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
|
||||
# API Base URL (default is fine for most users)
|
||||
OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
|
||||
|
||||
# OpenAI official path (canonical GPT-5.6 Sol); needs an account with quota
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# Alibaba Model Studio (DashScope) — eligible equivalent-provider path with
|
||||
# hosted web_search + hosted code_interpreter on the Responses API.
|
||||
# Get your API key from: https://www.alibabacloud.com/help/en/model-studio/get-api-key
|
||||
# Use https://dashscope.aliyuncs.com/compatible-mode/v1 for a China-region key.
|
||||
DASHSCOPE_API_KEY=your-dashscope-api-key
|
||||
DASHSCOPE_BASE_URL=https://dashscope-intl.aliyuncs.com/compatible-mode/v1
|
||||
DASHSCOPE_MODEL=qwen3.7-plus
|
||||
|
||||
# Model Configuration
|
||||
# GPT-5 model identifier on OpenRouter
|
||||
MODEL_NAME=openai/gpt-5.6-sol
|
||||
|
||||
# Request Configuration
|
||||
DEFAULT_TEMPERATURE=0.3
|
||||
DEFAULT_MAX_TOKENS=4000
|
||||
DEFAULT_TOOL_CHOICE=auto
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=INFO
|
||||
|
||||
# Rate Limiting
|
||||
RATE_LIMIT_RPM=20
|
||||
|
||||
# Retry Configuration
|
||||
MAX_RETRIES=3
|
||||
RETRY_DELAY=1.0
|
||||
|
||||
# Tool-specific Configuration
|
||||
WEB_SEARCH_MAX_RESULTS=5
|
||||
CODE_INTERPRETER_TIMEOUT=30
|
||||
@@ -0,0 +1,249 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Example showing the exact OpenRouter GPT-5 request format matching the Go implementation
|
||||
"""
|
||||
|
||||
import json
|
||||
import requests
|
||||
import os
|
||||
from typing import Dict, Any
|
||||
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv()
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
def make_gpt5_openrouter_request(
|
||||
api_key: str,
|
||||
system_prompt: str,
|
||||
user_prompt: str,
|
||||
reasoning_effort: str = "low"
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Make a GPT-5 request using the exact format from the Go implementation
|
||||
|
||||
This matches the GPT5OpenRouterRequest structure from the Go code
|
||||
"""
|
||||
|
||||
# Build messages (matching Go implementation)
|
||||
messages = [
|
||||
{
|
||||
"role": "system",
|
||||
"content": system_prompt
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": user_prompt
|
||||
}
|
||||
]
|
||||
|
||||
# Build web search tool configuration (matching Go GPT5OpenRouterWebSearchTool)
|
||||
web_search_tool = {
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium",
|
||||
"user_location": {
|
||||
"type": "approximate",
|
||||
"country": "US"
|
||||
}
|
||||
}
|
||||
|
||||
# Build request with OpenRouter-specific parameters (matching Go GPT5OpenRouterRequest)
|
||||
request_body = {
|
||||
"model": "openai/gpt-5.6-sol", # Default from Go code
|
||||
"messages": messages,
|
||||
"tools": [web_search_tool],
|
||||
"tool_choice": "auto",
|
||||
"parallel_tool_calls": True,
|
||||
"reasoning": {
|
||||
"effort": reasoning_effort,
|
||||
"generate_summary": False
|
||||
},
|
||||
"background": False,
|
||||
"stream": False # Can be set to True for streaming
|
||||
}
|
||||
|
||||
print("="*60)
|
||||
print("GPT-5 OpenRouter Request (matching Go implementation):")
|
||||
print("="*60)
|
||||
print(json.dumps(request_body, indent=2))
|
||||
print("="*60)
|
||||
|
||||
# Set headers (matching Go implementation)
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {api_key}"
|
||||
}
|
||||
|
||||
# Make the request
|
||||
url = "https://openrouter.ai/api/v1/chat/completions"
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
url,
|
||||
headers=headers,
|
||||
json=request_body,
|
||||
timeout=600 # Match Go timeout
|
||||
)
|
||||
|
||||
print(f"\nResponse Status: {response.status_code}")
|
||||
|
||||
if response.status_code == 200:
|
||||
response_data = response.json()
|
||||
|
||||
# Log usage (matching Go logging)
|
||||
if "usage" in response_data:
|
||||
usage = response_data["usage"]
|
||||
input_tokens = usage.get(
|
||||
"prompt_tokens", usage.get("input_tokens", 0)
|
||||
)
|
||||
output_tokens = usage.get(
|
||||
"completion_tokens", usage.get("output_tokens", 0)
|
||||
)
|
||||
input_details = usage.get(
|
||||
"prompt_tokens_details", usage.get("input_tokens_details")
|
||||
)
|
||||
output_details = usage.get(
|
||||
"completion_tokens_details", usage.get("output_tokens_details")
|
||||
)
|
||||
print("\nGPT-5 OpenRouter Usage:")
|
||||
print(f" Input: {input_tokens} tokens", end="")
|
||||
if isinstance(input_details, dict):
|
||||
print(f" (cached: {input_details.get('cached_tokens', 0)})")
|
||||
else:
|
||||
print()
|
||||
|
||||
print(f" Output: {output_tokens} tokens", end="")
|
||||
if isinstance(output_details, dict):
|
||||
print(f" (reasoning: {output_details.get('reasoning_tokens', 0)})")
|
||||
else:
|
||||
print()
|
||||
|
||||
print(f" Total: {usage.get('total_tokens', 0)}")
|
||||
|
||||
return response_data
|
||||
else:
|
||||
print(f"\nError: {response.text}")
|
||||
return {"error": response.text, "status_code": response.status_code}
|
||||
|
||||
except Exception as e:
|
||||
print(f"\nException: {str(e)}")
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
def demonstrate_streaming_response():
|
||||
"""
|
||||
Demonstrate how streaming would work (matching Go handleStreamingResponse)
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("Streaming Response Handler (pseudo-code matching Go):")
|
||||
print("="*60)
|
||||
|
||||
streaming_code = '''
|
||||
def handle_streaming_response(response):
|
||||
"""
|
||||
Handle streaming responses from GPT-5 OpenRouter API
|
||||
Matches Go handleStreamingResponse function
|
||||
"""
|
||||
content_builder = []
|
||||
reasoning_builder = []
|
||||
reasoning_token_count = 0
|
||||
|
||||
for line in response.iter_lines():
|
||||
if not line:
|
||||
continue
|
||||
|
||||
line_str = line.decode('utf-8')
|
||||
|
||||
if not line_str.startswith("data: "):
|
||||
continue
|
||||
|
||||
data = line_str[6:] # Remove "data: " prefix
|
||||
|
||||
if data == "[DONE]":
|
||||
break
|
||||
|
||||
try:
|
||||
chunk = json.loads(data)
|
||||
|
||||
if "choices" in chunk and len(chunk["choices"]) > 0:
|
||||
delta = chunk["choices"][0].get("delta", {})
|
||||
|
||||
# Check for reasoning content
|
||||
if "reasoning_content" in delta:
|
||||
reasoning = delta["reasoning_content"]
|
||||
reasoning_builder.append(reasoning)
|
||||
reasoning_token_count += 1
|
||||
print(f"🧠 [GPT-5 THINKING] {reasoning}")
|
||||
|
||||
# Check for regular content
|
||||
if "content" in delta:
|
||||
content = delta["content"]
|
||||
content_builder.append(content)
|
||||
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
final_content = "".join(content_builder)
|
||||
return final_content
|
||||
'''
|
||||
print(streaming_code)
|
||||
|
||||
|
||||
def main():
|
||||
"""
|
||||
Main demonstration
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print(" GPT-5 OpenRouter Request Format Demo")
|
||||
print(" Exact match with Go implementation")
|
||||
print("="*60)
|
||||
|
||||
# Get API key from environment
|
||||
api_key = os.getenv("OPENROUTER_API_KEY")
|
||||
|
||||
if not api_key:
|
||||
print("\n❌ Error: OPENROUTER_API_KEY not found in environment")
|
||||
print("Please set: export OPENROUTER_API_KEY=your-openrouter-api-key")
|
||||
return
|
||||
|
||||
# Example prompts
|
||||
system_prompt = "You are a helpful AI assistant with web search capabilities."
|
||||
user_prompt = "What are the latest developments in artificial intelligence?"
|
||||
|
||||
print("\n1. Making request with LOW reasoning effort:")
|
||||
print("-"*60)
|
||||
result_low = make_gpt5_openrouter_request(
|
||||
api_key=api_key,
|
||||
system_prompt=system_prompt,
|
||||
user_prompt=user_prompt,
|
||||
reasoning_effort="low"
|
||||
)
|
||||
|
||||
if "choices" in result_low:
|
||||
content = result_low["choices"][0]["message"]["content"]
|
||||
print(f"\nResponse preview: {content[:200]}...")
|
||||
|
||||
print("\n2. Making request with HIGH reasoning effort:")
|
||||
print("-"*60)
|
||||
result_high = make_gpt5_openrouter_request(
|
||||
api_key=api_key,
|
||||
system_prompt=system_prompt,
|
||||
user_prompt="Explain the implications of quantum computing on cryptography",
|
||||
reasoning_effort="high"
|
||||
)
|
||||
|
||||
if "choices" in result_high:
|
||||
content = result_high["choices"][0]["message"]["content"]
|
||||
print(f"\nResponse preview: {content[:200]}...")
|
||||
|
||||
# Show streaming handler
|
||||
demonstrate_streaming_response()
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Demo complete! This shows the exact request format from Go.")
|
||||
print("="*60)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,116 @@
|
||||
# Go vs Python Implementation Comparison
|
||||
|
||||
This document shows how the Python implementation exactly matches the Go implementation for GPT-5 OpenRouter API calls.
|
||||
|
||||
## Request Structure Comparison
|
||||
|
||||
### Go Implementation
|
||||
```go
|
||||
// From the provided Go code
|
||||
webSearchTool := GPT5OpenRouterWebSearchTool{
|
||||
Type: "web_search",
|
||||
SearchContextSize: "medium",
|
||||
UserLocation: map[string]interface{}{
|
||||
"type": "approximate",
|
||||
"country": "US",
|
||||
},
|
||||
}
|
||||
|
||||
request := GPT5OpenRouterRequest{
|
||||
Model: c.model,
|
||||
Messages: messages,
|
||||
Tools: []GPT5OpenRouterWebSearchTool{webSearchTool},
|
||||
ToolChoice: "auto",
|
||||
ParallelToolCalls: true,
|
||||
Reasoning: &GPT5OpenRouterReasoning{
|
||||
Effort: reasoningEffort,
|
||||
GenerateSummary: false,
|
||||
},
|
||||
Background: false,
|
||||
Stream: false,
|
||||
}
|
||||
```
|
||||
|
||||
### Python Implementation
|
||||
```python
|
||||
# From agent.py
|
||||
web_search_tool = {
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium",
|
||||
"user_location": {
|
||||
"type": "approximate",
|
||||
"country": "US"
|
||||
}
|
||||
}
|
||||
|
||||
request_body = {
|
||||
"model": self.model,
|
||||
"messages": messages,
|
||||
"tools": [web_search_tool],
|
||||
"tool_choice": "auto",
|
||||
"parallel_tool_calls": True,
|
||||
"reasoning": {
|
||||
"effort": reasoning_effort,
|
||||
"generate_summary": False
|
||||
},
|
||||
"background": False,
|
||||
"stream": False
|
||||
}
|
||||
```
|
||||
|
||||
## Key Matching Points
|
||||
|
||||
1. **Tool Structure**: Both implementations use the same tool structure with `type: "web_search"` and additional configuration fields.
|
||||
|
||||
2. **Request Parameters**: Identical parameters including:
|
||||
- `model`
|
||||
- `messages`
|
||||
- `tools` (array of web_search tools)
|
||||
- `tool_choice: "auto"`
|
||||
- `parallel_tool_calls: true/True`
|
||||
- `reasoning` with effort and generate_summary
|
||||
- `background: false/False`
|
||||
- `stream: false/False`
|
||||
|
||||
3. **Headers**: Both use simple headers:
|
||||
```go
|
||||
// Go
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", fmt.Sprintf("Bearer %s", c.apiKey))
|
||||
```
|
||||
|
||||
```python
|
||||
# Python
|
||||
headers = {
|
||||
"Content-Type": "application/json",
|
||||
"Authorization": f"Bearer {self.api_key}"
|
||||
}
|
||||
```
|
||||
|
||||
4. **Model Default**: Both default to `openai/gpt-5.6-sol`
|
||||
|
||||
5. **Reasoning Levels**: Both support "low", "medium", and "high" reasoning effort
|
||||
|
||||
## Usage Comparison
|
||||
|
||||
### Go
|
||||
```go
|
||||
client := NewGPT5OpenRouterClientAdapter(apiKey, baseURL, model)
|
||||
response, err := client.CallGPT5(ctx, systemPrompt, userPrompt, "medium")
|
||||
```
|
||||
|
||||
### Python
|
||||
```python
|
||||
agent = GPT5NativeAgent(api_key, base_url, model)
|
||||
result = agent.process_request(user_request, use_tools=True, reasoning_effort="medium")
|
||||
```
|
||||
|
||||
## Response Handling
|
||||
|
||||
Both implementations:
|
||||
- Handle streaming and non-streaming responses
|
||||
- Log token usage including cached and reasoning tokens
|
||||
- Extract content from the response choices
|
||||
- Handle errors with appropriate status codes
|
||||
|
||||
The Python implementation is a direct port of the Go implementation, ensuring complete compatibility with the OpenRouter GPT-5 API.
|
||||
@@ -0,0 +1,444 @@
|
||||
"""
|
||||
Main entry point for GPT-5 Native Tools Agent
|
||||
Interactive CLI for using web_search and code_interpreter tools
|
||||
"""
|
||||
|
||||
import sys
|
||||
import json
|
||||
import logging
|
||||
from typing import Optional
|
||||
from agent import GPT5NativeAgent, GPT5AgentChain
|
||||
from config import Config
|
||||
import argparse
|
||||
|
||||
# Set up logging
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, Config.LOG_LEVEL),
|
||||
format=Config.LOG_FORMAT
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class InteractiveCLI:
|
||||
"""Interactive command-line interface for GPT-5 Agent"""
|
||||
|
||||
def __init__(self, backend: str = None, model: str = None):
|
||||
"""Initialize the CLI"""
|
||||
if not Config.validate(backend):
|
||||
raise ValueError("Invalid configuration. Please check your .env file")
|
||||
api_key, base_url, resolved_model = Config.resolve(backend, model)
|
||||
self.agent = GPT5NativeAgent(
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
model=resolved_model,
|
||||
)
|
||||
self.backend = backend or Config.BACKEND
|
||||
|
||||
self.commands = {
|
||||
"/help": self.show_help,
|
||||
"/clear": self.clear_history,
|
||||
"/history": self.show_history,
|
||||
"/tools": self.toggle_tools,
|
||||
"/search": self.search_mode,
|
||||
"/code": self.code_mode,
|
||||
"/analyze": self.analyze_mode,
|
||||
"/config": self.show_config,
|
||||
"/reasoning": self.set_reasoning_effort,
|
||||
"/exit": self.exit_cli,
|
||||
"/quit": self.exit_cli,
|
||||
}
|
||||
|
||||
self.use_tools = True
|
||||
self.tool_choice = "auto"
|
||||
self.reasoning_effort = "low" # Default reasoning effort
|
||||
|
||||
def show_help(self):
|
||||
"""Display help information"""
|
||||
help_text = """
|
||||
Commands:
|
||||
/help - Show this help message
|
||||
/clear - Clear conversation history
|
||||
/history - Show conversation history
|
||||
/tools - Toggle tools on/off
|
||||
/search - Enter web search mode
|
||||
/code - Enter code interpreter mode
|
||||
/analyze - Combined search + analysis mode
|
||||
/config - Show current configuration
|
||||
/reasoning - Set reasoning effort (low/medium/high)
|
||||
/exit - Exit the application
|
||||
|
||||
Native Tools:
|
||||
• web_search - Search the internet for real-time info
|
||||
• code_interpreter - Execute Python code and analyze
|
||||
|
||||
Usage:
|
||||
Simply type your request and the agent will use
|
||||
appropriate tools automatically.
|
||||
|
||||
Examples:
|
||||
"东盟 10 国首都之间，距离最近的两个首都是？给出你的详细分析推理过程。"
|
||||
"搜索最近一年比特币的价格，计算收益率、最大回撤、年化波动等重要指标"
|
||||
"""
|
||||
print(help_text)
|
||||
|
||||
def clear_history(self):
|
||||
"""Clear conversation history"""
|
||||
self.agent.clear_history()
|
||||
print("✅ Conversation history cleared")
|
||||
|
||||
def show_history(self):
|
||||
"""Display conversation history"""
|
||||
history = self.agent.get_history()
|
||||
if not history:
|
||||
print("📭 No conversation history")
|
||||
return
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("CONVERSATION HISTORY")
|
||||
print("="*60)
|
||||
|
||||
for i, msg in enumerate(history, 1):
|
||||
role = msg["role"].upper()
|
||||
content = msg["content"][:200] + "..." if len(msg["content"]) > 200 else msg["content"]
|
||||
print(f"\n[{i}] {role}:\n{content}")
|
||||
|
||||
print("="*60)
|
||||
|
||||
def toggle_tools(self):
|
||||
"""Toggle tool usage on/off"""
|
||||
self.use_tools = not self.use_tools
|
||||
status = "enabled" if self.use_tools else "disabled"
|
||||
print(f"🔧 Tools {status}")
|
||||
|
||||
def search_mode(self):
|
||||
"""Enter web search mode"""
|
||||
print("\n🔍 Web Search Mode")
|
||||
print("Enter your search query (or 'back' to return):")
|
||||
|
||||
query = input("> ").strip()
|
||||
if query.lower() == "back":
|
||||
return
|
||||
|
||||
request = f"Search the web for: {query}"
|
||||
self._process_request(request, force_tools=True)
|
||||
|
||||
def code_mode(self):
|
||||
"""Enter code interpreter mode"""
|
||||
print("\n💻 Code Interpreter Mode")
|
||||
print("Enter your code or computational request (or 'back' to return):")
|
||||
|
||||
request = input("> ").strip()
|
||||
if request.lower() == "back":
|
||||
return
|
||||
|
||||
enhanced_request = f"Use the code interpreter to: {request}"
|
||||
self._process_request(enhanced_request, force_tools=True)
|
||||
|
||||
def analyze_mode(self):
|
||||
"""Combined search and analysis mode"""
|
||||
print("\n🔬 Search & Analyze Mode")
|
||||
print("Enter topic to research and analyze (or 'back' to return):")
|
||||
|
||||
topic = input("> ").strip()
|
||||
if topic.lower() == "back":
|
||||
return
|
||||
|
||||
print("\nOptional: Enter Python code for analysis (press Enter to skip):")
|
||||
code = input("> ").strip()
|
||||
|
||||
if code:
|
||||
result = self.agent.search_and_analyze(topic, code)
|
||||
else:
|
||||
result = self.agent.search_and_analyze(topic)
|
||||
|
||||
self._display_result(result)
|
||||
|
||||
def show_config(self):
|
||||
"""Display current configuration"""
|
||||
Config.display()
|
||||
print(f"\nCurrent Settings:")
|
||||
print(f" Tools Enabled: {self.use_tools}")
|
||||
print(f" Tool Choice: {self.tool_choice}")
|
||||
print(f" Reasoning Effort: {self.reasoning_effort}")
|
||||
|
||||
def set_reasoning_effort(self):
|
||||
"""Set the reasoning effort level"""
|
||||
print("\n🧠 Set Reasoning Effort")
|
||||
print("Options: low, medium, high")
|
||||
print(f"Current: {self.reasoning_effort}")
|
||||
|
||||
effort = input("Enter new effort level: ").strip().lower()
|
||||
if effort in ["low", "medium", "high"]:
|
||||
self.reasoning_effort = effort
|
||||
print(f"✅ Reasoning effort set to: {effort}")
|
||||
else:
|
||||
print(f"❌ Invalid effort level. Must be low, medium, or high")
|
||||
|
||||
def exit_cli(self):
|
||||
"""Exit the application"""
|
||||
print("\n👋 Goodbye!")
|
||||
sys.exit(0)
|
||||
|
||||
def _process_request(self, request: str, force_tools: bool = False):
|
||||
"""
|
||||
Process a user request
|
||||
|
||||
Args:
|
||||
request: User request
|
||||
force_tools: Force tool usage regardless of settings
|
||||
"""
|
||||
use_tools = force_tools or self.use_tools
|
||||
|
||||
result = self.agent.process_request(
|
||||
request,
|
||||
use_tools=use_tools,
|
||||
tool_choice=self.tool_choice if use_tools else "none",
|
||||
temperature=Config.DEFAULT_TEMPERATURE,
|
||||
max_tokens=Config.DEFAULT_MAX_TOKENS,
|
||||
reasoning_effort=self.reasoning_effort
|
||||
)
|
||||
|
||||
self._display_result(result)
|
||||
|
||||
def _display_result(self, result: dict):
|
||||
"""
|
||||
Display the result of a request
|
||||
|
||||
Args:
|
||||
result: Result dictionary from agent
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
|
||||
if result["success"]:
|
||||
# Display tool usage
|
||||
if result["tool_calls"]:
|
||||
print("🔧 Tools Used:")
|
||||
for tool in result["tool_calls"]:
|
||||
print(f" • {tool.get('type', 'unknown_tool')}")
|
||||
print()
|
||||
|
||||
# Display response
|
||||
print("📝 Response:")
|
||||
print("-"*60)
|
||||
print(result["response"])
|
||||
print("-"*60)
|
||||
|
||||
# Display token usage
|
||||
if result.get("usage"):
|
||||
usage = result["usage"]
|
||||
total = usage.get("total_tokens", 0)
|
||||
if total:
|
||||
print(f"\n📊 Tokens used: {total}")
|
||||
else:
|
||||
print(f"❌ Error: {result.get('error', 'Unknown error')}")
|
||||
|
||||
print("="*60)
|
||||
|
||||
def run(self):
|
||||
"""Run the interactive CLI"""
|
||||
print("\n" + "="*60)
|
||||
print(" 🤖 GPT-5 Native Tools Agent")
|
||||
print(f" Responses API backend: {self.backend}")
|
||||
print("="*60)
|
||||
|
||||
self.show_help()
|
||||
|
||||
while True:
|
||||
try:
|
||||
print("\n💬 Enter your request (or /help for commands):")
|
||||
user_input = input("> ").strip()
|
||||
|
||||
if not user_input:
|
||||
continue
|
||||
|
||||
# Check for commands
|
||||
if user_input.startswith("/"):
|
||||
command = user_input.split()[0].lower()
|
||||
if command in self.commands:
|
||||
self.commands[command]()
|
||||
else:
|
||||
print(f"❌ Unknown command: {command}")
|
||||
print("Type /help for available commands")
|
||||
else:
|
||||
# Process as regular request
|
||||
self._process_request(user_input)
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n\n⚠️ Interrupted. Type /exit to quit or continue chatting.")
|
||||
except Exception as e:
|
||||
logger.error(f"Error: {str(e)}")
|
||||
print(f"❌ An error occurred: {str(e)}")
|
||||
|
||||
|
||||
def _run_single(args):
|
||||
"""执行单次请求（single / dry-run 模式），打印可读轨迹并按需保存结果。"""
|
||||
# dry-run 只组装请求体、不联网，因此无需真实 API Key
|
||||
api_key, base_url, model = Config.resolve(args.backend, args.model)
|
||||
api_key = api_key or ("DRYRUN-PLACEHOLDER" if args.dry_run else "")
|
||||
|
||||
agent = GPT5NativeAgent(
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
model=model,
|
||||
)
|
||||
|
||||
result = agent.process_request(
|
||||
args.request,
|
||||
use_tools=not args.no_tools,
|
||||
temperature=Config.DEFAULT_TEMPERATURE,
|
||||
max_tokens=Config.DEFAULT_MAX_TOKENS,
|
||||
reasoning_effort=args.reasoning,
|
||||
verbosity=args.verbosity,
|
||||
dry_run=args.dry_run
|
||||
)
|
||||
|
||||
# dry-run：打印将要发送给模型的完整请求体（原生工具定义 + 参数）
|
||||
if result.get("dry_run"):
|
||||
print("\n" + "=" * 60)
|
||||
print("🧪 Dry-run：以下是发送给 GPT-5 的请求体（未联网）")
|
||||
print("=" * 60)
|
||||
print(f"Model: {result['model']}")
|
||||
print(f"任务: {args.request}")
|
||||
print("-" * 60)
|
||||
print(json.dumps(result["request"], indent=2, ensure_ascii=False))
|
||||
print("=" * 60)
|
||||
elif result["success"]:
|
||||
print("\n" + "=" * 60)
|
||||
print("📝 Response:")
|
||||
print("-" * 60)
|
||||
print(result["response"])
|
||||
print("-" * 60)
|
||||
usage = result.get("usage") or {}
|
||||
if usage:
|
||||
print(
|
||||
f"📊 Tokens - Input: {usage.get('input_tokens', 'N/A')}, "
|
||||
f"Output: {usage.get('output_tokens', 'N/A')}, "
|
||||
f"Reasoning: {usage.get('output_tokens_details', {}).get('reasoning_tokens', 0)}, "
|
||||
f"Total: {usage.get('total_tokens', 'N/A')}"
|
||||
)
|
||||
print("=" * 60)
|
||||
else:
|
||||
print(f"❌ Error: {result.get('error')}")
|
||||
|
||||
# 按需将完整结果（含轨迹/请求体）保存为 JSON，便于复盘
|
||||
if args.output:
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, indent=2, ensure_ascii=False)
|
||||
print(f"💾 结果已保存到: {args.output}")
|
||||
|
||||
if not result["success"]:
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def main():
|
||||
"""主入口：解析命令行参数并分派到交互 / 单次 / 测试模式。"""
|
||||
parser = argparse.ArgumentParser(
|
||||
description="GPT-5 原生工具 Agent —— 演示实验 1.3：网络搜索 + 代码解释器的原生 Deep Research 能力",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""示例：
|
||||
python main.py # 交互模式（默认）
|
||||
python main.py --mode single --request "东盟 10 国首都之间距离最近的两个首都是？"
|
||||
python main.py --mode single --request "分析比特币近一月走势" --reasoning high --verbosity high
|
||||
python main.py --mode single --request "..." --output result.json
|
||||
python main.py --dry-run --request "..." # 离线查看请求体（原生工具定义），无需 API Key
|
||||
python main.py --mode test --test basic # 运行指定联网手动用例
|
||||
""",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--mode",
|
||||
choices=["interactive", "single", "test"],
|
||||
default="interactive",
|
||||
help="运行模式：interactive 交互对话（默认）/ single 单次请求 / test 联网手动用例",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--request",
|
||||
type=str,
|
||||
help="single / dry-run 模式下的任务或查询内容",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--backend",
|
||||
choices=["openai", "openrouter", "dashscope"],
|
||||
default=Config.BACKEND,
|
||||
help="Responses API backend; openai is the exact canonical path, dashscope is the eligible equivalent-provider path",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--model",
|
||||
type=str,
|
||||
default=None,
|
||||
help=f"覆盖模型名称（默认取配置 {Config.MODEL_NAME}）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reasoning",
|
||||
choices=["none", "low", "medium", "high", "xhigh", "max"],
|
||||
default="low",
|
||||
help="推理力度 Reasoning Effort（low/medium/high，默认 low）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--verbosity",
|
||||
choices=["low", "medium", "high"],
|
||||
default=None,
|
||||
help="输出详略程度 Verbosity（low/medium/high，默认跟随模型）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--no-tools",
|
||||
action="store_true",
|
||||
help="禁用原生工具（web_search / code_interpreter）",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=str,
|
||||
default=None,
|
||||
help="将完整结果（含轨迹 / 请求体）保存为 JSON 文件的路径",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="离线组装并打印请求体（含原生工具定义），不调用 API、无需 API Key",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--test",
|
||||
type=str,
|
||||
help="test 模式下运行指定联网手动用例（basic/analysis/complex/code/reasoning/search_analyze/chain）",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# dry-run：离线路径，跳过 API Key 校验
|
||||
if args.dry_run:
|
||||
if not args.request:
|
||||
print("❌ --dry-run 需要配合 --request 使用")
|
||||
sys.exit(1)
|
||||
_run_single(args)
|
||||
return
|
||||
|
||||
# 其余模式需要有效配置
|
||||
if not Config.validate(args.backend):
|
||||
print("❌ 配置错误！")
|
||||
print("请配置所选 backend 对应的 OPENAI_API_KEY / OPENROUTER_API_KEY / DASHSCOPE_API_KEY")
|
||||
print("\n示例 .env：")
|
||||
print("DASHSCOPE_API_KEY=your-dashscope-api-key")
|
||||
sys.exit(1)
|
||||
|
||||
if args.mode == "interactive":
|
||||
cli = InteractiveCLI(args.backend, args.model)
|
||||
cli.run()
|
||||
|
||||
elif args.mode == "single":
|
||||
if not args.request:
|
||||
print("❌ single 模式需要 --request 参数")
|
||||
sys.exit(1)
|
||||
_run_single(args)
|
||||
|
||||
elif args.mode == "test":
|
||||
from tests.manual.agent_cases import TestGPT5Agent, run_single_test
|
||||
|
||||
if args.test:
|
||||
run_single_test(args.test)
|
||||
else:
|
||||
tester = TestGPT5Agent()
|
||||
tester.run_all_tests()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,161 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Quick Start Demo for GPT-5 Native Tools Agent
|
||||
Demonstrates basic usage of web_search and code_interpreter tools
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
from agent import GPT5NativeAgent
|
||||
from config import Config
|
||||
|
||||
def demo_web_search():
|
||||
"""Demonstrate web search capability"""
|
||||
print("\n" + "="*60)
|
||||
print("DEMO: Web Search Tool")
|
||||
print("="*60)
|
||||
|
||||
agent = GPT5NativeAgent(
|
||||
api_key=Config.OPENROUTER_API_KEY,
|
||||
base_url=Config.OPENROUTER_BASE_URL
|
||||
)
|
||||
|
||||
result = agent.process_request(
|
||||
"What are the latest developments in GPT-5 and its capabilities?",
|
||||
use_tools=True,
|
||||
reasoning_effort="low"
|
||||
)
|
||||
|
||||
if result["success"]:
|
||||
print("\n✅ Web Search Result:")
|
||||
print(result["response"][:500] + "...")
|
||||
if result["tool_calls"]:
|
||||
print(f"\n🔧 Tools used: {len(result['tool_calls'])}")
|
||||
else:
|
||||
print(f"❌ Error: {result['error']}")
|
||||
|
||||
def demo_code_interpreter():
|
||||
"""Demonstrate code generation and analysis capability"""
|
||||
print("\n" + "="*60)
|
||||
print("DEMO: Code Generation and Analysis")
|
||||
print("="*60)
|
||||
|
||||
agent = GPT5NativeAgent(
|
||||
api_key=Config.OPENROUTER_API_KEY,
|
||||
base_url=Config.OPENROUTER_BASE_URL
|
||||
)
|
||||
|
||||
result = agent.process_request(
|
||||
"""Create Python code to:
|
||||
1. Generate the first 20 Fibonacci numbers
|
||||
2. Calculate their sum and average
|
||||
3. Find the golden ratio approximation using consecutive pairs
|
||||
4. Explain the mathematical significance""",
|
||||
use_tools=True,
|
||||
reasoning_effort="medium"
|
||||
)
|
||||
|
||||
if result["success"]:
|
||||
print("\n✅ Code and Analysis Result:")
|
||||
print(result["response"][:500] + "...")
|
||||
if result["tool_calls"]:
|
||||
print(f"\n🔧 Tools used: {len(result['tool_calls'])}")
|
||||
else:
|
||||
print(f"❌ Error: {result['error']}")
|
||||
|
||||
def demo_combined_tools():
|
||||
"""Demonstrate using both tools together"""
|
||||
print("\n" + "="*60)
|
||||
print("DEMO: Combined Web Search + Code Analysis")
|
||||
print("="*60)
|
||||
|
||||
agent = GPT5NativeAgent(
|
||||
api_key=Config.OPENROUTER_API_KEY,
|
||||
base_url=Config.OPENROUTER_BASE_URL
|
||||
)
|
||||
|
||||
result = agent.search_and_analyze(
|
||||
topic="Current S&P 500 performance and major tech stocks",
|
||||
analysis_code="""
|
||||
# Analyze market data
|
||||
import random
|
||||
import statistics
|
||||
|
||||
# Simulate stock prices based on search results
|
||||
stocks = {
|
||||
'AAPL': [175 + random.uniform(-5, 5) for _ in range(10)],
|
||||
'GOOGL': [140 + random.uniform(-3, 3) for _ in range(10)],
|
||||
'MSFT': [380 + random.uniform(-8, 8) for _ in range(10)]
|
||||
}
|
||||
|
||||
# Calculate metrics
|
||||
for symbol, prices in stocks.items():
|
||||
avg = statistics.mean(prices)
|
||||
vol = statistics.stdev(prices)
|
||||
trend = "↑" if prices[-1] > prices[0] else "↓"
|
||||
print(f"{symbol}: Avg=${avg:.2f}, Volatility=${vol:.2f}, Trend={trend}")
|
||||
"""
|
||||
)
|
||||
|
||||
if result["success"]:
|
||||
print("\n✅ Combined Analysis Result:")
|
||||
print(result["response"][:500] + "...")
|
||||
if result["tool_calls"]:
|
||||
print(f"\n🔧 Tools used: {len(result['tool_calls'])}")
|
||||
else:
|
||||
print(f"❌ Error: {result['error']}")
|
||||
|
||||
def main():
|
||||
"""Run all demos"""
|
||||
print("\n" + "="*60)
|
||||
print(" GPT-5 Native Tools Agent - Quick Start Demo")
|
||||
print("="*60)
|
||||
|
||||
# Check configuration
|
||||
if not Config.validate():
|
||||
print("\n❌ Configuration Error!")
|
||||
print("Please set up your .env file with OPENROUTER_API_KEY")
|
||||
print("\nSteps:")
|
||||
print("1. Copy env.example to .env")
|
||||
print("2. Add your OpenRouter API key")
|
||||
print("3. Get a key at: https://openrouter.ai/keys")
|
||||
sys.exit(1)
|
||||
|
||||
print("\n✅ Configuration valid")
|
||||
print(f"Using model: {Config.MODEL_NAME}")
|
||||
|
||||
# Ask user which demo to run
|
||||
print("\nSelect demo to run:")
|
||||
print("1. Web Search only")
|
||||
print("2. Code Generation and Analysis")
|
||||
print("3. Combined Tools")
|
||||
print("4. All demos")
|
||||
|
||||
choice = input("\nEnter choice (1-4): ").strip()
|
||||
|
||||
if choice == "1":
|
||||
demo_web_search()
|
||||
elif choice == "2":
|
||||
demo_code_interpreter()
|
||||
elif choice == "3":
|
||||
demo_combined_tools()
|
||||
elif choice == "4":
|
||||
demo_web_search()
|
||||
demo_code_interpreter()
|
||||
demo_combined_tools()
|
||||
else:
|
||||
print("Invalid choice. Running all demos...")
|
||||
demo_web_search()
|
||||
demo_code_interpreter()
|
||||
demo_combined_tools()
|
||||
|
||||
print("\n" + "="*60)
|
||||
print("Demo complete! 🎉")
|
||||
print("\nNext steps:")
|
||||
print("- Run 'python main.py' for interactive mode")
|
||||
print("- Run 'python main.py --mode test' for live manual cases")
|
||||
print("- Check README.md for more examples")
|
||||
print("="*60)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,24 @@
|
||||
# GPT-5 Native Tools Agent Requirements
|
||||
|
||||
# Core dependencies
|
||||
openai>=1.35.0 # OpenAI client library (works with OpenRouter)
|
||||
python-dotenv>=1.0.0 # Environment variable management
|
||||
requests>=2.31.0 # HTTP requests for OpenRouter API calls
|
||||
pydantic>=2.5.0 # Data validation and settings management
|
||||
|
||||
# Development and testing
|
||||
pytest>=7.4.0 # Testing framework
|
||||
pytest-asyncio>=0.21.0 # Async test support
|
||||
black>=23.0.0 # Code formatting
|
||||
flake8>=6.1.0 # Linting
|
||||
mypy>=1.7.0 # Type checking
|
||||
|
||||
# Data processing (used by code_interpreter examples)
|
||||
numpy>=1.24.0 # Numerical computing
|
||||
pandas>=2.0.0 # Data analysis
|
||||
matplotlib>=3.7.0 # Plotting (for code interpreter visualizations)
|
||||
|
||||
# Utilities
|
||||
rich>=13.5.0 # Rich terminal output
|
||||
tqdm>=4.66.0 # Progress bars
|
||||
colorama>=0.4.6 # Cross-platform colored terminal text
|
||||
@@ -0,0 +1,447 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run Experiment 1-3 on a hosted web-search + code-execution Responses API.
|
||||
|
||||
Acceptance policy (author-mandated, 2026-07-31): the experiment's essence is
|
||||
model-directed multi-round web search + hosted code execution, clarification
|
||||
before tools, and a current answer with authoritative sources. The canonical
|
||||
OpenAI GPT-5.6 Sol path remains the reference implementation, but acceptance
|
||||
is NOT gated on the official OpenAI account: any provider whose Responses API
|
||||
genuinely closes the search/code loop server-side (currently Alibaba Model
|
||||
Studio DashScope ``qwen3.7-plus``) is an eligible acceptance backend. The
|
||||
OpenRouter route stays a diagnostic and is never accepted.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable, List, Optional, Tuple
|
||||
|
||||
from agent import GPT5NativeAgent
|
||||
from config import Config
|
||||
|
||||
|
||||
ASEAN_TASK = """Research the current official capitals and reliable coordinates
|
||||
for the ten ASEAN member states. You must use hosted web search and cite the
|
||||
sources. Then you must use the hosted Python tool—not mental arithmetic—to
|
||||
enumerate all 45 capital pairs with the haversine formula and identify the
|
||||
closest pair and distance. Include the coordinates, formula assumptions,
|
||||
calculation result, retrieval date, and clickable citations. Do not say Python
|
||||
was used unless a code_interpreter_call completes."""
|
||||
|
||||
AMBIGUOUS_TASK = "搜索最近一个月的比特币走势，做技术分析。"
|
||||
CLARIFICATION_REPLY = (
|
||||
"使用 CoinGecko 的 BTC/USD 日线收盘价；分析 MA7、MA20、RSI14、MACD(12,26,9)、"
|
||||
"区间收益和最大回撤，如代码环境支持请绘制收盘价走势图。请搜索数据并用托管 "
|
||||
"Python 工具实际计算，再给出含来源的报告和交易建议。"
|
||||
)
|
||||
|
||||
# Backends whose runs may close the experiment, in priority order. The
|
||||
# OpenRouter proxy is diagnostic-only and never appears here.
|
||||
ACCEPTANCE_BACKENDS = ("openai", "dashscope")
|
||||
|
||||
# Independent reference: standard coordinates of the ten ASEAN capitals,
|
||||
# used to verify the model's computed nearest pair without trusting it.
|
||||
ASEAN_CAPITAL_COORDS: Dict[str, Tuple[float, float]] = {
|
||||
"Bandar Seri Begawan": (4.9031, 114.9398),
|
||||
"Phnom Penh": (11.5564, 104.9282),
|
||||
"Jakarta": (-6.2088, 106.8456),
|
||||
"Vientiane": (17.9757, 102.6331),
|
||||
"Kuala Lumpur": (3.1390, 101.6869),
|
||||
"Naypyidaw": (19.7633, 96.0785),
|
||||
"Manila": (14.5995, 120.9842),
|
||||
"Singapore": (1.3521, 103.8198),
|
||||
"Bangkok": (13.7563, 100.5018),
|
||||
"Hanoi": (21.0278, 105.8342),
|
||||
}
|
||||
|
||||
|
||||
def haversine_km(a: Tuple[float, float], b: Tuple[float, float]) -> float:
|
||||
radius = 6371.0088
|
||||
lat1, lon1 = map(math.radians, a)
|
||||
lat2, lon2 = map(math.radians, b)
|
||||
dlat, dlon = lat2 - lat1, lon2 - lon1
|
||||
h = math.sin(dlat / 2) ** 2 + math.cos(lat1) * math.cos(lat2) * math.sin(dlon / 2) ** 2
|
||||
return 2 * radius * math.asin(math.sqrt(h))
|
||||
|
||||
|
||||
def independent_asean_reference() -> Dict[str, Any]:
|
||||
"""Locally computed ground truth for the ASEAN nearest-pair check."""
|
||||
pairs = [
|
||||
(haversine_km(ca, cb), a, b)
|
||||
for (a, ca), (b, cb) in itertools.combinations(ASEAN_CAPITAL_COORDS.items(), 2)
|
||||
]
|
||||
distance, first, second = min(pairs)
|
||||
return {
|
||||
"pair": sorted([first, second]),
|
||||
"distance_km": round(distance, 1),
|
||||
"pair_count": len(pairs),
|
||||
"coordinates": ASEAN_CAPITAL_COORDS,
|
||||
}
|
||||
|
||||
|
||||
def git_value(*args: str) -> str | None:
|
||||
try:
|
||||
return subprocess.check_output(
|
||||
["git", *args], text=True, stderr=subprocess.DEVNULL
|
||||
).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def output_types(result: Dict[str, Any]) -> List[str]:
|
||||
return [item.get("type") for item in result.get("output_items") or []]
|
||||
|
||||
|
||||
def completed_calls(result: Dict[str, Any], kind: str) -> List[Dict[str, Any]]:
|
||||
return [
|
||||
item
|
||||
for item in result.get("output_items") or []
|
||||
if item.get("type") == kind and item.get("status") == "completed"
|
||||
]
|
||||
|
||||
|
||||
def url_citations(result: Dict[str, Any]) -> List[Dict[str, Any]]:
|
||||
return [
|
||||
item for item in result.get("citations") or [] if item.get("type") == "url_citation"
|
||||
]
|
||||
|
||||
|
||||
def model_identity_exact(result: Dict[str, Any]) -> bool:
|
||||
"""The returned model must be exactly the requested model."""
|
||||
requested = (result.get("requested_model") or result.get("model") or "").removeprefix(
|
||||
"openai/"
|
||||
)
|
||||
returned = (result.get("model") or "").removeprefix("openai/")
|
||||
return bool(requested) and requested == returned
|
||||
|
||||
|
||||
def validate_asean(
|
||||
result: Dict[str, Any], reference: Optional[Dict[str, Any]] = None
|
||||
) -> Dict[str, Any]:
|
||||
answer = result.get("response") or ""
|
||||
reference = reference or independent_asean_reference()
|
||||
pair_city, other_city = reference["pair"]
|
||||
checks = {
|
||||
"request_succeeded": result.get("success") is True,
|
||||
"model_identity_exact": model_identity_exact(result),
|
||||
"web_search_completed": bool(completed_calls(result, "web_search_call")),
|
||||
"code_interpreter_completed": bool(
|
||||
completed_calls(result, "code_interpreter_call")
|
||||
),
|
||||
"url_citations_present": len(url_citations(result)) >= 2,
|
||||
"closest_pair_matches_independent_reference": (
|
||||
pair_city.lower() in answer.lower() and other_city.lower() in answer.lower()
|
||||
),
|
||||
"distance_reported": any(unit in answer.lower() for unit in ("km", "公里", "千米")),
|
||||
}
|
||||
return {
|
||||
"checks": checks,
|
||||
"passed": all(checks.values()),
|
||||
"output_types": output_types(result),
|
||||
"independent_reference": {
|
||||
"pair": reference["pair"],
|
||||
"distance_km": reference["distance_km"],
|
||||
"pair_count": reference["pair_count"],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def is_clarifying_question(result: Dict[str, Any]) -> bool:
|
||||
text = result.get("response") or ""
|
||||
return result.get("success") is True and not result.get("tool_calls") and (
|
||||
"?" in text or "？" in text
|
||||
)
|
||||
|
||||
|
||||
def validate_clarification(
|
||||
first: Dict[str, Any], second: Dict[str, Any] | None
|
||||
) -> Dict[str, Any]:
|
||||
followup_text = (second or {}).get("response") or ""
|
||||
lowered = followup_text.lower()
|
||||
checks = {
|
||||
"first_turn_clarified_before_tools": is_clarifying_question(first),
|
||||
"continuation_used_previous_response_id": bool(
|
||||
second and second.get("request", {}).get("previous_response_id") == first.get("response_id")
|
||||
),
|
||||
"followup_succeeded": bool(second and second.get("success")),
|
||||
"followup_web_search_completed": bool(
|
||||
second and completed_calls(second, "web_search_call")
|
||||
),
|
||||
"followup_code_interpreter_completed": bool(
|
||||
second and completed_calls(second, "code_interpreter_call")
|
||||
),
|
||||
"followup_citations_present": bool(second and url_citations(second)),
|
||||
"followup_reports_ma_rsi_macd": all(
|
||||
token in lowered for token in ("ma", "rsi", "macd")
|
||||
),
|
||||
}
|
||||
return {"checks": checks, "passed": all(checks.values())}
|
||||
|
||||
|
||||
def total_usage(results: Iterable[Dict[str, Any] | None]) -> Dict[str, Any]:
|
||||
totals: Dict[str, float] = {
|
||||
"input_tokens": 0,
|
||||
"output_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
"reported_cost_usd": 0.0,
|
||||
}
|
||||
cost_reported = False
|
||||
for result in results:
|
||||
usage = (result or {}).get("usage") or {}
|
||||
for name in ("input_tokens", "output_tokens", "total_tokens"):
|
||||
totals[name] += int(usage.get(name) or 0)
|
||||
if usage.get("cost") is not None:
|
||||
cost_reported = True
|
||||
totals["reported_cost_usd"] += float(usage["cost"])
|
||||
totals["reported_cost_available"] = cost_reported
|
||||
if not cost_reported:
|
||||
totals["reported_cost_usd"] = None
|
||||
return totals
|
||||
|
||||
|
||||
def run_backend(backend: str, reasoning: str) -> Dict[str, Any]:
|
||||
key, base_url, model = Config.resolve(backend)
|
||||
if not key:
|
||||
return {"backend": backend, "started": False, "error": "credential_missing"}
|
||||
|
||||
asean_agent = GPT5NativeAgent(key, base_url=base_url, model=model)
|
||||
asean = asean_agent.process_request(
|
||||
ASEAN_TASK,
|
||||
reasoning_effort=reasoning,
|
||||
verbosity="high",
|
||||
max_tokens=16000,
|
||||
)
|
||||
|
||||
clarification_agent = GPT5NativeAgent(key, base_url=base_url, model=model)
|
||||
first = clarification_agent.process_request(
|
||||
AMBIGUOUS_TASK,
|
||||
reasoning_effort="medium",
|
||||
verbosity="medium",
|
||||
max_tokens=4000,
|
||||
)
|
||||
second = None
|
||||
if is_clarifying_question(first):
|
||||
second = clarification_agent.process_request(
|
||||
CLARIFICATION_REPLY,
|
||||
reasoning_effort=reasoning,
|
||||
verbosity="high",
|
||||
max_tokens=16000,
|
||||
)
|
||||
|
||||
return {
|
||||
"backend": backend,
|
||||
"started": True,
|
||||
"base_url": base_url,
|
||||
"requested_model": model,
|
||||
"asean": asean,
|
||||
"asean_validation": validate_asean(asean),
|
||||
"clarification": {
|
||||
"ambiguous_task": AMBIGUOUS_TASK,
|
||||
"first": first,
|
||||
"user_reply": CLARIFICATION_REPLY if second else None,
|
||||
"second": second,
|
||||
"validation": validate_clarification(first, second),
|
||||
},
|
||||
"api_turns": asean_agent.api_turns + clarification_agent.api_turns,
|
||||
"usage": total_usage((asean, first, second)),
|
||||
}
|
||||
|
||||
|
||||
def acceptance(runs: List[Dict[str, Any]]) -> Dict[str, Any]:
|
||||
"""Multi-provider policy: any eligible backend may close the experiment."""
|
||||
per_backend = {}
|
||||
for run in runs:
|
||||
backend = run.get("backend")
|
||||
if not run.get("started"):
|
||||
per_backend[backend] = {"started": False, "error": run.get("error")}
|
||||
continue
|
||||
per_backend[backend] = {
|
||||
"started": True,
|
||||
"requested_model": run.get("requested_model"),
|
||||
"asean_passed": run.get("asean_validation", {}).get("passed") is True,
|
||||
"clarification_passed": run.get("clarification", {})
|
||||
.get("validation", {})
|
||||
.get("passed")
|
||||
is True,
|
||||
}
|
||||
accepting = next(
|
||||
(
|
||||
backend
|
||||
for backend in ACCEPTANCE_BACKENDS
|
||||
if per_backend.get(backend, {}).get("asean_passed")
|
||||
and per_backend.get(backend, {}).get("clarification_passed")
|
||||
),
|
||||
None,
|
||||
)
|
||||
eligible_attempted = [
|
||||
backend for backend in ACCEPTANCE_BACKENDS if backend in per_backend
|
||||
]
|
||||
return {
|
||||
"policy": (
|
||||
"multi-provider: acceptance is not gated on the official OpenAI "
|
||||
"account; any provider whose Responses API closes the hosted "
|
||||
"search + code-execution loop server-side is eligible"
|
||||
),
|
||||
"eligible_acceptance_backends": list(ACCEPTANCE_BACKENDS),
|
||||
"eligible_backends_attempted": eligible_attempted,
|
||||
"acceptance_backend": accepting,
|
||||
"per_backend": per_backend,
|
||||
"openrouter_is_diagnostic_not_acceptance": "openrouter" in per_backend,
|
||||
"passed": accepting is not None,
|
||||
"reference_docs": [
|
||||
"https://developers.openai.com/api/docs/guides/tools-web-search",
|
||||
"https://developers.openai.com/api/docs/guides/tools-code-interpreter",
|
||||
"https://help.aliyun.com/zh/model-studio/qwen-code-interpreter",
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def write_json(path: Path, value: Dict[str, Any]) -> str:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
payload = json.dumps(value, ensure_ascii=False, indent=2) + "\n"
|
||||
path.write_text(payload, encoding="utf-8")
|
||||
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def assert_credential_free(payloads: Iterable[str]) -> None:
|
||||
"""Refuse to write evidence that embeds any configured API key."""
|
||||
secrets = [
|
||||
value
|
||||
for value in (
|
||||
Config.OPENAI_API_KEY,
|
||||
Config.OPENROUTER_API_KEY,
|
||||
Config.DASHSCOPE_API_KEY,
|
||||
os.getenv("MOONSHOT_API_KEY", ""),
|
||||
os.getenv("KIMI_API_KEY", ""),
|
||||
os.getenv("ARK_API_KEY", ""),
|
||||
os.getenv("SILICONFLOW_API_KEY", ""),
|
||||
os.getenv("GEMINI_API_KEY", ""),
|
||||
)
|
||||
if value
|
||||
]
|
||||
for payload in payloads:
|
||||
for secret in secrets:
|
||||
if secret in payload:
|
||||
raise SystemExit(
|
||||
"Refusing to write evidence: an API key value appears in the payload"
|
||||
)
|
||||
if "authorization" in payload.lower() and "bearer" in payload.lower():
|
||||
raise SystemExit(
|
||||
"Refusing to write evidence: an Authorization header appears in the payload"
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument(
|
||||
"--backends",
|
||||
nargs="+",
|
||||
choices=["openai", "openrouter", "dashscope"],
|
||||
default=["openai", "dashscope"],
|
||||
)
|
||||
parser.add_argument(
|
||||
"--reasoning", choices=["low", "medium", "high", "xhigh", "max"], default="high"
|
||||
)
|
||||
parser.add_argument("--output-dir", type=Path)
|
||||
args = parser.parse_args()
|
||||
|
||||
runs = [run_backend(backend, args.reasoning) for backend in args.backends]
|
||||
stamp = datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
output_dir = args.output_dir or Path("validation") / "runs" / f"real_{stamp}"
|
||||
evidence = {
|
||||
"schema_version": "1.1",
|
||||
"experiment_id": "1-3",
|
||||
"evidence_mode": "real_api",
|
||||
"created_at": datetime.now(timezone.utc).isoformat(),
|
||||
"canonical_source": "book/chapter1.md#实验-1-3-gpt-5-6-原生-deep-research-能力",
|
||||
"host": {
|
||||
"platform": platform.platform(),
|
||||
"python": sys.version,
|
||||
"machine": platform.machine(),
|
||||
},
|
||||
"repository": {
|
||||
"commit": git_value("rev-parse", "HEAD"),
|
||||
"branch": git_value("branch", "--show-current"),
|
||||
"worktree_dirty": bool(git_value("status", "--porcelain")),
|
||||
},
|
||||
"credentials_recorded": False,
|
||||
"independent_asean_reference": independent_asean_reference(),
|
||||
"runs": runs,
|
||||
}
|
||||
evidence["acceptance"] = acceptance(runs)
|
||||
|
||||
receipts = {
|
||||
"schema_version": "1.0",
|
||||
"experiment_id": "1-3",
|
||||
"created_at": evidence["created_at"],
|
||||
"note": "Raw credential-free provider turns; no API keys or Authorization headers.",
|
||||
"turns": [
|
||||
{"backend": run.get("backend"), "api_turns": run.get("api_turns") or []}
|
||||
for run in runs
|
||||
],
|
||||
}
|
||||
|
||||
evidence_json = json.dumps(evidence, ensure_ascii=False, indent=2)
|
||||
receipts_json = json.dumps(receipts, ensure_ascii=False, indent=2)
|
||||
assert_credential_free((evidence_json, receipts_json))
|
||||
|
||||
evidence_path = output_dir / "evidence.json"
|
||||
evidence_digest = write_json(evidence_path, evidence)
|
||||
receipts_digest = write_json(output_dir / "receipts.json", receipts)
|
||||
(output_dir / "evidence.sha256").write_text(
|
||||
f"{evidence_digest} evidence.json\n", encoding="utf-8"
|
||||
)
|
||||
(output_dir / "receipts.sha256").write_text(
|
||||
f"{receipts_digest} receipts.json\n", encoding="utf-8"
|
||||
)
|
||||
manifest = {
|
||||
"schema_version": "1.0",
|
||||
"experiment_id": "1-3",
|
||||
"run_id": output_dir.name,
|
||||
"created_at": evidence["created_at"],
|
||||
"artifacts": {
|
||||
"evidence.json": {"sha256": evidence_digest},
|
||||
"receipts.json": {"sha256": receipts_digest},
|
||||
},
|
||||
"inputs": {
|
||||
"canonical_source": evidence["canonical_source"],
|
||||
"backends": args.backends,
|
||||
"reasoning": args.reasoning,
|
||||
},
|
||||
"repository": evidence["repository"],
|
||||
"acceptance_passed": evidence["acceptance"]["passed"],
|
||||
"acceptance_backend": evidence["acceptance"]["acceptance_backend"],
|
||||
}
|
||||
manifest_digest = write_json(output_dir / "manifest.json", manifest)
|
||||
|
||||
Path("validation").mkdir(exist_ok=True)
|
||||
shutil.copyfile(evidence_path, Path("validation/latest.json"))
|
||||
latest = json.loads(Path("validation/latest.json").read_text(encoding="utf-8"))
|
||||
latest["artifact_hashes"] = {
|
||||
"evidence.json": evidence_digest,
|
||||
"receipts.json": receipts_digest,
|
||||
"manifest.json": manifest_digest,
|
||||
"run_dir": str(output_dir),
|
||||
}
|
||||
Path("validation/latest.json").write_text(
|
||||
json.dumps(latest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
|
||||
)
|
||||
|
||||
print(json.dumps(evidence["acceptance"], ensure_ascii=False, indent=2))
|
||||
print(f"Evidence: {evidence_path}")
|
||||
return 0 if evidence["acceptance"]["passed"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,89 @@
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import example_request
|
||||
|
||||
|
||||
ROOT = Path(__file__).parent
|
||||
|
||||
|
||||
def _import_config_with(value: str | None) -> subprocess.CompletedProcess[str]:
|
||||
env = os.environ.copy()
|
||||
if value is None:
|
||||
env.pop("DEFAULT_MAX_TOKENS", None)
|
||||
else:
|
||||
env["DEFAULT_MAX_TOKENS"] = value
|
||||
return subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-c",
|
||||
"from config import Config; print(repr(Config.DEFAULT_MAX_TOKENS))",
|
||||
],
|
||||
cwd=ROOT,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
text=True,
|
||||
)
|
||||
|
||||
|
||||
def test_default_max_tokens_import_accepts_only_values_int_can_parse():
|
||||
for value in (None, "", " ", "4000.0", "abc", "²", "-1", "+1"):
|
||||
result = _import_config_with(value)
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert result.stdout.strip() == "None"
|
||||
|
||||
result = _import_config_with(" 4000 ")
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert result.stdout.strip() == "4000"
|
||||
|
||||
|
||||
def test_chat_completions_usage_uses_chat_token_and_detail_keys(monkeypatch, capsys):
|
||||
payload = {
|
||||
"usage": {
|
||||
"prompt_tokens": 123,
|
||||
"completion_tokens": 45,
|
||||
"total_tokens": 168,
|
||||
"prompt_tokens_details": {"cached_tokens": 7},
|
||||
"completion_tokens_details": {"reasoning_tokens": 9},
|
||||
}
|
||||
}
|
||||
|
||||
class FakeResponse:
|
||||
status_code = 200
|
||||
|
||||
def json(self):
|
||||
return payload
|
||||
|
||||
monkeypatch.setattr(
|
||||
example_request.requests,
|
||||
"post",
|
||||
lambda *args, **kwargs: FakeResponse(),
|
||||
)
|
||||
|
||||
result = example_request.make_gpt5_openrouter_request("key", "system", "user")
|
||||
output = capsys.readouterr().out
|
||||
|
||||
assert result == payload
|
||||
assert "Input: 123 tokens (cached: 7)" in output
|
||||
assert "Output: 45 tokens (reasoning: 9)" in output
|
||||
assert "Total: 168" in output
|
||||
|
||||
payload = {
|
||||
"usage": {
|
||||
"input_tokens": 210,
|
||||
"output_tokens": 34,
|
||||
"total_tokens": 244,
|
||||
"input_tokens_details": {"cached_tokens": 11},
|
||||
"output_tokens_details": {"reasoning_tokens": 13},
|
||||
}
|
||||
}
|
||||
result = example_request.make_gpt5_openrouter_request("key", "system", "user")
|
||||
output = capsys.readouterr().out
|
||||
|
||||
assert result == payload
|
||||
assert "Input: 210 tokens (cached: 11)" in output
|
||||
assert "Output: 34 tokens (reasoning: 13)" in output
|
||||
assert "Total: 244" in output
|
||||
@@ -0,0 +1,192 @@
|
||||
from agent import GPT5NativeAgent
|
||||
from config import Config
|
||||
from run_experiment_1_3 import (
|
||||
acceptance,
|
||||
independent_asean_reference,
|
||||
validate_asean,
|
||||
validate_clarification,
|
||||
)
|
||||
|
||||
DASHSCOPE_URL = "https://dashscope-intl.aliyuncs.com/compatible-mode/v1"
|
||||
|
||||
|
||||
def test_request_uses_official_responses_tool_shapes():
|
||||
agent = GPT5NativeAgent("key")
|
||||
request = agent._build_responses_request(
|
||||
"task", reasoning_effort="max", verbosity="high"
|
||||
)
|
||||
assert request["reasoning"] == {"effort": "max"}
|
||||
assert request["text"] == {"verbosity": "high"}
|
||||
assert request["tools"] == [
|
||||
{"type": "web_search", "search_context_size": "medium"},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {"type": "auto", "memory_limit": "4g"},
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def test_dashscope_request_uses_hosted_tool_shapes_and_streaming():
|
||||
agent = GPT5NativeAgent("key", base_url=DASHSCOPE_URL, model="qwen3.7-plus")
|
||||
assert agent.provider == "dashscope"
|
||||
request = agent._build_responses_request(
|
||||
"task", reasoning_effort="high", verbosity="high"
|
||||
)
|
||||
# DashScope runs thinking natively: no reasoning.effort or text.verbosity,
|
||||
# and streaming is mandatory because its gateway drops idle connections.
|
||||
assert "reasoning" not in request
|
||||
assert "text" not in request
|
||||
assert request["stream"] is True
|
||||
assert request["tools"] == [
|
||||
{"type": "web_search"},
|
||||
{"type": "code_interpreter"},
|
||||
]
|
||||
|
||||
|
||||
def test_config_resolves_dashscope_backend():
|
||||
key, base_url, model = Config.resolve("dashscope")
|
||||
assert base_url == DASHSCOPE_URL
|
||||
assert model == Config.DASHSCOPE_MODEL
|
||||
assert isinstance(key, str)
|
||||
|
||||
|
||||
def test_dashscope_citations_from_web_search_sources():
|
||||
agent = GPT5NativeAgent("key", base_url=DASHSCOPE_URL, model="qwen3.7-plus")
|
||||
response = {
|
||||
"output": [
|
||||
{
|
||||
"type": "web_search_call",
|
||||
"status": "completed",
|
||||
"action": {
|
||||
"query": "ASEAN capitals",
|
||||
"sources": [
|
||||
{"type": "url", "url": "https://asean.test/one"},
|
||||
{"type": "url", "url": "https://asean.test/two"},
|
||||
],
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
citations = agent._citations(response)
|
||||
assert citations == [
|
||||
{"type": "url_citation", "url": "https://asean.test/one"},
|
||||
{"type": "url_citation", "url": "https://asean.test/two"},
|
||||
]
|
||||
|
||||
def test_dashscope_citations_from_web_search_string_url_sources():
|
||||
agent = GPT5NativeAgent("key", base_url=DASHSCOPE_URL, model="qwen3.7-plus")
|
||||
response = {
|
||||
"output": [
|
||||
{
|
||||
"type": "web_search_call",
|
||||
"status": "completed",
|
||||
"action": {
|
||||
"query": "ASEAN capitals",
|
||||
"sources": [
|
||||
"https://asean.test/one",
|
||||
"https://asean.test/two",
|
||||
],
|
||||
},
|
||||
}
|
||||
]
|
||||
}
|
||||
citations = agent._citations(response)
|
||||
assert citations == [
|
||||
{"type": "url_citation", "url": "https://asean.test/one"},
|
||||
{"type": "url_citation", "url": "https://asean.test/two"},
|
||||
]
|
||||
|
||||
|
||||
def test_independent_asean_reference_is_kuala_lumpur_singapore():
|
||||
reference = independent_asean_reference()
|
||||
assert reference["pair"] == ["Kuala Lumpur", "Singapore"]
|
||||
assert reference["pair_count"] == 45
|
||||
assert 250 < reference["distance_km"] < 400
|
||||
|
||||
|
||||
def test_asean_acceptance_requires_both_completed_hosted_tools():
|
||||
result = {
|
||||
"success": True,
|
||||
"requested_model": "gpt-5.6-sol",
|
||||
"model": "gpt-5.6-sol",
|
||||
"response": "Singapore and Kuala Lumpur are 316 km apart.",
|
||||
"output_items": [
|
||||
{"type": "web_search_call", "status": "completed"},
|
||||
{"type": "code_interpreter_call", "status": "completed"},
|
||||
],
|
||||
"citations": [
|
||||
{"type": "url_citation", "url": "https://one.test"},
|
||||
{"type": "url_citation", "url": "https://two.test"},
|
||||
],
|
||||
}
|
||||
assert validate_asean(result)["passed"] is True
|
||||
result["output_items"] = result["output_items"][:1]
|
||||
assert validate_asean(result)["passed"] is False
|
||||
|
||||
|
||||
def test_asean_acceptance_rejects_model_substitution():
|
||||
result = {
|
||||
"success": True,
|
||||
"requested_model": "qwen3.7-plus",
|
||||
"model": "qwen3.7-flash",
|
||||
"response": "Singapore and Kuala Lumpur are 316 km apart.",
|
||||
"output_items": [
|
||||
{"type": "web_search_call", "status": "completed"},
|
||||
{"type": "code_interpreter_call", "status": "completed"},
|
||||
],
|
||||
"citations": [
|
||||
{"type": "url_citation", "url": "https://one.test"},
|
||||
{"type": "url_citation", "url": "https://two.test"},
|
||||
],
|
||||
}
|
||||
assert validate_asean(result)["checks"]["model_identity_exact"] is False
|
||||
assert validate_asean(result)["passed"] is False
|
||||
|
||||
|
||||
def test_clarification_requires_no_tools_then_linked_tool_run():
|
||||
first = {
|
||||
"success": True,
|
||||
"response": "Which source and indicators do you prefer?",
|
||||
"tool_calls": [],
|
||||
"response_id": "resp_1",
|
||||
}
|
||||
second = {
|
||||
"success": True,
|
||||
"request": {"previous_response_id": "resp_1"},
|
||||
"response": "MA7, MA20, RSI14 and MACD(12,26,9) were computed.",
|
||||
"output_items": [
|
||||
{"type": "web_search_call", "status": "completed"},
|
||||
{"type": "code_interpreter_call", "status": "completed"},
|
||||
],
|
||||
"citations": [{"type": "url_citation"}],
|
||||
}
|
||||
assert validate_clarification(first, second)["passed"] is True
|
||||
second_without_indicators = dict(second, response="Here is the report.")
|
||||
assert validate_clarification(first, second_without_indicators)["passed"] is False
|
||||
|
||||
|
||||
def test_acceptance_is_multi_provider_not_openai_gated():
|
||||
passing_run = {
|
||||
"backend": "dashscope",
|
||||
"started": True,
|
||||
"requested_model": "qwen3.7-plus",
|
||||
"asean_validation": {"passed": True},
|
||||
"clarification": {"validation": {"passed": True}},
|
||||
}
|
||||
blocked_openai = {"backend": "openai", "started": True,
|
||||
"requested_model": "gpt-5.6-sol",
|
||||
"asean_validation": {"passed": False},
|
||||
"clarification": {"validation": {"passed": False}}}
|
||||
result = acceptance([blocked_openai, passing_run])
|
||||
assert result["passed"] is True
|
||||
assert result["acceptance_backend"] == "dashscope"
|
||||
# OpenAI keeps priority when both pass.
|
||||
blocked_openai["asean_validation"] = {"passed": True}
|
||||
blocked_openai["clarification"] = {"validation": {"passed": True}}
|
||||
result = acceptance([blocked_openai, passing_run])
|
||||
assert result["acceptance_backend"] == "openai"
|
||||
# OpenRouter alone can never close the experiment.
|
||||
openrouter_run = dict(passing_run, backend="openrouter")
|
||||
result = acceptance([openrouter_run])
|
||||
assert result["passed"] is False
|
||||
assert result["openrouter_is_diagnostic_not_acceptance"] is True
|
||||
@@ -0,0 +1,363 @@
|
||||
"""
|
||||
Live manual cases for GPT-5 Native Tools Agent.
|
||||
|
||||
These cases demonstrate web_search with the OpenRouter format and require
|
||||
OPENROUTER_API_KEY.
|
||||
"""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
from typing import Dict, Any, List
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
PROJECT_ROOT = Path(__file__).resolve().parents[2]
|
||||
if str(PROJECT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(PROJECT_ROOT))
|
||||
|
||||
from agent import GPT5NativeAgent, GPT5AgentChain
|
||||
from config import Config
|
||||
|
||||
# Set up logging
|
||||
logging.basicConfig(
|
||||
level=getattr(logging, Config.LOG_LEVEL),
|
||||
format=Config.LOG_FORMAT
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TestGPT5Agent:
|
||||
"""Live manual case suite for GPT-5 Native Tools Agent"""
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize manual case suite"""
|
||||
if not Config.validate():
|
||||
raise ValueError("Invalid configuration. Please check your .env file")
|
||||
|
||||
self.agent = GPT5NativeAgent(
|
||||
api_key=Config.OPENROUTER_API_KEY,
|
||||
base_url=Config.OPENROUTER_BASE_URL,
|
||||
model=Config.MODEL_NAME
|
||||
)
|
||||
self.results = []
|
||||
|
||||
def test_web_search_basic(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Test Case 1: Basic web search
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 1: Basic Web Search")
|
||||
print("="*60)
|
||||
|
||||
request = """Search for the latest information about GPT-5 capabilities and features."""
|
||||
|
||||
result = self.agent.process_request(
|
||||
request,
|
||||
use_tools=True,
|
||||
reasoning_effort="low"
|
||||
)
|
||||
|
||||
self._print_result(result)
|
||||
return result
|
||||
|
||||
def test_web_search_with_analysis(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Test Case 2: Web search with analysis request
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 2: Web Search with Analysis")
|
||||
print("="*60)
|
||||
|
||||
request = """Search for current cryptocurrency market trends and Bitcoin price.
|
||||
Then analyze the data to identify patterns and provide insights."""
|
||||
|
||||
result = self.agent.process_request(
|
||||
request,
|
||||
use_tools=True,
|
||||
reasoning_effort="medium"
|
||||
)
|
||||
|
||||
self._print_result(result)
|
||||
return result
|
||||
|
||||
def test_complex_research(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Test Case 3: Complex research task
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 3: Complex Research Task")
|
||||
print("="*60)
|
||||
|
||||
request = """Research the current state of renewable energy adoption globally.
|
||||
Find statistics on solar, wind, and hydroelectric capacity.
|
||||
Analyze growth trends and project future adoption rates.
|
||||
Provide a comprehensive summary with data-driven insights."""
|
||||
|
||||
result = self.agent.process_request(
|
||||
request,
|
||||
use_tools=True,
|
||||
reasoning_effort="high"
|
||||
)
|
||||
|
||||
self._print_result(result)
|
||||
return result
|
||||
|
||||
def test_search_and_code(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Test Case 4: Search and code generation
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 4: Search and Code Generation")
|
||||
print("="*60)
|
||||
|
||||
request = """Search for the latest Python web frameworks in 2025.
|
||||
Then create a simple comparison table and sample code for the top 3 frameworks."""
|
||||
|
||||
result = self.agent.process_request(
|
||||
request,
|
||||
use_tools=True,
|
||||
reasoning_effort="medium"
|
||||
)
|
||||
|
||||
self._print_result(result)
|
||||
return result
|
||||
|
||||
def test_reasoning_efforts(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Test Case 5: Compare different reasoning efforts
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 5: Reasoning Effort Comparison")
|
||||
print("="*60)
|
||||
|
||||
request = "What are the implications of quantum computing on current encryption methods?"
|
||||
|
||||
results = []
|
||||
for effort in ["low", "medium", "high"]:
|
||||
print(f"\n--- Testing with {effort} reasoning effort ---")
|
||||
result = self.agent.process_request(
|
||||
request,
|
||||
use_tools=True,
|
||||
reasoning_effort=effort
|
||||
)
|
||||
self._print_result(result)
|
||||
results.append({
|
||||
"effort": effort,
|
||||
"result": result
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
def test_search_and_analyze_method(self) -> Dict[str, Any]:
|
||||
"""
|
||||
Test Case 6: Using the search_and_analyze convenience method
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 6: Search and Analyze Method")
|
||||
print("="*60)
|
||||
|
||||
analysis_code = """
|
||||
# Analyze stock market data
|
||||
import statistics
|
||||
|
||||
# Sample data processing
|
||||
prices = [100, 102, 98, 105, 103, 107, 104]
|
||||
returns = [(prices[i] - prices[i-1])/prices[i-1] * 100 for i in range(1, len(prices))]
|
||||
|
||||
avg_return = statistics.mean(returns)
|
||||
volatility = statistics.stdev(returns)
|
||||
|
||||
print(f"Average Return: {avg_return:.2f}%")
|
||||
print(f"Volatility: {volatility:.2f}%")
|
||||
"""
|
||||
|
||||
result = self.agent.search_and_analyze(
|
||||
topic="Current S&P 500 performance and market outlook for 2025",
|
||||
analysis_code=analysis_code
|
||||
)
|
||||
|
||||
self._print_result(result)
|
||||
return result
|
||||
|
||||
def test_agent_chain(self) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Test Case 7: Chain multiple requests
|
||||
"""
|
||||
print("\n" + "="*60)
|
||||
print("TEST 7: Agent Chain")
|
||||
print("="*60)
|
||||
|
||||
chain = GPT5AgentChain(self.agent)
|
||||
|
||||
# Step 1: Research
|
||||
chain.add_step(
|
||||
"Search for information about the latest AI developments in 2025",
|
||||
use_tools=True,
|
||||
reasoning_effort="low"
|
||||
)
|
||||
|
||||
# Step 2: Deep dive
|
||||
chain.add_step(
|
||||
"Based on the previous findings, search for more details about the most promising AI breakthrough",
|
||||
use_tools=True,
|
||||
reasoning_effort="medium"
|
||||
)
|
||||
|
||||
# Step 3: Analysis
|
||||
chain.add_step(
|
||||
"Analyze the impact of these AI developments on various industries",
|
||||
use_tools=True,
|
||||
reasoning_effort="high"
|
||||
)
|
||||
|
||||
results = chain.execute()
|
||||
|
||||
for i, step_result in enumerate(results, 1):
|
||||
print(f"\n--- Chain Step {i} ---")
|
||||
self._print_result(step_result["result"])
|
||||
|
||||
return results
|
||||
|
||||
def _print_result(self, result: Dict[str, Any]):
|
||||
"""
|
||||
Pretty print test result
|
||||
|
||||
Args:
|
||||
result: Test result dictionary
|
||||
"""
|
||||
if result["success"]:
|
||||
print(f"\n✅ Test Passed")
|
||||
print(f"\nResponse Preview:")
|
||||
print("-"*60)
|
||||
response = result["response"]
|
||||
if len(response) > 500:
|
||||
print(response[:500] + "...")
|
||||
else:
|
||||
print(response)
|
||||
print("-"*60)
|
||||
|
||||
if result.get("usage"):
|
||||
usage = result["usage"]
|
||||
print(f"\n📊 Token Usage:")
|
||||
print(f" - Input: {usage.get('input_tokens', 'N/A')}")
|
||||
print(f" - Output: {usage.get('output_tokens', 'N/A')}")
|
||||
print(f" - Total: {usage.get('total_tokens', 'N/A')}")
|
||||
if usage.get("input_tokens_details"):
|
||||
print(f" - Cached: {usage['input_tokens_details'].get('cached_tokens', 0)}")
|
||||
if usage.get("output_tokens_details"):
|
||||
print(f" - Reasoning: {usage['output_tokens_details'].get('reasoning_tokens', 0)}")
|
||||
else:
|
||||
print(f"\n❌ Test Failed")
|
||||
print(f"Error: {result.get('error', 'Unknown error')}")
|
||||
|
||||
def run_all_tests(self):
|
||||
"""Run all live manual cases"""
|
||||
print("\n" + "="*60)
|
||||
print("RUNNING GPT-5 NATIVE TOOLS MANUAL CASES")
|
||||
print(f"Timestamp: {datetime.now().isoformat()}")
|
||||
print(f"Model: {Config.MODEL_NAME}")
|
||||
print("="*60)
|
||||
|
||||
case_methods = [
|
||||
("Basic Web Search", self.test_web_search_basic),
|
||||
("Web Search with Analysis", self.test_web_search_with_analysis),
|
||||
("Complex Research", self.test_complex_research),
|
||||
("Search and Code", self.test_search_and_code),
|
||||
("Reasoning Efforts", self.test_reasoning_efforts),
|
||||
("Search and Analyze Method", self.test_search_and_analyze_method),
|
||||
("Agent Chain", self.test_agent_chain)
|
||||
]
|
||||
|
||||
results_summary = []
|
||||
|
||||
for case_name, case_method in case_methods:
|
||||
try:
|
||||
print(f"\n🧪 Running: {case_name}")
|
||||
result = case_method()
|
||||
|
||||
# Handle different result types
|
||||
if isinstance(result, list):
|
||||
# For tests that return multiple results
|
||||
if all(isinstance(r, dict) and "result" in r for r in result):
|
||||
success = all(r["result"]["success"] for r in result)
|
||||
else:
|
||||
success = all(r.get("success", False) for r in result if isinstance(r, dict))
|
||||
else:
|
||||
success = result.get("success", False)
|
||||
|
||||
results_summary.append({
|
||||
"case": case_name,
|
||||
"success": success,
|
||||
"result": result
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Manual case {case_name} failed with exception: {str(e)}")
|
||||
results_summary.append({
|
||||
"case": case_name,
|
||||
"success": False,
|
||||
"error": str(e)
|
||||
})
|
||||
|
||||
# Print summary
|
||||
print("\n" + "="*60)
|
||||
print("MANUAL CASE SUMMARY")
|
||||
print("="*60)
|
||||
|
||||
passed = sum(1 for r in results_summary if r["success"])
|
||||
total = len(results_summary)
|
||||
|
||||
for result in results_summary:
|
||||
status = "✅ PASS" if result["success"] else "❌ FAIL"
|
||||
print(f"{result['case']}: {status}")
|
||||
|
||||
print(f"\nTotal: {passed}/{total} manual cases passed")
|
||||
print("="*60)
|
||||
|
||||
return results_summary
|
||||
|
||||
|
||||
def run_single_test(test_name: str = "basic"):
|
||||
"""
|
||||
Run a single live manual case
|
||||
|
||||
Args:
|
||||
test_name: Name of manual case to run
|
||||
"""
|
||||
tester = TestGPT5Agent()
|
||||
|
||||
test_map = {
|
||||
"basic": tester.test_web_search_basic,
|
||||
"analysis": tester.test_web_search_with_analysis,
|
||||
"complex": tester.test_complex_research,
|
||||
"code": tester.test_search_and_code,
|
||||
"reasoning": tester.test_reasoning_efforts,
|
||||
"search_analyze": tester.test_search_and_analyze_method,
|
||||
"chain": tester.test_agent_chain
|
||||
}
|
||||
|
||||
if test_name in test_map:
|
||||
test_map[test_name]()
|
||||
else:
|
||||
print(f"Unknown test: {test_name}")
|
||||
print(f"Available tests: {', '.join(test_map.keys())}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Check configuration first
|
||||
Config.display()
|
||||
|
||||
if not Config.validate():
|
||||
print("\n❌ Configuration validation failed!")
|
||||
print("Please set up your .env file with OPENROUTER_API_KEY")
|
||||
sys.exit(1)
|
||||
|
||||
# Run manual cases
|
||||
if len(sys.argv) > 1:
|
||||
# Run specific test
|
||||
run_single_test(sys.argv[1])
|
||||
else:
|
||||
# Run all tests
|
||||
tester = TestGPT5Agent()
|
||||
tester.run_all_tests()
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,482 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"experiment_id": "1-3",
|
||||
"evidence_mode": "real_api",
|
||||
"created_at": "2026-07-29T15:54:59.247653+00:00",
|
||||
"canonical_source": "book/chapter1.md#实验-1-3-gpt-5-6-原生-deep-research-能力",
|
||||
"host": {
|
||||
"platform": "macOS-26.3-arm64-arm-64bit",
|
||||
"python": "3.11.4 (main, Jul 5 2023, 08:40:20) [Clang 14.0.6 ]",
|
||||
"machine": "arm64"
|
||||
},
|
||||
"repository": {
|
||||
"commit": "4a7f37cf278bd15948c409f14533017c4c7fbc29",
|
||||
"branch": "main",
|
||||
"worktree_dirty": true
|
||||
},
|
||||
"credentials_recorded": false,
|
||||
"runs": [
|
||||
{
|
||||
"backend": "openai",
|
||||
"started": true,
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"requested_model": "gpt-5.6-sol",
|
||||
"asean": {
|
||||
"success": false,
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
},
|
||||
"response": null,
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "Research the current official capitals and reliable coordinates\nfor the ten ASEAN member states. You must use hosted web search and cite the\nsources. Then you must use the hosted Python tool—not mental arithmetic—to\nenumerate all 45 capital pairs with the haversine formula and identify the\nclosest pair and distance. Include the coordinates, formula assumptions,\ncalculation result, retrieval date, and clickable citations. Do not say Python\nwas used unless a code_interpreter_call completes.",
|
||||
"reasoning": {
|
||||
"effort": "high"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "high"
|
||||
},
|
||||
"max_output_tokens": 16000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"raw_response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"usage": {},
|
||||
"model": "gpt-5.6-sol",
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"elapsed_seconds": 3.478408
|
||||
},
|
||||
"asean_validation": {
|
||||
"checks": {
|
||||
"request_succeeded": false,
|
||||
"exact_gpt_5_6_sol": true,
|
||||
"web_search_completed": false,
|
||||
"code_interpreter_completed": false,
|
||||
"url_citations_present": false,
|
||||
"closest_pair_reported": false,
|
||||
"distance_reported": false
|
||||
},
|
||||
"passed": false,
|
||||
"output_types": []
|
||||
},
|
||||
"clarification": {
|
||||
"ambiguous_task": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"first": {
|
||||
"success": false,
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
},
|
||||
"response": null,
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"reasoning": {
|
||||
"effort": "medium"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "medium"
|
||||
},
|
||||
"max_output_tokens": 4000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"raw_response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"usage": {},
|
||||
"model": "gpt-5.6-sol",
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"elapsed_seconds": 3.775702
|
||||
},
|
||||
"user_reply": null,
|
||||
"second": null,
|
||||
"validation": {
|
||||
"checks": {
|
||||
"first_turn_clarified_before_tools": false,
|
||||
"continuation_used_previous_response_id": false,
|
||||
"followup_succeeded": false,
|
||||
"followup_web_search_completed": false,
|
||||
"followup_code_interpreter_completed": false,
|
||||
"followup_citations_present": false
|
||||
},
|
||||
"passed": false
|
||||
}
|
||||
},
|
||||
"api_turns": [
|
||||
{
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "Research the current official capitals and reliable coordinates\nfor the ten ASEAN member states. You must use hosted web search and cite the\nsources. Then you must use the hosted Python tool—not mental arithmetic—to\nenumerate all 45 capital pairs with the haversine formula and identify the\nclosest pair and distance. Include the coordinates, formula assumptions,\ncalculation result, retrieval date, and clickable citations. Do not say Python\nwas used unless a code_interpreter_call completes.",
|
||||
"reasoning": {
|
||||
"effort": "high"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "high"
|
||||
},
|
||||
"max_output_tokens": 16000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"http_status": 429,
|
||||
"response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"elapsed_seconds": 3.478408
|
||||
},
|
||||
{
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"reasoning": {
|
||||
"effort": "medium"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "medium"
|
||||
},
|
||||
"max_output_tokens": 4000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"http_status": 429,
|
||||
"response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"elapsed_seconds": 3.775702
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"input_tokens": 0,
|
||||
"output_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
"reported_cost_usd": null,
|
||||
"reported_cost_available": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"backend": "openrouter",
|
||||
"started": true,
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"requested_model": "openai/gpt-5.6-sol",
|
||||
"asean": {
|
||||
"success": false,
|
||||
"error": {
|
||||
"message": "User not found.",
|
||||
"code": 401
|
||||
},
|
||||
"response": null,
|
||||
"request": {
|
||||
"model": "openai/gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "Research the current official capitals and reliable coordinates\nfor the ten ASEAN member states. You must use hosted web search and cite the\nsources. Then you must use the hosted Python tool—not mental arithmetic—to\nenumerate all 45 capital pairs with the haversine formula and identify the\nclosest pair and distance. Include the coordinates, formula assumptions,\ncalculation result, retrieval date, and clickable citations. Do not say Python\nwas used unless a code_interpreter_call completes.",
|
||||
"reasoning": {
|
||||
"effort": "high"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "high"
|
||||
},
|
||||
"max_output_tokens": 16000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"raw_response": {
|
||||
"error": {
|
||||
"message": "User not found.",
|
||||
"code": 401
|
||||
}
|
||||
},
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"usage": {},
|
||||
"model": "openai/gpt-5.6-sol",
|
||||
"provider": "openrouter",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"elapsed_seconds": 1.115963
|
||||
},
|
||||
"asean_validation": {
|
||||
"checks": {
|
||||
"request_succeeded": false,
|
||||
"exact_gpt_5_6_sol": true,
|
||||
"web_search_completed": false,
|
||||
"code_interpreter_completed": false,
|
||||
"url_citations_present": false,
|
||||
"closest_pair_reported": false,
|
||||
"distance_reported": false
|
||||
},
|
||||
"passed": false,
|
||||
"output_types": []
|
||||
},
|
||||
"clarification": {
|
||||
"ambiguous_task": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"first": {
|
||||
"success": false,
|
||||
"error": {
|
||||
"message": "User not found.",
|
||||
"code": 401
|
||||
},
|
||||
"response": null,
|
||||
"request": {
|
||||
"model": "openai/gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"reasoning": {
|
||||
"effort": "medium"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "medium"
|
||||
},
|
||||
"max_output_tokens": 4000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"raw_response": {
|
||||
"error": {
|
||||
"message": "User not found.",
|
||||
"code": 401
|
||||
}
|
||||
},
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"usage": {},
|
||||
"model": "openai/gpt-5.6-sol",
|
||||
"provider": "openrouter",
|
||||
"base_url": "https://openrouter.ai/api/v1",
|
||||
"elapsed_seconds": 1.025438
|
||||
},
|
||||
"user_reply": null,
|
||||
"second": null,
|
||||
"validation": {
|
||||
"checks": {
|
||||
"first_turn_clarified_before_tools": false,
|
||||
"continuation_used_previous_response_id": false,
|
||||
"followup_succeeded": false,
|
||||
"followup_web_search_completed": false,
|
||||
"followup_code_interpreter_completed": false,
|
||||
"followup_citations_present": false
|
||||
},
|
||||
"passed": false
|
||||
}
|
||||
},
|
||||
"api_turns": [
|
||||
{
|
||||
"request": {
|
||||
"model": "openai/gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "Research the current official capitals and reliable coordinates\nfor the ten ASEAN member states. You must use hosted web search and cite the\nsources. Then you must use the hosted Python tool—not mental arithmetic—to\nenumerate all 45 capital pairs with the haversine formula and identify the\nclosest pair and distance. Include the coordinates, formula assumptions,\ncalculation result, retrieval date, and clickable citations. Do not say Python\nwas used unless a code_interpreter_call completes.",
|
||||
"reasoning": {
|
||||
"effort": "high"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "high"
|
||||
},
|
||||
"max_output_tokens": 16000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"http_status": 401,
|
||||
"response": {
|
||||
"error": {
|
||||
"message": "User not found.",
|
||||
"code": 401
|
||||
}
|
||||
},
|
||||
"elapsed_seconds": 1.115963
|
||||
},
|
||||
{
|
||||
"request": {
|
||||
"model": "openai/gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"reasoning": {
|
||||
"effort": "medium"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "medium"
|
||||
},
|
||||
"max_output_tokens": 4000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"http_status": 401,
|
||||
"response": {
|
||||
"error": {
|
||||
"message": "User not found.",
|
||||
"code": 401
|
||||
}
|
||||
},
|
||||
"elapsed_seconds": 1.025438
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"input_tokens": 0,
|
||||
"output_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
"reported_cost_usd": null,
|
||||
"reported_cost_available": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"acceptance": {
|
||||
"checks": {
|
||||
"official_openai_asean_passed": false,
|
||||
"official_openai_clarification_passed": false
|
||||
},
|
||||
"passed": false,
|
||||
"openrouter_is_diagnostic_not_official_acceptance": true,
|
||||
"official_docs": [
|
||||
"https://developers.openai.com/api/docs/models/gpt-5.6-sol",
|
||||
"https://developers.openai.com/api/docs/guides/tools-web-search",
|
||||
"https://developers.openai.com/api/docs/guides/tools-code-interpreter",
|
||||
"https://developers.openai.com/api/docs/guides/model-guidance?model=gpt-5.6-sol"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
530316ce5e4d5de004b8076dd57a364bc0173aeb371dbc87f09f4fa913d11942 evidence.json
|
||||
@@ -0,0 +1,264 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"experiment_id": "1-3",
|
||||
"evidence_mode": "real_api",
|
||||
"created_at": "2026-07-29T19:39:13.370389+00:00",
|
||||
"canonical_source": "book/chapter1.md#实验-1-3-gpt-5-6-原生-deep-research-能力",
|
||||
"host": {
|
||||
"platform": "macOS-26.3-arm64-arm-64bit",
|
||||
"python": "3.11.4 (main, Jul 5 2023, 08:40:20) [Clang 14.0.6 ]",
|
||||
"machine": "arm64"
|
||||
},
|
||||
"repository": {
|
||||
"commit": "4a7f37cf278bd15948c409f14533017c4c7fbc29",
|
||||
"branch": "main",
|
||||
"worktree_dirty": true
|
||||
},
|
||||
"credentials_recorded": false,
|
||||
"runs": [
|
||||
{
|
||||
"backend": "openai",
|
||||
"started": true,
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"requested_model": "gpt-5.6-sol",
|
||||
"asean": {
|
||||
"success": false,
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
},
|
||||
"response": null,
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "Research the current official capitals and reliable coordinates\nfor the ten ASEAN member states. You must use hosted web search and cite the\nsources. Then you must use the hosted Python tool—not mental arithmetic—to\nenumerate all 45 capital pairs with the haversine formula and identify the\nclosest pair and distance. Include the coordinates, formula assumptions,\ncalculation result, retrieval date, and clickable citations. Do not say Python\nwas used unless a code_interpreter_call completes.",
|
||||
"reasoning": {
|
||||
"effort": "high"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "high"
|
||||
},
|
||||
"max_output_tokens": 16000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"raw_response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"usage": {},
|
||||
"model": "gpt-5.6-sol",
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"elapsed_seconds": 3.900986
|
||||
},
|
||||
"asean_validation": {
|
||||
"checks": {
|
||||
"request_succeeded": false,
|
||||
"exact_gpt_5_6_sol": true,
|
||||
"web_search_completed": false,
|
||||
"code_interpreter_completed": false,
|
||||
"url_citations_present": false,
|
||||
"closest_pair_reported": false,
|
||||
"distance_reported": false
|
||||
},
|
||||
"passed": false,
|
||||
"output_types": []
|
||||
},
|
||||
"clarification": {
|
||||
"ambiguous_task": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"first": {
|
||||
"success": false,
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
},
|
||||
"response": null,
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"reasoning": {
|
||||
"effort": "medium"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "medium"
|
||||
},
|
||||
"max_output_tokens": 4000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"raw_response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"tool_calls": [],
|
||||
"citations": [],
|
||||
"usage": {},
|
||||
"model": "gpt-5.6-sol",
|
||||
"provider": "openai",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"elapsed_seconds": 3.851438
|
||||
},
|
||||
"user_reply": null,
|
||||
"second": null,
|
||||
"validation": {
|
||||
"checks": {
|
||||
"first_turn_clarified_before_tools": false,
|
||||
"continuation_used_previous_response_id": false,
|
||||
"followup_succeeded": false,
|
||||
"followup_web_search_completed": false,
|
||||
"followup_code_interpreter_completed": false,
|
||||
"followup_citations_present": false
|
||||
},
|
||||
"passed": false
|
||||
}
|
||||
},
|
||||
"api_turns": [
|
||||
{
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "Research the current official capitals and reliable coordinates\nfor the ten ASEAN member states. You must use hosted web search and cite the\nsources. Then you must use the hosted Python tool—not mental arithmetic—to\nenumerate all 45 capital pairs with the haversine formula and identify the\nclosest pair and distance. Include the coordinates, formula assumptions,\ncalculation result, retrieval date, and clickable citations. Do not say Python\nwas used unless a code_interpreter_call completes.",
|
||||
"reasoning": {
|
||||
"effort": "high"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "high"
|
||||
},
|
||||
"max_output_tokens": 16000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"http_status": 429,
|
||||
"response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"elapsed_seconds": 3.900986
|
||||
},
|
||||
{
|
||||
"request": {
|
||||
"model": "gpt-5.6-sol",
|
||||
"instructions": "You are a deep-research assistant. Use hosted web search for\ncurrent facts and cite sources. Use the hosted Python/code-interpreter tool for\nquantitative analysis; do not claim a calculation was run unless the response\ncontains a completed code_interpreter_call. Ask a concise clarifying question\nbefore research when a material user preference is genuinely ambiguous.",
|
||||
"input": "搜索最近一个月的比特币走势，做技术分析。",
|
||||
"reasoning": {
|
||||
"effort": "medium"
|
||||
},
|
||||
"background": false,
|
||||
"store": true,
|
||||
"text": {
|
||||
"verbosity": "medium"
|
||||
},
|
||||
"max_output_tokens": 4000,
|
||||
"tools": [
|
||||
{
|
||||
"type": "web_search",
|
||||
"search_context_size": "medium"
|
||||
},
|
||||
{
|
||||
"type": "code_interpreter",
|
||||
"container": {
|
||||
"type": "auto",
|
||||
"memory_limit": "4g"
|
||||
}
|
||||
}
|
||||
],
|
||||
"tool_choice": "auto"
|
||||
},
|
||||
"http_status": 429,
|
||||
"response": {
|
||||
"error": {
|
||||
"message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",
|
||||
"type": "insufficient_quota",
|
||||
"param": null,
|
||||
"code": "insufficient_quota"
|
||||
}
|
||||
},
|
||||
"elapsed_seconds": 3.851438
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"input_tokens": 0,
|
||||
"output_tokens": 0,
|
||||
"total_tokens": 0,
|
||||
"reported_cost_usd": null,
|
||||
"reported_cost_available": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"acceptance": {
|
||||
"checks": {
|
||||
"official_openai_asean_passed": false,
|
||||
"official_openai_clarification_passed": false
|
||||
},
|
||||
"passed": false,
|
||||
"openrouter_is_diagnostic_not_official_acceptance": false,
|
||||
"official_docs": [
|
||||
"https://developers.openai.com/api/docs/models/gpt-5.6-sol",
|
||||
"https://developers.openai.com/api/docs/guides/tools-web-search",
|
||||
"https://developers.openai.com/api/docs/guides/tools-code-interpreter",
|
||||
"https://developers.openai.com/api/docs/guides/model-guidance?model=gpt-5.6-sol"
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
e2364d62b234e9691d0f69d09ea24335b9d685e17178577dfe1bade3573dfa1a evidence.json
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1 @@
|
||||
aab1864446613463c936154eca2fc65df8496869c65d4c17dc1f12628eaff3cc evidence.json
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"experiment_id": "1-3",
|
||||
"run_id": "real_20260731T170529Z",
|
||||
"created_at": "2026-07-31T17:05:29.580761+00:00",
|
||||
"artifacts": {
|
||||
"evidence.json": {
|
||||
"sha256": "aab1864446613463c936154eca2fc65df8496869c65d4c17dc1f12628eaff3cc"
|
||||
},
|
||||
"receipts.json": {
|
||||
"sha256": "9247d69b5915ceff7d27b9dbc0752d63791fc26309f8d69e17844508058edcbe"
|
||||
}
|
||||
},
|
||||
"inputs": {
|
||||
"canonical_source": "book/chapter1.md#实验-1-3-gpt-5-6-原生-deep-research-能力",
|
||||
"backends": [
|
||||
"openai",
|
||||
"dashscope"
|
||||
],
|
||||
"reasoning": "high"
|
||||
},
|
||||
"repository": {
|
||||
"commit": "8c2b7f55e4bd2e5348b15b823dd23a179ef3decf",
|
||||
"branch": "codex/exp1-3-multiprovider-20260731",
|
||||
"worktree_dirty": true
|
||||
},
|
||||
"acceptance_passed": true,
|
||||
"acceptance_backend": "dashscope"
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user