ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,427 @@
|
||||
# Multi-dimensional Model Benchmarking / 多维度模型性能基准 / 实验 7-10
|
||||
|
||||
## English
|
||||
|
||||
This directory contains two layers. `demo.py` is the short interactive sampler;
|
||||
`campaign.py` is the **complete, resumable Experiment 7-10 campaign**. The latter
|
||||
supports OpenAI-compatible, native Anthropic, and native Gemini APIs and records
|
||||
every real request in SQLite.
|
||||
|
||||
The complete campaign supports:
|
||||
- the 8K / 32K / 128K input × 512 / 2048 output workload matrix;
|
||||
- at least 100 requests per provider/model/cell;
|
||||
- exact TTFT, end-to-end latency, thinking TTFT, reported reasoning length,
|
||||
token usage, cache hits, and error classification;
|
||||
- a 168-hour hourly availability monitor with outage duration, MTTR, and longest
|
||||
continuous availability analysis;
|
||||
- a measured concurrency ramp with RPM and input/output TPM saturation;
|
||||
- cached/input/output pricing and a six-round Agent cost trace;
|
||||
- an explicit same-model/different-provider comparison group (DeepSeek V4 Flash on the official DeepSeek API and SiliconFlow);
|
||||
- resumability through unique request cells in SQLite and a strict completion audit.
|
||||
|
||||
The quick sampler still supports:
|
||||
- **Concurrency stress testing**: sweep concurrency to identify rate limits and observe metric curves.
|
||||
- **Offline mock mode** (`--mock`): synthetic data pipeline for verifying aggregation logic without API keys/network.
|
||||
|
||||
Synthetic mode is never accepted by `campaign.py` and cannot populate the
|
||||
official evidence database.
|
||||
|
||||
## Full campaign
|
||||
|
||||
Review `campaign_config.json` before a cost-sensitive run. Pricing fields are
|
||||
deliberately explicit; `analysis.py` refuses to declare the campaign complete
|
||||
while any cached/input/output rate is missing.
|
||||
|
||||
```bash
|
||||
# Real API integration smoke (small scope is visibly labelled)
|
||||
python campaign.py workload --smoke --requests 1 \
|
||||
--context-tokens 256 --output-tokens 128 \
|
||||
--provider 'Ark/doubao-seed-1.6' \
|
||||
--campaign-id integration-smoke --db results/integration-smoke.sqlite3
|
||||
|
||||
# Official standardized workload: defaults are 3 contexts × 2 outputs × N=100
|
||||
python campaign.py workload --campaign-id release-2026-07
|
||||
|
||||
# Actual RPM/TPM ramp
|
||||
python campaign.py rate-limit --campaign-id release-2026-07
|
||||
|
||||
# Six-turn stable-prefix Agent cost/cache trace
|
||||
python campaign.py agent-cost --campaign-id release-2026-07
|
||||
|
||||
# Keep this process alive for one week; every hourly cell is resumable
|
||||
python campaign.py availability --duration-hours 168 --interval-seconds 3600 \
|
||||
--campaign-id release-2026-07
|
||||
|
||||
python analysis.py --campaign-id release-2026-07
|
||||
```
|
||||
|
||||
`analysis.py` writes JSON and Markdown reports and prints
|
||||
`Official completion: True` only after every manuscript requirement has direct
|
||||
database evidence. A smoke run is useful validation, but it can never satisfy
|
||||
the 100-request or 168-hour gates.
|
||||
|
||||
The completion audit requires every configured provider—not merely one working
|
||||
provider—to have the complete workload, 169 hourly boundary probes spanning 168
|
||||
hours, rate ramp, and six-round cost trace. Pricing is accepted only when input,
|
||||
cached-input, and output rates are all pinned together with an authoritative
|
||||
`source_url` and `as_of` date. Prices remain in the provider's published native
|
||||
currency. A non-USD price also requires `usd_per_currency_unit`, `fx_source_url`,
|
||||
and `fx_as_of` before the cross-provider USD cost gate can pass; CNY values are
|
||||
never copied into USD-labelled fields. A null rate cannot pass simply because a
|
||||
smoke run reported zero tokens in that category. The same-model provider group must also
|
||||
have successful official workload cells on both endpoints. The configured
|
||||
identifiers are `deepseek-v4-flash` on the official API and
|
||||
`deepseek-ai/DeepSeek-V4-Flash` on SiliconFlow. Ark's
|
||||
`deepseek-v4-flash-260425` remains a third independent deployment in the wider
|
||||
provider matrix, but it is not substituted for either comparison arm.
|
||||
|
||||
## Metric definitions
|
||||
|
||||
| Metric | Meaning | How measured |
|
||||
|---|---|---|
|
||||
| Success rate | Availability | successful request count / total |
|
||||
| TTFT | Time to first token | stream first non-empty chunk - request start |
|
||||
| End-to-end latency | complete response time | request start -> final chunk |
|
||||
| Throughput (tokens/s) | generation speed | output token count / (end-to-end - TTFT) |
|
||||
| p50 / p95 / p99 | latency percentiles | interpolation over successful requests |
|
||||
| std | standard deviation | per-provider latency dispersion |
|
||||
| aggregate throughput / RPS | batch throughput metrics | aggregated output token rate and request rate |
|
||||
|
||||
If `usage.completion_tokens` is unavailable, token count falls back to chunk-count approximation with a documented caveat.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 6 environment
|
||||
uv sync --locked --python 3.12 --extra ch6
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch6]"
|
||||
|
||||
cd chapter7/model-benchmark
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
cp env.example .env
|
||||
# or export OPENAI_API_KEY=... MOONSHOT_API_KEY=... ARK_API_KEY=...
|
||||
|
||||
python demo.py
|
||||
```
|
||||
|
||||
### Common parameters
|
||||
|
||||
```bash
|
||||
python demo.py --list
|
||||
python demo.py --num-requests 20 --concurrency 5
|
||||
python demo.py --serial
|
||||
python demo.py --max-tokens 256
|
||||
```
|
||||
|
||||
## Specify custom endpoint/model
|
||||
|
||||
Use `--base-url`, `--model`, and `--api-key-env` to test a new provider without changing `DEFAULT_PROVIDERS`.
|
||||
|
||||
```bash
|
||||
python demo.py --base-url https://api.deepseek.com --model deepseek-chat \
|
||||
--api-key-env DEEPSEEK_API_KEY --name "DeepSeek官方/deepseek-chat"
|
||||
```
|
||||
|
||||
## Concurrency sweep
|
||||
|
||||
```bash
|
||||
python demo.py --model gpt-5.6-luna --concurrency-sweep 1,2,4,8,16 --num-requests 100
|
||||
```
|
||||
|
||||
As concurrency increases, p95/p99/std generally get worse, and success rate may drop due to rate limits; aggregate throughput/RPS usually rises then plateaus.
|
||||
|
||||
## Metrics and export
|
||||
|
||||
```bash
|
||||
python demo.py --metrics ttft,throughput
|
||||
python demo.py --output result.json
|
||||
```
|
||||
|
||||
## Offline mock validation
|
||||
|
||||
```bash
|
||||
python demo.py --mock
|
||||
python demo.py --mock --concurrency-sweep 1,2,4,8,16
|
||||
```
|
||||
|
||||
This validates full aggregation logic with synthetic numbers labelled `[SYNTHETIC]`.
|
||||
|
||||
## Default providers
|
||||
|
||||
`DEFAULT_PROVIDERS` includes the keys that are present in environment:
|
||||
|
||||
- OpenAI-compatible entries (gpt-5.6-luna)
|
||||
- Moonshot / doubao (explicit base_url + key)
|
||||
|
||||
OpenRouter fallback behavior:
|
||||
- If `OPENAI_API_KEY` is missing, OpenAI-style entries can still run via OpenRouter (`OPENROUTER_API_KEY`), with model id mapping.
|
||||
- For gpt-5.x, OpenRouter is preferred when `OPENROUTER_API_KEY` exists.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `benchmark.py` | core benchmark core: provider config, streaming measure, concurrency scheduling, aggregation |
|
||||
| `demo.py` | CLI, parameter parsing, reporting and mock mode |
|
||||
| `campaign.py` | full provider adapters, exact workloads, scheduler, rate ramp, cache trace, SQLite persistence |
|
||||
| `analysis.py` | p50/p95/p99/std, outages/MTTR, RPM/TPM, cost, and completion audit |
|
||||
| `campaign_config.json` | provider/model/workload/pricing inputs; no credentials |
|
||||
| `test_campaign.py` | deterministic native/OpenAI adapter, persistence, and analysis tests |
|
||||
| `requirements.txt` | dependencies |
|
||||
| `env.example` | env templates |
|
||||
|
||||
## Operational boundaries
|
||||
|
||||
- `demo.py` retains low-cost defaults; it is not the full experiment.
|
||||
- The official campaign is intentionally expensive and takes at least seven days.
|
||||
- Provider pricing changes over time. Pin the rates used for a decision in
|
||||
`campaign_config.json`; missing prices remain visible as an incomplete gate.
|
||||
- TTFT depends heavily on geography/network.
|
||||
- Offline mock is for method validation only, not production decisions.
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
# 多维度模型性能基准测试（实验 7-10 配套代码）
|
||||
|
||||
对多个 OpenAI 兼容的 LLM API 提供商做横向基准测试，一条命令跑出
|
||||
**TTFT / 端到端延迟 / 吞吐 / 标准差 / p50 / p95 / p99 / 成功率** 的多维度对比表，
|
||||
为模型选型提供实测依据。还支持**并发压测**（逐档加压找限流点，看指标随并发的变化）
|
||||
与**离线自检**（`--mock` 合成数据，无需 key/网络即可验证指标聚合）。
|
||||
|
||||
对应《深入理解 AI Agent》第 6 章 **实验 7-10：多维度模型性能基准测试**。
|
||||
|
||||
## 目的
|
||||
|
||||
本目录现在分为两层：`demo.py` 保留几分钟即可运行的低成本抽样；
|
||||
`campaign.py` 则完整实现正文要求的长期实验。完整路径包含
|
||||
**8K/32K/128K × 512/2048、每格至少 100 次请求、一周逐小时探测、
|
||||
故障分组与 MTTR、并发爬坡实测 RPM/TPM、思考长度/延迟、缓存/输入/输出
|
||||
三类价格与典型多轮 Agent 成本**。每次真实请求写入 SQLite，进程中断后可继续。
|
||||
`analysis.py` 会逐项审计证据；只跑小样本或留下未填写的价格时不会误报完成。
|
||||
|
||||
### 完整实验命令
|
||||
|
||||
```bash
|
||||
# 小规模真实 API 集成验证（不会被当成正式结果）
|
||||
python campaign.py workload --smoke --requests 1 \
|
||||
--context-tokens 256 --output-tokens 128 \
|
||||
--provider 'Ark/doubao-seed-1.6' \
|
||||
--campaign-id integration-smoke --db results/integration-smoke.sqlite3
|
||||
|
||||
# 正式负载矩阵（默认每格 N=100）
|
||||
python campaign.py workload --campaign-id release-2026-07
|
||||
|
||||
# 逐级并发实测 RPM / TPM 上限
|
||||
python campaign.py rate-limit --campaign-id release-2026-07
|
||||
|
||||
# 多轮 Agent 缓存与成本轨迹
|
||||
python campaign.py agent-cost --campaign-id release-2026-07
|
||||
|
||||
# 正式一周可用性监控
|
||||
python campaign.py availability --duration-hours 168 --interval-seconds 3600 \
|
||||
--campaign-id release-2026-07
|
||||
|
||||
python analysis.py --campaign-id release-2026-07
|
||||
```
|
||||
|
||||
正式运行前必须在 `campaign_config.json` 中固定本次决策采用的公开价格。
|
||||
价格保留提供商发布的原始币种；非美元价格还必须固定带日期和来源的汇率，才能进入
|
||||
跨提供商美元成本比较。程序不会把人民币数字直接写入美元字段。任何 input /
|
||||
cached input / output 单价为空，完成审计都会明确失败，而不会用猜测价格填补。
|
||||
|
||||
## 指标定义
|
||||
|
||||
| 指标 | 含义 | 怎么测的 |
|
||||
| --- | --- | --- |
|
||||
| 成功率（可用性） | 成功请求数 / 总请求数 | 单次请求任何异常（超时/限流/网络错误/空响应）都计为失败，不中断整表 |
|
||||
| TTFT | 首个 token 到达延迟 | 流式读取，记录第一个"有内容" chunk 到达的时刻 − 请求发出时刻 |
|
||||
| 端到端延迟 | 请求发出到响应结束的总耗时 | 最后一个 chunk 时刻 − 请求发出时刻 |
|
||||
| 吞吐（tokens/s） | 生成阶段的输出速度 | 输出 token 数 / (端到端 − TTFT)，剥离首 token 等待，反映纯解码速度 |
|
||||
| p50 / p95 / p99 | 延迟的中位数 / 95 / 99 分位 | 对同一 (provider, model) 的多次成功请求排序后线性插值；p95、p99 高说明长尾重、体验不稳 |
|
||||
| 标准差（std） | 延迟的离散程度 | 样本标准差；书中强调"高延迟方差意味着用户体验不稳定" |
|
||||
| 聚合吞吐 / RPS | 整批的总吞吐 | 并发压测时：全部成功请求的输出 token 总数 / 整批墙钟耗时（RPS 为成功请求数 / 墙钟）；随并发上升先增后趋平，触及服务端上限即触顶 |
|
||||
|
||||
> 输出 token 数优先取服务端回传的精确 `usage.completion_tokens`；
|
||||
> 若服务不返回 usage，则以流式 chunk 数近似计数（会略微偏高，已在代码注释标明）。
|
||||
|
||||
## 运行
|
||||
|
||||
```bash
|
||||
# 在仓库根目录使用统一的第 6 章环境
|
||||
uv sync --locked --python 3.12 --extra ch6
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch6]"
|
||||
|
||||
cd chapter7/model-benchmark
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
# 配置 key：只需填手上有的，未设置的提供商会自动跳过
|
||||
cp env.example .env # 然后编辑 .env
|
||||
# 或直接 export OPENAI_API_KEY=... MOONSHOT_API_KEY=... ARK_API_KEY=...
|
||||
|
||||
python demo.py # 一条命令跑出对比表
|
||||
```
|
||||
|
||||
常用参数：
|
||||
|
||||
```bash
|
||||
python demo.py --list # 仅列出将测试的提供商
|
||||
python demo.py --num-requests 20 --concurrency 5 # 加大样本与并发
|
||||
python demo.py --serial # 串行发送（并发=1，看无竞争下的基线延迟）
|
||||
python demo.py --max-tokens 256 # 生成更长响应，更充分地测吞吐
|
||||
```
|
||||
|
||||
默认参数（`N=10/家, 并发=3, max_tokens=64`）单次全跑成本约几分钱。
|
||||
要接近书中"每配置 ≥100 次请求"的统计口径，把 `--num-requests` 调到 100 即可
|
||||
（注意成本与限流会同步上升）。
|
||||
|
||||
### 指定任意 OpenAI 兼容端点（不改代码测新模型/新提供商）
|
||||
|
||||
书中要求"对同一模型测试不同 API 提供商（如 DeepSeek 官方 vs SiliconFlow）"。
|
||||
用 `--base-url / --model / --api-key-env` 即可直接指定单个端点，无需改 `DEFAULT_PROVIDERS`：
|
||||
|
||||
```bash
|
||||
python demo.py --base-url https://api.deepseek.com --model deepseek-chat \
|
||||
--api-key-env DEEPSEEK_API_KEY --name "DeepSeek官方/deepseek-chat"
|
||||
# 换个 base_url、保持同一 model，即可对比"同模型不同提供商"
|
||||
```
|
||||
|
||||
### 并发压测：逐步加压找限流点
|
||||
|
||||
书中实验 7-10 要求"通过逐步提升并发量来找到限流点，记录 RPM/TPM 上限"。
|
||||
`--concurrency-sweep` 对同一模型逐档加压，产出一张随并发变化的指标表
|
||||
（p50/p95/p99/std/成功率/RPS/聚合吞吐）：
|
||||
|
||||
```bash
|
||||
python demo.py --model gpt-5.6-luna --concurrency-sweep 1,2,4,8,16 --num-requests 100
|
||||
```
|
||||
|
||||
随着并发上升，单请求延迟长尾（p95/p99/std）通常变差、可用性可能因限流下降，
|
||||
而**聚合吞吐（tokens/s）与 RPS 先升后趋平**——趋平点即服务端的实际吞吐上限。
|
||||
|
||||
### 选择要显示的指标 / 导出结果
|
||||
|
||||
```bash
|
||||
python demo.py --metrics ttft,throughput # 主表只看 TTFT 与吞吐（成功率始终显示）
|
||||
python demo.py --output result.json # 完整结果（含 p50/p95/p99/std）写入 JSON
|
||||
```
|
||||
|
||||
### 离线自检（`--mock`，无需 key/网络）
|
||||
|
||||
用**合成（synthetic）数据**跑通整条指标聚合链路，便于在没有 API key 或无网络时
|
||||
验证 p50/p95/p99/std/可用性/聚合吞吐的计算是否正确。**输出数字全部为伪随机合成，
|
||||
带 `[SYNTHETIC]` 标注，绝非真实基准，切勿用于选型。**
|
||||
|
||||
```bash
|
||||
python demo.py --mock # 合成横向对比表
|
||||
python demo.py --mock --concurrency-sweep 1,2,4,8,16 # 合成并发压测表
|
||||
```
|
||||
|
||||
一次合成并发压测的输出（`--mock --concurrency-sweep 1,2,4,8,16 --num-requests 100`，
|
||||
**数字为合成，仅演示趋势**）：
|
||||
|
||||
```
|
||||
并发 | 成功率 | TTFT_p50 | TTFT_p95 | 端到端p50 | 端到端p95 | 端到端p99 | 端到端std | RPS | 聚合吞吐
|
||||
-----+----------------+----------+----------+-----------+-----------+-----------+-----------+------+----------
|
||||
1 | 99/100 (99%) | 301ms | 514ms | 0.73s | 1.04s | 1.16s | 0.13s | 1.3 | 49.8 t/s
|
||||
2 | 100/100 (100%) | 335ms | 570ms | 0.79s | 1.07s | 1.19s | 0.15s | 2.5 | 94.4 t/s
|
||||
4 | 98/100 (98%) | 381ms | 617ms | 0.83s | 1.11s | 1.19s | 0.16s | 4.7 | 180.0 t/s
|
||||
8 | 92/100 (92%) | 523ms | 932ms | 0.96s | 1.53s | 1.67s | 0.25s | 8.0 | 305.3 t/s
|
||||
16 | 97/100 (97%) | 878ms | 1487ms | 1.30s | 1.97s | 2.37s | 0.35s | 11.9 | 441.0 t/s
|
||||
```
|
||||
|
||||
可见随并发上升：端到端 p95/p99 与 std 走高（长尾变差），聚合吞吐持续增长（尚未触顶）。
|
||||
真实端点上这条曲线会在某个并发处趋平并伴随可用性下降——那就是限流点。
|
||||
|
||||
## 默认测试的提供商
|
||||
|
||||
代码里 `DEFAULT_PROVIDERS` 默认只跑**手上有有效 key**的提供商（OpenAI 一个 key 测多个模型）：
|
||||
|
||||
| 展示名 | 模型 | base_url | key 环境变量 |
|
||||
| --- | --- | --- | --- |
|
||||
| OpenAI/gpt-5.6-luna | gpt-5.6-luna | （官方默认，可回退 OpenRouter） | OPENAI_API_KEY |
|
||||
| Moonshot/moonshot-v1-8k | moonshot-v1-8k | https://api.moonshot.cn/v1 | MOONSHOT_API_KEY |
|
||||
| Doubao/doubao-1.5-pro-32k | doubao-1-5-pro-32k-250115 | https://ark.cn-beijing.volces.com/api/v3 | ARK_API_KEY |
|
||||
|
||||
> **OpenRouter 回退**：`OpenAI/*` 这几条（base_url 为空的 OpenAI 原生条目）在未设置
|
||||
> `OPENAI_API_KEY` 时会自动改走 **OpenRouter**（`OPENROUTER_API_KEY`，模型名映射为
|
||||
> `openai/*`）。`gpt-5.x` 直连 OpenAI 需组织实名认证，因此只要设置了 `OPENROUTER_API_KEY`
|
||||
> 就优先走 OpenRouter。带专属 `base_url` 的条目（Kimi/豆包）不参与回退。
|
||||
|
||||
**提供商列表是可配置的**：在 `benchmark.py` 的 `DEFAULT_PROVIDERS` 里追加
|
||||
`ProviderConfig(...)` 即可扩展。所有提供商都走同一套 OpenAI 兼容协议，
|
||||
只是 `base_url` 与 `model` 不同——这正是可以"同一模型对比不同提供商"
|
||||
（如书中提到的 DeepSeek 官方 vs SiliconFlow）的原因。
|
||||
|
||||
## 真实运行结果（示例）
|
||||
|
||||
以下是一次真实运行的输出（`python demo.py --num-requests 10 --concurrency 3`，
|
||||
测试机在中国大陆网络环境，`2026-07`）。**数字为真实测得，非虚构**；
|
||||
不同网络/时段会有波动，请以自己跑出的结果为准。
|
||||
|
||||
```
|
||||
Provider/Model | 成功率 | TTFT均值 | TTFT_p95 | 端到端均值 | 端到端p95 | 吞吐 | 输出tok
|
||||
--------------------------+--------------+----------+----------+------------+-----------+-----------+--------
|
||||
OpenAI/gpt-5.6-luna | 10/10 (100%) | 1360ms | 2334ms | 1.73s | 2.54s | 174.9 t/s | 26
|
||||
Moonshot/moonshot-v1-8k | 10/10 (100%) | 530ms | 671ms | 0.89s | 1.07s | 92.1 t/s | 32
|
||||
Doubao/doubao-1.5-pro-32k | 10/10 (100%) | 1097ms | 1409ms | 2.32s | 2.91s | 36.2 t/s | 44
|
||||
```
|
||||
|
||||
## 结论（基于上面这次运行）
|
||||
|
||||
- **可用性**：本次三家全部 10/10（100%）成功。可用性差异往往要在更大样本、
|
||||
更高并发或更长时间窗口下才暴露——这正是书中强调"一周每小时探测"的原因。
|
||||
代码已把单点失败设计成"记为可用性下降、不中断整表"，便于长时间挂机采样。
|
||||
- **首 token 延迟（TTFT）**：本测试机在国内网络下，Kimi 的 TTFT（~530ms）明显低于
|
||||
跨境访问的 OpenAI/gpt-5.6-luna（~1.36s）；豆包 TTFT（~1.1s）略低于 OpenAI 但端到端更长。
|
||||
**TTFT 强依赖网络位置**——同一份代码在美国机房跑，OpenAI 的 TTFT 会大幅下降。
|
||||
- **吞吐**：本次 gpt-5.6-luna（175 t/s）> Kimi（92 t/s）> 豆包（36 t/s）。
|
||||
吞吐决定长响应的等待时间，与 TTFT 是两个独立维度。
|
||||
- **稳定性（p95）**：看 p95 与均值的差距。gpt-5.6-luna 跨境访问，TTFT p95(2.33s)/均值(1.36s)
|
||||
拉开较大，长尾更重；Kimi 的 p95 与均值最接近，本次最稳。
|
||||
- **选型启示**：不存在"全面最优"的一家——延迟、吞吐、可用性、价格是**多维权衡**。
|
||||
面向国内用户的实时交互场景，低 TTFT 的本地化服务体验更好；
|
||||
批处理/长文本生成则更看重吞吐与单价。**务必在你自己的部署网络环境下实测**，
|
||||
不要直接照搬第三方监测平台（如 Artificial Analysis）的数字。
|
||||
|
||||
## 文件说明
|
||||
|
||||
| 文件 | 作用 |
|
||||
| --- | --- |
|
||||
| `benchmark.py` | 核心：提供商配置、单次流式测量、并发调度、指标聚合（含 p99/std/聚合吞吐）、并发扫描 `sweep_concurrency`、合成数据 `synthetic_summary` |
|
||||
| `demo.py` | 命令行入口：解析参数、跑测试（含并发压测 / `--mock` 离线自检）、打印对比表、导出 JSON |
|
||||
| `requirements.txt` | 依赖（openai SDK + 可选 python-dotenv） |
|
||||
| `env.example` | key 配置模板 |
|
||||
|
||||
## 注意事项
|
||||
|
||||
- **成本控制**：默认 `max_tokens=64`、`N=10`，全跑成本极低。调大参数前请留意计费。
|
||||
- **限流**：把并发或 N 调很大时可能触发提供商 RPM/TPM 限流，届时会以失败形式
|
||||
计入可用性下降——这本身也是一种"实测限流阈值"的方式（书中实验 7-10 的一环）。
|
||||
- **TTFT 与网络强相关**：跨境访问的服务 TTFT 会显著偏高，结论需结合部署地点解读。
|
||||
- **OpenRouter 回退**：未设置 `OPENAI_API_KEY` 时，`OpenAI/*` 条目自动经 OpenRouter 路由
|
||||
（需 `OPENROUTER_API_KEY`，`gpt-*` 映射为 `openai/*`）；`gpt-5.x` 只要有 `OPENROUTER_API_KEY`
|
||||
即优先走 OpenRouter（直连需实名认证）。其它提供商（DEEPSEEK / SILICONFLOW 等）如需启用，
|
||||
在 `DEFAULT_PROVIDERS` 中补充配置并设置对应环境变量即可。
|
||||
@@ -0,0 +1,864 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Analyze the SQLite evidence produced by the full Experiment 7-10 campaign."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import sqlite3
|
||||
import statistics
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable, Sequence
|
||||
|
||||
from campaign import (
|
||||
DEFAULT_CONFIG,
|
||||
DEFAULT_DB,
|
||||
PROMPT_SCHEMA_VERSION,
|
||||
Price,
|
||||
Provider,
|
||||
execution_config_fingerprint,
|
||||
load_config,
|
||||
)
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def percentile(values: Sequence[float], q: float) -> float | None:
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
if len(ordered) == 1:
|
||||
return ordered[0]
|
||||
position = q * (len(ordered) - 1)
|
||||
lower = math.floor(position)
|
||||
upper = math.ceil(position)
|
||||
if lower == upper:
|
||||
return ordered[lower]
|
||||
fraction = position - lower
|
||||
return ordered[lower] + (ordered[upper] - ordered[lower]) * fraction
|
||||
|
||||
|
||||
def describe(values: Iterable[float | int | None]) -> dict[str, float | int | None]:
|
||||
present = [float(value) for value in values if value is not None]
|
||||
if not present:
|
||||
return {"n": 0, "mean": None, "std": None, "p50": None, "p95": None, "p99": None}
|
||||
return {
|
||||
"n": len(present),
|
||||
"mean": statistics.fmean(present),
|
||||
"std": statistics.stdev(present) if len(present) > 1 else 0.0,
|
||||
"p50": percentile(present, 0.50),
|
||||
"p95": percentile(present, 0.95),
|
||||
"p99": percentile(present, 0.99),
|
||||
}
|
||||
|
||||
|
||||
def parse_time(value: str) -> datetime:
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def grouped(rows: Sequence[dict[str, Any]], keys: Sequence[str]):
|
||||
result: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
|
||||
for row in rows:
|
||||
result[tuple(row[key] for key in keys)].append(row)
|
||||
return result
|
||||
|
||||
|
||||
def summarize_workloads(rows: Sequence[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
keys = ("provider", "model", "target_context_tokens", "target_output_tokens", "concurrency")
|
||||
result = []
|
||||
for identity, items in sorted(grouped(rows, keys).items()):
|
||||
successes = [row for row in items if row["ok"]]
|
||||
wall = [row["e2e_s"] for row in successes if row["e2e_s"] is not None]
|
||||
visible_output = [
|
||||
int(row.get("visible_output_tokens") or max(
|
||||
0, int(row.get("output_tokens") or 0) - int(row.get("reasoning_tokens") or 0)
|
||||
))
|
||||
for row in successes
|
||||
]
|
||||
generation = [
|
||||
visible / max(row["e2e_s"] - row["ttft_s"], 1e-9)
|
||||
for row, visible in zip(successes, visible_output)
|
||||
if visible and row["e2e_s"] is not None and row["ttft_s"] is not None
|
||||
]
|
||||
input_prefill = [
|
||||
row["input_tokens"] / max(row["ttft_s"], 1e-9)
|
||||
for row in successes
|
||||
if row["input_tokens"] and row["ttft_s"] is not None
|
||||
]
|
||||
target_output = identity[3]
|
||||
result.append({
|
||||
**dict(zip(keys, identity)),
|
||||
"requests": len(items),
|
||||
"successes": len(successes),
|
||||
"success_rate": len(successes) / len(items),
|
||||
"errors": dict(sorted(_counts(row["error_type"] or "unknown" for row in items if not row["ok"]).items())),
|
||||
"ttft_s": describe(row["ttft_s"] for row in successes),
|
||||
"e2e_s": describe(wall),
|
||||
"input_prefill_throughput_tokens_s": describe(input_prefill),
|
||||
"output_throughput_tokens_s": describe(generation),
|
||||
"actual_input_tokens": describe(row["input_tokens"] for row in successes),
|
||||
"actual_output_tokens": describe(row["output_tokens"] for row in successes),
|
||||
"visible_output_tokens": describe(visible_output),
|
||||
"output_length_attainment_rate": (
|
||||
sum(value >= 0.95 * target_output for value in visible_output) / len(successes)
|
||||
if successes else 0.0
|
||||
),
|
||||
"reasoning_tokens": describe(row["reasoning_tokens"] for row in successes),
|
||||
"thinking_ttft_s": describe(row["thinking_ttft_s"] for row in successes),
|
||||
})
|
||||
return result
|
||||
|
||||
|
||||
def _counts(values: Iterable[str]) -> dict[str, int]:
|
||||
result: dict[str, int] = defaultdict(int)
|
||||
for value in values:
|
||||
result[value] += 1
|
||||
return dict(result)
|
||||
|
||||
|
||||
def availability_summary(rows: Sequence[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
result = []
|
||||
for (provider, model), items in sorted(grouped(rows, ("provider", "model")).items()):
|
||||
ordered = sorted(items, key=lambda row: row["scheduled_at_utc"] or row["started_at_utc"])
|
||||
outages: list[dict[str, Any]] = []
|
||||
outage_start: datetime | None = None
|
||||
outage_errors: list[str] = []
|
||||
availability_start: datetime | None = None
|
||||
continuous: list[float] = []
|
||||
for row in ordered:
|
||||
current = parse_time(row["scheduled_at_utc"] or row["started_at_utc"])
|
||||
if row["ok"]:
|
||||
if outage_start is not None:
|
||||
outages.append({
|
||||
"started_at_utc": outage_start.isoformat(),
|
||||
"recovered_at_utc": current.isoformat(),
|
||||
"duration_s": (current - outage_start).total_seconds(),
|
||||
"errors": dict(_counts(outage_errors)),
|
||||
"open": False,
|
||||
})
|
||||
outage_start = None
|
||||
outage_errors = []
|
||||
if availability_start is None:
|
||||
availability_start = current
|
||||
else:
|
||||
if availability_start is not None:
|
||||
continuous.append((current - availability_start).total_seconds())
|
||||
availability_start = None
|
||||
if outage_start is None:
|
||||
outage_start = current
|
||||
outage_errors.append(row["error_type"] or "unknown")
|
||||
if ordered:
|
||||
last = parse_time(ordered[-1]["scheduled_at_utc"] or ordered[-1]["started_at_utc"])
|
||||
if availability_start is not None:
|
||||
continuous.append((last - availability_start).total_seconds())
|
||||
if outage_start is not None:
|
||||
outages.append({
|
||||
"started_at_utc": outage_start.isoformat(),
|
||||
"recovered_at_utc": None,
|
||||
"duration_s": (last - outage_start).total_seconds(),
|
||||
"errors": dict(_counts(outage_errors)),
|
||||
"open": True,
|
||||
})
|
||||
recovered = [outage["duration_s"] for outage in outages if not outage["open"]]
|
||||
successes = sum(row["ok"] for row in ordered)
|
||||
result.append({
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"probes": len(ordered),
|
||||
"successes": successes,
|
||||
"uptime": successes / len(ordered) if ordered else None,
|
||||
"failure_rate": 1 - successes / len(ordered) if ordered else None,
|
||||
"error_types": dict(_counts(row["error_type"] or "unknown" for row in ordered if not row["ok"])),
|
||||
"outages": outages,
|
||||
"outage_count": len(outages),
|
||||
"mttr_s": statistics.fmean(recovered) if recovered else None,
|
||||
"longest_continuous_availability_s": max(continuous, default=0.0),
|
||||
"observed_start_utc": (ordered[0]["scheduled_at_utc"] or ordered[0]["started_at_utc"]) if ordered else None,
|
||||
"observed_end_utc": (ordered[-1]["scheduled_at_utc"] or ordered[-1]["started_at_utc"]) if ordered else None,
|
||||
})
|
||||
return result
|
||||
|
||||
|
||||
def rate_limit_summary(
|
||||
batches: Sequence[dict[str, Any]],
|
||||
observations: Sequence[dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
errors = {
|
||||
identity: dict(_counts(
|
||||
row["error_type"] or "unknown"
|
||||
for row in items if not row["ok"]
|
||||
))
|
||||
for identity, items in grouped(
|
||||
observations, ("provider", "model", "concurrency")
|
||||
).items()
|
||||
}
|
||||
result = []
|
||||
for row in sorted(batches, key=lambda item: (item["provider"], item["concurrency"])):
|
||||
wall = row["wall_s"]
|
||||
result.append({
|
||||
"provider": row["provider"],
|
||||
"model": row["model"],
|
||||
"concurrency": row["concurrency"],
|
||||
"requests": row["requested"],
|
||||
"successes": row["succeeded"],
|
||||
"success_rate": row["succeeded"] / row["requested"] if row["requested"] else None,
|
||||
"measured_rpm": row["succeeded"] / wall * 60 if wall else None,
|
||||
"measured_input_tpm": row["input_tokens"] / wall * 60 if wall else None,
|
||||
"measured_output_tpm": row["output_tokens"] / wall * 60 if wall else None,
|
||||
"wall_s": wall,
|
||||
"errors": errors.get((row["provider"], row["model"], row["concurrency"]), {}),
|
||||
})
|
||||
return result
|
||||
|
||||
|
||||
def cost_summary(
|
||||
rows: Sequence[dict[str, Any]],
|
||||
providers: dict[str, Provider],
|
||||
) -> list[dict[str, Any]]:
|
||||
result = []
|
||||
for (provider_name, model, phase), items in sorted(grouped(rows, ("provider", "model", "phase")).items()):
|
||||
provider = providers[provider_name]
|
||||
price = provider.pricing
|
||||
uncached = sum(max(0, row["input_tokens"] - row["cached_input_tokens"]) for row in items)
|
||||
cached = sum(row["cached_input_tokens"] for row in items)
|
||||
output = sum(row["output_tokens"] for row in items)
|
||||
unpriced: dict[str, int] = {}
|
||||
native_cost = 0.0
|
||||
for label, tokens, rate in (
|
||||
("input", uncached, price.input_per_million),
|
||||
("cached_input", cached, price.cached_input_per_million),
|
||||
("output", output, price.output_per_million),
|
||||
):
|
||||
if tokens and rate is None:
|
||||
unpriced[label] = tokens
|
||||
elif rate is not None:
|
||||
native_cost += tokens * rate / 1_000_000
|
||||
no_cache_native = None
|
||||
if price.input_per_million is not None and price.output_per_million is not None:
|
||||
no_cache_native = (
|
||||
(uncached + cached) * price.input_per_million
|
||||
+ output * price.output_per_million
|
||||
) / 1_000_000
|
||||
priced = not unpriced and price.native_rates_complete
|
||||
usd_conversion = 1.0 if price.currency == "USD" else price.usd_per_currency_unit
|
||||
comparable_usd = priced and price.usd_conversion_complete
|
||||
measured_usd = native_cost * usd_conversion if comparable_usd else None
|
||||
no_cache_usd = (
|
||||
no_cache_native * usd_conversion
|
||||
if comparable_usd and no_cache_native is not None else None
|
||||
)
|
||||
result.append({
|
||||
"provider": provider_name,
|
||||
"model": model,
|
||||
"phase": phase,
|
||||
"requests": len(items),
|
||||
"uncached_input_tokens": uncached,
|
||||
"cached_input_tokens": cached,
|
||||
"output_tokens": output,
|
||||
"currency": price.currency,
|
||||
"measured_cost_native": native_cost if priced else None,
|
||||
"no_cache_counterfactual_native": no_cache_native if priced else None,
|
||||
"cache_savings_native": (
|
||||
no_cache_native - native_cost
|
||||
if priced and no_cache_native is not None else None
|
||||
),
|
||||
"usd_per_currency_unit": usd_conversion if comparable_usd else None,
|
||||
"measured_cost_usd": measured_usd,
|
||||
"no_cache_counterfactual_usd": no_cache_usd,
|
||||
"cache_savings_usd": (
|
||||
no_cache_usd - measured_usd
|
||||
if no_cache_usd is not None and measured_usd is not None else None
|
||||
),
|
||||
"unpriced_tokens": unpriced,
|
||||
"native_pricing_complete": priced,
|
||||
"pricing_complete": comparable_usd,
|
||||
"pricing_status": price.status,
|
||||
"pricing_blocker": price.blocker,
|
||||
})
|
||||
return result
|
||||
|
||||
|
||||
def external_benchmark_comparison(
|
||||
config: dict[str, Any],
|
||||
workload: Sequence[dict[str, Any]],
|
||||
) -> list[dict[str, Any]]:
|
||||
result = []
|
||||
for reference in config.get("external_benchmark_references", []):
|
||||
context = reference.get("context_tokens", 32768)
|
||||
output = reference.get("output_tokens", 512)
|
||||
measured = next((
|
||||
row for row in workload
|
||||
if row["provider"] == reference.get("provider")
|
||||
and row["model"] == reference.get("model")
|
||||
and row["target_context_tokens"] == context
|
||||
and row["target_output_tokens"] == output
|
||||
), None)
|
||||
reference_metrics = reference.get("metrics", {})
|
||||
measured_metrics = None
|
||||
deltas = None
|
||||
if measured:
|
||||
measured_metrics = {
|
||||
"ttft_p50_s": measured["ttft_s"]["p50"],
|
||||
"output_throughput_p50_tokens_s": measured["output_throughput_tokens_s"]["p50"],
|
||||
}
|
||||
deltas = {
|
||||
key: (
|
||||
measured_metrics[key] - float(reference_metrics[key])
|
||||
if measured_metrics.get(key) is not None and key in reference_metrics
|
||||
else None
|
||||
)
|
||||
for key in measured_metrics
|
||||
}
|
||||
result.append({
|
||||
"provider": reference.get("provider"),
|
||||
"model": reference.get("model"),
|
||||
"context_tokens": context,
|
||||
"output_tokens": output,
|
||||
"source_url": reference.get("source_url"),
|
||||
"as_of": reference.get("as_of"),
|
||||
"reference_metrics": reference_metrics,
|
||||
"measured_metrics": measured_metrics,
|
||||
"measured_minus_reference": deltas,
|
||||
})
|
||||
return result
|
||||
|
||||
|
||||
def completion_audit(
|
||||
config: dict[str, Any],
|
||||
workload: Sequence[dict[str, Any]],
|
||||
availability: Sequence[dict[str, Any]],
|
||||
rates: Sequence[dict[str, Any]],
|
||||
costs: Sequence[dict[str, Any]],
|
||||
observations: Sequence[dict[str, Any]],
|
||||
metadata: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
expected_contexts = set(config["workload"]["context_tokens"])
|
||||
expected_outputs = set(config["workload"]["output_tokens"])
|
||||
exact_workload_design = (
|
||||
expected_contexts == {8192, 32768, 131072}
|
||||
and expected_outputs == {512, 2048}
|
||||
and int(config["workload"].get("requests_per_cell", 0)) >= 100
|
||||
)
|
||||
exact_availability_design = (
|
||||
float(config["availability"].get("duration_hours", 0)) >= 168
|
||||
and float(config["availability"].get("interval_seconds", float("inf"))) <= 3600
|
||||
and int(config["availability"].get("requests_per_probe", 0)) >= 1
|
||||
)
|
||||
configured_rate_levels = list(config["rate_limit"].get("concurrency_levels", []))
|
||||
exact_rate_ramp_design = (
|
||||
len(configured_rate_levels) >= 2
|
||||
and configured_rate_levels == sorted(set(configured_rate_levels))
|
||||
and int(config["rate_limit"].get("requests_per_level", 0)) >= 100
|
||||
)
|
||||
workload_cells = {
|
||||
(row["provider"], row["target_context_tokens"], row["target_output_tokens"]): row
|
||||
for row in workload
|
||||
}
|
||||
providers = [raw["name"] for raw in config["providers"]]
|
||||
missing_cells = []
|
||||
undersampled = []
|
||||
unsuccessful_cells = []
|
||||
for provider in providers:
|
||||
for context in expected_contexts:
|
||||
for output in expected_outputs:
|
||||
row = workload_cells.get((provider, context, output))
|
||||
if row is None:
|
||||
missing_cells.append({"provider": provider, "context": context, "output": output})
|
||||
elif row["requests"] < max(
|
||||
100, int(config["workload"].get("requests_per_cell", 100))
|
||||
):
|
||||
undersampled.append({
|
||||
"provider": provider, "context": context, "output": output,
|
||||
"requests": row["requests"],
|
||||
})
|
||||
elif row["successes"] == 0:
|
||||
unsuccessful_cells.append({
|
||||
"provider": provider, "context": context, "output": output,
|
||||
})
|
||||
minimum_attainment = float(config["workload"].get("minimum_output_attainment_rate", 0.95))
|
||||
output_attainment_gaps = [
|
||||
{
|
||||
"provider": row["provider"],
|
||||
"context": row["target_context_tokens"],
|
||||
"output": row["target_output_tokens"],
|
||||
"attainment_rate": row.get("output_length_attainment_rate", 0.0),
|
||||
"required": minimum_attainment,
|
||||
}
|
||||
for row in workload
|
||||
if row.get("output_length_attainment_rate", 0.0) < minimum_attainment
|
||||
]
|
||||
availability_hours = []
|
||||
availability_by_provider = {row["provider"]: row for row in availability}
|
||||
for row in availability:
|
||||
if row["observed_start_utc"] and row["observed_end_utc"]:
|
||||
hours = (
|
||||
parse_time(row["observed_end_utc"]) - parse_time(row["observed_start_utc"])
|
||||
).total_seconds() / 3600
|
||||
availability_hours.append({"provider": row["provider"], "hours": hours})
|
||||
required_availability_hours = float(config["availability"]["duration_hours"])
|
||||
expected_probe_count = int(
|
||||
required_availability_hours * 3600
|
||||
/ float(config["availability"]["interval_seconds"])
|
||||
) + 1
|
||||
missing_availability_providers = sorted(
|
||||
set(providers) - set(availability_by_provider)
|
||||
)
|
||||
short_availability = [
|
||||
row for row in availability_hours
|
||||
if row["hours"] < required_availability_hours
|
||||
]
|
||||
undersampled_availability = [
|
||||
{
|
||||
"provider": provider,
|
||||
"probes": availability_by_provider[provider]["probes"],
|
||||
"required_probes": expected_probe_count,
|
||||
}
|
||||
for provider in providers
|
||||
if provider in availability_by_provider
|
||||
and availability_by_provider[provider]["probes"] < expected_probe_count
|
||||
]
|
||||
expected_levels = set(config["rate_limit"]["concurrency_levels"])
|
||||
measured_levels = defaultdict(set)
|
||||
undersampled_rate_levels = []
|
||||
for row in rates:
|
||||
measured_levels[row["provider"]].add(row["concurrency"])
|
||||
if row["requests"] < config["rate_limit"]["requests_per_level"]:
|
||||
undersampled_rate_levels.append({
|
||||
"provider": row["provider"], "concurrency": row["concurrency"],
|
||||
"requests": row["requests"],
|
||||
})
|
||||
missing_rate_levels = {
|
||||
provider: sorted(expected_levels - measured_levels[provider])
|
||||
for provider in providers if expected_levels - measured_levels[provider]
|
||||
}
|
||||
rate_limit_boundaries = {}
|
||||
missing_rate_limit_boundaries = []
|
||||
for provider in providers:
|
||||
provider_rows = sorted(
|
||||
(row for row in rates if row["provider"] == provider),
|
||||
key=lambda row: row["concurrency"],
|
||||
)
|
||||
boundary = next(
|
||||
(
|
||||
row for row in provider_rows
|
||||
if row.get("errors", {}).get("rate_limit", 0) > 0
|
||||
),
|
||||
None,
|
||||
)
|
||||
if boundary:
|
||||
rate_limit_boundaries[provider] = {
|
||||
"first_rate_limited_concurrency": boundary["concurrency"],
|
||||
"last_pre_limit_rpm": next(
|
||||
(
|
||||
row["measured_rpm"] for row in reversed(provider_rows)
|
||||
if row["concurrency"] < boundary["concurrency"]
|
||||
and row["successes"] > 0
|
||||
),
|
||||
None,
|
||||
),
|
||||
"last_pre_limit_input_tpm": next(
|
||||
(
|
||||
row["measured_input_tpm"] for row in reversed(provider_rows)
|
||||
if row["concurrency"] < boundary["concurrency"]
|
||||
and row["successes"] > 0
|
||||
),
|
||||
None,
|
||||
),
|
||||
}
|
||||
else:
|
||||
missing_rate_limit_boundaries.append(provider)
|
||||
cost_rounds = defaultdict(int)
|
||||
for row in observations:
|
||||
if row["phase"] == "agent_cost":
|
||||
cost_rounds[row["provider"]] += 1
|
||||
missing_cost_traces = {
|
||||
provider: config["agent_cost"]["rounds"] - cost_rounds[provider]
|
||||
for provider in providers if cost_rounds[provider] < config["agent_cost"]["rounds"]
|
||||
}
|
||||
incomplete_pricing = [
|
||||
row["provider"] for row in costs
|
||||
if row["phase"] == "agent_cost" and not row["pricing_complete"]
|
||||
]
|
||||
pricing_config_gaps = []
|
||||
for raw in config["providers"]:
|
||||
pricing = raw.get("pricing", {})
|
||||
missing = [
|
||||
field for field in (
|
||||
"input_per_million",
|
||||
"cached_input_per_million",
|
||||
"output_per_million",
|
||||
"currency",
|
||||
"source_url",
|
||||
"as_of",
|
||||
)
|
||||
if pricing.get(field) in (None, "")
|
||||
]
|
||||
currency = pricing.get("currency")
|
||||
if currency and currency != "USD":
|
||||
missing.extend(
|
||||
field for field in (
|
||||
"usd_per_currency_unit", "fx_source_url", "fx_as_of"
|
||||
)
|
||||
if pricing.get(field) in (None, "")
|
||||
)
|
||||
if missing:
|
||||
pricing_config_gaps.append({
|
||||
"provider": raw["name"],
|
||||
"missing": sorted(set(missing)),
|
||||
"status": pricing.get("status", "unresolved"),
|
||||
"blocker": pricing.get("blocker"),
|
||||
})
|
||||
elif (
|
||||
pricing.get("status") not in {"verified", "verified_native", "verified_with_fx"}
|
||||
or not str(pricing.get("source_url", "")).startswith(("https://", "http://"))
|
||||
or (currency != "USD" and not str(pricing.get("fx_source_url", "")).startswith(("https://", "http://")))
|
||||
):
|
||||
pricing_config_gaps.append({
|
||||
"provider": raw["name"],
|
||||
"missing": [],
|
||||
"status": pricing.get("status", "unresolved"),
|
||||
"blocker": pricing.get("blocker") or "pricing provenance/status is not validated",
|
||||
})
|
||||
missing_pricing_rows = [
|
||||
provider for provider in providers
|
||||
if not any(row["provider"] == provider and row["phase"] == "agent_cost" for row in costs)
|
||||
]
|
||||
thinking_providers = [
|
||||
raw["name"] for raw in config["providers"]
|
||||
if raw.get("thinking_budget_tokens") or any(
|
||||
tag in raw["model"].casefold() for tag in ("thinking", "reason", "kimi-k3", "gpt-5")
|
||||
)
|
||||
]
|
||||
thinking_measured = {
|
||||
provider for provider in thinking_providers
|
||||
if any(
|
||||
row["provider"] == provider
|
||||
and row["ok"]
|
||||
and (row["reasoning_tokens"] > 0 or row["thinking_ttft_s"] is not None)
|
||||
for row in observations
|
||||
)
|
||||
}
|
||||
missing_thinking_metrics = sorted(set(thinking_providers) - thinking_measured)
|
||||
successful_workload_rows = [
|
||||
row for row in observations if row["phase"] == "workload" and row["ok"]
|
||||
]
|
||||
missing_raw_evidence = [
|
||||
row["cell_id"] for row in successful_workload_rows
|
||||
if not row.get("prompt_sha256")
|
||||
or not row.get("output_sha256")
|
||||
or row.get("output_text") in (None, "")
|
||||
]
|
||||
required_families = set(config.get("required_model_families", []))
|
||||
observed_families = {
|
||||
raw.get("model_family") for raw in config["providers"] if raw.get("model_family")
|
||||
}
|
||||
missing_model_families = sorted(required_families - observed_families)
|
||||
external_reference_gaps = []
|
||||
references = config.get("external_benchmark_references", [])
|
||||
if not references:
|
||||
external_reference_gaps.append({"reason": "no dated external benchmark reference configured"})
|
||||
for reference in references:
|
||||
missing = [
|
||||
field for field in ("source_url", "as_of", "provider", "model", "metrics")
|
||||
if not reference.get(field)
|
||||
]
|
||||
metrics = reference.get("metrics") or {}
|
||||
missing.extend(
|
||||
f"metrics.{field}"
|
||||
for field in ("ttft_p50_s", "output_throughput_p50_tokens_s")
|
||||
if metrics.get(field) is None
|
||||
)
|
||||
if reference.get("source_url") and not str(reference["source_url"]).startswith(("https://", "http://")):
|
||||
missing.append("source_url_http")
|
||||
if missing:
|
||||
external_reference_gaps.append({
|
||||
"provider": reference.get("provider"),
|
||||
"model": reference.get("model"),
|
||||
"missing": missing,
|
||||
})
|
||||
continue
|
||||
matching = [
|
||||
row for row in workload
|
||||
if row["provider"] == reference["provider"]
|
||||
and row["model"] == reference["model"]
|
||||
and row["target_context_tokens"] == reference.get("context_tokens", 32768)
|
||||
and row["target_output_tokens"] == reference.get("output_tokens", 512)
|
||||
and row["successes"] > 0
|
||||
]
|
||||
if not matching:
|
||||
external_reference_gaps.append({
|
||||
"provider": reference["provider"],
|
||||
"model": reference["model"],
|
||||
"reason": "no successful matching workload cell for external comparison",
|
||||
})
|
||||
expected_fingerprint = execution_config_fingerprint(config)
|
||||
metadata_matches = bool(
|
||||
metadata
|
||||
and metadata.get("execution_config_fingerprint") == expected_fingerprint
|
||||
and metadata.get("prompt_schema_version") == PROMPT_SCHEMA_VERSION
|
||||
)
|
||||
same_model_provider_gaps = []
|
||||
for group in config.get("same_model_provider_groups", []):
|
||||
group_providers = group.get("providers", [])
|
||||
if len(group_providers) < 2:
|
||||
same_model_provider_gaps.append({
|
||||
"logical_model": group.get("logical_model"),
|
||||
"reason": "comparison group must name at least two providers",
|
||||
})
|
||||
continue
|
||||
for provider in group_providers:
|
||||
for context in expected_contexts:
|
||||
for output in expected_outputs:
|
||||
row = workload_cells.get((provider, context, output))
|
||||
if (
|
||||
row is None
|
||||
or row["requests"] < 100
|
||||
or row["successes"] == 0
|
||||
):
|
||||
same_model_provider_gaps.append({
|
||||
"logical_model": group.get("logical_model"),
|
||||
"provider": provider,
|
||||
"context": context,
|
||||
"output": output,
|
||||
"reason": "missing successful official workload cell",
|
||||
})
|
||||
checks = {
|
||||
"configuration_matches_exact_8k_32k_128k_x_512_2048_design": exact_workload_design,
|
||||
"all_8k_32k_128k_x_512_2048_cells_present": not missing_cells,
|
||||
"at_least_100_requests_per_workload_cell": not undersampled and not missing_cells,
|
||||
"at_least_one_success_per_workload_cell": not unsuccessful_cells and not missing_cells,
|
||||
"visible_output_length_target_attained": not output_attainment_gaps and not missing_cells,
|
||||
"availability_observed_for_at_least_168_hours": (
|
||||
not missing_availability_providers
|
||||
and not short_availability
|
||||
and not undersampled_availability
|
||||
),
|
||||
"availability_schedule_is_hourly_for_seven_days": exact_availability_design,
|
||||
"rate_limit_ramp_configuration_is_progressive": exact_rate_ramp_design,
|
||||
"all_rate_limit_levels_measured": not missing_rate_levels and not undersampled_rate_levels,
|
||||
"rate_limit_boundary_identified": not missing_rate_limit_boundaries,
|
||||
"multi_round_agent_cost_trace_present": not missing_cost_traces,
|
||||
"cached_input_output_pricing_complete": (
|
||||
not incomplete_pricing
|
||||
and not missing_pricing_rows
|
||||
and not pricing_config_gaps
|
||||
),
|
||||
"thinking_length_or_latency_measured": not missing_thinking_metrics,
|
||||
"raw_request_response_evidence_present": (
|
||||
bool(successful_workload_rows) and not missing_raw_evidence
|
||||
),
|
||||
"required_model_families_covered": bool(required_families) and not missing_model_families,
|
||||
"external_monitoring_reference_compared": not external_reference_gaps,
|
||||
"execution_config_fingerprint_matches": metadata_matches,
|
||||
"same_model_compared_across_providers": (
|
||||
bool(config.get("same_model_provider_groups"))
|
||||
and not same_model_provider_gaps
|
||||
),
|
||||
}
|
||||
return {
|
||||
"official_complete": all(checks.values()),
|
||||
"checks": checks,
|
||||
"missing_workload_cells": missing_cells,
|
||||
"undersampled_workload_cells": undersampled,
|
||||
"unsuccessful_workload_cells": unsuccessful_cells,
|
||||
"output_attainment_gaps": output_attainment_gaps,
|
||||
"availability_hours": availability_hours,
|
||||
"missing_availability_providers": missing_availability_providers,
|
||||
"short_availability": short_availability,
|
||||
"undersampled_availability": undersampled_availability,
|
||||
"missing_rate_levels": missing_rate_levels,
|
||||
"undersampled_rate_levels": undersampled_rate_levels,
|
||||
"rate_limit_boundaries": rate_limit_boundaries,
|
||||
"missing_rate_limit_boundaries": missing_rate_limit_boundaries,
|
||||
"missing_cost_traces": missing_cost_traces,
|
||||
"incomplete_pricing": incomplete_pricing,
|
||||
"pricing_config_gaps": pricing_config_gaps,
|
||||
"missing_pricing_rows": missing_pricing_rows,
|
||||
"missing_thinking_metrics": missing_thinking_metrics,
|
||||
"missing_raw_evidence_cell_ids": missing_raw_evidence,
|
||||
"missing_model_families": missing_model_families,
|
||||
"external_reference_gaps": external_reference_gaps,
|
||||
"expected_execution_config_fingerprint": expected_fingerprint,
|
||||
"observed_campaign_metadata": metadata,
|
||||
"same_model_provider_gaps": same_model_provider_gaps,
|
||||
}
|
||||
|
||||
|
||||
def load_rows(connection: sqlite3.Connection, campaign_id: str, table: str) -> list[dict[str, Any]]:
|
||||
rows = connection.execute(
|
||||
f"SELECT * FROM {table} WHERE campaign_id = ?", (campaign_id,)
|
||||
).fetchall()
|
||||
return [dict(row) for row in rows]
|
||||
|
||||
|
||||
def analyze(db: Path, config_path: Path, campaign_id: str) -> dict[str, Any]:
|
||||
config = load_config(config_path)
|
||||
providers = {
|
||||
raw["name"]: Provider.from_dict(dict(raw)) for raw in config["providers"]
|
||||
}
|
||||
connection = sqlite3.connect(db)
|
||||
connection.row_factory = sqlite3.Row
|
||||
try:
|
||||
observations = load_rows(connection, campaign_id, "observations")
|
||||
batches = load_rows(connection, campaign_id, "batches")
|
||||
try:
|
||||
metadata_row = connection.execute(
|
||||
"SELECT * FROM campaign_metadata WHERE campaign_id = ?", (campaign_id,)
|
||||
).fetchone()
|
||||
except sqlite3.OperationalError:
|
||||
metadata_row = None
|
||||
metadata = dict(metadata_row) if metadata_row else None
|
||||
finally:
|
||||
connection.close()
|
||||
workload = summarize_workloads([row for row in observations if row["phase"] == "workload"])
|
||||
availability = availability_summary([row for row in observations if row["phase"] == "availability"])
|
||||
rate_observations = [row for row in observations if row["phase"] == "rate_limit"]
|
||||
rates = rate_limit_summary(
|
||||
[row for row in batches if row["phase"] == "rate_limit"],
|
||||
rate_observations,
|
||||
)
|
||||
costs = cost_summary(observations, providers)
|
||||
external_comparison = external_benchmark_comparison(config, workload)
|
||||
return {
|
||||
"schema_version": "1.0",
|
||||
"campaign_id": campaign_id,
|
||||
"database": str(db),
|
||||
"campaign_metadata": metadata,
|
||||
"observation_count": len(observations),
|
||||
"workload": workload,
|
||||
"availability": availability,
|
||||
"rate_limits": rates,
|
||||
"costs": costs,
|
||||
"external_benchmark_comparison": external_comparison,
|
||||
"completion_audit": completion_audit(
|
||||
config, workload, availability, rates, costs, observations, metadata
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def fmt(value: Any, digits: int = 3) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
if isinstance(value, float):
|
||||
return f"{value:.{digits}f}"
|
||||
return str(value)
|
||||
|
||||
|
||||
def fmt_pct(value: float | None, digits: int = 1) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
return f"{value:.{digits}%}"
|
||||
|
||||
|
||||
def markdown(report: dict[str, Any]) -> str:
|
||||
lines = [
|
||||
f"# Experiment 7-10 campaign: `{report['campaign_id']}`",
|
||||
"",
|
||||
f"Observations: **{report['observation_count']}**",
|
||||
f"Official completion: **{report['completion_audit']['official_complete']}**",
|
||||
"",
|
||||
"## Standard workloads",
|
||||
"",
|
||||
"| Provider | Context | Output | N | Success | TTFT p50/p95/p99 (s) | E2E p50/p95/p99 (s) | Input tok/s p50 | Visible output tok/s p50 | Output attainment | Reasoning tok p50 | Thinking TTFT p50 (s) |",
|
||||
"|---|---:|---:|---:|---:|---|---|---:|---:|---:|---:|---:|",
|
||||
]
|
||||
for row in report["workload"]:
|
||||
ttft, e2e = row["ttft_s"], row["e2e_s"]
|
||||
lines.append(
|
||||
f"| {row['provider']} | {row['target_context_tokens']} | {row['target_output_tokens']} | "
|
||||
f"{row['requests']} | {fmt_pct(row['success_rate'], 1)} | "
|
||||
f"{fmt(ttft['p50'])}/{fmt(ttft['p95'])}/{fmt(ttft['p99'])} | "
|
||||
f"{fmt(e2e['p50'])}/{fmt(e2e['p95'])}/{fmt(e2e['p99'])} | "
|
||||
f"{fmt(row['input_prefill_throughput_tokens_s']['p50'], 1)} | "
|
||||
f"{fmt(row['output_throughput_tokens_s']['p50'], 1)} | "
|
||||
f"{fmt_pct(row['output_length_attainment_rate'], 1)} | "
|
||||
f"{fmt(row['reasoning_tokens']['p50'], 1)} | "
|
||||
f"{fmt(row['thinking_ttft_s']['p50'])} |"
|
||||
)
|
||||
lines.extend([
|
||||
"", "## Availability", "",
|
||||
"| Provider | Probes | Uptime | Outages | MTTR (s) | Longest available (h) |",
|
||||
"|---|---:|---:|---:|---:|---:|",
|
||||
])
|
||||
for row in report["availability"]:
|
||||
lines.append(
|
||||
f"| {row['provider']} | {row['probes']} | {fmt_pct(row['uptime'], 2)} | "
|
||||
f"{row['outage_count']} | {fmt(row['mttr_s'])} | "
|
||||
f"{row['longest_continuous_availability_s'] / 3600:.2f} |"
|
||||
)
|
||||
lines.extend([
|
||||
"", "## Measured rate-limit ramp", "",
|
||||
"| Provider | Concurrency | Success | RPM | Input TPM | Output TPM |",
|
||||
"|---|---:|---:|---:|---:|---:|",
|
||||
])
|
||||
for row in report["rate_limits"]:
|
||||
lines.append(
|
||||
f"| {row['provider']} | {row['concurrency']} | {fmt_pct(row['success_rate'], 1)} | "
|
||||
f"{fmt(row['measured_rpm'], 1)} | {fmt(row['measured_input_tpm'], 1)} | "
|
||||
f"{fmt(row['measured_output_tpm'], 1)} |"
|
||||
)
|
||||
lines.extend([
|
||||
"", "## Native and comparable costs", "",
|
||||
"| Provider | Phase | Requests | Native cost | Currency | USD cost | Cache savings (native) | Unpriced usage |",
|
||||
"|---|---|---:|---:|---|---:|---:|---|",
|
||||
])
|
||||
for row in report["costs"]:
|
||||
lines.append(
|
||||
f"| {row['provider']} | {row['phase']} | {row['requests']} | "
|
||||
f"{fmt(row['measured_cost_native'], 6)} | {row['currency'] or '—'} | "
|
||||
f"{fmt(row['measured_cost_usd'], 6)} | "
|
||||
f"{fmt(row['cache_savings_native'], 6)} | "
|
||||
f"{json.dumps(row['unpriced_tokens'], ensure_ascii=False)} |"
|
||||
)
|
||||
lines.extend(["", "## External monitoring comparison", ""])
|
||||
if report["external_benchmark_comparison"]:
|
||||
lines.extend(["```json", json.dumps(
|
||||
report["external_benchmark_comparison"], ensure_ascii=False, indent=2
|
||||
), "```"])
|
||||
else:
|
||||
lines.append("No dated external monitoring reference is configured; this manuscript gate is incomplete.")
|
||||
lines.extend([
|
||||
"", "## Completion audit", "", "```json",
|
||||
json.dumps(report["completion_audit"], ensure_ascii=False, indent=2),
|
||||
"```", "",
|
||||
])
|
||||
return "\n".join(lines)
|
||||
def export_campaign_summary(
|
||||
db_path: Path, config_path: Path | None = None, campaign_id: str = "experiment-7-10"
|
||||
) -> dict[str, Any]:
|
||||
"""Export structured JSON and Markdown summary metrics for a campaign database."""
|
||||
cfg = config_path or DEFAULT_CONFIG
|
||||
report = analyze(db_path, cfg, campaign_id)
|
||||
md_text = markdown(report)
|
||||
return {
|
||||
"report": report,
|
||||
"markdown": md_text,
|
||||
"official_complete": report["completion_audit"]["official_complete"],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Analyze full Experiment 7-10 campaign evidence")
|
||||
parser.add_argument("--db", type=Path, default=DEFAULT_DB)
|
||||
parser.add_argument("--config", type=Path, default=DEFAULT_CONFIG)
|
||||
parser.add_argument("--campaign-id", default="experiment-7-10")
|
||||
parser.add_argument("--json", type=Path, default=HERE / "results" / "campaign_report.json")
|
||||
parser.add_argument("--markdown", type=Path, default=HERE / "results" / "campaign_report.md")
|
||||
args = parser.parse_args()
|
||||
report = analyze(args.db, args.config, args.campaign_id)
|
||||
args.json.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.markdown.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.json.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
args.markdown.write_text(markdown(report), encoding="utf-8")
|
||||
print(f"Wrote {args.json} and {args.markdown}")
|
||||
print(f"Official completion: {report['completion_audit']['official_complete']}")
|
||||
return 0 if report["completion_audit"]["official_complete"] else 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
多维度模型性能基准测试（实验 7-10 配套代码）
|
||||
|
||||
对多个 OpenAI 兼容的 LLM API 提供商，测量以下核心指标：
|
||||
- TTFT（Time To First Token，首个 token 到达延迟）
|
||||
- 端到端延迟（发出请求到接收完整响应）
|
||||
- 吞吐（tokens/s，按生成的输出 token 计；并发下另给聚合吞吐 / RPS）
|
||||
- 标准差 / p50 / p95 / p99 延迟分位数（方差大意味着体验不稳定）
|
||||
- 可用性 / 成功率（失败即计入可用性下降，不中断整表）
|
||||
|
||||
支持两种模式：
|
||||
- 单档位对比：多提供商横向对比表（默认）。
|
||||
- 并发扫描（压测）：对同一模型逐步提升并发，观察延迟长尾与聚合吞吐随并发的变化。
|
||||
|
||||
实现要点：
|
||||
- 使用 openai SDK 的流式接口（stream=True）来精确测量 TTFT。
|
||||
- 通过 base_url 复用同一套 OpenAI 兼容协议，适配 Kimi / 豆包等国产 API。
|
||||
- 单点请求失败被捕获并记录，不影响同一 (provider, model) 的其它请求，
|
||||
也不影响其它 provider —— 这样一次运行就能测出"可用性"这一维度。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
import random
|
||||
import statistics
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# OpenRouter 回退：对「OpenAI 原生」条目（base_url 为空）在缺主 key 时改走 OpenRouter。
|
||||
# gpt-5.x 直连 OpenAI 需组织实名认证，只要有 OPENROUTER_API_KEY 就优先走 OpenRouter。
|
||||
# ---------------------------------------------------------------------------
|
||||
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
|
||||
|
||||
def _to_openrouter_model(model: str) -> str:
|
||||
"""把模型名映射成 OpenRouter id：含 '/' 视为原生 id；gpt-* -> openai/*；
|
||||
claude-* -> anthropic/claude-opus-4.8；其余回退到 openai/gpt-5.6-luna。"""
|
||||
if "/" in model:
|
||||
return model
|
||||
if model.startswith("gpt-"):
|
||||
return "openai/" + model
|
||||
if model.startswith("claude-"):
|
||||
return "anthropic/claude-opus-4.8"
|
||||
return "openai/gpt-5.6-luna"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 提供商配置
|
||||
# ---------------------------------------------------------------------------
|
||||
@dataclass
|
||||
class ProviderConfig:
|
||||
"""单个待测 (提供商, 模型) 配置。"""
|
||||
|
||||
name: str # 展示名，例如 "OpenAI/gpt-5.6-luna"
|
||||
model: str # 传给 API 的模型名
|
||||
api_key_env: str # 读取 API key 的环境变量名
|
||||
base_url: Optional[str] = None # OpenAI 官方留空；其它填各自 base_url
|
||||
|
||||
def api_key(self) -> Optional[str]:
|
||||
return os.environ.get(self.api_key_env)
|
||||
|
||||
def _openrouter_key(self) -> Optional[str]:
|
||||
return os.environ.get("OPENROUTER_API_KEY", "").strip() or None
|
||||
|
||||
def resolve(self) -> tuple[Optional[str], Optional[str], str, bool]:
|
||||
"""解析实际使用的 (api_key, base_url, model, 是否经 OpenRouter)。
|
||||
|
||||
仅「OpenAI 原生」条目（base_url 为空）参与回退；带专属 base_url 的条目
|
||||
（如 Kimi/豆包）保持不变。回退规则：
|
||||
- gpt-5.x 且有 OPENROUTER_API_KEY -> 优先走 OpenRouter（直连需实名认证）；
|
||||
- 否则主 key 存在 -> 直连，模型名不变；
|
||||
- 否则（OpenAI 原生 + 有 OPENROUTER_API_KEY）-> 走 OpenRouter，模型名映射。
|
||||
"""
|
||||
primary = self.api_key()
|
||||
openai_native = self.base_url is None
|
||||
orkey = self._openrouter_key() if openai_native else None
|
||||
prefer_or = bool(orkey) and self.model.startswith("gpt-5")
|
||||
|
||||
if not prefer_or and primary:
|
||||
return primary, self.base_url, self.model, False
|
||||
if orkey:
|
||||
return orkey, OPENROUTER_BASE_URL, _to_openrouter_model(self.model), True
|
||||
return primary, self.base_url, self.model, False
|
||||
|
||||
def is_available(self) -> bool:
|
||||
"""主 key 存在即可测；OpenAI 原生条目在缺主 key 时可回退 OpenRouter。"""
|
||||
if self.api_key():
|
||||
return True
|
||||
return self.base_url is None and self._openrouter_key() is not None
|
||||
|
||||
|
||||
# 默认只跑"手上有有效 key"的三家提供商。
|
||||
# 需要扩展时，往这里追加 ProviderConfig 即可（例如 DeepSeek 官方 vs SiliconFlow 对比）。
|
||||
DEFAULT_PROVIDERS: list[ProviderConfig] = [
|
||||
# OpenAI 官方（一个 key 测多个模型，观察同厂不同规格的差异）
|
||||
# gpt-5.6-luna 为当前廉价旗舰；无 OPENAI_API_KEY 时自动经 OpenRouter 路由
|
||||
# （openai/gpt-5.6-luna），gpt-5.x 只要有 OPENROUTER_API_KEY 就优先走 OpenRouter。
|
||||
ProviderConfig(
|
||||
name="OpenAI/gpt-5.6-luna",
|
||||
model="gpt-5.6-luna",
|
||||
api_key_env="OPENAI_API_KEY",
|
||||
),
|
||||
# 月之暗面 Kimi（OpenAI 兼容）
|
||||
ProviderConfig(
|
||||
name="Moonshot/moonshot-v1-8k",
|
||||
model="moonshot-v1-8k",
|
||||
api_key_env="MOONSHOT_API_KEY",
|
||||
base_url="https://api.moonshot.cn/v1",
|
||||
),
|
||||
# 字节豆包 / 火山方舟（OpenAI 兼容）
|
||||
ProviderConfig(
|
||||
name="Doubao/doubao-1.5-pro-32k",
|
||||
model="doubao-1-5-pro-32k-250115",
|
||||
api_key_env="ARK_API_KEY",
|
||||
base_url="https://ark.cn-beijing.volces.com/api/v3",
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 单次请求测量
|
||||
# ---------------------------------------------------------------------------
|
||||
@dataclass
|
||||
class RequestResult:
|
||||
"""一次流式请求的测量结果。"""
|
||||
|
||||
ok: bool
|
||||
ttft: Optional[float] = None # 首 token 延迟（秒）
|
||||
latency: Optional[float] = None # 端到端延迟（秒）
|
||||
completion_tokens: Optional[int] = None # 生成的输出 token 数
|
||||
throughput: Optional[float] = None # 输出吞吐（tokens/s）
|
||||
error: Optional[str] = None # 失败原因（可用性下降时记录）
|
||||
|
||||
|
||||
def measure_once(
|
||||
client: OpenAI,
|
||||
model: str,
|
||||
prompt: str,
|
||||
max_tokens: int,
|
||||
timeout: float,
|
||||
) -> RequestResult:
|
||||
"""
|
||||
发起一次流式请求并测量各项指标。
|
||||
|
||||
任何异常都被捕获为一次"失败"，用于统计可用性 —— 绝不向上抛出，
|
||||
以免单点故障中断整表测试。
|
||||
"""
|
||||
start = time.perf_counter()
|
||||
first_token_at: Optional[float] = None
|
||||
completion_tokens = 0
|
||||
try:
|
||||
stream = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
max_tokens=max_tokens,
|
||||
temperature=0.0,
|
||||
stream=True,
|
||||
# 请求用量统计（部分 OpenAI 兼容服务支持；不支持时下方回退到计数）
|
||||
stream_options={"include_usage": True},
|
||||
timeout=timeout,
|
||||
)
|
||||
|
||||
reported_tokens: Optional[int] = None
|
||||
for chunk in stream:
|
||||
# 首个"有内容"的 chunk 到达时刻即 TTFT
|
||||
if chunk.choices:
|
||||
delta = chunk.choices[0].delta
|
||||
content = getattr(delta, "content", None)
|
||||
if content:
|
||||
if first_token_at is None:
|
||||
first_token_at = time.perf_counter()
|
||||
completion_tokens += 1 # 回退计数：以流式 chunk 近似 token 数
|
||||
# 若服务在末尾回传了精确 usage，则以其为准
|
||||
usage = getattr(chunk, "usage", None)
|
||||
if usage is not None:
|
||||
reported_tokens = getattr(usage, "completion_tokens", None)
|
||||
|
||||
end = time.perf_counter()
|
||||
|
||||
if first_token_at is None:
|
||||
# 拿到了响应但没有任何内容 token，视为失败
|
||||
return RequestResult(ok=False, error="empty response (no content token)")
|
||||
|
||||
final_tokens = reported_tokens if reported_tokens else completion_tokens
|
||||
latency = end - start
|
||||
ttft = first_token_at - start
|
||||
# 吞吐按"生成阶段"计：输出 token 数 / (端到端 - 首 token 延迟)
|
||||
gen_time = max(latency - ttft, 1e-6)
|
||||
throughput = final_tokens / gen_time if final_tokens else 0.0
|
||||
|
||||
return RequestResult(
|
||||
ok=True,
|
||||
ttft=ttft,
|
||||
latency=latency,
|
||||
completion_tokens=final_tokens,
|
||||
throughput=throughput,
|
||||
)
|
||||
except Exception as exc: # noqa: BLE001 —— 故意兜底，任何错误都记为可用性下降
|
||||
return RequestResult(ok=False, error=f"{type(exc).__name__}: {exc}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 聚合结果
|
||||
# ---------------------------------------------------------------------------
|
||||
@dataclass
|
||||
class ProviderSummary:
|
||||
provider: str
|
||||
model: str
|
||||
total: int
|
||||
success: int
|
||||
results: list[RequestResult] = field(default_factory=list)
|
||||
errors: list[str] = field(default_factory=list)
|
||||
concurrency: int = 1 # 本次批次使用的并发数（并发扫描时用于标注行）
|
||||
wall_time: float = 0.0 # 整批请求的墙钟耗时（秒），用于算聚合吞吐/RPS
|
||||
|
||||
@property
|
||||
def availability(self) -> float:
|
||||
return self.success / self.total if self.total else 0.0
|
||||
|
||||
@property
|
||||
def rps(self) -> Optional[float]:
|
||||
"""吞吐（请求/秒）：成功请求数 / 整批墙钟耗时。并发越高一般越大，直到触顶。"""
|
||||
if self.wall_time <= 0:
|
||||
return None
|
||||
return self.success / self.wall_time
|
||||
|
||||
@property
|
||||
def agg_throughput(self) -> Optional[float]:
|
||||
"""聚合输出吞吐（tokens/s）：全部成功请求的输出 token 总数 / 整批墙钟耗时。"""
|
||||
if self.wall_time <= 0:
|
||||
return None
|
||||
total_tokens = sum(
|
||||
r.completion_tokens for r in self.results
|
||||
if r.ok and r.completion_tokens
|
||||
)
|
||||
return total_tokens / self.wall_time if total_tokens else 0.0
|
||||
|
||||
def _vals(self, attr: str) -> list[float]:
|
||||
return [getattr(r, attr) for r in self.results if r.ok and getattr(r, attr) is not None]
|
||||
|
||||
@staticmethod
|
||||
def _pct(values: list[float], q: float) -> Optional[float]:
|
||||
"""线性插值分位数；样本过少时退化为最大/最小值。"""
|
||||
if not values:
|
||||
return None
|
||||
s = sorted(values)
|
||||
if len(s) == 1:
|
||||
return s[0]
|
||||
pos = q * (len(s) - 1)
|
||||
lo = int(pos)
|
||||
hi = min(lo + 1, len(s) - 1)
|
||||
frac = pos - lo
|
||||
return s[lo] + (s[hi] - s[lo]) * frac
|
||||
|
||||
def stat(self, attr: str, kind: str) -> Optional[float]:
|
||||
vals = self._vals(attr)
|
||||
if not vals:
|
||||
return None
|
||||
if kind == "mean":
|
||||
return statistics.mean(vals)
|
||||
if kind == "std":
|
||||
# 标准差：样本 <2 时无从谈起，返回 0 而非报错
|
||||
return statistics.stdev(vals) if len(vals) >= 2 else 0.0
|
||||
if kind == "p50":
|
||||
return self._pct(vals, 0.50)
|
||||
if kind == "p95":
|
||||
return self._pct(vals, 0.95)
|
||||
if kind == "p99":
|
||||
return self._pct(vals, 0.99)
|
||||
raise ValueError(kind)
|
||||
|
||||
|
||||
def benchmark_provider(
|
||||
cfg: ProviderConfig,
|
||||
prompt: str,
|
||||
num_requests: int,
|
||||
concurrency: int,
|
||||
max_tokens: int,
|
||||
timeout: float,
|
||||
) -> ProviderSummary:
|
||||
"""对单个提供商发起 num_requests 次请求（并发 concurrency）。"""
|
||||
# 这是延迟基准：显式关闭 SDK 自动重试（max_retries=0），让一次超时/挂起的
|
||||
# 请求被如实记为「失败」（计入可用性下降），而不是被静默重试从而拉高延迟、
|
||||
# 掩盖真实故障。每次请求仍带 per-call timeout（见 measure_once）。
|
||||
# 再加一个客户端级 timeout 作为兜底，避免个别请求永久挂起拖死线程池。
|
||||
# 解析实际使用的凭据/端点/模型（OpenAI 原生条目缺 key 时回退 OpenRouter）。
|
||||
api_key, base_url, model, via_openrouter = cfg.resolve()
|
||||
if via_openrouter:
|
||||
print(f" （回退 OpenRouter：{cfg.model} -> {model}）", flush=True)
|
||||
client = OpenAI(
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
timeout=timeout,
|
||||
max_retries=0,
|
||||
)
|
||||
results: list[RequestResult] = []
|
||||
|
||||
batch_start = time.perf_counter()
|
||||
if concurrency <= 1:
|
||||
for _ in range(num_requests):
|
||||
results.append(measure_once(client, model, prompt, max_tokens, timeout))
|
||||
else:
|
||||
with ThreadPoolExecutor(max_workers=concurrency) as pool:
|
||||
futures = [
|
||||
pool.submit(measure_once, client, model, prompt, max_tokens, timeout)
|
||||
for _ in range(num_requests)
|
||||
]
|
||||
for fut in as_completed(futures):
|
||||
results.append(fut.result())
|
||||
wall_time = time.perf_counter() - batch_start
|
||||
|
||||
success = sum(1 for r in results if r.ok)
|
||||
errors = [r.error for r in results if not r.ok and r.error]
|
||||
return ProviderSummary(
|
||||
provider=cfg.name,
|
||||
model=model,
|
||||
total=num_requests,
|
||||
success=success,
|
||||
results=results,
|
||||
errors=errors,
|
||||
concurrency=concurrency,
|
||||
wall_time=wall_time,
|
||||
)
|
||||
|
||||
|
||||
def run_benchmark(
|
||||
providers: list[ProviderConfig],
|
||||
prompt: str,
|
||||
num_requests: int,
|
||||
concurrency: int,
|
||||
max_tokens: int,
|
||||
timeout: float,
|
||||
) -> list[ProviderSummary]:
|
||||
"""依次对每个提供商跑基准测试（提供商之间串行，单提供商内部并发）。"""
|
||||
summaries: list[ProviderSummary] = []
|
||||
for cfg in providers:
|
||||
print(f" → 正在测试 {cfg.name} "
|
||||
f"(model={cfg.model}, N={num_requests}, 并发={concurrency}) ...", flush=True)
|
||||
summary = benchmark_provider(
|
||||
cfg, prompt, num_requests, concurrency, max_tokens, timeout
|
||||
)
|
||||
print(f" 完成：成功 {summary.success}/{summary.total}", flush=True)
|
||||
summaries.append(summary)
|
||||
return summaries
|
||||
|
||||
|
||||
def sweep_concurrency(
|
||||
cfg: ProviderConfig,
|
||||
prompt: str,
|
||||
num_requests: int,
|
||||
concurrency_levels: list[int],
|
||||
max_tokens: int,
|
||||
timeout: float,
|
||||
) -> list[ProviderSummary]:
|
||||
"""
|
||||
压测：对同一 (provider, model) 逐步提升并发，返回每个并发档位的汇总。
|
||||
|
||||
对应书中"通过逐步提升并发量来找到限流点，记录 RPM/TPM 上限"——
|
||||
随着并发上升，单请求延迟（p95）会变差、可用性可能因限流而下降，
|
||||
而聚合吞吐（RPS / tokens·s⁻¹）会先升后平（触及服务端上限即触顶）。
|
||||
"""
|
||||
summaries: list[ProviderSummary] = []
|
||||
for c in concurrency_levels:
|
||||
print(f" → {cfg.name} @ 并发={c} (N={num_requests}) ...", flush=True)
|
||||
summary = benchmark_provider(cfg, prompt, num_requests, c, max_tokens, timeout)
|
||||
print(f" 完成：成功 {summary.success}/{summary.total}, "
|
||||
f"墙钟 {summary.wall_time:.2f}s", flush=True)
|
||||
summaries.append(summary)
|
||||
return summaries
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 合成（synthetic）数据：仅供离线演示指标聚合，绝非真实基准
|
||||
# ---------------------------------------------------------------------------
|
||||
def synthetic_summary(
|
||||
provider: str,
|
||||
model: str,
|
||||
num_requests: int,
|
||||
concurrency: int,
|
||||
*,
|
||||
base_ttft: float = 0.30,
|
||||
base_gen_throughput: float = 90.0,
|
||||
fail_rate: float = 0.0,
|
||||
seed: int = 0,
|
||||
) -> ProviderSummary:
|
||||
"""
|
||||
用伪随机数生成一批"看起来像真实测量"的 RequestResult，用于：
|
||||
1) 在没有 API key / 没有网络时验证指标聚合数学（p50/p95/p99/std/可用性）；
|
||||
2) 演示并发上升时延迟长尾变差、可用性可能下降的趋势。
|
||||
|
||||
⚠️ 生成的所有数字都是合成的，不代表任何真实模型/提供商的性能。
|
||||
并发越高，用一个简单的排队模型抬高 TTFT 与端到端延迟，仅为呈现趋势。
|
||||
"""
|
||||
rng = random.Random(seed + concurrency * 1000)
|
||||
# 并发放大系数：并发越高，排队等待越久（简单线性 + 抖动模型）
|
||||
contention = 1.0 + 0.12 * max(concurrency - 1, 0)
|
||||
|
||||
results: list[RequestResult] = []
|
||||
total_tokens = 0
|
||||
sum_latency = 0.0
|
||||
for _ in range(num_requests):
|
||||
# 高并发下失败率随之升高（模拟限流），封顶 60%
|
||||
eff_fail = min(fail_rate * contention, 0.60)
|
||||
if rng.random() < eff_fail:
|
||||
results.append(RequestResult(ok=False, error="synthetic: rate_limited (429)"))
|
||||
continue
|
||||
# TTFT：对数正态形状，右偏（长尾），再乘并发放大
|
||||
ttft = base_ttft * contention * rng.lognormvariate(0.0, 0.35)
|
||||
gen_tp = max(base_gen_throughput * rng.uniform(0.75, 1.15), 1.0)
|
||||
tokens = rng.randint(28, 48)
|
||||
gen_time = tokens / gen_tp
|
||||
latency = ttft + gen_time
|
||||
total_tokens += tokens
|
||||
sum_latency += latency
|
||||
results.append(RequestResult(
|
||||
ok=True,
|
||||
ttft=ttft,
|
||||
latency=latency,
|
||||
completion_tokens=tokens,
|
||||
throughput=gen_tp,
|
||||
))
|
||||
|
||||
success = sum(1 for r in results if r.ok)
|
||||
# 合成墙钟：把成功请求的总延迟按并发均摊，得到一个自洽的批次耗时
|
||||
wall_time = max(sum_latency / max(concurrency, 1), 1e-6)
|
||||
errors = [r.error for r in results if not r.ok and r.error]
|
||||
return ProviderSummary(
|
||||
provider=provider,
|
||||
model=model,
|
||||
total=num_requests,
|
||||
success=success,
|
||||
results=results,
|
||||
errors=errors,
|
||||
concurrency=concurrency,
|
||||
wall_time=wall_time,
|
||||
)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,242 @@
|
||||
{
|
||||
"providers": [
|
||||
{
|
||||
"name": "OpenAI/gpt-5.6-luna",
|
||||
"model": "gpt-5.6-luna",
|
||||
"api_key_env": "OPENAI_API_KEY",
|
||||
"base_url": "https://api.openai.com/v1",
|
||||
"protocol": "openai",
|
||||
"model_family": "GPT",
|
||||
"access_class": "closed",
|
||||
"max_output_field": "max_completion_tokens",
|
||||
"thinking_budget_tokens": 0,
|
||||
"pricing": {
|
||||
"input_per_million": 1.0,
|
||||
"cached_input_per_million": 0.1,
|
||||
"output_per_million": 6.0,
|
||||
"currency": "USD",
|
||||
"source_url": "https://openai.com/api/pricing/",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "verified"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Anthropic/claude-opus-4.8",
|
||||
"model": "claude-opus-4-8",
|
||||
"api_key_env": "ANTHROPIC_API_KEY",
|
||||
"protocol": "anthropic",
|
||||
"model_family": "Claude",
|
||||
"access_class": "closed",
|
||||
"thinking_budget_tokens": 1024,
|
||||
"pricing": {
|
||||
"input_per_million": 5.0,
|
||||
"cached_input_per_million": 0.5,
|
||||
"output_per_million": 25.0,
|
||||
"currency": "USD",
|
||||
"source_url": "https://platform.claude.com/docs/en/about-claude/pricing",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "verified"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Google/gemini-3.5-flash",
|
||||
"model": "gemini-3.5-flash",
|
||||
"api_key_env": "GEMINI_API_KEY",
|
||||
"protocol": "gemini",
|
||||
"model_family": "Gemini",
|
||||
"access_class": "closed",
|
||||
"thinking_budget_tokens": 1024,
|
||||
"pricing": {
|
||||
"input_per_million": 1.5,
|
||||
"cached_input_per_million": 0.15,
|
||||
"output_per_million": 9.0,
|
||||
"currency": "USD",
|
||||
"source_url": "https://ai.google.dev/gemini-api/docs/pricing",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "verified"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Moonshot/kimi-k3",
|
||||
"model": "kimi-k3",
|
||||
"api_key_env": "MOONSHOT_API_KEY",
|
||||
"base_url": "https://api.moonshot.cn/v1",
|
||||
"protocol": "openai",
|
||||
"model_family": "Kimi",
|
||||
"access_class": "open-weight",
|
||||
"thinking_budget_tokens": 1024,
|
||||
"extra_body": {"reasoning_effort": "low"},
|
||||
"pricing": {
|
||||
"input_per_million": 20.0,
|
||||
"cached_input_per_million": 2.0,
|
||||
"output_per_million": 100.0,
|
||||
"currency": "CNY",
|
||||
"source_url": "https://platform.kimi.com/docs/pricing/chat-k3.md",
|
||||
"as_of": "2026-07-29",
|
||||
"usd_per_currency_unit": 0.1477922077922078,
|
||||
"fx_source_url": "https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml",
|
||||
"fx_as_of": "2026-07-29",
|
||||
"status": "verified_with_fx"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Ark/doubao-seed-1.6",
|
||||
"model": "doubao-seed-1-6-250615",
|
||||
"api_key_env": "ARK_API_KEY",
|
||||
"base_url": "https://ark.cn-beijing.volces.com/api/v3",
|
||||
"protocol": "openai",
|
||||
"model_family": "Doubao",
|
||||
"access_class": "closed",
|
||||
"thinking_budget_tokens": 0,
|
||||
"extra_body": {"thinking": {"type": "disabled"}},
|
||||
"pricing": {
|
||||
"input_per_million": null,
|
||||
"cached_input_per_million": null,
|
||||
"output_per_million": null,
|
||||
"currency": "CNY",
|
||||
"source_url": "https://www.volcengine.com/docs/82379/1099320",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "unresolved",
|
||||
"blocker": "The exact public cached-input and output price for endpoint doubao-seed-1-6-250615 has not been extracted from an authoritative model row."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Ark/Qwen3-32B",
|
||||
"model": "qwen3-32b-20250429",
|
||||
"api_key_env": "ARK_API_KEY",
|
||||
"base_url": "https://ark.cn-beijing.volces.com/api/v3",
|
||||
"protocol": "openai",
|
||||
"model_family": "Qwen",
|
||||
"access_class": "open-weight",
|
||||
"thinking_budget_tokens": 0,
|
||||
"pricing": {
|
||||
"input_per_million": null,
|
||||
"cached_input_per_million": null,
|
||||
"output_per_million": null,
|
||||
"currency": "CNY",
|
||||
"source_url": "https://www.volcengine.com/docs/82379/1099320",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "unresolved",
|
||||
"blocker": "The exact public price for Ark endpoint qwen3-32b-20250429 has not been matched to an authoritative model row."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Ark/DeepSeek-V4-Flash",
|
||||
"model": "deepseek-v4-flash-260425",
|
||||
"api_key_env": "ARK_API_KEY",
|
||||
"base_url": "https://ark.cn-beijing.volces.com/api/v3",
|
||||
"protocol": "openai",
|
||||
"model_family": "DeepSeek",
|
||||
"access_class": "open-weight",
|
||||
"thinking_budget_tokens": 0,
|
||||
"pricing": {
|
||||
"input_per_million": null,
|
||||
"cached_input_per_million": null,
|
||||
"output_per_million": null,
|
||||
"currency": "CNY",
|
||||
"source_url": "https://www.volcengine.com/docs/82379/1099320",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "unresolved",
|
||||
"blocker": "The exact public price for Ark endpoint deepseek-v4-flash-260425 has not been matched to an authoritative model row."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "DeepSeek official/DeepSeek-V4-Flash",
|
||||
"model": "deepseek-v4-flash",
|
||||
"api_key_env": "DEEPSEEK_API_KEY",
|
||||
"base_url": "https://api.deepseek.com/v1",
|
||||
"protocol": "openai",
|
||||
"model_family": "DeepSeek",
|
||||
"access_class": "open-weight",
|
||||
"thinking_budget_tokens": 0,
|
||||
"pricing": {
|
||||
"input_per_million": 0.14,
|
||||
"cached_input_per_million": 0.0028,
|
||||
"output_per_million": 0.28,
|
||||
"currency": "USD",
|
||||
"source_url": "https://api-docs.deepseek.com/quick_start/pricing",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "verified"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "SiliconFlow/Qwen3-32B",
|
||||
"model": "Qwen/Qwen3-32B",
|
||||
"api_key_env": "SILICONFLOW_API_KEY",
|
||||
"base_url": "https://api.siliconflow.cn/v1",
|
||||
"protocol": "openai",
|
||||
"model_family": "Qwen",
|
||||
"access_class": "open-weight",
|
||||
"thinking_budget_tokens": 0,
|
||||
"pricing": {
|
||||
"input_per_million": null,
|
||||
"cached_input_per_million": null,
|
||||
"output_per_million": null,
|
||||
"currency": "CNY",
|
||||
"source_url": "https://siliconflow.cn/pricing",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "unresolved",
|
||||
"blocker": "The exact Qwen/Qwen3-32B cached-input and output price has not been extracted from an authoritative public row."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "SiliconFlow/DeepSeek-V4-Flash",
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash",
|
||||
"api_key_env": "SILICONFLOW_API_KEY",
|
||||
"base_url": "https://api.siliconflow.cn/v1",
|
||||
"protocol": "openai",
|
||||
"model_family": "DeepSeek",
|
||||
"access_class": "open-weight",
|
||||
"thinking_budget_tokens": 0,
|
||||
"pricing": {
|
||||
"input_per_million": 1.0,
|
||||
"cached_input_per_million": 0.02,
|
||||
"output_per_million": 2.0,
|
||||
"currency": "CNY",
|
||||
"source_url": "https://siliconflow.cn/pricing",
|
||||
"as_of": "2026-07-29",
|
||||
"usd_per_currency_unit": 0.1477922077922078,
|
||||
"fx_source_url": "https://www.ecb.europa.eu/stats/eurofxref/eurofxref-daily.xml",
|
||||
"fx_as_of": "2026-07-29",
|
||||
"status": "verified_with_fx"
|
||||
}
|
||||
}
|
||||
],
|
||||
"required_model_families": ["GPT", "Claude", "Gemini", "Doubao", "Qwen", "Kimi", "DeepSeek"],
|
||||
"external_benchmark_references": [],
|
||||
"same_model_provider_groups": [
|
||||
{
|
||||
"logical_model": "DeepSeek-V4-Flash",
|
||||
"providers": [
|
||||
"DeepSeek official/DeepSeek-V4-Flash",
|
||||
"SiliconFlow/DeepSeek-V4-Flash"
|
||||
]
|
||||
}
|
||||
],
|
||||
"workload": {
|
||||
"context_tokens": [8192, 32768, 131072],
|
||||
"output_tokens": [512, 2048],
|
||||
"requests_per_cell": 100,
|
||||
"concurrency": 4,
|
||||
"minimum_output_attainment_rate": 0.95
|
||||
},
|
||||
"availability": {
|
||||
"duration_hours": 168,
|
||||
"interval_seconds": 3600,
|
||||
"requests_per_probe": 1,
|
||||
"context_tokens": 1024,
|
||||
"output_tokens": 64
|
||||
},
|
||||
"rate_limit": {
|
||||
"concurrency_levels": [1, 2, 4, 8, 16, 32],
|
||||
"requests_per_level": 100,
|
||||
"context_tokens": 8192,
|
||||
"output_tokens": 512
|
||||
},
|
||||
"agent_cost": {
|
||||
"rounds": 6,
|
||||
"initial_context_tokens": 8192,
|
||||
"tokens_added_per_round": 2048,
|
||||
"output_tokens": 512
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,409 @@
|
||||
"""
|
||||
demo.py —— 一条命令跑出多提供商性能对比表 / 并发压测表。
|
||||
|
||||
用法：
|
||||
python demo.py # 使用默认参数，多提供商横向对比
|
||||
python demo.py --num-requests 20 --concurrency 5
|
||||
python demo.py --serial # 串行发送（并发=1）
|
||||
python demo.py --list # 仅列出将要测试的提供商
|
||||
|
||||
# 指定任意一个 OpenAI 兼容端点（不改代码即可测新模型/新提供商）：
|
||||
python demo.py --base-url https://api.deepseek.com --model deepseek-chat \
|
||||
--api-key-env DEEPSEEK_API_KEY
|
||||
|
||||
# 并发压测：对同一模型逐步提升并发，找限流点、看延迟长尾随并发的变化：
|
||||
python demo.py --model gpt-5.6-luna --concurrency-sweep 1,2,4,8
|
||||
|
||||
# 离线自检（无需 key/网络）：用合成数据跑通指标聚合数学
|
||||
python demo.py --mock
|
||||
python demo.py --mock --concurrency-sweep 1,2,4,8,16
|
||||
|
||||
默认只测"手上有有效 key"的提供商（OpenAI / Kimi / 豆包）。
|
||||
未设置对应环境变量的提供商会被自动跳过。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
|
||||
# 若安装了 python-dotenv 且存在 .env，则自动加载（可选，不强制）
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
|
||||
from benchmark import (
|
||||
DEFAULT_PROVIDERS,
|
||||
ProviderConfig,
|
||||
ProviderSummary,
|
||||
run_benchmark,
|
||||
sweep_concurrency,
|
||||
synthetic_summary,
|
||||
)
|
||||
|
||||
|
||||
# 短 prompt：控制成本，同时保证有稳定的输出用于测吞吐。
|
||||
DEFAULT_PROMPT = "用一句话解释什么是大语言模型。"
|
||||
|
||||
# 主对比表可选的指标族（成功率始终显示）。--metrics 用逗号选择子集。
|
||||
METRIC_KEYS = ["ttft", "e2e", "throughput", "tokens"]
|
||||
|
||||
|
||||
def _fmt(v, unit: str = "", scale: float = 1.0, digits: int = 1) -> str:
|
||||
"""把可能为 None 的数值格式化为对齐的字符串。"""
|
||||
if v is None:
|
||||
return " N/A"
|
||||
return f"{v * scale:.{digits}f}{unit}"
|
||||
|
||||
|
||||
def _render_table(headers: list[str], rows: list[list[str]]) -> None:
|
||||
"""按中文宽度对齐打印一张表。"""
|
||||
def width(text: str) -> int:
|
||||
return sum(2 if ord(c) > 127 else 1 for c in text)
|
||||
|
||||
cols = len(headers)
|
||||
col_w = [width(headers[i]) for i in range(cols)]
|
||||
for row in rows:
|
||||
for i in range(cols):
|
||||
col_w[i] = max(col_w[i], width(row[i]))
|
||||
|
||||
def pad(text: str, w: int) -> str:
|
||||
return text + " " * (w - width(text))
|
||||
|
||||
sep = "-+-".join("-" * col_w[i] for i in range(cols))
|
||||
print()
|
||||
print(" | ".join(pad(headers[i], col_w[i]) for i in range(cols)))
|
||||
print(sep)
|
||||
for row in rows:
|
||||
print(" | ".join(pad(row[i], col_w[i]) for i in range(cols)))
|
||||
print()
|
||||
|
||||
|
||||
def _print_errors(summaries: list[ProviderSummary]) -> None:
|
||||
"""打印失败明细，便于定位可用性问题。"""
|
||||
if not any(s.errors for s in summaries):
|
||||
return
|
||||
print("失败请求明细（可用性下降原因）：")
|
||||
for s in summaries:
|
||||
if s.errors:
|
||||
for e in s.errors[:3]:
|
||||
print(f" - {s.provider}: {e}")
|
||||
if len(s.errors) > 3:
|
||||
print(f" ... 以及另外 {len(s.errors) - 3} 条同类错误")
|
||||
print()
|
||||
|
||||
|
||||
def print_table(summaries: list[ProviderSummary], metrics: list[str]) -> None:
|
||||
"""打印多提供商横向对比表（成功率 + 所选指标族）。"""
|
||||
headers = ["Provider/Model", "成功率"]
|
||||
for m in metrics:
|
||||
if m == "ttft":
|
||||
headers += ["TTFT均值", "TTFT_p95"]
|
||||
elif m == "e2e":
|
||||
headers += ["端到端均值", "端到端p95"]
|
||||
elif m == "throughput":
|
||||
headers += ["吞吐"]
|
||||
elif m == "tokens":
|
||||
headers += ["输出tok"]
|
||||
|
||||
rows: list[list[str]] = []
|
||||
for s in summaries:
|
||||
row = [
|
||||
s.provider,
|
||||
f"{s.success}/{s.total} ({s.availability * 100:.0f}%)",
|
||||
]
|
||||
for m in metrics:
|
||||
if m == "ttft":
|
||||
row += [_fmt(s.stat("ttft", "mean"), "ms", 1000, 0),
|
||||
_fmt(s.stat("ttft", "p95"), "ms", 1000, 0)]
|
||||
elif m == "e2e":
|
||||
row += [_fmt(s.stat("latency", "mean"), "s", 1, 2),
|
||||
_fmt(s.stat("latency", "p95"), "s", 1, 2)]
|
||||
elif m == "throughput":
|
||||
row += [_fmt(s.stat("throughput", "mean"), " t/s", 1, 1)]
|
||||
elif m == "tokens":
|
||||
row += [_fmt(s.stat("completion_tokens", "mean"), "", 1, 0)]
|
||||
rows.append(row)
|
||||
|
||||
_render_table(headers, rows)
|
||||
_print_errors(summaries)
|
||||
|
||||
|
||||
def print_sweep_table(summaries: list[ProviderSummary]) -> None:
|
||||
"""
|
||||
打印并发压测表：每一行是一个并发档位，展示延迟长尾（p50/p95/p99/std）、
|
||||
可用性与聚合吞吐（RPS / tokens·s⁻¹）随并发的变化。
|
||||
"""
|
||||
headers = [
|
||||
"并发", "成功率", "TTFT_p50", "TTFT_p95",
|
||||
"端到端p50", "端到端p95", "端到端p99", "端到端std",
|
||||
"RPS", "聚合吞吐",
|
||||
]
|
||||
rows: list[list[str]] = []
|
||||
for s in summaries:
|
||||
rows.append([
|
||||
str(s.concurrency),
|
||||
f"{s.success}/{s.total} ({s.availability * 100:.0f}%)",
|
||||
_fmt(s.stat("ttft", "p50"), "ms", 1000, 0),
|
||||
_fmt(s.stat("ttft", "p95"), "ms", 1000, 0),
|
||||
_fmt(s.stat("latency", "p50"), "s", 1, 2),
|
||||
_fmt(s.stat("latency", "p95"), "s", 1, 2),
|
||||
_fmt(s.stat("latency", "p99"), "s", 1, 2),
|
||||
_fmt(s.stat("latency", "std"), "s", 1, 2),
|
||||
_fmt(s.rps, "", 1, 1),
|
||||
_fmt(s.agg_throughput, " t/s", 1, 1),
|
||||
])
|
||||
_render_table(headers, rows)
|
||||
_print_errors(summaries)
|
||||
|
||||
|
||||
def summary_to_dict(s: ProviderSummary) -> dict:
|
||||
"""把一个汇总序列化为可 JSON 落盘的结构（供 --output 使用）。"""
|
||||
def stats(attr: str) -> dict:
|
||||
return {
|
||||
k: s.stat(attr, k)
|
||||
for k in ("mean", "std", "p50", "p95", "p99")
|
||||
}
|
||||
|
||||
return {
|
||||
"provider": s.provider,
|
||||
"model": s.model,
|
||||
"concurrency": s.concurrency,
|
||||
"total": s.total,
|
||||
"success": s.success,
|
||||
"availability": s.availability,
|
||||
"wall_time_s": s.wall_time,
|
||||
"rps": s.rps,
|
||||
"agg_throughput_tps": s.agg_throughput,
|
||||
"ttft_s": stats("ttft"),
|
||||
"latency_s": stats("latency"),
|
||||
"throughput_tps": stats("throughput"),
|
||||
"completion_tokens_mean": s.stat("completion_tokens", "mean"),
|
||||
"errors": s.errors[:20],
|
||||
}
|
||||
|
||||
|
||||
def write_output(path: str, meta: dict, summaries: list[ProviderSummary]) -> None:
|
||||
payload = {"meta": meta, "results": [summary_to_dict(s) for s in summaries]}
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
print(f"结果已写入：{path}")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="多维度模型性能基准测试（实验 7-10）：TTFT / 端到端 / 吞吐 / p50·p95·p99·std / 可用性",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
)
|
||||
parser.add_argument("--num-requests", type=int, default=10,
|
||||
help="每个档位的请求次数（默认 10，控制成本；书中口径 ≥100）")
|
||||
parser.add_argument("--concurrency", type=int, default=3,
|
||||
help="单档位并发数（默认 3；与 --concurrency-sweep 二选一）")
|
||||
parser.add_argument("--serial", action="store_true",
|
||||
help="串行发送（等价于 --concurrency 1，看无竞争下的基线延迟）")
|
||||
parser.add_argument("--concurrency-sweep", type=str, default=None, metavar="1,2,4,8",
|
||||
help="并发压测：逗号分隔的并发档位列表，对同一模型逐档加压找限流点")
|
||||
parser.add_argument("--max-tokens", type=int, default=64,
|
||||
help="每次请求生成的最大 token 数（默认 64，控制成本）")
|
||||
parser.add_argument("--timeout", type=float, default=60.0,
|
||||
help="单次请求超时（秒），超时记为可用性下降")
|
||||
parser.add_argument("--prompt", type=str, default=DEFAULT_PROMPT,
|
||||
help="测试用的短 prompt")
|
||||
parser.add_argument("--metrics", type=str, default="all",
|
||||
help="主对比表显示的指标族，逗号分隔，可选 "
|
||||
"ttft/e2e/throughput/tokens 或 all（默认 all；成功率始终显示）")
|
||||
parser.add_argument("--output", type=str, default=None, metavar="FILE.json",
|
||||
help="把完整结果（含 p50/p95/p99/std）写入 JSON 文件")
|
||||
parser.add_argument("--list", action="store_true",
|
||||
help="仅列出将测试的提供商后退出")
|
||||
|
||||
# 指定任意单个 OpenAI 兼容端点（不改代码即可测新提供商/新模型）
|
||||
grp = parser.add_argument_group("自定义端点（指定后只测这一个，忽略默认提供商列表）")
|
||||
grp.add_argument("--base-url", type=str, default=None,
|
||||
help="OpenAI 兼容端点的 base_url（OpenAI 官方留空）")
|
||||
grp.add_argument("--model", type=str, default=None,
|
||||
help="要测试的模型名（如 gpt-5.6-luna / deepseek-chat）")
|
||||
grp.add_argument("--api-key-env", type=str, default="OPENAI_API_KEY",
|
||||
help="读取 API key 的环境变量名（默认 OPENAI_API_KEY）")
|
||||
grp.add_argument("--name", type=str, default=None,
|
||||
help="该端点在表格中的展示名（默认用 model 名）")
|
||||
|
||||
parser.add_argument("--mock", action="store_true",
|
||||
help="离线自检：用合成（synthetic）数据跑通指标聚合，"
|
||||
"不发任何网络请求、不需要 key（数字为合成，非真实基准）")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def resolve_metrics(raw: str) -> list[str]:
|
||||
if raw.strip().lower() == "all":
|
||||
return list(METRIC_KEYS)
|
||||
chosen = [m.strip() for m in raw.split(",") if m.strip()]
|
||||
bad = [m for m in chosen if m not in METRIC_KEYS]
|
||||
if bad:
|
||||
raise SystemExit(f"未知指标：{', '.join(bad)}；可选：{', '.join(METRIC_KEYS)} 或 all")
|
||||
return chosen
|
||||
|
||||
|
||||
def build_providers(args: argparse.Namespace) -> tuple[list[ProviderConfig], list[ProviderConfig]]:
|
||||
"""
|
||||
返回 (available, skipped)。
|
||||
若指定了 --base-url 或 --model，则构造单个自定义提供商（覆盖默认列表）。
|
||||
"""
|
||||
if args.base_url or args.model:
|
||||
if not args.model:
|
||||
raise SystemExit("使用自定义端点时必须提供 --model")
|
||||
cfg = ProviderConfig(
|
||||
name=args.name or f"custom/{args.model}",
|
||||
model=args.model,
|
||||
api_key_env=args.api_key_env,
|
||||
base_url=args.base_url,
|
||||
)
|
||||
available = [cfg] if cfg.is_available() else []
|
||||
skipped = [] if cfg.is_available() else [cfg]
|
||||
return available, skipped
|
||||
|
||||
available = [p for p in DEFAULT_PROVIDERS if p.is_available()]
|
||||
skipped = [p for p in DEFAULT_PROVIDERS if not p.is_available()]
|
||||
return available, skipped
|
||||
|
||||
|
||||
def run_mock(args: argparse.Namespace, metrics: list[str]) -> None:
|
||||
"""用合成数据演示指标聚合，无需 key/网络。"""
|
||||
print("=" * 72)
|
||||
print("多维度模型性能基准测试（实验 7-10）—— 合成数据自检模式 [SYNTHETIC]")
|
||||
print("=" * 72)
|
||||
print("⚠️ 以下所有数字均为合成（伪随机）生成，仅用于验证指标聚合数学，")
|
||||
print(" 不代表任何真实模型/提供商/网络环境的性能，切勿作为选型依据。")
|
||||
print("-" * 72)
|
||||
|
||||
name = args.name or (args.model and f"custom/{args.model}") or "mock/demo-model"
|
||||
model = args.model or "demo-model"
|
||||
|
||||
if args.concurrency_sweep:
|
||||
levels = parse_sweep_levels(args.concurrency_sweep)
|
||||
print(f"并发压测（合成）：{name} 档位={levels} N={args.num_requests}/档")
|
||||
summaries = [
|
||||
synthetic_summary(name, model, args.num_requests, c, fail_rate=0.02, seed=42)
|
||||
for c in levels
|
||||
]
|
||||
print_sweep_table(summaries)
|
||||
print("解读：并发上升 → 端到端 p95/p99 与 std 走高（长尾变差），")
|
||||
print(" 可用性因限流下降，聚合吞吐先升后趋平（触及服务端上限即触顶）。")
|
||||
else:
|
||||
concurrency = 1 if args.serial else args.concurrency
|
||||
print(f"单档位对比（合成）：并发={concurrency} N={args.num_requests}/家")
|
||||
# 造三个"提供商"，参数不同以体现横向差异
|
||||
summaries = [
|
||||
synthetic_summary("mockA/fast-low-ttft", "fast", args.num_requests,
|
||||
concurrency, base_ttft=0.20, base_gen_throughput=110, seed=1),
|
||||
synthetic_summary("mockB/balanced", "balanced", args.num_requests,
|
||||
concurrency, base_ttft=0.35, base_gen_throughput=85, seed=2),
|
||||
synthetic_summary("mockC/high-throughput", "hi-tp", args.num_requests,
|
||||
concurrency, base_ttft=0.55, base_gen_throughput=140,
|
||||
fail_rate=0.05, seed=3),
|
||||
]
|
||||
print_table(summaries, metrics)
|
||||
|
||||
if args.output:
|
||||
write_output(args.output, {"mode": "mock-synthetic", "note": "数字为合成，非真实基准"},
|
||||
summaries)
|
||||
|
||||
|
||||
def parse_sweep_levels(raw: str) -> list[int]:
|
||||
try:
|
||||
levels = [int(x) for x in raw.split(",") if x.strip()]
|
||||
except ValueError:
|
||||
raise SystemExit(f"--concurrency-sweep 需为逗号分隔的整数，如 1,2,4,8；收到：{raw!r}")
|
||||
levels = [c for c in levels if c >= 1]
|
||||
if not levels:
|
||||
raise SystemExit("--concurrency-sweep 至少需要一个 ≥1 的并发档位")
|
||||
return levels
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
metrics = resolve_metrics(args.metrics)
|
||||
|
||||
if args.mock:
|
||||
run_mock(args, metrics)
|
||||
return
|
||||
|
||||
available, skipped = build_providers(args)
|
||||
|
||||
print("=" * 72)
|
||||
print("多维度模型性能基准测试（实验 7-10）")
|
||||
print("=" * 72)
|
||||
if skipped:
|
||||
for p in skipped:
|
||||
print(f"[跳过] {p.name} —— 未设置环境变量 {p.api_key_env}")
|
||||
if not available:
|
||||
print("没有任何可用提供商：请设置对应 API key 环境变量，")
|
||||
print("或用 --mock 在无 key 情况下离线验证指标聚合。")
|
||||
return
|
||||
|
||||
print(f"待测提供商：{', '.join(p.name for p in available)}")
|
||||
|
||||
# ---- 并发压测模式 ----
|
||||
if args.concurrency_sweep:
|
||||
levels = parse_sweep_levels(args.concurrency_sweep)
|
||||
print(f"模式：并发压测（逐档加压找限流点） 档位={levels}")
|
||||
print(f"参数：N={args.num_requests}/档, max_tokens={args.max_tokens}, "
|
||||
f"timeout={args.timeout}s")
|
||||
print(f"Prompt：{args.prompt!r}")
|
||||
if args.list:
|
||||
return
|
||||
all_summaries: list[ProviderSummary] = []
|
||||
for cfg in available:
|
||||
print("-" * 72)
|
||||
print(f"压测 {cfg.name}:")
|
||||
summaries = sweep_concurrency(
|
||||
cfg, args.prompt, args.num_requests, levels,
|
||||
args.max_tokens, args.timeout,
|
||||
)
|
||||
print_sweep_table(summaries)
|
||||
all_summaries.extend(summaries)
|
||||
if args.output:
|
||||
write_output(args.output,
|
||||
{"mode": "concurrency-sweep", "levels": levels}, all_summaries)
|
||||
return
|
||||
|
||||
# ---- 单档位横向对比模式（默认，保持原行为）----
|
||||
concurrency = 1 if args.serial else args.concurrency
|
||||
print(f"参数：N={args.num_requests}/家, 并发={concurrency}, "
|
||||
f"max_tokens={args.max_tokens}, timeout={args.timeout}s")
|
||||
print(f"Prompt：{args.prompt!r}")
|
||||
|
||||
if args.list:
|
||||
return
|
||||
|
||||
print("-" * 72)
|
||||
summaries = run_benchmark(
|
||||
providers=available,
|
||||
prompt=args.prompt,
|
||||
num_requests=args.num_requests,
|
||||
concurrency=concurrency,
|
||||
max_tokens=args.max_tokens,
|
||||
timeout=args.timeout,
|
||||
)
|
||||
|
||||
print_table(summaries, metrics)
|
||||
|
||||
print("指标说明：")
|
||||
print(" 成功率 = 成功请求数 / 总请求数（可用性维度）")
|
||||
print(" TTFT = 首个 token 到达延迟（流式测得），越低越流畅")
|
||||
print(" 端到端 = 请求发出到响应结束的总耗时")
|
||||
print(" 吞吐 = 输出 token 数 / 生成阶段耗时（tokens/s）")
|
||||
print(" p95 = 95 分位延迟，反映长尾/稳定性（方差大则体验不稳）")
|
||||
print(" 提示 = 加 --concurrency-sweep 1,2,4,8 可做并发压测，看指标随并发的变化")
|
||||
|
||||
if args.output:
|
||||
write_output(args.output,
|
||||
{"mode": "single", "concurrency": concurrency}, summaries)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,28 @@
|
||||
# 复制为 .env 或直接 export 到环境变量。
|
||||
# 只需填你手上有的 key，未填的提供商会被自动跳过。
|
||||
|
||||
# OpenAI 官方（用于 gpt-5.6-luna / kimi-k3 / doubao-1.6）
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# 原生 Claude / Gemini 适配器（完整 campaign 可直接横向比较，不经中转）
|
||||
ANTHROPIC_API_KEY=
|
||||
GEMINI_API_KEY=
|
||||
|
||||
# OpenRouter 回退：未设置 OPENAI_API_KEY 时，OpenAI 原生条目自动改走 OpenRouter
|
||||
# （模型名映射 gpt-* -> openai/*）。gpt-5.x 直连需组织实名认证，只要设置了本 key
|
||||
# 就会优先走 OpenRouter。
|
||||
# OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
|
||||
# 月之暗面 Kimi（base_url 已在代码中配置为 https://api.moonshot.cn/v1）
|
||||
MOONSHOT_API_KEY=your-moonshot-api-key
|
||||
|
||||
# 字节火山方舟 / 豆包（base_url 已配置为 https://ark.cn-beijing.volces.com/api/v3）
|
||||
ARK_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
||||
|
||||
# DeepSeek 官方与 SiliconFlow。完整实验使用同一个 DeepSeek-V4-Flash
|
||||
# 权重族做“官方端点 vs 第三方端点”的受控提供商对比。
|
||||
DEEPSEEK_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxx
|
||||
SILICONFLOW_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxx
|
||||
|
||||
# 完整实验的模型、负载和三类 token 单价统一固定在 campaign_config.json。
|
||||
# 不要把凭据写入该 JSON；这里只通过环境变量注入。
|
||||
@@ -0,0 +1,321 @@
|
||||
"""Rate Ramp Benchmark for LLM Endpoints (Chapter 6).
|
||||
|
||||
Simulates multi-concurrency load testing (1 to 50 req/s) against LLM endpoints,
|
||||
measuring 429 rate limit backoff curves, TTFT percentiles (p50, p95, p99),
|
||||
error rates, and compiling N=100 evidence packages.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import datetime, timezone
|
||||
import math
|
||||
import random
|
||||
from typing import Any, Callable, Dict, List, Optional, Sequence, Union
|
||||
|
||||
|
||||
def utc_timestamp() -> str:
|
||||
"""Return ISO 8601 formatted UTC timestamp."""
|
||||
return datetime.now(timezone.utc).isoformat(timespec="milliseconds")
|
||||
|
||||
|
||||
def calculate_percentile(values: Sequence[float], percentile: float) -> float:
|
||||
"""Calculate percentile (0-100) using linear interpolation."""
|
||||
if not values:
|
||||
return 0.0
|
||||
sorted_vals = sorted(values)
|
||||
if len(sorted_vals) == 1:
|
||||
return float(sorted_vals[0])
|
||||
|
||||
p = max(0.0, min(100.0, percentile))
|
||||
k = (len(sorted_vals) - 1) * (p / 100.0)
|
||||
f = math.floor(k)
|
||||
c = math.ceil(k)
|
||||
|
||||
if f == c:
|
||||
return float(sorted_vals[int(k)])
|
||||
|
||||
d0 = sorted_vals[int(f)] * (c - k)
|
||||
d1 = sorted_vals[int(c)] * (k - f)
|
||||
return round(float(d0 + d1), 4)
|
||||
|
||||
|
||||
class RateRampBenchmark:
|
||||
"""Simulates or executes multi-concurrency rate-ramping load tests on LLM endpoints.
|
||||
|
||||
Measures:
|
||||
- 429 rate limit backoff curves (attempts, rate limit hits, backoff delays).
|
||||
- Time To First Token (TTFT) percentiles: p50, p95, p99.
|
||||
- Error rates across load levels.
|
||||
- Compiles N=100 evidence packages.
|
||||
"""
|
||||
|
||||
def __init__(self, config: Optional[dict[str, Any]] = None) -> None:
|
||||
self.config = self._parse_config(config or {})
|
||||
|
||||
def _parse_config(self, config: dict[str, Any]) -> dict[str, Any]:
|
||||
start_rate = int(config.get("start_rate", 1))
|
||||
end_rate = int(config.get("end_rate", 50))
|
||||
step_rate = int(config.get("step_rate", 5))
|
||||
|
||||
if "rates" in config and isinstance(config["rates"], (list, tuple)) and config["rates"]:
|
||||
rates = [int(r) for r in config["rates"]]
|
||||
start_rate = rates[0]
|
||||
end_rate = rates[-1]
|
||||
else:
|
||||
if start_rate <= end_rate:
|
||||
step = max(1, step_rate)
|
||||
rates = list(range(start_rate, end_rate + 1, step))
|
||||
else:
|
||||
step = -max(1, abs(step_rate))
|
||||
rates = list(range(start_rate, end_rate - 1, step))
|
||||
if not rates or rates[-1] != end_rate:
|
||||
rates.append(end_rate)
|
||||
return {
|
||||
"start_rate": start_rate,
|
||||
"end_rate": end_rate,
|
||||
"step_rate": step_rate,
|
||||
"rates": rates,
|
||||
"requests_per_step": int(config.get("requests_per_step", 15)),
|
||||
"sample_size": int(config.get("sample_size", 100)),
|
||||
"endpoint_url": str(config.get("endpoint_url", "https://api.openai.com/v1/chat/completions")),
|
||||
"model": str(config.get("model", "gpt-4o")),
|
||||
"max_backoff_sec": float(config.get("max_backoff_sec", 8.0)),
|
||||
"request_fn": config.get("request_fn"),
|
||||
}
|
||||
|
||||
def simulate_request(
|
||||
self, target_rate: int, concurrency: int, request_idx: int
|
||||
) -> dict[str, Any]:
|
||||
"""Simulate a single endpoint request under load when no live request_fn is provided."""
|
||||
# Seed deterministically for test consistency
|
||||
rng = random.Random(target_rate * 1000 + request_idx)
|
||||
|
||||
# Base TTFT increases slightly with rate/concurrency
|
||||
base_ttft = 0.05 + (target_rate / 100.0) * 0.35 + rng.uniform(0.01, 0.05)
|
||||
ttft_sec = round(base_ttft, 4)
|
||||
|
||||
# 429 probability ramps up as target_rate exceeds 25 req/s
|
||||
prob_429 = max(0.0, (target_rate - 20) / 40.0) if target_rate > 20 else 0.0
|
||||
is_429 = rng.random() < prob_429
|
||||
|
||||
prob_5xx = 0.03 if target_rate > 40 else 0.01
|
||||
is_5xx = not is_429 and (rng.random() < prob_5xx)
|
||||
|
||||
if is_429:
|
||||
status_code = 429
|
||||
retry_count = rng.randint(1, 3)
|
||||
backoff_sec = round(min(self.config["max_backoff_sec"], 0.4 * (2 ** (retry_count - 1)) + rng.uniform(0.05, 0.2)), 4)
|
||||
error_type = "rate_limit_429"
|
||||
elif is_5xx:
|
||||
status_code = 500
|
||||
retry_count = 0
|
||||
backoff_sec = 0.0
|
||||
error_type = "server_error_500"
|
||||
else:
|
||||
status_code = 200
|
||||
retry_count = 0
|
||||
backoff_sec = 0.0
|
||||
error_type = None
|
||||
|
||||
total_latency_sec = round(ttft_sec + rng.uniform(0.1, 0.3) + backoff_sec, 4)
|
||||
|
||||
return {
|
||||
"request_id": f"req-{target_rate:02d}-{request_idx:03d}",
|
||||
"timestamp": utc_timestamp(),
|
||||
"target_rate": target_rate,
|
||||
"concurrency": concurrency,
|
||||
"status_code": status_code,
|
||||
"ttft_sec": ttft_sec,
|
||||
"total_latency_sec": total_latency_sec,
|
||||
"backoff_sec": backoff_sec,
|
||||
"retry_count": retry_count,
|
||||
"error_type": error_type,
|
||||
}
|
||||
|
||||
def compile_evidence_package(
|
||||
self, records: Sequence[dict[str, Any]], sample_size: int = 100
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Compile exactly sample_size (default N=100) evidence items uniformly sampled from raw records."""
|
||||
if not records or not sample_size or sample_size <= 0:
|
||||
return []
|
||||
|
||||
valid_records = [r for r in records if isinstance(r, dict)]
|
||||
if not valid_records:
|
||||
return []
|
||||
|
||||
if len(valid_records) <= sample_size:
|
||||
return [dict(r) for r in valid_records]
|
||||
|
||||
# Uniformly sample across the records to cover all rate tiers
|
||||
step = len(valid_records) / float(sample_size)
|
||||
indices = [int(i * step) for i in range(sample_size)]
|
||||
return [dict(valid_records[idx]) for idx in indices]
|
||||
|
||||
def calculate_backoff_curves(
|
||||
self, records: Sequence[dict[str, Any]]
|
||||
) -> dict[str, Any]:
|
||||
"""Compute 429 rate limit backoff curve metrics by request rate level."""
|
||||
by_rate: dict[int, dict[str, Any]] = {}
|
||||
total_429_backoff_time = 0.0
|
||||
max_backoff = 0.0
|
||||
total_429 = 0
|
||||
total_429_backoff_count = 0
|
||||
|
||||
grouped: dict[int, list[dict[str, Any]]] = {}
|
||||
for r in records:
|
||||
if not isinstance(r, dict):
|
||||
continue
|
||||
rate = int(r.get("target_rate", 0) or 0)
|
||||
grouped.setdefault(rate, []).append(r)
|
||||
|
||||
for rate in sorted(grouped.keys()):
|
||||
step_recs = grouped[rate]
|
||||
cnt = len(step_recs)
|
||||
hits_429 = sum(1 for r in step_recs if isinstance(r, dict) and r.get("status_code") == 429)
|
||||
|
||||
backoffs_429 = [
|
||||
float(r.get("backoff_sec") or 0.0)
|
||||
for r in step_recs
|
||||
if isinstance(r, dict)
|
||||
and r.get("status_code") == 429
|
||||
and float(r.get("backoff_sec") or 0.0) > 0
|
||||
]
|
||||
avg_backoff = round(sum(backoffs_429) / len(backoffs_429), 4) if backoffs_429 else 0.0
|
||||
step_max_backoff = max(backoffs_429, default=0.0)
|
||||
|
||||
total_429 += hits_429
|
||||
total_429_backoff_count += len(backoffs_429)
|
||||
total_429_backoff_time += sum(backoffs_429)
|
||||
max_backoff = max(max_backoff, step_max_backoff)
|
||||
|
||||
by_rate[rate] = {
|
||||
"total_requests": cnt,
|
||||
"429_count": hits_429,
|
||||
"backoff_ratio": round(hits_429 / cnt, 4) if cnt > 0 else 0.0,
|
||||
"avg_backoff_sec": avg_backoff,
|
||||
"max_backoff_sec": round(step_max_backoff, 4),
|
||||
}
|
||||
|
||||
overall_avg_backoff = (
|
||||
round(total_429_backoff_time / total_429_backoff_count, 4)
|
||||
if total_429_backoff_count > 0
|
||||
else 0.0
|
||||
)
|
||||
|
||||
return {
|
||||
"by_rate": by_rate,
|
||||
"overall_avg_backoff_sec": overall_avg_backoff,
|
||||
"total_backoff_time_sec": round(total_429_backoff_time, 4),
|
||||
"max_backoff_observed_sec": round(max_backoff, 4),
|
||||
"total_429_count": total_429,
|
||||
}
|
||||
|
||||
def run(self, config: Optional[dict[str, Any]] = None) -> dict[str, Any]:
|
||||
"""Execute rate ramp benchmark and return structured benchmark metrics."""
|
||||
if config is not None:
|
||||
self.config = self._parse_config(config)
|
||||
|
||||
rates = self.config["rates"]
|
||||
reqs_per_step = self.config["requests_per_step"]
|
||||
request_fn: Optional[Callable] = self.config["request_fn"]
|
||||
|
||||
all_records: list[dict[str, Any]] = []
|
||||
ramp_steps_summary: list[dict[str, Any]] = []
|
||||
|
||||
for rate in rates:
|
||||
concurrency = rate
|
||||
step_records: list[dict[str, Any]] = []
|
||||
|
||||
for i in range(reqs_per_step):
|
||||
if callable(request_fn):
|
||||
rec = request_fn(rate, concurrency, i)
|
||||
else:
|
||||
rec = self.simulate_request(rate, concurrency, i)
|
||||
step_records.append(rec)
|
||||
all_records.append(rec)
|
||||
|
||||
ttfts = [float(r["ttft_sec"]) for r in step_records if isinstance(r, dict) and r.get("ttft_sec") is not None]
|
||||
hits_429 = sum(1 for r in step_records if isinstance(r, dict) and r.get("status_code") == 429)
|
||||
other_errs = sum(
|
||||
1
|
||||
for r in step_records
|
||||
if not isinstance(r, dict)
|
||||
or r.get("status_code") not in (200, 429)
|
||||
)
|
||||
successes = sum(1 for r in step_records if isinstance(r, dict) and r.get("status_code") == 200)
|
||||
|
||||
# Only throttled (429) requests contribute backoff time to averages.
|
||||
backoff_secs = [
|
||||
float(r.get("backoff_sec", 0.0) or 0.0)
|
||||
for r in step_records
|
||||
if isinstance(r, dict)
|
||||
and r.get("status_code") == 429
|
||||
and float(r.get("backoff_sec", 0.0) or 0.0) > 0
|
||||
]
|
||||
avg_backoff = (
|
||||
round(sum(backoff_secs) / len(backoff_secs), 4)
|
||||
if backoff_secs
|
||||
else 0.0
|
||||
)
|
||||
|
||||
ramp_steps_summary.append(
|
||||
{
|
||||
"rate_req_per_sec": rate,
|
||||
"concurrency": concurrency,
|
||||
"total_requests": len(step_records),
|
||||
"successful_requests": successes,
|
||||
"rate_limit_errors": hits_429,
|
||||
"other_errors": other_errs,
|
||||
"error_rate": round((hits_429 + other_errs) / max(1, len(step_records)), 4),
|
||||
"ttft_p50": calculate_percentile(ttfts, 50),
|
||||
"ttft_p95": calculate_percentile(ttfts, 95),
|
||||
"ttft_p99": calculate_percentile(ttfts, 99),
|
||||
"avg_backoff_sec": avg_backoff,
|
||||
}
|
||||
)
|
||||
|
||||
all_ttfts = [float(r.get("ttft_sec", 0.0) or 0.0) for r in all_records if isinstance(r, dict)]
|
||||
total_reqs = len(all_records)
|
||||
total_429 = sum(1 for r in all_records if isinstance(r, dict) and r.get("status_code") == 429)
|
||||
total_other = sum(
|
||||
1 for r in all_records if isinstance(r, dict) and r.get("status_code") not in (200, 429)
|
||||
)
|
||||
total_success = sum(1 for r in all_records if isinstance(r, dict) and r.get("status_code") == 200)
|
||||
|
||||
backoff_curves = self.calculate_backoff_curves(all_records)
|
||||
evidence_package = self.compile_evidence_package(
|
||||
all_records, sample_size=self.config["sample_size"]
|
||||
)
|
||||
|
||||
overall_metrics = {
|
||||
"total_requests": total_reqs,
|
||||
"successful_requests": total_success,
|
||||
"total_errors": total_429 + total_other,
|
||||
"error_rate": round((total_429 + total_other) / max(1, total_reqs), 4),
|
||||
"rate_limit_429_count": total_429,
|
||||
"ttft_p50": calculate_percentile(all_ttfts, 50),
|
||||
"ttft_p95": calculate_percentile(all_ttfts, 95),
|
||||
"ttft_p99": calculate_percentile(all_ttfts, 99),
|
||||
"avg_backoff_sec": backoff_curves["overall_avg_backoff_sec"],
|
||||
}
|
||||
|
||||
return {
|
||||
"config": {
|
||||
"start_rate": self.config["start_rate"],
|
||||
"end_rate": self.config["end_rate"],
|
||||
"step_rate": self.config["step_rate"],
|
||||
"sample_size": self.config["sample_size"],
|
||||
"endpoint_url": self.config["endpoint_url"],
|
||||
"model": self.config["model"],
|
||||
},
|
||||
"ramp_steps": ramp_steps_summary,
|
||||
"overall_metrics": overall_metrics,
|
||||
"backoff_curves": backoff_curves,
|
||||
"evidence_package": evidence_package,
|
||||
}
|
||||
|
||||
|
||||
def run_benchmark(config: Optional[dict[str, Any]] = None) -> dict[str, Any]:
|
||||
"""Entrypoint function to run rate ramp benchmark and return structured metrics."""
|
||||
bench = RateRampBenchmark(config)
|
||||
return bench.run()
|
||||
@@ -0,0 +1,7 @@
|
||||
openai>=1.30.0
|
||||
anthropic>=0.59.0
|
||||
google-genai>=1.30.0
|
||||
tiktoken>=0.7.0
|
||||
pytest>=8.0.0
|
||||
# 可选：自动加载 .env 文件（demo.py 里做了软依赖，未安装也能跑）
|
||||
python-dotenv>=1.0.0
|
||||
@@ -0,0 +1,425 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
from analysis import availability_summary, completion_audit, markdown, percentile, summarize_workloads
|
||||
from campaign import (
|
||||
CampaignStore,
|
||||
Observation,
|
||||
Price,
|
||||
PromptFactory,
|
||||
Provider,
|
||||
error_details,
|
||||
execution_config_fingerprint,
|
||||
measure_anthropic,
|
||||
measure_gemini,
|
||||
measure_stream,
|
||||
)
|
||||
|
||||
|
||||
class FakeCompletions:
|
||||
def create(self, **kwargs):
|
||||
assert kwargs["stream"] is True
|
||||
usage = SimpleNamespace(
|
||||
prompt_tokens=100,
|
||||
completion_tokens=12,
|
||||
prompt_tokens_details=SimpleNamespace(cached_tokens=40),
|
||||
completion_tokens_details=SimpleNamespace(reasoning_tokens=5),
|
||||
)
|
||||
return iter([
|
||||
SimpleNamespace(
|
||||
id="request-1", usage=None,
|
||||
choices=[SimpleNamespace(
|
||||
finish_reason=None,
|
||||
delta=SimpleNamespace(content=None, reasoning_content="think"),
|
||||
)],
|
||||
),
|
||||
SimpleNamespace(
|
||||
id="request-1", usage=None,
|
||||
choices=[SimpleNamespace(
|
||||
finish_reason=None,
|
||||
delta=SimpleNamespace(content="answer", reasoning_content=None),
|
||||
)],
|
||||
),
|
||||
SimpleNamespace(id="request-1", usage=usage, choices=[]),
|
||||
])
|
||||
|
||||
|
||||
def fake_client():
|
||||
return SimpleNamespace(chat=SimpleNamespace(completions=FakeCompletions()))
|
||||
|
||||
|
||||
def provider():
|
||||
return Provider(
|
||||
name="test", model="test-model", api_key_env="UNUSED", base_url="https://example.test/v1",
|
||||
pricing=Price(1.0, 0.1, 2.0, currency="USD", source_url="https://example.test", as_of="2026-07-29", status="verified"),
|
||||
)
|
||||
|
||||
|
||||
def observation(cell: str, ok: bool, scheduled: str, **overrides):
|
||||
values = dict(
|
||||
campaign_id="campaign", phase="availability", cell_id=cell,
|
||||
provider="test", model="test-model", scheduled_at_utc=scheduled,
|
||||
started_at_utc=scheduled, ended_at_utc=scheduled,
|
||||
target_context_tokens=100, target_output_tokens=10, concurrency=1,
|
||||
request_index=0, ok=ok,
|
||||
)
|
||||
values.update(overrides)
|
||||
return Observation(**values)
|
||||
|
||||
|
||||
def test_prompt_factory_hits_exact_reference_token_count():
|
||||
factory = PromptFactory()
|
||||
text = factory.build(256, 64)
|
||||
assert len(factory.encoding.encode(text)) == 256
|
||||
assert "64 tokens" in text
|
||||
|
||||
|
||||
def test_measure_stream_records_usage_cache_reasoning_and_hash():
|
||||
row = measure_stream(
|
||||
provider(), campaign_id="c", phase="workload", cell_id="id", prompt="hello",
|
||||
target_context_tokens=10, target_output_tokens=12, concurrency=1,
|
||||
request_index=0, client=fake_client(),
|
||||
)
|
||||
assert row.ok is True
|
||||
assert row.input_tokens == 100
|
||||
assert row.cached_input_tokens == 40
|
||||
assert row.output_tokens == 12
|
||||
assert row.visible_output_tokens == 7
|
||||
assert row.reasoning_tokens == 5
|
||||
assert row.thinking_ttft_s is not None
|
||||
assert row.output_sha256
|
||||
assert row.output_text == "answer"
|
||||
assert row.prompt_sha256
|
||||
|
||||
|
||||
def test_store_is_resumable_and_ignores_duplicate_cell(tmp_path: Path):
|
||||
store = CampaignStore(tmp_path / "campaign.sqlite3")
|
||||
row = observation("same", True, "2026-01-01T00:00:00+00:00")
|
||||
store.add(row)
|
||||
store.add(row)
|
||||
assert store.has("same")
|
||||
count = store.connection.execute("SELECT count(*) FROM observations").fetchone()[0]
|
||||
store.close()
|
||||
assert count == 1
|
||||
|
||||
|
||||
def test_campaign_binding_rejects_changed_execution_but_allows_repricing(tmp_path: Path):
|
||||
import pytest
|
||||
|
||||
config = {
|
||||
"providers": [{
|
||||
"name": "p", "model": "m", "api_key_env": "KEY",
|
||||
"protocol": "openai", "pricing": {"input_per_million": 1.0},
|
||||
}],
|
||||
"workload": {"context_tokens": [8192], "output_tokens": [512], "requests_per_cell": 100},
|
||||
"availability": {"duration_hours": 168, "interval_seconds": 3600},
|
||||
"rate_limit": {"concurrency_levels": [1, 2], "requests_per_level": 100},
|
||||
"agent_cost": {"rounds": 2},
|
||||
}
|
||||
store = CampaignStore(tmp_path / "campaign.sqlite3")
|
||||
first = store.bind_campaign("c", config)
|
||||
repriced = {**config, "providers": [{
|
||||
**config["providers"][0], "pricing": {"input_per_million": 2.0},
|
||||
}]}
|
||||
assert store.bind_campaign("c", repriced) == first
|
||||
changed = {**config, "workload": {**config["workload"], "output_tokens": [2048]}}
|
||||
assert execution_config_fingerprint(changed) != first
|
||||
with pytest.raises(RuntimeError, match="bound to execution fingerprint"):
|
||||
store.bind_campaign("c", changed)
|
||||
store.close()
|
||||
|
||||
|
||||
def test_batch_counts_accumulate_across_resumed_invocations(tmp_path: Path):
|
||||
store = CampaignStore(tmp_path / "campaign.sqlite3")
|
||||
payload = {
|
||||
"batch_id": "b", "campaign_id": "c", "phase": "workload",
|
||||
"provider": "p", "model": "m", "target_context_tokens": 8192,
|
||||
"target_output_tokens": 512, "concurrency": 4,
|
||||
"requested": 1, "succeeded": 1, "input_tokens": 8192,
|
||||
"output_tokens": 512, "wall_s": 1.0,
|
||||
"started_at_utc": "2026-01-01T00:00:00+00:00",
|
||||
"ended_at_utc": "2026-01-01T00:00:01+00:00",
|
||||
}
|
||||
store.add_batch(payload)
|
||||
store.add_batch({
|
||||
**payload, "requested": 99, "succeeded": 98,
|
||||
"input_tokens": 99 * 8192, "output_tokens": 98 * 512,
|
||||
"wall_s": 10.0,
|
||||
})
|
||||
row = store.connection.execute(
|
||||
"SELECT requested, succeeded, input_tokens, output_tokens, wall_s FROM batches"
|
||||
).fetchone()
|
||||
assert tuple(row) == (100, 99, 100 * 8192, 99 * 512, 11.0)
|
||||
store.close()
|
||||
|
||||
|
||||
def test_availability_groups_failures_and_computes_mttr():
|
||||
rows = [
|
||||
observation("1", True, "2026-01-01T00:00:00+00:00"),
|
||||
observation("2", False, "2026-01-01T01:00:00+00:00", error_type="provider_5xx"),
|
||||
observation("3", False, "2026-01-01T02:00:00+00:00", error_type="provider_5xx"),
|
||||
observation("4", True, "2026-01-01T03:00:00+00:00"),
|
||||
]
|
||||
summary = availability_summary([row.__dict__ | {"ok": int(row.ok)} for row in rows])[0]
|
||||
assert summary["uptime"] == 0.5
|
||||
assert summary["outage_count"] == 1
|
||||
assert summary["mttr_s"] == 7200
|
||||
|
||||
|
||||
def test_percentile_and_workload_summary_include_output_attainment():
|
||||
assert percentile([1, 2, 3], 0.5) == 2
|
||||
rows = [
|
||||
observation(
|
||||
"1", True, "2026-01-01T00:00:00+00:00", phase="workload",
|
||||
target_output_tokens=100, input_tokens=200, output_tokens=100,
|
||||
ttft_s=0.1, e2e_s=1.1,
|
||||
).__dict__,
|
||||
observation(
|
||||
"2", True, "2026-01-01T00:01:00+00:00", phase="workload",
|
||||
target_output_tokens=100, input_tokens=200, output_tokens=90,
|
||||
ttft_s=0.2, e2e_s=1.2,
|
||||
).__dict__,
|
||||
]
|
||||
summary = summarize_workloads(rows)[0]
|
||||
assert summary["requests"] == 2
|
||||
assert summary["output_length_attainment_rate"] == 0.5
|
||||
|
||||
|
||||
def test_error_classification_detects_rate_limit():
|
||||
exc = RuntimeError("429 rate limit exceeded")
|
||||
_, category, _ = error_details(exc)
|
||||
assert category == "rate_limit"
|
||||
|
||||
|
||||
def test_error_classification_does_not_mislabel_exhausted_quota_as_rate_limit():
|
||||
exc = RuntimeError("429 insufficient_quota: check billing")
|
||||
_, category, _ = error_details(exc)
|
||||
assert category == "quota_or_balance"
|
||||
|
||||
|
||||
class FakeAnthropicStream:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *_args):
|
||||
return False
|
||||
|
||||
def __iter__(self):
|
||||
return iter([
|
||||
SimpleNamespace(
|
||||
type="content_block_delta",
|
||||
delta=SimpleNamespace(type="thinking_delta", thinking="reason"),
|
||||
),
|
||||
SimpleNamespace(
|
||||
type="content_block_delta",
|
||||
delta=SimpleNamespace(type="text_delta", text="answer"),
|
||||
),
|
||||
])
|
||||
|
||||
def get_final_message(self):
|
||||
usage = SimpleNamespace(
|
||||
input_tokens=60, output_tokens=10,
|
||||
cache_creation_input_tokens=20, cache_read_input_tokens=20,
|
||||
)
|
||||
return SimpleNamespace(
|
||||
id="anthropic-1", usage=usage, content=[], stop_reason="end_turn"
|
||||
)
|
||||
|
||||
|
||||
def test_native_anthropic_adapter_records_cache_and_ttft():
|
||||
client = SimpleNamespace(
|
||||
messages=SimpleNamespace(stream=lambda **_kwargs: FakeAnthropicStream())
|
||||
)
|
||||
row = measure_anthropic(
|
||||
Provider("anthropic", "claude-test", "UNUSED", protocol="anthropic"),
|
||||
campaign_id="c", phase="workload", cell_id="a", prompt="hello",
|
||||
target_context_tokens=10, target_output_tokens=10, concurrency=1,
|
||||
request_index=0, client=client,
|
||||
)
|
||||
assert row.ok
|
||||
assert row.input_tokens == 100
|
||||
assert row.cached_input_tokens == 20
|
||||
assert row.output_tokens == 10
|
||||
assert row.visible_output_tokens > 0
|
||||
assert row.thinking_ttft_s is not None
|
||||
|
||||
|
||||
def test_native_gemini_adapter_records_thought_tokens():
|
||||
usage = SimpleNamespace(
|
||||
prompt_token_count=50, cached_content_token_count=10,
|
||||
candidates_token_count=8, thoughts_token_count=4,
|
||||
)
|
||||
parts = [
|
||||
SimpleNamespace(text="reason", thought=True),
|
||||
SimpleNamespace(text="answer", thought=False),
|
||||
]
|
||||
chunk = SimpleNamespace(
|
||||
usage_metadata=usage,
|
||||
candidates=[SimpleNamespace(
|
||||
finish_reason="STOP",
|
||||
content=SimpleNamespace(parts=parts),
|
||||
)],
|
||||
)
|
||||
client = SimpleNamespace(
|
||||
models=SimpleNamespace(generate_content_stream=lambda **_kwargs: iter([chunk]))
|
||||
)
|
||||
row = measure_gemini(
|
||||
Provider("gemini", "gemini-test", "UNUSED", protocol="gemini"),
|
||||
campaign_id="c", phase="workload", cell_id="g", prompt="hello",
|
||||
target_context_tokens=10, target_output_tokens=10, concurrency=1,
|
||||
request_index=0, client=client,
|
||||
)
|
||||
assert row.ok
|
||||
assert row.input_tokens == 50
|
||||
assert row.cached_input_tokens == 10
|
||||
assert row.reasoning_tokens == 4
|
||||
assert row.output_tokens == 12
|
||||
assert row.visible_output_tokens == 8
|
||||
|
||||
|
||||
def test_completion_requires_every_provider_availability_and_pinned_prices():
|
||||
config = {
|
||||
"workload": {"context_tokens": [8192], "output_tokens": [512]},
|
||||
"availability": {"duration_hours": 168, "interval_seconds": 3600},
|
||||
"rate_limit": {"concurrency_levels": [1], "requests_per_level": 1},
|
||||
"agent_cost": {"rounds": 1},
|
||||
"providers": [
|
||||
{
|
||||
"name": "provider-a", "model": "shared-model",
|
||||
"pricing": {
|
||||
"input_per_million": 1.0,
|
||||
"cached_input_per_million": 0.1,
|
||||
"output_per_million": 2.0,
|
||||
"currency": "USD",
|
||||
"source_url": "https://example.test/pricing",
|
||||
"as_of": "2026-07-29",
|
||||
"status": "verified",
|
||||
},
|
||||
},
|
||||
{
|
||||
"name": "provider-b", "model": "shared-model",
|
||||
"pricing": {
|
||||
"input_per_million": None,
|
||||
"cached_input_per_million": None,
|
||||
"output_per_million": None,
|
||||
"currency": None,
|
||||
"source_url": None,
|
||||
"as_of": None,
|
||||
"status": "unresolved",
|
||||
"blocker": "exact public model price not found",
|
||||
},
|
||||
},
|
||||
],
|
||||
"same_model_provider_groups": [{
|
||||
"logical_model": "shared-model",
|
||||
"providers": ["provider-a", "provider-b"],
|
||||
}],
|
||||
}
|
||||
workload = [{
|
||||
"provider": "provider-a", "target_context_tokens": 8192,
|
||||
"target_output_tokens": 512, "requests": 100, "successes": 100,
|
||||
}]
|
||||
availability = [{
|
||||
"provider": "provider-a", "model": "shared-model", "probes": 169,
|
||||
"observed_start_utc": "2026-01-01T00:00:00+00:00",
|
||||
"observed_end_utc": "2026-01-08T00:00:00+00:00",
|
||||
}]
|
||||
|
||||
audit = completion_audit(config, workload, availability, [], [], [])
|
||||
|
||||
assert audit["checks"]["configuration_matches_exact_8k_32k_128k_x_512_2048_design"] is False
|
||||
assert audit["checks"]["availability_observed_for_at_least_168_hours"] is False
|
||||
assert audit["missing_availability_providers"] == ["provider-b"]
|
||||
assert audit["checks"]["cached_input_output_pricing_complete"] is False
|
||||
assert audit["pricing_config_gaps"][0]["provider"] == "provider-b"
|
||||
assert audit["checks"]["same_model_compared_across_providers"] is False
|
||||
assert audit["same_model_provider_gaps"]
|
||||
|
||||
|
||||
def test_non_usd_pricing_requires_dated_fx_for_comparable_cost():
|
||||
price = Price(
|
||||
20.0, 2.0, 100.0,
|
||||
currency="CNY",
|
||||
source_url="https://example.test/cny-pricing",
|
||||
as_of="2026-07-29",
|
||||
status="verified_native",
|
||||
)
|
||||
assert price.native_rates_complete is True
|
||||
assert price.usd_conversion_complete is False
|
||||
|
||||
converted = Price(
|
||||
20.0, 2.0, 100.0,
|
||||
currency="CNY",
|
||||
source_url="https://example.test/cny-pricing",
|
||||
as_of="2026-07-29",
|
||||
usd_per_currency_unit=0.139,
|
||||
fx_source_url="https://example.test/fx",
|
||||
fx_as_of="2026-07-29",
|
||||
status="verified_with_fx",
|
||||
)
|
||||
assert converted.usd_conversion_complete is True
|
||||
|
||||
|
||||
def test_markdown_tolerates_null_percentage_fields() -> None:
|
||||
report = {
|
||||
"campaign_id": "exp7-10-test",
|
||||
"observation_count": 0,
|
||||
"completion_audit": {"official_complete": False},
|
||||
"workload": [
|
||||
{
|
||||
"provider": "openai",
|
||||
"target_context_tokens": 8192,
|
||||
"target_output_tokens": 512,
|
||||
"requests": 0,
|
||||
"success_rate": None,
|
||||
"ttft_s": {"p50": None, "p95": None, "p99": None},
|
||||
"e2e_s": {"p50": None, "p95": None, "p99": None},
|
||||
"input_prefill_throughput_tokens_s": {"p50": None},
|
||||
"output_throughput_tokens_s": {"p50": None},
|
||||
"output_length_attainment_rate": None,
|
||||
"reasoning_tokens": {"p50": None},
|
||||
"thinking_ttft_s": {"p50": None},
|
||||
}
|
||||
],
|
||||
"availability": [
|
||||
{
|
||||
"provider": "openai",
|
||||
"probes": 0,
|
||||
"uptime": None,
|
||||
"outage_count": 0,
|
||||
"mttr_s": None,
|
||||
"longest_continuous_availability_s": 0.0,
|
||||
}
|
||||
],
|
||||
"rate_limits": [
|
||||
{
|
||||
"provider": "openai",
|
||||
"concurrency": 1,
|
||||
"success_rate": None,
|
||||
"measured_rpm": None,
|
||||
"measured_input_tpm": None,
|
||||
"measured_output_tpm": None,
|
||||
}
|
||||
],
|
||||
"costs": [],
|
||||
"external_benchmark_comparison": [],
|
||||
}
|
||||
|
||||
result = markdown(report)
|
||||
assert "| openai | 8192 | 512 | 0 | — | —/—/— | —/—/— | — | — | — | — | — |" in result
|
||||
assert "| openai | 0 | — | 0 | — | 0.00 |" in result
|
||||
assert "| openai | 1 | — | — | — | — |" in result
|
||||
|
||||
def test_export_campaign_summary_returns_report_and_markdown(tmp_path: Path) -> None:
|
||||
from analysis import export_campaign_summary
|
||||
|
||||
store = CampaignStore(tmp_path / "summary.sqlite3")
|
||||
store.bind_campaign("exp-test", {"providers": [], "workload": {}, "availability": {}, "rate_limit": {}, "agent_cost": {}})
|
||||
store.close()
|
||||
|
||||
exported = export_campaign_summary(tmp_path / "summary.sqlite3", campaign_id="exp-test")
|
||||
assert "report" in exported
|
||||
assert "markdown" in exported
|
||||
assert exported["official_complete"] is False
|
||||
Reference in New Issue
Block a user