ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,4 @@
|
||||
output/
|
||||
.env
|
||||
__pycache__/
|
||||
*.pyc
|
||||
@@ -0,0 +1,293 @@
|
||||
# TTS Quality Evaluation Pipeline / TTS 质量评估流水线
|
||||
|
||||
## English
|
||||
|
||||
This project implements an end-to-end benchmark pipeline for TTS quality across multiple providers and configurations. The same source scripts are synthesized, then evaluated with an LLM-as-a-Judge rubric.
|
||||
|
||||
It compares:
|
||||
- provider differences (OpenAI, ElevenLabs, Fish Audio, Minimax, Doubao)
|
||||
- model / voice / speed settings
|
||||
- objective speech metrics and rubric-based subjective dimensions
|
||||
|
||||
The workflow is fully reproducible and can run offline checks when API keys are unavailable.
|
||||
|
||||
### Goals
|
||||
|
||||
Answer practical questions such as:
|
||||
- How much difference exists between `tts-1` and `tts-1-hd`?
|
||||
- What is the quality cost of changing voice or speed (for example 1.5x)?
|
||||
|
||||
The pipeline answers these through a single command and produces a structured comparison report.
|
||||
|
||||
### Evaluation dimensions
|
||||
|
||||
The acceptance path sends both the synthesized audio and a fixed real reference
|
||||
clip to an audio-capable judge and records the manuscript's exact four dimensions:
|
||||
|
||||
- Accuracy: omissions, substitutions, additions, numbers, names, and polyphones
|
||||
- Naturalness: machine artifacts, pauses, emphasis, rhythm, and fluency
|
||||
- Emotional expression: match between audible delivery and the requested emotion
|
||||
- Voice consistency: speaker similarity against the simultaneously supplied reference audio
|
||||
|
||||
CER-based objective metrics are computed with normalized transcript comparison.
|
||||
|
||||
### Provider support
|
||||
|
||||
- TTS synthesis is implemented for multiple providers (OpenAI via SDK, others via REST).
|
||||
- Default run covers 4 OpenAI configurations with only `OPENAI_API_KEY`.
|
||||
- `--providers` enables cross-provider comparisons.
|
||||
- Missing key -> that provider is skipped; the benchmark continues.
|
||||
|
||||
### Judge/backend details
|
||||
|
||||
- The manuscript-grade path is retained under the backward-compatible `--gemini` flag. It directly sends both clips through the configured Google Gemini, OpenRouter, or Mistral Voxtral audio route; no route substitutes transcripts for either clip.
|
||||
- Optional `--with-asr` adds Whisper/CER as a secondary objective measure.
|
||||
- The transcript-only LLM path remains a diagnostic fallback and is explicitly marked incomplete because it cannot judge emotion or speaker identity.
|
||||
|
||||
### Files
|
||||
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `config.py` | providers, model pricing, configs, corpus |
|
||||
| `pipeline.py` | synthesis, ffprobe duration, transcription, CER, rubric scoring |
|
||||
| `demo.py` | command entry, run grid, output summaries |
|
||||
| `tests/` | offline regression tests for judge-response robustness |
|
||||
| `requirements.txt` / `env.example` | dependencies and env template |
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 6 environment
|
||||
uv sync --locked --python 3.12 --extra ch6
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch6]"
|
||||
|
||||
cd chapter7/tts-quality-eval
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
brew install ffmpeg
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
python demo.py
|
||||
python demo.py --quick
|
||||
python demo.py --extra
|
||||
python demo.py --providers openai,fishaudio --gemini --fresh
|
||||
python demo.py --providers openai,fishaudio --gemini --with-asr
|
||||
python demo.py --fresh
|
||||
python demo.py --providers openai,minimax,elevenlabs
|
||||
python demo.py --text "2026年营收增长37.5%"
|
||||
python demo.py --judge-model gpt-5.6-luna
|
||||
python demo.py --output ./runs/exp1
|
||||
python demo.py --list-providers
|
||||
python demo.py --dump-rubric
|
||||
```
|
||||
|
||||
Outputs are under `output/` (audio) and `output/results.json` (structured results).
|
||||
|
||||
### Tests
|
||||
|
||||
```bash
|
||||
# From the repository root, include dev tools for pytest
|
||||
uv sync --locked --python 3.12 --extra ch6 --extra dev
|
||||
source .venv/bin/activate
|
||||
cd chapter7/tts-quality-eval
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
### Robustness notes
|
||||
|
||||
- Required-key (`OPENAI_API_KEY`) and ffprobe checks fail fast with clear instructions; a provider-specific missing key only marks that provider's cells as failed without stopping the run.
|
||||
- A single failed (provider, text) cell does not stop the full run.
|
||||
- OpenAI SDK is configured with retries.
|
||||
|
||||
### Limitations
|
||||
|
||||
- `--gemini` requires `GEMINI_API_KEY`, `OPENROUTER_API_KEY`, or `MISTRAL_API_KEY` plus a real reference-audio file; the default is the immutable
|
||||
Chapter 9 Fish S1 reference clip and its SHA-256 is saved in the report.
|
||||
- CER is optional and depends on Whisper accuracy; it is not substituted for direct listening.
|
||||
- Scores are comparative experimental measurements, not absolute quality certification.
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
# 实验 7-6：全自动 TTS 质量评估流水线
|
||||
|
||||
配套《深入理解 AI Agent》第 6 章「实验 7-6 ★★：构建全自动 TTS 质量评估流水线」。
|
||||
|
||||
用多个 **TTS provider / 配置**（OpenAI、ElevenLabs、Fish Audio、Minimax、豆包，或同
|
||||
一家的不同 model / voice / speed）合成同一组带挑战性的参考文本，再用
|
||||
**多模态 LLM-as-a-Judge** 的思路对合成语音按 **Rubric** 逐维度打分，最后汇总成一张
|
||||
**对比表**，反映不同 provider / 配置在准确性 / 自然度上的优劣。
|
||||
|
||||
## 目的
|
||||
|
||||
回答工程中的实际问题：*同一段文本，`tts-1` 和 `tts-1-hd` 有多大差距？换 voice、把
|
||||
语速调到 1.5x 会牺牲多少质量？* 本 demo 把这类对比做成**一条命令跑通、可复现**的流水线。
|
||||
|
||||
## 评审维度与 Rubric
|
||||
|
||||
对每条合成语音，音频多模态评审模型同时接收**合成语音、原文、目标情感和固定参考语音**，按正文
|
||||
精确规定的四维 Rubric 逐项 1–5 分打分：
|
||||
|
||||
| 维度 | 含义 |
|
||||
|------|------|
|
||||
| 准确性 | 直接听辨漏读/错读/添读，以及数字、专名和多音字 |
|
||||
| 自然度 | 流畅度、机器感、停顿、重音与韵律是否符合人类习惯 |
|
||||
| 情感表达 | 语调、语速和强调是否符合中性、兴奋、悲伤、疑问等目标情感 |
|
||||
| 音色一致性 | 与同时提供的固定参考语音比较说话人音色 |
|
||||
|
||||
客观指标 **CER（字错误率）/ 字准确率**：把 Whisper 回译文本与原文归一化（去标点空白、
|
||||
统一大小写）后做字符级编辑距离，`CER = 编辑距离 / 参考字数`，`字准确率 = 1 - CER`。
|
||||
中文按**字级**计算（等价于书中 WER 的可懂度维度）。
|
||||
|
||||
## Provider 适配说明
|
||||
|
||||
- **TTS 合成（多 provider）**：对应书中「接入主流服务：OpenAI、ElevenLabs、Fish Audio、
|
||||
Minimax、豆包」。每个 provider 按各家公开 REST 接口实现（OpenAI 走官方 SDK，其余走内置
|
||||
`urllib`，无额外依赖）。默认（不加 `--providers`）只跑 OpenAI 的 4 个配置，保证单个
|
||||
`OPENAI_API_KEY` 即可零配置跑通；`--providers openai,minimax,...` 做跨服务商横向对比。
|
||||
各 provider 所需环境变量与 voice 字段语义见 `python demo.py --list-providers`。
|
||||
|
||||
| provider | 环境变量 | voice 语义 |
|
||||
|----------|----------|-----------|
|
||||
| `openai` | `OPENAI_API_KEY` | alloy/nova…；model=tts-1 / tts-1-hd / gpt-4o-mini-tts |
|
||||
| `elevenlabs` | `ELEVENLABS_API_KEY` | voice_id；model 默认 eleven_multilingual_v2 |
|
||||
| `fishaudio` | `FISH_API_KEY`（别名 `FISHAUDIO_API_KEY`） | reference_id（留空用默认音色） |
|
||||
| `minimax` | `MINIMAX_API_KEY`（可选 `MINIMAX_REGION`） | voice_id；model 默认 speech-2.8-hd（另有 speech-2.8-turbo） |
|
||||
| `doubao` | `DOUBAO_APP_ID` + `DOUBAO_ACCESS_TOKEN` | voice_type |
|
||||
|
||||
> 说明：本仓库的 **OpenAI 与 Fish Audio** 路径已有端到端保存证据；其余三家按各自公开 REST 文档实现，请用自己
|
||||
> 账号可用的 voice/model 覆盖 `config.PROVIDER_CONFIGS` 后使用。缺对应 key 时该 provider
|
||||
> 的行会被记为失败，**不中断整表**。
|
||||
- **诊断回退（非验收）**：可用 Whisper（`whisper-1`）把合成语音回译成文本算 CER，再用
|
||||
文本模型基于「转写文本 + 时长 + 语速 + CER」打分；该路径听不到音频，情感表达和音色
|
||||
一致性会明确记为 0，因此不能作为实验 7-6 的完成证据。
|
||||
转写时用简体中文提示语引导 Whisper 输出简体，避免繁体字形差异虚高 CER。
|
||||
**凭据/回退**：TTS 合成与 Whisper 回译必须走 **OpenAI 直连**（`OPENAI_API_KEY`，
|
||||
OpenRouter 不提供音频/转写）；**仅 LLM Rubric 的 chat 评审支持 OpenRouter 回退**——
|
||||
`gpt-5.x` 直连需组织实名认证，故只要设置了 `OPENROUTER_API_KEY`，评审就优先走
|
||||
OpenRouter（`gpt-*` 映射为 `openai/*`）。
|
||||
- **质量评审（正文验收路径）**：`--gemini`（保留的兼容参数名）让**音频多模态模型同时听合成音频和参考音频**
|
||||
（原文 + 目标情感 + 两段音频 + Rubric 一起输入）。程序先尝试 `GEMINI_API_KEY` 的
|
||||
Google 直连；若直连凭据不可用但有 `OPENROUTER_API_KEY`，则把两段原始音频以
|
||||
`input_audio` 发送给 OpenRouter；若前两路不可用且配置了 `MISTRAL_API_KEY`，再用
|
||||
Mistral 原生 data-URL `input_audio` 格式把同两段 MP3 交给 `voxtral-small-latest`。
|
||||
可用 `TTS_AUDIO_JUDGE_MODEL` / `TTS_MISTRAL_AUDIO_JUDGE_MODEL` 覆盖模型。三条路径都会记录实际模型和脱敏的
|
||||
provider attempt，不会把 key 写入结果。
|
||||
|
||||
> `--gemini` 才是正文方案。程序默认复用第 9 章固定的真实参考片段，并把参考片段与每条
|
||||
> 合成音频的 SHA-256 都写入结果。回译路径只是故障诊断，不能冒充音频评审。
|
||||
|
||||
## 文件
|
||||
|
||||
| 文件 | 说明 |
|
||||
|------|------|
|
||||
| `config.py` | 模型名与单价、provider 注册表（`PROVIDERS` / `PROVIDER_CONFIGS`）、TTS 配置集合、测试语料 |
|
||||
| `pipeline.py` | 多 provider 合成分发 / ffprobe 时长 / Whisper 回译 / CER 计算 / LLM Rubric / Gemini、OpenRouter、Voxtral 双音频评审 |
|
||||
| `demo.py` | 入口：多配置 × 多语料跑全流程，打印逐条明细 + 对比汇总表 |
|
||||
| `tests/` | 离线回归测试，覆盖评审响应健壮性 |
|
||||
| `requirements.txt` / `env.example` | 依赖与环境变量示例 |
|
||||
|
||||
## 运行
|
||||
|
||||
```bash
|
||||
# 在仓库根目录使用统一的第 6 章环境
|
||||
uv sync --locked --python 3.12 --extra ch6
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch6]"
|
||||
|
||||
cd chapter7/tts-quality-eval
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
brew install ffmpeg # 提供 ffprobe（时长探测）
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
python demo.py # 诊断回退：4 个 OpenAI 配置 × 6 条语料
|
||||
python demo.py --quick # 只用前 2 条语料，快速冒烟
|
||||
python demo.py --extra # 额外加入 gpt-4o-mini-tts 配置
|
||||
python demo.py --providers openai,fishaudio --gemini --fresh
|
||||
python demo.py --providers openai,fishaudio --gemini --with-asr
|
||||
python demo.py --providers openai,fishaudio --gemini --limit 4
|
||||
python demo.py --fresh # 忽略已有音频全部重合成
|
||||
|
||||
# 多 provider / 自定义输入（新增）
|
||||
python demo.py --providers openai,minimax,elevenlabs # 跨服务商横向对比（需各自 key）
|
||||
python demo.py --text '2026年营收增长37.5%' # 用一段自定义文本替换语料库
|
||||
python demo.py --judge-model gpt-5.6-luna # 覆盖 LLM 评审模型
|
||||
python demo.py --output ./runs/exp1 # 自定义输出目录
|
||||
|
||||
# 离线（无需任何 API key）
|
||||
python demo.py --list-providers # 查看所有 provider 及配置状态
|
||||
python demo.py --dump-rubric # 查看 Rubric 维度定义
|
||||
```
|
||||
|
||||
完整参数见 `python demo.py --help`（全中文）。合成音频写入 `output/`（已被 `.gitignore`
|
||||
忽略），结构化结果写入 `output/results.json`（可用 `--output` 改目录）。
|
||||
**幂等**：默认复用已存在的音频，重复运行不会重复合成。
|
||||
|
||||
## 测试
|
||||
|
||||
```bash
|
||||
# 在仓库根目录安装 pytest 等开发工具
|
||||
uv sync --locked --python 3.12 --extra ch6 --extra dev
|
||||
source .venv/bin/activate
|
||||
cd chapter7/tts-quality-eval
|
||||
python -m pytest tests
|
||||
```
|
||||
|
||||
## 当前真实验收状态（2026-07-30）
|
||||
|
||||
[`validation/mistral_multimodal_20260730/results.json`](validation/mistral_multimodal_20260730/results.json)
|
||||
与同目录的 [`manifest.json`](validation/mistral_multimodal_20260730/manifest.json) 保存当前
|
||||
完整验收：OpenAI `tts-1/alloy` 与 Fish S1 两个真实合成 provider，覆盖数字、多音字、
|
||||
长句和兴奋情感四类文本，共 8/8 单元。每个单元把候选 MP3 与固定真实参考 MP3 一起交给
|
||||
Mistral `voxtral-small-latest`，四维分数均为 1–5 整数；Fish 四维均分为
|
||||
5.00/4.00/4.00/3.00，OpenAI 为 5.00/4.00/3.75/2.75。manifest 会复核结果、参考音频、
|
||||
八段候选音频和前序合成结果的 SHA-256；合成音频是前序真实 provider 运行的留存产物，
|
||||
不是在 OpenAI 余额耗尽后伪造的新合成。
|
||||
|
||||
早期 [`real_multimodal_20260730`](validation/real_multimodal_20260730/manifest.json) 与
|
||||
[`audio_fallback_probe_20260730`](validation/audio_fallback_probe_20260730/manifest.json)
|
||||
仍保留 Google key 无效、OpenRouter 401 和 OpenAI 新合成余额不足的负面证据；它们是
|
||||
故障历史，不再代表当前 Voxtral 直接听评的验收状态。
|
||||
|
||||
## 测试语料
|
||||
|
||||
6 条覆盖数字/百分比/日期、多音字（行/长/重/还）、长句新闻文体、专有名词与兴奋情感、
|
||||
悲伤内容、疑问句升调。可在 `config.py` 的 `CORPUS` 中增删。
|
||||
|
||||
## 健壮性
|
||||
|
||||
- 缺 `OPENAI_API_KEY` 立即清晰报错退出；缺 `ffprobe` 给出安装提示。
|
||||
- 单个（配置, 语料）在合成/转写/评审任一步失败，只把该条记为失败，**不中断整表**，
|
||||
汇总表按成功条数聚合。
|
||||
- OpenAI 客户端带自动重试（`max_retries=5`）缓解偶发网络抖动。
|
||||
- ffprobe 调用检查返回码与输出可解析性。
|
||||
|
||||
## 局限
|
||||
|
||||
- 不加 `--gemini` 的回译评审看不到音频，只是诊断模式；它会显式标记实验未验收。
|
||||
- 默认参考音频来自第 9 章 Fish S1 固定媒体库。更换参考说话人时必须通过
|
||||
`--reference-audio` 明确提供，并保留结果中的内容哈希。
|
||||
- CER 依赖 Whisper 转写质量，Whisper 自身错误会引入噪声；数字/专名可能因书写形式
|
||||
（阿拉伯数字 vs 中文数字）产生非发音性差异。
|
||||
- Rubric 由 LLM 打分，存在评审模型偏好；分数用于**相对对比**而非绝对基准。
|
||||
@@ -0,0 +1,203 @@
|
||||
"""实验 7-6：全自动 TTS 质量评估流水线 —— 配置与测试语料。
|
||||
|
||||
本模块集中管理：
|
||||
- 用到的 OpenAI 模型名与计费单价（仅供参考成本估算）；
|
||||
- 多个 TTS「配置」（model / voice / speed 的组合，作为待对比的对象）；
|
||||
- 一组带挑战性的参考文本（数字 / 多音字 / 长句 / 专有名词 + 情感）。
|
||||
"""
|
||||
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 模型名（均为 OpenAI，读 OPENAI_API_KEY）。
|
||||
# ---------------------------------------------------------------------------
|
||||
WHISPER_MODEL = "whisper-1" # 语音转写（回译），用于计算 WER/字准确率（须走 OpenAI 直连）
|
||||
JUDGE_MODEL = "gpt-5.6-luna" # LLM Rubric 评审模型（当前廉价旗舰；chat 调用可回退 OpenRouter）
|
||||
|
||||
# 可选的 Gemini 音频评审（书中方案）。默认用当前廉价旗舰 gemini-3.5-flash（已验证支持
|
||||
# 音频输入，能直接「听」合成语音）。模型名可能随时间过期，运行时会通过 REST /models
|
||||
# 探测校正。仅当 --gemini 开启时才会用到。
|
||||
GEMINI_MODEL_DEFAULT = "gemini-3.5-flash"
|
||||
|
||||
# 计费单价（美元），仅用于打印粗略成本，不影响评分。数值随官方调整可能变化。
|
||||
PRICE = {
|
||||
"tts-1": 15.0 / 1_000_000, # $ / 字符
|
||||
"tts-1-hd": 30.0 / 1_000_000, # $ / 字符
|
||||
"gpt-4o-mini-tts": 12.0 / 1_000_000,
|
||||
"whisper-1": 0.006 / 60, # $ / 秒
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class TTSConfig:
|
||||
"""一个待评估的 TTS 配置。name 需在整表内唯一。
|
||||
|
||||
provider 指明合成走哪个服务商（openai / elevenlabs / fishaudio / minimax /
|
||||
doubao）。model / voice / speed 的语义由各 provider 自行解释：例如 elevenlabs
|
||||
的 voice 是 voice_id，fishaudio 的 voice 是 reference_id（可留空用默认音色）。
|
||||
"""
|
||||
|
||||
name: str
|
||||
model: str
|
||||
voice: str
|
||||
speed: float = 1.0
|
||||
provider: str = "openai"
|
||||
|
||||
def supports_speed(self) -> bool:
|
||||
# 只有部分 provider/模型支持 speed 参数；不支持时忽略该字段。
|
||||
if self.provider == "openai":
|
||||
return self.model in ("tts-1", "tts-1-hd")
|
||||
return self.provider in ("minimax", "doubao")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 多 provider 注册表（对应书中「接入主流服务：OpenAI、ElevenLabs、Fish Audio、
|
||||
# Minimax、豆包」）。每个 provider 声明所需环境变量与一个代表性配置，便于跨服务商
|
||||
# 横向对比。除 OpenAI 外均按各家公开 REST 接口实现，缺 key 时该 provider 的行会被
|
||||
# 记为失败而不影响整表（见 demo.py）。
|
||||
# ---------------------------------------------------------------------------
|
||||
# 环境变量别名：同一凭据可能有多个历史/惯用名，任意一个被设置即视为已配置。
|
||||
ENV_ALIASES = {
|
||||
"FISH_API_KEY": ("FISH_API_KEY", "FISHAUDIO_API_KEY"),
|
||||
}
|
||||
|
||||
|
||||
def env_get(name: str) -> str:
|
||||
"""读取环境变量，支持 ENV_ALIASES 中登记的别名，返回第一个非空值（已 strip）。"""
|
||||
import os
|
||||
for n in ENV_ALIASES.get(name, (name,)):
|
||||
val = os.environ.get(n, "").strip()
|
||||
if val:
|
||||
return val
|
||||
return ""
|
||||
|
||||
|
||||
@dataclass
|
||||
class ProviderInfo:
|
||||
key: str # 内部标识（--providers 用）
|
||||
label: str # 展示名
|
||||
env: tuple # 该 provider 合成所需的环境变量名
|
||||
note: str # 一句话说明 voice 字段语义等
|
||||
|
||||
def configured(self) -> bool:
|
||||
return all(env_get(e) for e in self.env)
|
||||
|
||||
|
||||
PROVIDERS = {
|
||||
"openai": ProviderInfo(
|
||||
"openai", "OpenAI", ("OPENAI_API_KEY",),
|
||||
"voice=alloy/nova/…，model=tts-1/tts-1-hd/gpt-4o-mini-tts；本仓库唯一端到端验证过的 provider。",
|
||||
),
|
||||
"elevenlabs": ProviderInfo(
|
||||
"elevenlabs", "ElevenLabs", ("ELEVENLABS_API_KEY",),
|
||||
"voice=voice_id，model 默认 eleven_multilingual_v2（多语言/中文）。",
|
||||
),
|
||||
"fishaudio": ProviderInfo(
|
||||
"fishaudio", "Fish Audio", ("FISH_API_KEY",),
|
||||
"voice=reference_id（留空用默认音色），走 /v1/tts；key 亦可用别名 FISHAUDIO_API_KEY。",
|
||||
),
|
||||
"minimax": ProviderInfo(
|
||||
"minimax", "Minimax", ("MINIMAX_API_KEY",),
|
||||
"voice=voice_id，model 默认 speech-2.8-hd（另有 speech-2.8-turbo）；Bearer 鉴权，"
|
||||
"MINIMAX_REGION 选 global(api.minimax.io)/cn(api.minimaxi.com)。",
|
||||
),
|
||||
"doubao": ProviderInfo(
|
||||
"doubao", "豆包（火山引擎）", ("DOUBAO_APP_ID", "DOUBAO_ACCESS_TOKEN"),
|
||||
"voice=voice_type，走 openspeech.bytedance.com；鉴权头为 'Bearer;{token}'。",
|
||||
),
|
||||
}
|
||||
|
||||
# 各 provider 的代表性配置（--providers 选中时，每个 provider 取这一条参与对比）。
|
||||
# 非 OpenAI 的 voice/model 取各家常见默认值，可在此按账号可用音色调整。
|
||||
PROVIDER_CONFIGS = {
|
||||
# Reuse the identical default-grid identity so a cross-provider campaign
|
||||
# can audit an existing OpenAI artifact instead of synthesizing it twice.
|
||||
"openai": TTSConfig("tts1-alloy-1.0", provider="openai", model="tts-1", voice="alloy"),
|
||||
"elevenlabs": TTSConfig("elevenlabs-multi", provider="elevenlabs",
|
||||
model="eleven_multilingual_v2", voice="21m00Tcm4TlvDq8ikWAM"),
|
||||
# This immutable reference ID is the same real source voice used to build
|
||||
# Chapter 9's checked-in 24-clip Fish S1 library. Accounts that cannot
|
||||
# access it can supply FISH_REFERENCE_ID explicitly; an empty/default
|
||||
# voice would make the voice-consistency arm scientifically meaningless.
|
||||
"fishaudio": TTSConfig(
|
||||
"fishaudio-s1-clone",
|
||||
provider="fishaudio",
|
||||
model="s1",
|
||||
voice=os.getenv("FISH_REFERENCE_ID", "6df3c1e14c9440e9ac978556536bf116"),
|
||||
),
|
||||
"minimax": TTSConfig("minimax-hd", provider="minimax",
|
||||
model="speech-2.8-hd", voice="male-qn-qingse"),
|
||||
"doubao": TTSConfig("doubao-tts", provider="doubao",
|
||||
model="volcano_tts", voice="zh_female_qingxin"),
|
||||
}
|
||||
|
||||
|
||||
# 默认对比的配置集合：覆盖 model（tts-1 vs tts-1-hd）、voice、speed 三个维度，
|
||||
# 便于观察不同配置在准确性/自然度上的差异。默认全部走 OpenAI 以保证零额外配置跑通。
|
||||
TTS_CONFIGS = [
|
||||
TTSConfig("tts1-alloy-1.0", model="tts-1", voice="alloy", speed=1.0),
|
||||
TTSConfig("tts1hd-alloy-1.0", model="tts-1-hd", voice="alloy", speed=1.0),
|
||||
TTSConfig("tts1-nova-1.0", model="tts-1", voice="nova", speed=1.0),
|
||||
TTSConfig("tts1-alloy-1.5", model="tts-1", voice="alloy", speed=1.5),
|
||||
]
|
||||
|
||||
# 可选加入（--extra 开启）：gpt-4o-mini-tts。默认不加入以保证一定跑通。
|
||||
EXTRA_CONFIGS = [
|
||||
TTSConfig("4omini-nova-1.0", model="gpt-4o-mini-tts", voice="nova", speed=1.0),
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class Sample:
|
||||
"""一条参考文本 + 期望情感标签（供 Rubric 情感维度参考）。"""
|
||||
|
||||
id: str
|
||||
text: str
|
||||
challenge: str # 该样本主要考察的挑战点
|
||||
emotion: str = "中性"
|
||||
|
||||
|
||||
# 多样化测试语料：数字/日期、多音字、长句、专有名词+情感。
|
||||
CORPUS = [
|
||||
Sample(
|
||||
id="num",
|
||||
text="2026年第三季度营收增长了37.5%，同比提升12个百分点。",
|
||||
challenge="数字/百分比/日期",
|
||||
emotion="中性",
|
||||
),
|
||||
Sample(
|
||||
id="polyphone",
|
||||
text="银行行长正在重新调整这件事的重点，长此以往，还得还清所有欠款。",
|
||||
challenge="多音字（行/长/重/还）",
|
||||
emotion="中性",
|
||||
),
|
||||
Sample(
|
||||
id="long",
|
||||
text="据报道，随着人工智能技术的快速发展，越来越多的企业开始将大语言模型"
|
||||
"应用于客户服务、内容创作和数据分析等场景，从而显著提升了运营效率。",
|
||||
challenge="长句/新闻文体",
|
||||
emotion="中性",
|
||||
),
|
||||
Sample(
|
||||
id="emotion",
|
||||
text="太棒了！OpenAI 刚刚发布的新模型在 GAIA 基准测试上表现惊人！",
|
||||
challenge="专有名词 + 感叹情感",
|
||||
emotion="兴奋",
|
||||
),
|
||||
Sample(
|
||||
id="sad",
|
||||
text="很遗憾地通知您，救援队今天仍然没有找到失踪的登山者。",
|
||||
challenge="悲伤情感/低语速低语调",
|
||||
emotion="悲伤",
|
||||
),
|
||||
Sample(
|
||||
id="question",
|
||||
text="请问您希望把明天下午三点的预约改到星期五上午吗？",
|
||||
challenge="对话文体/疑问句升调",
|
||||
emotion="礼貌询问",
|
||||
),
|
||||
]
|
||||
@@ -0,0 +1,418 @@
|
||||
"""实验 7-6：全自动 TTS 质量评估流水线 —— 一条命令跑通。
|
||||
|
||||
python demo.py # 默认 4 个 OpenAI 配置 x 4 条语料
|
||||
python demo.py --providers openai,minimax # 跨服务商横向对比
|
||||
python demo.py --text '一段话' # 自定义文本
|
||||
python demo.py --gemini # 评审改用多模态模型直接听两段音频
|
||||
python demo.py --quick # 只用前 2 条语料，快速冒烟
|
||||
python demo.py --list-providers # 离线：查看 provider 及配置状态
|
||||
python demo.py --dump-rubric # 离线：查看 Rubric 维度定义
|
||||
|
||||
流程：多 provider TTS 合成 -> ffprobe 时长 -> Whisper 回译 -> CER/字准确率
|
||||
-> LLM/多模态音频 Rubric 打分 -> 打印逐条明细 + 配置对比汇总表。
|
||||
幂等：音频写入 output/ 并复用（除非 --fresh）。完整参数见 `python demo.py --help`。
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import traceback
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from statistics import mean
|
||||
|
||||
import config
|
||||
import pipeline
|
||||
|
||||
OUT_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "output")
|
||||
DEFAULT_REFERENCE_AUDIO = str(
|
||||
Path(__file__).resolve().parents[2]
|
||||
/ "chapter9"
|
||||
/ "controllable-tts"
|
||||
/ "reference_audio"
|
||||
/ "neutral_normal_formal.mp3"
|
||||
)
|
||||
|
||||
|
||||
def load_env():
|
||||
"""极简 .env 加载（不引第三方依赖）。"""
|
||||
path = os.path.join(os.path.dirname(os.path.abspath(__file__)), ".env")
|
||||
if os.path.exists(path):
|
||||
for line in open(path, encoding="utf-8"):
|
||||
line = line.strip()
|
||||
if line and not line.startswith("#") and "=" in line:
|
||||
k, v = line.split("=", 1)
|
||||
os.environ.setdefault(k.strip(), v.strip())
|
||||
|
||||
|
||||
def audio_path(cfg_name: str, sample_id: str) -> str:
|
||||
return os.path.join(OUT_DIR, f"{cfg_name}__{sample_id}.mp3")
|
||||
|
||||
|
||||
def evaluate_one(
|
||||
cfg,
|
||||
sample,
|
||||
use_gemini: bool,
|
||||
fresh: bool,
|
||||
judge_model: str = None,
|
||||
reference_audio: str = "",
|
||||
with_asr: bool = False,
|
||||
) -> dict:
|
||||
"""对单个 (配置, 语料) 跑完整链路。任一步失败返回 error 记录，不抛出。"""
|
||||
rec = {"config": cfg.name, "sample": sample.id, "challenge": sample.challenge,
|
||||
"provider": getattr(cfg, "provider", "openai"), "ok": False, "error": None}
|
||||
path = audio_path(cfg.name, sample.id)
|
||||
stage = "synthesis"
|
||||
try:
|
||||
# 1) 合成（幂等：已存在且非 fresh 则复用）
|
||||
if fresh or not os.path.exists(path) or os.path.getsize(path) == 0:
|
||||
pipeline.synthesize(cfg, sample.text, path)
|
||||
# Preserve successful synthesis evidence even if a later ASR/judge
|
||||
# stage fails. This keeps provider progress auditable without
|
||||
# incorrectly marking the end-to-end cell complete.
|
||||
rec.update({
|
||||
"audio_path": os.path.relpath(path, OUT_DIR),
|
||||
"audio_sha256": pipeline.sha256_file(path),
|
||||
"audio_bytes": os.path.getsize(path),
|
||||
})
|
||||
# 2) 时长
|
||||
stage = "duration_probe"
|
||||
dur = pipeline.probe_duration(path)
|
||||
# 3) Optional objective ASR. Direct-audio Gemini judging does not
|
||||
# require OpenAI/Whisper and therefore keeps Fish-only runs possible.
|
||||
stage = "optional_asr"
|
||||
hyp = pipeline.transcribe(path) if (with_asr or not use_gemini) else None
|
||||
er = pipeline.char_error_rate(sample.text, hyp) if hyp is not None else None
|
||||
# 4) Rubric: manuscript acceptance requires direct audio plus a real
|
||||
# reference clip, not a transcript-only inference.
|
||||
stage = "multimodal_judge" if use_gemini else "text_judge"
|
||||
if use_gemini:
|
||||
rub = pipeline.judge_gemini_audio(
|
||||
sample.text, sample.emotion, path, reference_audio
|
||||
)
|
||||
else:
|
||||
rub = pipeline.judge_rubric(
|
||||
sample.text, sample.emotion, hyp or "", dur, er.cer if er else 0.0,
|
||||
model=judge_model,
|
||||
)
|
||||
rec.update({
|
||||
"ok": True,
|
||||
"duration": dur,
|
||||
"hypothesis": hyp,
|
||||
"cer": er.cer if er else None,
|
||||
"asr_accuracy": er.accuracy if er else None,
|
||||
"speed": (len(pipeline.normalize(sample.text)) / dur) if dur else 0.0,
|
||||
"scores": rub.scores, "reasons": rub.reasons,
|
||||
"judge_model": rub.judge_model,
|
||||
"evidence_mode": rub.evidence_mode,
|
||||
"judge_provider_attempts": rub.provider_attempts,
|
||||
})
|
||||
except Exception as e: # 单条失败不影响整表
|
||||
rec["failed_stage"] = stage
|
||||
rec["error"] = f"{type(e).__name__}: {e}"
|
||||
attempts = getattr(e, "provider_attempts", None)
|
||||
if attempts:
|
||||
rec["judge_provider_attempts"] = attempts
|
||||
return rec
|
||||
|
||||
|
||||
def fmt(x, nd=2):
|
||||
return f"{x:.{nd}f}" if isinstance(x, (int, float)) else str(x)
|
||||
|
||||
|
||||
def print_detail(rec, sample_text):
|
||||
head = f"[{rec['config']} | {rec['sample']}] {rec['challenge']}"
|
||||
if not rec["ok"]:
|
||||
print(f" {head}\n !! 失败: {rec['error']}")
|
||||
return
|
||||
print(f" {head}")
|
||||
print(f" 原文 : {sample_text}")
|
||||
if rec.get("hypothesis") is not None:
|
||||
print(f" 回译 : {rec['hypothesis']}")
|
||||
objective = ""
|
||||
if rec.get("cer") is not None:
|
||||
objective = (
|
||||
f" CER: {fmt(rec['cer'],3)}"
|
||||
f" ASR字准确率: {fmt(rec['asr_accuracy']*100,1)}%"
|
||||
)
|
||||
print(f" 时长 : {fmt(rec['duration'])}s 语速: {fmt(rec['speed'])} 字/秒{objective}")
|
||||
s, r = rec["scores"], rec["reasons"]
|
||||
for dim in pipeline.RUBRIC_DIMENSIONS:
|
||||
print(f" {dim:<4}: {s.get(dim,'-')}/5 {r.get(dim,'')}")
|
||||
|
||||
|
||||
def summarize(records):
|
||||
"""按配置聚合：平均 CER、平均字准确率、各 Rubric 维度均分、成功数。"""
|
||||
by_cfg = {}
|
||||
for rec in records:
|
||||
by_cfg.setdefault(rec["config"], []).append(rec)
|
||||
rows = []
|
||||
for cfg_name, recs in by_cfg.items():
|
||||
ok = [r for r in recs if r["ok"]]
|
||||
row = {"config": cfg_name, "n_ok": len(ok), "n": len(recs)}
|
||||
if ok:
|
||||
objective = [r for r in ok if r.get("cer") is not None]
|
||||
row["cer"] = mean(r["cer"] for r in objective) if objective else None
|
||||
row["asr_accuracy"] = (
|
||||
mean(r["asr_accuracy"] for r in objective) if objective else None
|
||||
)
|
||||
for dim in pipeline.RUBRIC_DIMENSIONS:
|
||||
row[dim] = mean(r["scores"].get(dim, 0) for r in ok)
|
||||
rows.append(row)
|
||||
# 按整体分降序、CER 升序排序
|
||||
rows.sort(
|
||||
key=lambda x: (
|
||||
-mean(x.get(dim, 0) for dim in pipeline.RUBRIC_DIMENSIONS),
|
||||
x.get("cer") if x.get("cer") is not None else 1,
|
||||
)
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def print_table(rows):
|
||||
cols = list(pipeline.RUBRIC_DIMENSIONS)
|
||||
header = (f"{'配置':<22}{'成功':>6}{'ASR准确率':>11}{'CER':>8}"
|
||||
+ "".join(f"{c:>9}" for c in cols))
|
||||
print(header)
|
||||
print("-" * 74)
|
||||
for r in rows:
|
||||
ok_str = f"{r['n_ok']}/{r['n']}"
|
||||
if not r.get("n_ok"):
|
||||
print(f"{r['config']:<22}{ok_str:>6} (全部失败)")
|
||||
continue
|
||||
acc = f"{r['asr_accuracy']*100:.1f}%" if r.get("asr_accuracy") is not None else "n/a"
|
||||
cer = f"{r['cer']:.3f}" if r.get("cer") is not None else "n/a"
|
||||
line = f"{r['config']:<22}{ok_str:>6}{acc:>11}{cer:>8}"
|
||||
line += "".join(f"{r.get(c,0):>9.2f}" for c in cols)
|
||||
print(line)
|
||||
|
||||
|
||||
def print_providers():
|
||||
"""离线打印所有可用 TTS provider 及其配置状态（无需任何 API key）。"""
|
||||
print("可用 TTS provider（书中：OpenAI / ElevenLabs / Fish Audio / Minimax / 豆包）：\n")
|
||||
for key, p in config.PROVIDERS.items():
|
||||
state = "已配置" if p.configured() else "未配置"
|
||||
env = " + ".join(p.env)
|
||||
print(f" [{key}] {p.label} ({state}；需 {env})")
|
||||
print(f" {p.note}")
|
||||
print("\n用 --providers openai,minimax 选择跨服务商横向对比（默认仅 OpenAI）。")
|
||||
print("非 OpenAI provider 需各自的 key（见 env.example）；缺 key 时该行记为失败，不中断整表。")
|
||||
|
||||
|
||||
def print_rubric():
|
||||
"""离线打印 Rubric 维度定义（无需任何 API key）。"""
|
||||
print("TTS 质量评估 Rubric（1-5 分，5 最好）：\n")
|
||||
for dim in pipeline.RUBRIC_DIMENSIONS:
|
||||
print(f" {dim}：{pipeline.RUBRIC_DESCRIPTIONS.get(dim, '')}")
|
||||
print("\n默认（Whisper 回译 + LLM）评审基于「转写文本 + 时长 + 语速 + CER」保守打分；")
|
||||
print("--gemini 同时提供合成音频与参考语音，覆盖正文全部四个维度。")
|
||||
|
||||
|
||||
def main():
|
||||
global OUT_DIR
|
||||
ap = argparse.ArgumentParser(
|
||||
description="全自动 TTS 质量评估流水线（实验 7-6）：多 provider 合成 + 多模态 LLM Rubric 评审",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="示例：\n"
|
||||
" python demo.py 默认 4 个 OpenAI 配置 × 4 条语料\n"
|
||||
" python demo.py --providers openai,minimax 跨服务商横向对比\n"
|
||||
" python demo.py --text '今天天气不错' --gemini 自定义文本 + Gemini 多模态评审\n"
|
||||
" python demo.py --list-providers 离线查看 provider 及配置状态\n"
|
||||
" python demo.py --dump-rubric 离线查看 Rubric 维度定义",
|
||||
)
|
||||
ap.add_argument("--text", metavar="文本",
|
||||
help="用一段自定义文本替换测试语料库（只评这一句）")
|
||||
ap.add_argument("--providers", metavar="列表",
|
||||
help="逗号分隔的 provider（openai,elevenlabs,fishaudio,minimax,doubao），"
|
||||
"每个取代表性配置做横向对比；默认仅 OpenAI 的多配置")
|
||||
ap.add_argument("--judge-model", metavar="模型", dest="judge_model",
|
||||
help=f"覆盖 LLM 评审模型（默认 {config.JUDGE_MODEL}）；--gemini 时不生效")
|
||||
ap.add_argument("--output", metavar="目录",
|
||||
help=f"输出目录（音频 + results.json），默认 {OUT_DIR}")
|
||||
ap.add_argument("--extra", action="store_true", help="额外加入 gpt-4o-mini-tts 配置")
|
||||
ap.add_argument(
|
||||
"--gemini",
|
||||
action="store_true",
|
||||
help="用 Gemini/OpenRouter/Voxtral 多模态路线直接听两段音频评审",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--reference-audio",
|
||||
default=DEFAULT_REFERENCE_AUDIO,
|
||||
help="Gemini 音色一致性对照的真实参考语音（默认复用第 9 章固定证据）",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--with-asr",
|
||||
action="store_true",
|
||||
help="Gemini 直听之外再运行 Whisper/CER；需要可用 OPENAI_API_KEY",
|
||||
)
|
||||
ap.add_argument("--quick", action="store_true", help="只用前 2 条语料快速冒烟")
|
||||
ap.add_argument("--limit", type=int, default=0, help="只用前 N 条语料（0 = 全部）")
|
||||
ap.add_argument("--fresh", action="store_true", help="忽略已有音频，全部重新合成")
|
||||
ap.add_argument("--list-providers", action="store_true", dest="list_providers",
|
||||
help="离线打印所有 TTS provider 及配置状态后退出（无需 key）")
|
||||
ap.add_argument("--dump-rubric", action="store_true", dest="dump_rubric",
|
||||
help="离线打印 Rubric 维度定义后退出（无需 key）")
|
||||
args = ap.parse_args()
|
||||
|
||||
load_env()
|
||||
|
||||
# 离线路径：不联网、不需要任何 key，打印后直接退出。
|
||||
if args.list_providers:
|
||||
print_providers()
|
||||
return
|
||||
if args.dump_rubric:
|
||||
print_rubric()
|
||||
return
|
||||
|
||||
if args.output:
|
||||
OUT_DIR = os.path.abspath(args.output)
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
|
||||
if not args.gemini and not os.environ.get("OPENAI_API_KEY", "").strip():
|
||||
print("错误：缺少 OPENAI_API_KEY（回译/默认评审需要）。请 export 或写入 .env 后重试。",
|
||||
file=sys.stderr)
|
||||
sys.exit(1)
|
||||
if (args.gemini
|
||||
and not os.environ.get("GEMINI_API_KEY", "").strip()
|
||||
and not os.environ.get("OPENROUTER_API_KEY", "").strip()
|
||||
and not os.environ.get("MISTRAL_API_KEY", "").strip()):
|
||||
print(
|
||||
"错误：--gemini 需要 GEMINI_API_KEY、OPENROUTER_API_KEY 或 MISTRAL_API_KEY。",
|
||||
file=sys.stderr,
|
||||
)
|
||||
sys.exit(1)
|
||||
if args.gemini and not os.path.isfile(args.reference_audio):
|
||||
print(f"错误：参考语音不存在：{args.reference_audio}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# 选择待对比的配置：--providers 优先（跨服务商），否则默认 OpenAI 多配置。
|
||||
if args.providers:
|
||||
configs = []
|
||||
for key in [p.strip() for p in args.providers.split(",") if p.strip()]:
|
||||
if key not in config.PROVIDER_CONFIGS:
|
||||
print(f"错误：未知 provider {key!r}。可用：{', '.join(config.PROVIDER_CONFIGS)}",
|
||||
file=sys.stderr)
|
||||
sys.exit(1)
|
||||
configs.append(config.PROVIDER_CONFIGS[key])
|
||||
else:
|
||||
configs = list(config.TTS_CONFIGS)
|
||||
if args.extra:
|
||||
configs += config.EXTRA_CONFIGS
|
||||
|
||||
if args.text:
|
||||
corpus = [config.Sample(id="custom", text=args.text,
|
||||
challenge="自定义文本", emotion="中性")]
|
||||
else:
|
||||
corpus = config.CORPUS[:2] if args.quick else config.CORPUS
|
||||
if args.limit:
|
||||
if args.limit < 0:
|
||||
print("错误：--limit 不能为负数。", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
corpus = corpus[:args.limit]
|
||||
|
||||
judge_model = args.judge_model or config.JUDGE_MODEL
|
||||
mode = ("多模态直接音频评审" if args.gemini
|
||||
else f"Whisper 回译 + LLM Rubric（{judge_model}）")
|
||||
providers_used = sorted({getattr(c, "provider", "openai") for c in configs})
|
||||
print("=" * 72)
|
||||
print(f"实验 7-6：全自动 TTS 质量评估流水线")
|
||||
print(f"评审模式：{mode}")
|
||||
print(f"参与 provider：{', '.join(providers_used)}")
|
||||
print(f"配置数：{len(configs)} 语料数：{len(corpus)} "
|
||||
f"共 {len(configs)*len(corpus)} 条待评估")
|
||||
print("=" * 72)
|
||||
|
||||
records = []
|
||||
t0 = time.time()
|
||||
for cfg in configs:
|
||||
print(f"\n### 配置 {cfg.name} (provider={getattr(cfg,'provider','openai')}, "
|
||||
f"model={cfg.model}, voice={cfg.voice}, speed={cfg.speed})")
|
||||
for sample in corpus:
|
||||
rec = evaluate_one(
|
||||
cfg,
|
||||
sample,
|
||||
args.gemini,
|
||||
args.fresh,
|
||||
judge_model=None if args.gemini else args.judge_model,
|
||||
reference_audio=args.reference_audio,
|
||||
with_asr=args.with_asr,
|
||||
)
|
||||
print_detail(rec, sample.text)
|
||||
records.append(rec)
|
||||
|
||||
rows = summarize(records)
|
||||
print("\n" + "=" * 72)
|
||||
print("配置对比汇总（按四维宏平均分降序）")
|
||||
print("=" * 72)
|
||||
print_table(rows)
|
||||
|
||||
ok = sum(1 for r in records if r["ok"])
|
||||
print(f"\n完成：{ok}/{len(records)} 条成功，耗时 {time.time()-t0:.1f}s。")
|
||||
|
||||
# 落盘结构化结果，便于二次分析
|
||||
out_json = os.path.join(OUT_DIR, "results.json")
|
||||
expected = len(configs) * len(corpus)
|
||||
exact_dims = set(pipeline.RUBRIC_DIMENSIONS)
|
||||
complete_records = [
|
||||
r for r in records
|
||||
if r.get("ok")
|
||||
and r.get("evidence_mode") == "direct-audio-with-reference"
|
||||
and set(r.get("scores", {})) == exact_dims
|
||||
and all(1 <= int(v) <= 5 for v in r.get("scores", {}).values())
|
||||
]
|
||||
reference_sha = (
|
||||
pipeline.sha256_file(args.reference_audio)
|
||||
if args.gemini and os.path.isfile(args.reference_audio)
|
||||
else None
|
||||
)
|
||||
payload = {
|
||||
"schema_version": "2.0",
|
||||
"experiment": "7-6",
|
||||
"generated_at_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"command_scope": {
|
||||
"providers": providers_used,
|
||||
"configurations": [
|
||||
{
|
||||
"name": c.name,
|
||||
"provider": c.provider,
|
||||
"model": c.model,
|
||||
"voice": c.voice,
|
||||
"speed": c.speed,
|
||||
}
|
||||
for c in configs
|
||||
],
|
||||
"corpus_ids": [s.id for s in corpus],
|
||||
"expected_records": expected,
|
||||
"direct_audio_judge": args.gemini,
|
||||
"optional_asr_enabled": args.with_asr,
|
||||
},
|
||||
"reference_audio": {
|
||||
"path": os.path.relpath(args.reference_audio, OUT_DIR) if args.gemini else None,
|
||||
"sha256": reference_sha,
|
||||
},
|
||||
"rubric_dimensions": pipeline.RUBRIC_DIMENSIONS,
|
||||
"completion": {
|
||||
"successful_records": ok,
|
||||
"direct_audio_four_dimension_records": len(complete_records),
|
||||
"expected_records": expected,
|
||||
"all_cells_complete": len(complete_records) == expected,
|
||||
"multi_provider": len(providers_used) >= 2,
|
||||
"manuscript_core_complete": (
|
||||
len(complete_records) == expected
|
||||
and len(providers_used) >= 2
|
||||
and bool(reference_sha)
|
||||
),
|
||||
},
|
||||
"records": records,
|
||||
"summary": rows,
|
||||
}
|
||||
with open(out_json, "w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
print(f"明细结果已写入 {out_json}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,27 @@
|
||||
# 复制为 .env 或直接 export。TTS 合成与 Whisper 回译必须走 OpenAI 直连，需下面这个 key。
|
||||
# 默认 LLM 评审模型为 gpt-5.6-luna（当前廉价旗舰）。
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# 可选：LLM Rubric 与 `--gemini` 直接音频评审的 OpenRouter 回退（TTS/Whisper 不行）。
|
||||
# 音频路径仍发送两段真实 MP3，不会退化成文本转写。gpt-5.x 直连 OpenAI 需组织实名认证。
|
||||
# OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
# TTS_AUDIO_JUDGE_MODEL=google/gemini-3.5-flash
|
||||
|
||||
# 可选：用 `python demo.py --gemini` 让 Google Gemini 直连听音频时设置；若不设置，
|
||||
# 也可以只用上面的 OPENROUTER_API_KEY 走支持 audio 输入的 Gemini 路由。
|
||||
# 默认 Gemini 评审模型为 gemini-3.5-flash（已验证支持音频输入）。
|
||||
# 默认（不加 --gemini）评审是 Whisper 回译 + gpt-5.6-luna Rubric，无需此 key。
|
||||
# GEMINI_API_KEY=your-gemini-key
|
||||
|
||||
# 可选：Gemini/OpenRouter 不可用时，用 Mistral Voxtral 直接听两段 MP3。
|
||||
# MISTRAL_API_KEY=your-mistral-key
|
||||
# TTS_MISTRAL_AUDIO_JUDGE_MODEL=voxtral-small-latest
|
||||
|
||||
# 可选：跨服务商横向对比时（`--providers openai,elevenlabs,minimax,...`）按需配置。
|
||||
# 未配置的 provider 对应的行会被记为失败，不影响整表。逐项见 `python demo.py --list-providers`。
|
||||
# ELEVENLABS_API_KEY=your-elevenlabs-key
|
||||
# FISH_API_KEY=your-fishaudio-key # 亦兼容别名 FISHAUDIO_API_KEY
|
||||
# MINIMAX_API_KEY=your-minimax-key # Bearer 鉴权，model 默认 speech-2.8-hd
|
||||
# MINIMAX_REGION=global # 可选：global->api.minimax.io，cn->api.minimaxi.com
|
||||
# DOUBAO_APP_ID=your-volcengine-app-id
|
||||
# DOUBAO_ACCESS_TOKEN=your-volcengine-access-token
|
||||
@@ -0,0 +1,825 @@
|
||||
"""TTS 质量评估流水线的核心步骤。
|
||||
|
||||
一条评估链路：
|
||||
合成(OpenAI TTS) -> 时长探测(ffprobe) -> 回译(Whisper) -> 计算 CER/字准确率
|
||||
-> LLM Rubric 打分(gpt-5.6-luna) [可选: Gemini 音频评审 gemini-3.5-flash]
|
||||
|
||||
说明：TTS 合成与 Whisper 回译必须走 OpenAI 直连；文本 Rubric 与直接听音频的
|
||||
多模态 Rubric 支持 Google Gemini、OpenRouter 与 Mistral Voxtral。每条路径都把
|
||||
两段真实音频交给音频模型，不会退化成转写文本评审。
|
||||
|
||||
所有对外函数都做了健壮性处理：单条失败抛出带上下文的异常，由 demo.py 捕获后
|
||||
在汇总表里记为失败，而不会中断整表。
|
||||
"""
|
||||
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
import config
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 客户端（带自动重试，缓解偶发的网络抖动）。
|
||||
# ---------------------------------------------------------------------------
|
||||
_client: Optional[OpenAI] = None
|
||||
|
||||
|
||||
def get_client() -> OpenAI:
|
||||
"""OpenAI 直连 client：用于 TTS 合成与 Whisper 回译（这两项不能走 OpenRouter）。"""
|
||||
global _client
|
||||
if _client is None:
|
||||
key = os.environ.get("OPENAI_API_KEY", "").strip()
|
||||
if not key:
|
||||
raise RuntimeError(
|
||||
"缺少 OPENAI_API_KEY（TTS 合成 / Whisper 回译需 OpenAI 直连）。"
|
||||
"请 `export OPENAI_API_KEY=your-openai-api-key` 或写入 .env。"
|
||||
)
|
||||
_client = OpenAI(api_key=key, max_retries=5, timeout=60.0)
|
||||
return _client
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# LLM Rubric 评审客户端：支持 OpenRouter 回退。
|
||||
# gpt-5.x 直连 OpenAI 需组织实名认证，只要有 OPENROUTER_API_KEY 就优先走 OpenRouter。
|
||||
# 注意：仅 chat 评审可回退；TTS / Whisper 仍需 OpenAI 直连（见 get_client）。
|
||||
# ---------------------------------------------------------------------------
|
||||
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
_judge_client: Optional[OpenAI] = None
|
||||
_judge_client_kind: str = ""
|
||||
|
||||
|
||||
def _to_openrouter_model(model: str) -> str:
|
||||
"""把模型名映射成 OpenRouter id：含 '/' 视为原生 id；gpt-* -> openai/*；
|
||||
claude-* -> anthropic/claude-opus-4.8；其余回退到 openai/gpt-5.6-luna。"""
|
||||
if "/" in model:
|
||||
return model
|
||||
if model.startswith("gpt-"):
|
||||
return "openai/" + model
|
||||
if model.startswith("claude-"):
|
||||
return "anthropic/claude-opus-4.8"
|
||||
return "openai/gpt-5.6-luna"
|
||||
|
||||
|
||||
def get_judge_client_and_model(model: str):
|
||||
"""构造 LLM 评审用的 client 并返回 (client, 实际模型名)。
|
||||
|
||||
回退：gpt-5.x 且有 OPENROUTER_API_KEY -> 优先 OpenRouter；否则有 OPENAI_API_KEY ->
|
||||
直连；否则有 OPENROUTER_API_KEY -> OpenRouter（模型名映射）；皆无 -> 清晰报错。
|
||||
"""
|
||||
global _judge_client, _judge_client_kind
|
||||
primary = os.environ.get("OPENAI_API_KEY", "").strip()
|
||||
orkey = os.environ.get("OPENROUTER_API_KEY", "").strip()
|
||||
prefer_or = bool(orkey) and model.startswith("gpt-5")
|
||||
|
||||
if not prefer_or and primary:
|
||||
if _judge_client_kind != "openai":
|
||||
_judge_client = OpenAI(api_key=primary, max_retries=5, timeout=60.0)
|
||||
_judge_client_kind = "openai"
|
||||
return _judge_client, model
|
||||
if orkey:
|
||||
if _judge_client_kind != "openrouter":
|
||||
_judge_client = OpenAI(base_url=OPENROUTER_BASE_URL, api_key=orkey,
|
||||
max_retries=5, timeout=60.0)
|
||||
_judge_client_kind = "openrouter"
|
||||
return _judge_client, _to_openrouter_model(model)
|
||||
if primary:
|
||||
if _judge_client_kind != "openai":
|
||||
_judge_client = OpenAI(api_key=primary, max_retries=5, timeout=60.0)
|
||||
_judge_client_kind = "openai"
|
||||
return _judge_client, model
|
||||
raise RuntimeError(
|
||||
"缺少 OPENAI_API_KEY / OPENROUTER_API_KEY，无法运行 LLM Rubric 评审。"
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1) TTS 合成（多 provider 分发）
|
||||
# ---------------------------------------------------------------------------
|
||||
def synthesize(cfg: config.TTSConfig, text: str, out_path: str) -> None:
|
||||
"""按 cfg.provider 分发到对应服务商合成语音，写入 out_path（mp3）。失败抛异常。
|
||||
|
||||
OpenAI 走官方 SDK；其余服务商按各家公开 REST 接口用内置 urllib 调用，
|
||||
不引入额外依赖。缺少对应 key 时抛出带上下文的异常，由上层记为该行失败。
|
||||
"""
|
||||
fn = _SYNTH_DISPATCH.get(cfg.provider)
|
||||
if fn is None:
|
||||
raise RuntimeError(
|
||||
f"未知 provider: {cfg.provider!r}（可选：{', '.join(_SYNTH_DISPATCH)}）"
|
||||
)
|
||||
audio = fn(cfg, text)
|
||||
if not audio:
|
||||
raise RuntimeError(f"{cfg.provider} TTS 返回空音频")
|
||||
with open(out_path, "wb") as f:
|
||||
f.write(audio)
|
||||
|
||||
|
||||
def _require_env(name: str) -> str:
|
||||
# 走 config.env_get 以支持环境变量别名（如 Fish 的 FISH_API_KEY / FISHAUDIO_API_KEY）。
|
||||
val = config.env_get(name)
|
||||
if not val:
|
||||
raise RuntimeError(f"缺少环境变量 {name}，无法用该 provider 合成。")
|
||||
return val
|
||||
|
||||
|
||||
def _http_post(url: str, body: dict, headers: dict, timeout: float = 90.0) -> bytes:
|
||||
"""POST JSON，返回原始响应字节。非 2xx 抛出带响应体片段的异常。"""
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
req = urllib.request.Request(
|
||||
url, data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json", **headers}, method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
return r.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
detail = e.read().decode("utf-8", "replace")[:300]
|
||||
raise RuntimeError(f"HTTP {e.code}: {detail}") from None
|
||||
|
||||
|
||||
def _synth_openai(cfg: config.TTSConfig, text: str) -> bytes:
|
||||
kwargs = dict(model=cfg.model, voice=cfg.voice, input=text)
|
||||
if cfg.supports_speed() and abs(cfg.speed - 1.0) > 1e-6:
|
||||
kwargs["speed"] = cfg.speed
|
||||
return get_client().audio.speech.create(**kwargs).content
|
||||
|
||||
|
||||
def _synth_elevenlabs(cfg: config.TTSConfig, text: str) -> bytes:
|
||||
key = _require_env("ELEVENLABS_API_KEY")
|
||||
voice = cfg.voice or "21m00Tcm4TlvDq8ikWAM"
|
||||
url = (f"https://api.elevenlabs.io/v1/text-to-speech/{voice}"
|
||||
f"?output_format=mp3_44100_128")
|
||||
body = {"text": text, "model_id": cfg.model or "eleven_multilingual_v2"}
|
||||
# ElevenLabs 返回原始 mp3 字节。
|
||||
return _http_post(url, body, {"xi-api-key": key, "Accept": "audio/mpeg"})
|
||||
|
||||
|
||||
def _synth_fishaudio(cfg: config.TTSConfig, text: str) -> bytes:
|
||||
key = _require_env("FISH_API_KEY")
|
||||
if not cfg.voice:
|
||||
raise RuntimeError(
|
||||
"Fish Audio voice consistency requires a real reference_id; "
|
||||
"set FISH_REFERENCE_ID."
|
||||
)
|
||||
# Use Fish's maintained SDK instead of assuming the REST response is raw
|
||||
# MP3. The current S1 endpoint streams MessagePack chunks and the SDK is
|
||||
# the provider-supported decoder for that wire format.
|
||||
from fish_audio_sdk import Session, TTSRequest
|
||||
|
||||
request = TTSRequest(text=text, reference_id=cfg.voice, format="mp3")
|
||||
return b"".join(Session(key).tts(request, backend=cfg.model or "s1"))
|
||||
|
||||
|
||||
# Minimax /v1/t2a_v2 uses Bearer auth and no longer takes a GroupId query
|
||||
# parameter. The global and mainland-China deployments live on separate hosts;
|
||||
# pick one via MINIMAX_REGION (defaults to the global api.minimax.io host).
|
||||
_MINIMAX_T2A_ENDPOINTS = {
|
||||
"global": "https://api.minimax.io/v1/t2a_v2",
|
||||
"cn": "https://api.minimaxi.com/v1/t2a_v2",
|
||||
}
|
||||
# Success criteria for the non-streaming t2a_v2 call: base_resp.status_code == 0
|
||||
# (request accepted) and data.status == 2 (synthesis finished).
|
||||
_MINIMAX_SUCCESS_CODE = 0
|
||||
_MINIMAX_STATUS_DONE = 2
|
||||
|
||||
|
||||
def _minimax_endpoint() -> str:
|
||||
"""Return the t2a_v2 endpoint for MINIMAX_REGION: cn -> api.minimaxi.com,
|
||||
otherwise the global api.minimax.io host."""
|
||||
region = os.environ.get("MINIMAX_REGION", "").strip().lower()
|
||||
if region in ("cn", "cn_zh", "china", "minimaxi"):
|
||||
return _MINIMAX_T2A_ENDPOINTS["cn"]
|
||||
return _MINIMAX_T2A_ENDPOINTS["global"]
|
||||
|
||||
|
||||
def _synth_minimax(cfg: config.TTSConfig, text: str) -> bytes:
|
||||
key = _require_env("MINIMAX_API_KEY")
|
||||
body = {
|
||||
"model": cfg.model or "speech-2.8-hd",
|
||||
"text": text,
|
||||
"stream": False,
|
||||
"voice_setting": {"voice_id": cfg.voice, "speed": cfg.speed},
|
||||
"audio_setting": {"format": "mp3", "sample_rate": 32000},
|
||||
}
|
||||
raw = _http_post(_minimax_endpoint(), body, {"Authorization": f"Bearer {key}"})
|
||||
data = json.loads(raw)
|
||||
# Validate the request-level return code first, then the synthesis status.
|
||||
base_resp = data.get("base_resp") or {}
|
||||
if base_resp.get("status_code") != _MINIMAX_SUCCESS_CODE:
|
||||
raise RuntimeError(f"Minimax t2a_v2 failed: base_resp={base_resp or data}")
|
||||
payload = data.get("data") or {}
|
||||
status = payload.get("status")
|
||||
hexstr = payload.get("audio")
|
||||
if status != _MINIMAX_STATUS_DONE or not hexstr:
|
||||
raise RuntimeError(
|
||||
f"Minimax returned no finished audio: status={status} base_resp={base_resp}"
|
||||
)
|
||||
# data.audio is a hex-encoded mp3 payload.
|
||||
return bytes.fromhex(hexstr)
|
||||
|
||||
|
||||
def _synth_doubao(cfg: config.TTSConfig, text: str) -> bytes:
|
||||
import uuid
|
||||
appid = _require_env("DOUBAO_APP_ID")
|
||||
token = _require_env("DOUBAO_ACCESS_TOKEN")
|
||||
body = {
|
||||
"app": {"appid": appid, "token": token,
|
||||
"cluster": cfg.model or "volcano_tts"},
|
||||
"user": {"uid": "tts-quality-eval"},
|
||||
"audio": {"voice_type": cfg.voice, "encoding": "mp3",
|
||||
"speed_ratio": cfg.speed},
|
||||
"request": {"reqid": str(uuid.uuid4()), "text": text, "operation": "query"},
|
||||
}
|
||||
# 火山引擎鉴权头是特殊的 'Bearer;{token}' 形式；音频为 base64 编码的 data 字段。
|
||||
raw = _http_post("https://openspeech.bytedance.com/api/v1/tts", body,
|
||||
{"Authorization": f"Bearer;{token}"})
|
||||
data = json.loads(raw)
|
||||
b64 = data.get("data")
|
||||
if not b64:
|
||||
raise RuntimeError(f"豆包无音频返回：code={data.get('code')} "
|
||||
f"message={data.get('message')}")
|
||||
return base64.b64decode(b64)
|
||||
|
||||
|
||||
_SYNTH_DISPATCH = {
|
||||
"openai": _synth_openai,
|
||||
"elevenlabs": _synth_elevenlabs,
|
||||
"fishaudio": _synth_fishaudio,
|
||||
"minimax": _synth_minimax,
|
||||
"doubao": _synth_doubao,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2) 时长探测（ffprobe）
|
||||
# ---------------------------------------------------------------------------
|
||||
def probe_duration(path: str) -> float:
|
||||
"""返回音频时长（秒）。ffprobe 缺失或出错时抛异常。"""
|
||||
if shutil.which("ffprobe") is None:
|
||||
raise RuntimeError("未找到 ffprobe，请安装 ffmpeg（macOS: brew install ffmpeg）。")
|
||||
proc = subprocess.run(
|
||||
["ffprobe", "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1", path],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise RuntimeError(f"ffprobe 失败: {proc.stderr.strip()}")
|
||||
out = proc.stdout.strip()
|
||||
try:
|
||||
return float(out)
|
||||
except ValueError:
|
||||
raise RuntimeError(f"ffprobe 输出无法解析为时长: {out!r}")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3) 回译（Whisper 转写）
|
||||
# ---------------------------------------------------------------------------
|
||||
# 用简体中文提示语引导 Whisper 输出简体，避免它偶尔返回繁体导致 CER 被字形差异
|
||||
# 虚高（那是转写脚本选择问题，不是 TTS 发音错误）。
|
||||
_ZH_PROMPT = "以下是普通话简体中文的句子。"
|
||||
|
||||
|
||||
def transcribe(path: str) -> str:
|
||||
with open(path, "rb") as f:
|
||||
tr = get_client().audio.transcriptions.create(
|
||||
model=config.WHISPER_MODEL, file=f, language="zh", prompt=_ZH_PROMPT,
|
||||
)
|
||||
return tr.text or ""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 4) 文本归一化 + 字错误率（中文用字级 CER，等价于书中所说 WER 的可懂度维度）
|
||||
# ---------------------------------------------------------------------------
|
||||
def normalize(text: str) -> str:
|
||||
"""去掉标点/空白，只保留 CJK / 字母 / 数字，并小写，便于逐字比较。"""
|
||||
text = text.lower()
|
||||
return "".join(ch for ch in text if ch.isalnum())
|
||||
|
||||
|
||||
def _edit_distance(a: str, b: str) -> int:
|
||||
"""Levenshtein 距离（字符级）。"""
|
||||
if a == b:
|
||||
return 0
|
||||
if not a:
|
||||
return len(b)
|
||||
if not b:
|
||||
return len(a)
|
||||
prev = list(range(len(b) + 1))
|
||||
for i, ca in enumerate(a, 1):
|
||||
cur = [i]
|
||||
for j, cb in enumerate(b, 1):
|
||||
cur.append(min(
|
||||
prev[j] + 1, # 删除
|
||||
cur[j - 1] + 1, # 插入
|
||||
prev[j - 1] + (ca != cb), # 替换
|
||||
))
|
||||
prev = cur
|
||||
return prev[-1]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ErrorRate:
|
||||
cer: float # 字错误率 = 编辑距离 / 参考字数
|
||||
accuracy: float # 字准确率 = 1 - cer（下限 0）
|
||||
edits: int
|
||||
ref_len: int
|
||||
|
||||
|
||||
def char_error_rate(reference: str, hypothesis: str) -> ErrorRate:
|
||||
ref = normalize(reference)
|
||||
hyp = normalize(hypothesis)
|
||||
if not ref:
|
||||
if not hyp:
|
||||
return ErrorRate(0.0, 1.0, 0, 0)
|
||||
dist = len(hyp)
|
||||
return ErrorRate(cer=float(dist), accuracy=0.0, edits=dist, ref_len=0)
|
||||
dist = _edit_distance(ref, hyp)
|
||||
cer = dist / len(ref)
|
||||
return ErrorRate(cer=cer, accuracy=max(0.0, 1.0 - cer), edits=dist, ref_len=len(ref))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 5) LLM Rubric 评审（默认，OpenAI 闭环）
|
||||
# ---------------------------------------------------------------------------
|
||||
RUBRIC_DIMENSIONS = ["准确性", "自然度", "情感表达", "音色一致性"]
|
||||
|
||||
# 维度说明（供 --dump-rubric 离线打印，也是评审 prompt 的依据）。括号内标注与书中
|
||||
# 四维度（准确性 / 自然度 / 情感表达 / 音色一致性）的对应关系。
|
||||
RUBRIC_DESCRIPTIONS = {
|
||||
"准确性": "逐字核对原文，检查漏读、错读、添读、数字、专名与多音字。",
|
||||
"自然度": "直接听语音的流畅度、机器感、停顿、重音和韵律是否符合人类习惯。",
|
||||
"情感表达": "语调、语速和强调是否符合中性、兴奋、悲伤或疑问等目标情感。",
|
||||
"音色一致性": "把合成语音与同时提供的参考语音比较，判断说话人音色是否一致。",
|
||||
}
|
||||
# The text-only judge remains a diagnostic fallback. It cannot complete the
|
||||
# manuscript experiment because it cannot hear emotion or compare a speaker.
|
||||
|
||||
_JUDGE_SYSTEM = """你是严格的 TTS（文本转语音）质量评审专家。
|
||||
你将拿到：原始参考文本、该文本的期望情感、由 Whisper 对合成语音回译得到的转写文本，
|
||||
以及从音频客观测得的时长、语速（字/秒）和字错误率（CER）。
|
||||
请据此对合成语音质量按 Rubric 逐维度打分（1-5 的整数，5 最好）。你无法听到音频，
|
||||
所以情感表达和音色一致性必须返回 0 并明确标记无法判定；本路径仅是诊断回退，不能验收实验：
|
||||
|
||||
- 准确性：转写与原文是否高度一致（漏字/错字/多字越多分越低；CER 越高分越低）。
|
||||
- 自然度：语速是否接近自然朗读（中文自然朗读约 4-6 字/秒；过快>7 或过慢<3 都不自然）。
|
||||
- 情感表达：返回 0，理由说明文本特征不足以判断真实语调。
|
||||
- 音色一致性：返回 0，理由说明没有听到参考语音和合成语音。
|
||||
|
||||
注意：你看不到音频本身，只能基于以上可测特征做保守、可解释的判断。
|
||||
只输出 JSON，格式：
|
||||
{"准确性": {"score": int, "reason": str},
|
||||
"自然度": {"score": int, "reason": str},
|
||||
"情感表达": {"score": int, "reason": str},
|
||||
"音色一致性": {"score": int, "reason": str}}
|
||||
reason 用一句简短中文说明。"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class RubricResult:
|
||||
scores: dict # 维度 -> int
|
||||
reasons: dict # 维度 -> str
|
||||
raw: str = ""
|
||||
judge_model: str = ""
|
||||
evidence_mode: str = ""
|
||||
provider_attempts: list = field(default_factory=list)
|
||||
|
||||
|
||||
class JudgeRouteError(RuntimeError):
|
||||
"""A sanitized multimodal-judge failure carrying every attempted route."""
|
||||
|
||||
def __init__(self, message: str, provider_attempts: list):
|
||||
super().__init__(message)
|
||||
self.provider_attempts = provider_attempts
|
||||
|
||||
|
||||
def judge_rubric(reference: str, emotion: str, hypothesis: str,
|
||||
duration: float, cer: float, model: Optional[str] = None) -> RubricResult:
|
||||
"""用评审模型（默认 gpt-5.6-luna）按 Rubric 打分。返回结构化分数 + 点评。
|
||||
|
||||
评审 chat 调用支持 OpenRouter 回退（见 get_judge_client_and_model）。"""
|
||||
chars = len(normalize(reference))
|
||||
speed = chars / duration if duration > 0 else 0.0
|
||||
user = (
|
||||
f"原始参考文本：{reference}\n"
|
||||
f"期望情感：{emotion}\n"
|
||||
f"Whisper 回译文本：{hypothesis}\n"
|
||||
f"音频时长：{duration:.2f} 秒\n"
|
||||
f"语速：{speed:.2f} 字/秒（参考文本 {chars} 个有效字符）\n"
|
||||
f"字错误率 CER：{cer:.3f}\n"
|
||||
)
|
||||
judge_client, judge_model = get_judge_client_and_model(model or config.JUDGE_MODEL)
|
||||
resp = judge_client.chat.completions.create(
|
||||
model=judge_model,
|
||||
messages=[{"role": "system", "content": _JUDGE_SYSTEM},
|
||||
{"role": "user", "content": user}],
|
||||
temperature=0.0,
|
||||
response_format={"type": "json_object"},
|
||||
)
|
||||
raw = resp.choices[0].message.content or "{}"
|
||||
data = json.loads(raw)
|
||||
scores, reasons = {}, {}
|
||||
for dim in RUBRIC_DIMENSIONS:
|
||||
item = data.get(dim, {})
|
||||
if isinstance(item, dict):
|
||||
scores[dim] = int(item.get("score") or 0) # score 缺失或为 null 时按 0 分
|
||||
reasons[dim] = str(item.get("reason", "")).strip()
|
||||
else: # 兼容模型直接返回数字（null 按 0 分）
|
||||
scores[dim] = int(item or 0)
|
||||
reasons[dim] = ""
|
||||
return RubricResult(
|
||||
scores=scores,
|
||||
reasons=reasons,
|
||||
raw=raw,
|
||||
judge_model=judge_model,
|
||||
evidence_mode="transcript-metrics-only-incomplete",
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 6) 可选：Gemini 多模态音频评审（书中方案）。用 REST，避免额外 SDK 依赖。
|
||||
# ---------------------------------------------------------------------------
|
||||
def _resolve_gemini_model(api_key: str) -> str:
|
||||
"""探测当前可用的 Gemini 模型，避免默认名过期。"""
|
||||
import urllib.request
|
||||
url = f"https://generativelanguage.googleapis.com/v1beta/models?key={api_key}"
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=20) as r:
|
||||
data = json.loads(r.read())
|
||||
models_list = data.get("models") or [] if isinstance(data, dict) else []
|
||||
names = [(m.get("name") or "").split("/")[-1] for m in models_list
|
||||
if isinstance(m, dict) and "generateContent" in (m.get("supportedGenerationMethods") or [])]
|
||||
# 优先默认的 gemini-3.5-flash（已验证支持音频输入），再退到 pro / 旧 flash 系列。
|
||||
for want in (config.GEMINI_MODEL_DEFAULT, "gemini-3.5-flash",
|
||||
"gemini-2.5-pro", "gemini-2.5-flash", "gemini-flash-latest"):
|
||||
if want in names:
|
||||
return want
|
||||
# 退而求其次：任意非 tts/image 的可用模型
|
||||
for n in names:
|
||||
if "tts" not in n and "image" not in n and "embedding" not in n:
|
||||
return n
|
||||
except Exception:
|
||||
pass
|
||||
return config.GEMINI_MODEL_DEFAULT
|
||||
|
||||
|
||||
def _parse_direct_audio_rubric(text: str, *, judge_model: str, provider_attempts: list) -> RubricResult:
|
||||
"""Validate a direct-audio judge response against the exact four dimensions."""
|
||||
parsed = json.loads(text)
|
||||
scores, reasons = {}, {}
|
||||
for dim in RUBRIC_DIMENSIONS:
|
||||
item = parsed.get(dim, {})
|
||||
scores[dim] = int(item.get("score") or 0) if isinstance(item, dict) else int(item or 0)
|
||||
reasons[dim] = str(item.get("reason", "")).strip() if isinstance(item, dict) else ""
|
||||
return RubricResult(
|
||||
scores=scores,
|
||||
reasons=reasons,
|
||||
raw=text,
|
||||
judge_model=judge_model,
|
||||
evidence_mode="direct-audio-with-reference",
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
|
||||
|
||||
def _message_text(data: dict) -> str:
|
||||
"""Extract text from OpenAI-compatible string or chunk-list content."""
|
||||
choices = data.get("choices") or []
|
||||
message_content = ((choices[0].get("message") or {}).get("content")) if choices else None
|
||||
if isinstance(message_content, list):
|
||||
return "".join(
|
||||
str(item.get("text", "")) for item in message_content if isinstance(item, dict)
|
||||
).strip()
|
||||
return str(message_content or "").strip()
|
||||
|
||||
|
||||
def _judge_mistral_audio(
|
||||
prompt: str,
|
||||
audio_b64: str,
|
||||
reference_audio_b64: str,
|
||||
*,
|
||||
provider_attempts: list,
|
||||
) -> RubricResult:
|
||||
"""Send both MP3s to Mistral Voxtral using its native data-URL chunks."""
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
key = os.environ.get("MISTRAL_API_KEY", "").strip()
|
||||
if not key:
|
||||
raise RuntimeError("缺少 MISTRAL_API_KEY，无法回退 Voxtral 音频评审。")
|
||||
model = os.environ.get("TTS_MISTRAL_AUDIO_JUDGE_MODEL", "voxtral-small-latest").strip()
|
||||
body = {
|
||||
"model": model,
|
||||
"temperature": 0.0,
|
||||
"response_format": {"type": "json_object"},
|
||||
"messages": [{"role": "user", "content": [
|
||||
{"type": "text", "text": prompt},
|
||||
{"type": "text", "text": "待评估合成语音（candidate）:"},
|
||||
{
|
||||
"type": "input_audio",
|
||||
"input_audio": "data:audio/mpeg;base64," + audio_b64,
|
||||
},
|
||||
{"type": "text", "text": "参考说话人语音（reference）:"},
|
||||
{
|
||||
"type": "input_audio",
|
||||
"input_audio": "data:audio/mpeg;base64," + reference_audio_b64,
|
||||
},
|
||||
]}],
|
||||
}
|
||||
req = urllib.request.Request(
|
||||
"https://api.mistral.ai/v1/chat/completions",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={
|
||||
"Authorization": f"Bearer {key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
for attempt in range(3):
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as response:
|
||||
data = json.loads(response.read())
|
||||
break
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", "replace")[:2000]
|
||||
error = f"Mistral Voxtral HTTP {exc.code}: {detail}"
|
||||
if exc.code >= 500 and attempt < 2:
|
||||
time.sleep(2 ** attempt)
|
||||
continue
|
||||
provider_attempts.append({
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": model,
|
||||
"status": "unavailable",
|
||||
"error": error,
|
||||
"attempts": attempt + 1,
|
||||
})
|
||||
raise JudgeRouteError(error, provider_attempts) from None
|
||||
text = _message_text(data)
|
||||
if not text:
|
||||
error = f"Mistral Voxtral 未返回评审文本：{data}"
|
||||
provider_attempts.append({
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": model,
|
||||
"status": "unavailable",
|
||||
"error": error,
|
||||
})
|
||||
raise JudgeRouteError(error, provider_attempts)
|
||||
provider_attempts.append({
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": model,
|
||||
"status": "ok",
|
||||
"attempts": attempt + 1,
|
||||
})
|
||||
return _parse_direct_audio_rubric(
|
||||
text,
|
||||
judge_model=f"mistral/{model}",
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
|
||||
|
||||
def _judge_openrouter_audio(
|
||||
prompt: str,
|
||||
audio_b64: str,
|
||||
reference_audio_b64: str,
|
||||
*,
|
||||
provider_attempts: list,
|
||||
) -> RubricResult:
|
||||
"""Send both audio clips to an audio-capable Gemini route on OpenRouter."""
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
key = os.environ.get("OPENROUTER_API_KEY", "").strip()
|
||||
if not key:
|
||||
raise RuntimeError("缺少 OPENROUTER_API_KEY，无法回退多模态音频评审。")
|
||||
model = os.environ.get("TTS_AUDIO_JUDGE_MODEL", "google/gemini-3.5-flash").strip()
|
||||
content = [
|
||||
{"type": "text", "text": prompt},
|
||||
{
|
||||
"type": "input_audio",
|
||||
"input_audio": {"data": audio_b64, "format": "mp3"},
|
||||
},
|
||||
{
|
||||
"type": "input_audio",
|
||||
"input_audio": {"data": reference_audio_b64, "format": "mp3"},
|
||||
},
|
||||
]
|
||||
body = {
|
||||
"model": model,
|
||||
"temperature": 0.0,
|
||||
"response_format": {"type": "json_object"},
|
||||
"messages": [{"role": "user", "content": content}],
|
||||
}
|
||||
req = urllib.request.Request(
|
||||
f"{OPENROUTER_BASE_URL}/chat/completions",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={
|
||||
"Authorization": f"Bearer {key}",
|
||||
"Content-Type": "application/json",
|
||||
},
|
||||
method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as response:
|
||||
data = json.loads(response.read())
|
||||
except urllib.error.HTTPError as exc:
|
||||
detail = exc.read().decode("utf-8", "replace")[:2000]
|
||||
error = f"OpenRouter audio HTTP {exc.code}: {detail}"
|
||||
provider_attempts.append({
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": model,
|
||||
"status": "unavailable",
|
||||
"error": error,
|
||||
})
|
||||
if os.environ.get("MISTRAL_API_KEY", "").strip():
|
||||
return _judge_mistral_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
raise JudgeRouteError(error, provider_attempts) from None
|
||||
text = _message_text(data)
|
||||
if not text:
|
||||
error = f"OpenRouter audio 未返回评审文本：{data}"
|
||||
provider_attempts.append({
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": model,
|
||||
"status": "unavailable",
|
||||
"error": error,
|
||||
})
|
||||
if os.environ.get("MISTRAL_API_KEY", "").strip():
|
||||
return _judge_mistral_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
raise JudgeRouteError(error, provider_attempts)
|
||||
provider_attempts.append({
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": model,
|
||||
"status": "ok",
|
||||
})
|
||||
return _parse_direct_audio_rubric(
|
||||
text,
|
||||
judge_model=f"openrouter/{model}",
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
|
||||
|
||||
def judge_gemini_audio(
|
||||
reference: str,
|
||||
emotion: str,
|
||||
audio_path: str,
|
||||
reference_audio_path: str,
|
||||
) -> RubricResult:
|
||||
"""让 Gemini 同时听合成音频与参考音频，执行正文四维 Rubric。
|
||||
|
||||
默认关闭；--gemini 开启。依次尝试已配置的 Google Gemini、OpenRouter 与
|
||||
Mistral Voxtral，失败抛异常由上层记为失败。
|
||||
"""
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
key = os.environ.get("GEMINI_API_KEY", "").strip()
|
||||
openrouter_key = os.environ.get("OPENROUTER_API_KEY", "").strip()
|
||||
mistral_key = os.environ.get("MISTRAL_API_KEY", "").strip()
|
||||
if not key and not openrouter_key and not mistral_key:
|
||||
raise RuntimeError(
|
||||
"缺少 GEMINI_API_KEY / OPENROUTER_API_KEY / MISTRAL_API_KEY，"
|
||||
"无法使用直接音频评审。"
|
||||
)
|
||||
with open(audio_path, "rb") as f:
|
||||
audio_b64 = base64.b64encode(f.read()).decode()
|
||||
if not reference_audio_path or not os.path.isfile(reference_audio_path):
|
||||
raise RuntimeError(
|
||||
f"音色一致性评估需要真实参考语音，文件不存在: {reference_audio_path!r}"
|
||||
)
|
||||
with open(reference_audio_path, "rb") as f:
|
||||
reference_audio_b64 = base64.b64encode(f.read()).decode()
|
||||
prompt = (
|
||||
"你是严格的 TTS 质量评审专家。你会收到两段音频：第一段是待评估的合成语音，"
|
||||
"第二段是参考说话人语音。请直接聆听并按正文四维 Rubric 独立打 1-5 整数分："
|
||||
"(1)准确性：逐字核对原文，检查漏读、错读、添读、数字、专名和多音字；"
|
||||
"(2)自然度：检查机器感、不自然停顿、流畅度、重音和韵律；"
|
||||
"(3)情感表达：检查语调、语速和强调是否符合期望情感；"
|
||||
"(4)音色一致性：只比较说话人音色，不要把内容或录音质量差异误当作不同说话人。"
|
||||
"每个理由必须引用一个可听见的具体观察。只输出 JSON："
|
||||
'{"准确性":{"score":int,"reason":str},"自然度":{"score":int,"reason":str},'
|
||||
'"情感表达":{"score":int,"reason":str},"音色一致性":{"score":int,"reason":str}}\n'
|
||||
f"合成语音原文：{reference}\n期望情感：{emotion}\n"
|
||||
"音频顺序：1=待评估合成语音；2=参考说话人语音。"
|
||||
)
|
||||
provider_attempts = []
|
||||
if not key:
|
||||
if openrouter_key:
|
||||
return _judge_openrouter_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
return _judge_mistral_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
model = _resolve_gemini_model(key)
|
||||
body = {
|
||||
"contents": [{"parts": [
|
||||
{"text": prompt},
|
||||
{"inline_data": {"mime_type": "audio/mp3", "data": audio_b64}},
|
||||
{"inline_data": {"mime_type": "audio/mp3", "data": reference_audio_b64}},
|
||||
]}],
|
||||
"generationConfig": {"temperature": 0.0, "responseMimeType": "application/json"},
|
||||
}
|
||||
url = (f"https://generativelanguage.googleapis.com/v1beta/models/"
|
||||
f"{model}:generateContent?key={key}")
|
||||
req = urllib.request.Request(
|
||||
url, data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json"}, method="POST",
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=90) as r:
|
||||
data = json.loads(r.read())
|
||||
except urllib.error.HTTPError as exc:
|
||||
# Preserve the provider's diagnostic while never serializing the key
|
||||
# (it only appears in the request URL, not this response excerpt).
|
||||
detail = exc.read().decode("utf-8", "replace")[:2000]
|
||||
direct_error = f"Gemini HTTP {exc.code}: {detail}"
|
||||
provider_attempts.append({
|
||||
"provider": "Google Gemini API",
|
||||
"model": model,
|
||||
"status": "unavailable",
|
||||
"error": direct_error,
|
||||
})
|
||||
if openrouter_key:
|
||||
return _judge_openrouter_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
if mistral_key:
|
||||
return _judge_mistral_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
raise JudgeRouteError(direct_error, provider_attempts) from None
|
||||
# Gemini 在安全拦截时不返回 candidates（或 candidate 无 content/parts），
|
||||
# 防御式取值并给出带 promptFeedback 的清晰错误，交由上层记为该条失败。
|
||||
candidates = data.get("candidates") or []
|
||||
parts = []
|
||||
if candidates:
|
||||
parts = (candidates[0].get("content") or {}).get("parts") or []
|
||||
if not parts or not parts[0].get("text"):
|
||||
error = f"Gemini 未返回评审文本：{data.get('promptFeedback') or data}"
|
||||
provider_attempts.append({
|
||||
"provider": "Google Gemini API",
|
||||
"model": model,
|
||||
"status": "unavailable",
|
||||
"error": error,
|
||||
})
|
||||
if openrouter_key:
|
||||
return _judge_openrouter_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
if mistral_key:
|
||||
return _judge_mistral_audio(
|
||||
prompt,
|
||||
audio_b64,
|
||||
reference_audio_b64,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
raise JudgeRouteError(error, provider_attempts)
|
||||
text = parts[0]["text"]
|
||||
provider_attempts.append({
|
||||
"provider": "Google Gemini API",
|
||||
"model": model,
|
||||
"status": "ok",
|
||||
})
|
||||
return _parse_direct_audio_rubric(
|
||||
text,
|
||||
judge_model=model,
|
||||
provider_attempts=provider_attempts,
|
||||
)
|
||||
|
||||
|
||||
def sha256_file(path: str) -> str:
|
||||
"""Return a content identity for an evidence audio file."""
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
@@ -0,0 +1,4 @@
|
||||
openai>=1.40.0
|
||||
python-dotenv>=1.0.0
|
||||
fish-audio-sdk>=1.3.0,<2
|
||||
# ffprobe 走系统命令；Gemini 音频评审走内置 urllib（REST）。
|
||||
@@ -0,0 +1,73 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "7-6",
|
||||
"status": "incomplete",
|
||||
"generated_at_utc": "2026-07-29T22:26:23.177040+00:00",
|
||||
"run_dir": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2",
|
||||
"git_commit": "4a7f37cf278bd15948c409f14533017c4c7fbc29",
|
||||
"command": "python demo.py --providers fishaudio --gemini --fresh --output RUN_DIR",
|
||||
"provider_receipt_count": 0,
|
||||
"status_reasons": [
|
||||
"Fish Audio synthesized all six real corpus clips, but Gemini rejected the configured credential, so zero four-dimensional judgments are valid.",
|
||||
"Only one synthesis provider completed; the manuscript requires a multi-provider comparison."
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/config.py",
|
||||
"bytes": 8485,
|
||||
"sha256": "512913faae41a62e8e2093fa855bbd19d62d3bbbd8b8bb049aac42febe5b90d3"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/pipeline.py",
|
||||
"bytes": 23173,
|
||||
"sha256": "e6ae0088d7649cd90a4bbd21b01dcb24f350d2708b2c747effd0d62c6c2ac29f"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/demo.py",
|
||||
"bytes": 16925,
|
||||
"sha256": "277f032794b10f419affd5800af2c3cf8b624d3fff756f5af98e53e2891bb6da"
|
||||
},
|
||||
{
|
||||
"path": "chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"bytes": 73977,
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2/fishaudio-s1-clone__emotion.mp3",
|
||||
"bytes": 95293,
|
||||
"sha256": "f124226905ed828674d2517ff67fe9f3b976f8d48e739ad2214088a459833a8f"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2/fishaudio-s1-clone__long.mp3",
|
||||
"bytes": 203127,
|
||||
"sha256": "1a0560b87d85699ee8e5635f4b50cac315d0621163ca27cf3b6740e1ba70129b"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2/fishaudio-s1-clone__num.mp3",
|
||||
"bytes": 103652,
|
||||
"sha256": "536bd78c6f39cbc24833ac7096b6c9c5cf7018273619b58f307fc9a012ea5f1a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2/fishaudio-s1-clone__polyphone.mp3",
|
||||
"bytes": 97801,
|
||||
"sha256": "10f0ef83d8440209a016f9ca0aa0de8c528c809cf94dbab0f61403edf24ab391"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2/fishaudio-s1-clone__question.mp3",
|
||||
"bytes": 78993,
|
||||
"sha256": "ee56d58c14fce8cdedf6027a738077aa3dcf10d713f09bf99d56871d4879ef82"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2/fishaudio-s1-clone__sad.mp3",
|
||||
"bytes": 88606,
|
||||
"sha256": "5a01f6d1b7f02e8c4c92d317a872f45672e20719ce99f2bbaa190f0a25a8051f"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/runs/exp7-6-fish-gemini-20260730-v2/results.json",
|
||||
"bytes": 7778,
|
||||
"sha256": "a3b1a78591db3e8bdccd35af7e7f2184cf164e6add6d0d42d77d9437924f5b17"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "7-6",
|
||||
"generated_at_utc": "2026-07-29T22:25:34.024892+00:00",
|
||||
"command_scope": {
|
||||
"providers": [
|
||||
"fishaudio"
|
||||
],
|
||||
"configurations": [
|
||||
{
|
||||
"name": "fishaudio-s1-clone",
|
||||
"provider": "fishaudio",
|
||||
"model": "s1",
|
||||
"voice": "6df3c1e14c9440e9ac978556536bf116",
|
||||
"speed": 1.0
|
||||
}
|
||||
],
|
||||
"corpus_ids": [
|
||||
"num",
|
||||
"polyphone",
|
||||
"long",
|
||||
"emotion",
|
||||
"sad",
|
||||
"question"
|
||||
],
|
||||
"expected_records": 6,
|
||||
"direct_audio_judge": true,
|
||||
"optional_asr_enabled": false
|
||||
},
|
||||
"reference_audio": {
|
||||
"path": "../../../../chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
},
|
||||
"rubric_dimensions": [
|
||||
"准确性",
|
||||
"自然度",
|
||||
"情感表达",
|
||||
"音色一致性"
|
||||
],
|
||||
"completion": {
|
||||
"successful_records": 0,
|
||||
"direct_audio_four_dimension_records": 0,
|
||||
"expected_records": 6,
|
||||
"all_cells_complete": false,
|
||||
"multi_provider": false,
|
||||
"manuscript_core_complete": false
|
||||
},
|
||||
"records": [
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n",
|
||||
"audio_path": "fishaudio-s1-clone__num.mp3",
|
||||
"audio_sha256": "536bd78c6f39cbc24833ac7096b6c9c5cf7018273619b58f307fc9a012ea5f1a",
|
||||
"audio_bytes": 103652,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n",
|
||||
"audio_path": "fishaudio-s1-clone__polyphone.mp3",
|
||||
"audio_sha256": "10f0ef83d8440209a016f9ca0aa0de8c528c809cf94dbab0f61403edf24ab391",
|
||||
"audio_bytes": 97801,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n",
|
||||
"audio_path": "fishaudio-s1-clone__long.mp3",
|
||||
"audio_sha256": "1a0560b87d85699ee8e5635f4b50cac315d0621163ca27cf3b6740e1ba70129b",
|
||||
"audio_bytes": 203127,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n",
|
||||
"audio_path": "fishaudio-s1-clone__emotion.mp3",
|
||||
"audio_sha256": "f124226905ed828674d2517ff67fe9f3b976f8d48e739ad2214088a459833a8f",
|
||||
"audio_bytes": 95293,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "sad",
|
||||
"challenge": "悲伤情感/低语速低语调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n",
|
||||
"audio_path": "fishaudio-s1-clone__sad.mp3",
|
||||
"audio_sha256": "5a01f6d1b7f02e8c4c92d317a872f45672e20719ce99f2bbaa190f0a25a8051f",
|
||||
"audio_bytes": 88606,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "question",
|
||||
"challenge": "对话文体/疑问句升调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n",
|
||||
"audio_path": "fishaudio-s1-clone__question.mp3",
|
||||
"audio_sha256": "ee56d58c14fce8cdedf6027a738077aa3dcf10d713f09bf99d56871d4879ef82",
|
||||
"audio_bytes": 78993,
|
||||
"failed_stage": "multimodal_judge"
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"n_ok": 0,
|
||||
"n": 6
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,166 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "7-6",
|
||||
"generated_at_utc": "2026-07-29T19:41:22.781006+00:00",
|
||||
"command_scope": {
|
||||
"providers": [
|
||||
"fishaudio",
|
||||
"openai"
|
||||
],
|
||||
"configurations": [
|
||||
{
|
||||
"name": "openai-alloy",
|
||||
"provider": "openai",
|
||||
"model": "tts-1",
|
||||
"voice": "alloy",
|
||||
"speed": 1.0
|
||||
},
|
||||
{
|
||||
"name": "fishaudio-s1-clone",
|
||||
"provider": "fishaudio",
|
||||
"model": "s1",
|
||||
"voice": "6df3c1e14c9440e9ac978556536bf116",
|
||||
"speed": 1.0
|
||||
}
|
||||
],
|
||||
"corpus_ids": [
|
||||
"num",
|
||||
"polyphone",
|
||||
"long",
|
||||
"emotion",
|
||||
"sad",
|
||||
"question"
|
||||
],
|
||||
"expected_records": 12,
|
||||
"direct_audio_judge": true,
|
||||
"optional_asr_enabled": false
|
||||
},
|
||||
"reference_audio": {
|
||||
"path": "../../../../chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
},
|
||||
"rubric_dimensions": [
|
||||
"准确性",
|
||||
"自然度",
|
||||
"情感表达",
|
||||
"音色一致性"
|
||||
],
|
||||
"completion": {
|
||||
"successful_records": 0,
|
||||
"direct_audio_four_dimension_records": 0,
|
||||
"expected_records": 12,
|
||||
"all_cells_complete": false,
|
||||
"multi_provider": true,
|
||||
"manuscript_core_complete": false
|
||||
},
|
||||
"records": [
|
||||
{
|
||||
"config": "openai-alloy",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}"
|
||||
},
|
||||
{
|
||||
"config": "openai-alloy",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}"
|
||||
},
|
||||
{
|
||||
"config": "openai-alloy",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}"
|
||||
},
|
||||
{
|
||||
"config": "openai-alloy",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}"
|
||||
},
|
||||
{
|
||||
"config": "openai-alloy",
|
||||
"sample": "sad",
|
||||
"challenge": "悲伤情感/低语速低语调",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}"
|
||||
},
|
||||
{
|
||||
"config": "openai-alloy",
|
||||
"sample": "question",
|
||||
"challenge": "对话文体/疑问句升调",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.', 'type': 'insufficient_quota', 'param': None, 'code': 'insufficient_quota'}}"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "HTTPError: HTTP Error 400: Bad Request"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "HTTPError: HTTP Error 400: Bad Request"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "HTTPError: HTTP Error 400: Bad Request"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "HTTPError: HTTP Error 400: Bad Request"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "sad",
|
||||
"challenge": "悲伤情感/低语速低语调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "HTTPError: HTTP Error 400: Bad Request"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "question",
|
||||
"challenge": "对话文体/疑问句升调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "HTTPError: HTTP Error 400: Bad Request"
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"config": "openai-alloy",
|
||||
"n_ok": 0,
|
||||
"n": 6
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"n_ok": 0,
|
||||
"n": 6
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
"""
|
||||
Regression tests for the Minimax t2a_v2 synthesis adapter (实验 7-6 TTS 质量评估).
|
||||
|
||||
Locks in the refreshed contract:
|
||||
- the request targets the /v1/t2a_v2 endpoint with Bearer auth and no GroupId
|
||||
query parameter, on the global host by default and the mainland-China host
|
||||
when MINIMAX_REGION selects it;
|
||||
- the default model is speech-2.8-hd;
|
||||
- the response is validated on base_resp.status_code, data.status and
|
||||
data.audio, so a bad return code or an unfinished status raises a clear
|
||||
RuntimeError instead of yielding empty/garbage audio.
|
||||
|
||||
Network is stubbed: pipeline._http_post is replaced with a fake that records the
|
||||
request and returns a canned JSON body.
|
||||
"""
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
import config
|
||||
import pipeline
|
||||
|
||||
|
||||
def _cfg(model="speech-2.8-hd"):
|
||||
return config.TTSConfig("minimax-test", provider="minimax",
|
||||
model=model, voice="male-qn-qingse")
|
||||
|
||||
|
||||
def _stub_http_post(monkeypatch, response: dict):
|
||||
"""Capture the outgoing request and return a canned decoded JSON body."""
|
||||
calls = {}
|
||||
|
||||
def fake_post(url, body, headers, timeout=90.0):
|
||||
calls["url"] = url
|
||||
calls["body"] = body
|
||||
calls["headers"] = headers
|
||||
return json.dumps(response).encode()
|
||||
|
||||
monkeypatch.setattr(pipeline, "_http_post", fake_post)
|
||||
return calls
|
||||
|
||||
|
||||
def _ok_response(audio_bytes: bytes):
|
||||
return {
|
||||
"data": {"audio": audio_bytes.hex(), "status": pipeline._MINIMAX_STATUS_DONE},
|
||||
"base_resp": {"status_code": pipeline._MINIMAX_SUCCESS_CODE, "status_msg": "success"},
|
||||
}
|
||||
|
||||
|
||||
def test_default_endpoint_is_global_bearer_no_groupid(monkeypatch):
|
||||
"""Global host, Bearer auth, no GroupId query param, default model."""
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "fake-key")
|
||||
monkeypatch.delenv("MINIMAX_REGION", raising=False)
|
||||
calls = _stub_http_post(monkeypatch, _ok_response(b"\xff\xfb\x10\x20"))
|
||||
|
||||
audio = pipeline._synth_minimax(_cfg(), "你好")
|
||||
|
||||
assert calls["url"] == "https://api.minimax.io/v1/t2a_v2"
|
||||
assert "GroupId" not in calls["url"]
|
||||
assert calls["headers"]["Authorization"] == "Bearer fake-key"
|
||||
assert calls["body"]["model"] == "speech-2.8-hd"
|
||||
assert audio == b"\xff\xfb\x10\x20"
|
||||
|
||||
|
||||
def test_cn_region_uses_minimaxi_host(monkeypatch):
|
||||
"""MINIMAX_REGION=cn routes to the mainland-China api.minimaxi.com host."""
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "fake-key")
|
||||
monkeypatch.setenv("MINIMAX_REGION", "cn")
|
||||
calls = _stub_http_post(monkeypatch, _ok_response(b"\x00\x01"))
|
||||
|
||||
pipeline._synth_minimax(_cfg(), "你好")
|
||||
|
||||
assert calls["url"] == "https://api.minimaxi.com/v1/t2a_v2"
|
||||
|
||||
|
||||
def test_empty_model_falls_back_to_default(monkeypatch):
|
||||
"""An empty cfg.model falls back to the current default speech-2.8-hd."""
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "fake-key")
|
||||
calls = _stub_http_post(monkeypatch, _ok_response(b"\x00"))
|
||||
|
||||
pipeline._synth_minimax(_cfg(model=""), "你好")
|
||||
|
||||
assert calls["body"]["model"] == "speech-2.8-hd"
|
||||
|
||||
|
||||
def test_nonzero_base_resp_raises(monkeypatch):
|
||||
"""base_resp.status_code != 0 raises a clear error instead of decoding junk."""
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "fake-key")
|
||||
_stub_http_post(monkeypatch, {
|
||||
"data": {},
|
||||
"base_resp": {"status_code": 1004, "status_msg": "authentication failed"},
|
||||
})
|
||||
with pytest.raises(RuntimeError, match="base_resp"):
|
||||
pipeline._synth_minimax(_cfg(), "你好")
|
||||
|
||||
|
||||
def test_unfinished_status_raises(monkeypatch):
|
||||
"""A non-finished data.status (with no audio) raises rather than returning empty audio."""
|
||||
monkeypatch.setenv("MINIMAX_API_KEY", "fake-key")
|
||||
_stub_http_post(monkeypatch, {
|
||||
"data": {"status": 1, "audio": ""},
|
||||
"base_resp": {"status_code": 0, "status_msg": "success"},
|
||||
})
|
||||
with pytest.raises(RuntimeError, match="finished audio"):
|
||||
pipeline._synth_minimax(_cfg(), "你好")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,9 @@
|
||||
"""Test import bootstrap for the tts-quality-eval experiment."""
|
||||
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
|
||||
EXPERIMENT_ROOT = Path(__file__).resolve().parents[1]
|
||||
if str(EXPERIMENT_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(EXPERIMENT_ROOT))
|
||||
@@ -0,0 +1,254 @@
|
||||
"""
|
||||
Regression tests for judge-response robustness (实验 7-6 TTS 质量评估).
|
||||
|
||||
Covers two failure classes on LLM/Gemini judge responses:
|
||||
- judge_rubric: judge returns "score": null (or a bare null dimension) -> int(None) TypeError
|
||||
- judge_gemini_audio: safety-blocked Gemini responses have no
|
||||
candidates/content/parts -> KeyError/IndexError instead of a clear error
|
||||
|
||||
Network is stubbed: the OpenAI-compatible judge client is replaced with a fake,
|
||||
and urllib.request.urlopen is monkeypatched for the Gemini REST call.
|
||||
"""
|
||||
import io
|
||||
import json
|
||||
|
||||
import pytest
|
||||
|
||||
import pipeline
|
||||
|
||||
|
||||
class _FakeMessage:
|
||||
content = "{}"
|
||||
|
||||
|
||||
class _FakeChoice:
|
||||
message = _FakeMessage()
|
||||
|
||||
|
||||
class _FakeResp:
|
||||
choices = [_FakeChoice()]
|
||||
|
||||
|
||||
class _FakeCompletions:
|
||||
@staticmethod
|
||||
def create(**kwargs):
|
||||
return _FakeResp()
|
||||
|
||||
|
||||
class _FakeChat:
|
||||
completions = _FakeCompletions()
|
||||
|
||||
|
||||
class _FakeClient:
|
||||
chat = _FakeChat()
|
||||
|
||||
|
||||
def _stub_judge(monkeypatch, payload: dict):
|
||||
_FakeMessage.content = json.dumps(payload, ensure_ascii=False)
|
||||
monkeypatch.setattr(
|
||||
pipeline, "get_judge_client_and_model", lambda model=None: (_FakeClient(), "fake-judge"))
|
||||
|
||||
|
||||
def test_judge_rubric_tolerates_null_score(monkeypatch):
|
||||
"""'score': null in a dimension dict is scored 0, not int(None) TypeError."""
|
||||
_stub_judge(monkeypatch, {
|
||||
"准确性": {"score": None, "reason": "无法判断"},
|
||||
"自然度": {"score": 4, "reason": "语速正常"},
|
||||
"情感表达": {"score": 0},
|
||||
"音色一致性": {"score": 0, "reason": "无法听到音频"},
|
||||
})
|
||||
rub = pipeline.judge_rubric("原文文本", "中性", "回译文本", 3.0, 0.05)
|
||||
assert rub.scores["准确性"] == 0
|
||||
assert rub.scores["自然度"] == 4
|
||||
assert rub.scores["音色一致性"] == 0
|
||||
|
||||
|
||||
def test_judge_rubric_tolerates_null_dimension(monkeypatch):
|
||||
"""A bare null dimension (non-dict) is scored 0, not int(None) TypeError."""
|
||||
_stub_judge(monkeypatch, {
|
||||
"准确性": None,
|
||||
"自然度": 4,
|
||||
"情感表达": 0,
|
||||
"音色一致性": 0,
|
||||
})
|
||||
rub = pipeline.judge_rubric("原文文本", "中性", "回译文本", 3.0, 0.05)
|
||||
assert rub.scores["准确性"] == 0
|
||||
assert rub.scores["自然度"] == 4
|
||||
|
||||
|
||||
class _FakeHTTPResp(io.BytesIO):
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
return False
|
||||
|
||||
|
||||
def _stub_gemini(monkeypatch, payload: dict):
|
||||
monkeypatch.setenv("GEMINI_API_KEY", "fake-key-for-test")
|
||||
monkeypatch.delenv("OPENROUTER_API_KEY", raising=False)
|
||||
monkeypatch.delenv("MISTRAL_API_KEY", raising=False)
|
||||
monkeypatch.setattr(pipeline, "_resolve_gemini_model", lambda key: "gemini-fake")
|
||||
monkeypatch.setattr("urllib.request.urlopen",
|
||||
lambda req, timeout=None: _FakeHTTPResp(json.dumps(payload).encode()))
|
||||
|
||||
|
||||
@pytest.mark.parametrize("payload", [
|
||||
{"promptFeedback": {"blockReason": "SAFETY"}}, # prompt 被拦截：无 candidates
|
||||
{"candidates": []}, # 生成被拦截：空 candidates
|
||||
{"candidates": [{"finishReason": "SAFETY", "index": 0}]}, # candidate 无 content
|
||||
])
|
||||
def test_judge_gemini_audio_blocked_raises_clear_error(monkeypatch, tmp_path, payload):
|
||||
"""Blocked/empty Gemini responses raise a clear RuntimeError, not KeyError/IndexError."""
|
||||
_stub_gemini(monkeypatch, payload)
|
||||
audio = tmp_path / "a.mp3"
|
||||
audio.write_bytes(b"\xff\xfb" + b"\x00" * 256)
|
||||
reference = tmp_path / "reference.mp3"
|
||||
reference.write_bytes(b"\xff\xfb" + b"\x01" * 256)
|
||||
with pytest.raises(RuntimeError, match="Gemini 未返回评审文本"):
|
||||
pipeline.judge_gemini_audio("原文", "中性", str(audio), str(reference))
|
||||
|
||||
|
||||
def test_judge_gemini_audio_parses_valid_response(monkeypatch, tmp_path):
|
||||
"""A normal Gemini response still parses (defensive navigation keeps working)."""
|
||||
inner = json.dumps({
|
||||
"准确性": {"score": 4, "reason": "ok"},
|
||||
"自然度": 4,
|
||||
"情感表达": None,
|
||||
"音色一致性": {"score": 5},
|
||||
}, ensure_ascii=False)
|
||||
_stub_gemini(monkeypatch, {
|
||||
"candidates": [{"content": {"parts": [{"text": inner}]}}],
|
||||
})
|
||||
audio = tmp_path / "a.mp3"
|
||||
audio.write_bytes(b"\xff\xfb" + b"\x00" * 256)
|
||||
reference = tmp_path / "reference.mp3"
|
||||
reference.write_bytes(b"\xff\xfb" + b"\x01" * 256)
|
||||
rub = pipeline.judge_gemini_audio("原文", "中性", str(audio), str(reference))
|
||||
assert rub.scores["准确性"] == 4
|
||||
assert rub.scores["情感表达"] == 0 # null score -> 0
|
||||
assert rub.scores["音色一致性"] == 5
|
||||
|
||||
|
||||
def test_judge_gemini_audio_falls_back_to_openrouter(monkeypatch, tmp_path):
|
||||
"""An unavailable direct key keeps both clips on a direct-audio fallback route."""
|
||||
monkeypatch.setenv("GEMINI_API_KEY", "invalid-direct-key")
|
||||
monkeypatch.setenv("OPENROUTER_API_KEY", "fake-openrouter-key")
|
||||
monkeypatch.setattr(pipeline, "_resolve_gemini_model", lambda key: "gemini-fake")
|
||||
|
||||
def _http_error(req, timeout=None):
|
||||
import urllib.error
|
||||
raise urllib.error.HTTPError(req.full_url, 400, "bad key", {}, io.BytesIO(b"invalid"))
|
||||
|
||||
monkeypatch.setattr("urllib.request.urlopen", _http_error)
|
||||
expected = pipeline.RubricResult(
|
||||
scores={dim: 4 for dim in pipeline.RUBRIC_DIMENSIONS},
|
||||
reasons={dim: "audible evidence" for dim in pipeline.RUBRIC_DIMENSIONS},
|
||||
judge_model="openrouter/google/gemini-3.5-flash",
|
||||
evidence_mode="direct-audio-with-reference",
|
||||
provider_attempts=[],
|
||||
)
|
||||
monkeypatch.setattr(pipeline, "_judge_openrouter_audio", lambda *args, **kwargs: expected)
|
||||
audio = tmp_path / "a.mp3"
|
||||
audio.write_bytes(b"\xff\xfb" + b"\x00" * 256)
|
||||
reference = tmp_path / "reference.mp3"
|
||||
reference.write_bytes(b"\xff\xfb" + b"\x01" * 256)
|
||||
|
||||
rub = pipeline.judge_gemini_audio("原文", "中性", str(audio), str(reference))
|
||||
assert rub is expected
|
||||
assert rub.evidence_mode == "direct-audio-with-reference"
|
||||
|
||||
|
||||
def test_openrouter_failure_preserves_both_route_attempts(monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("GEMINI_API_KEY", "invalid-direct-key")
|
||||
monkeypatch.setenv("OPENROUTER_API_KEY", "invalid-openrouter-key")
|
||||
monkeypatch.delenv("MISTRAL_API_KEY", raising=False)
|
||||
monkeypatch.setattr(pipeline, "_resolve_gemini_model", lambda key: "gemini-fake")
|
||||
|
||||
def _http_error(req, timeout=None):
|
||||
import urllib.error
|
||||
body = b"direct invalid" if "googleapis" in req.full_url else b"router invalid"
|
||||
code = 400 if "googleapis" in req.full_url else 401
|
||||
raise urllib.error.HTTPError(req.full_url, code, "unavailable", {}, io.BytesIO(body))
|
||||
|
||||
monkeypatch.setattr("urllib.request.urlopen", _http_error)
|
||||
audio = tmp_path / "a.mp3"
|
||||
audio.write_bytes(b"\xff\xfb" + b"\x00" * 256)
|
||||
reference = tmp_path / "reference.mp3"
|
||||
reference.write_bytes(b"\xff\xfb" + b"\x01" * 256)
|
||||
|
||||
with pytest.raises(pipeline.JudgeRouteError) as caught:
|
||||
pipeline.judge_gemini_audio("原文", "中性", str(audio), str(reference))
|
||||
assert [attempt["status"] for attempt in caught.value.provider_attempts] == [
|
||||
"unavailable", "unavailable"
|
||||
]
|
||||
assert [attempt["provider"] for attempt in caught.value.provider_attempts] == [
|
||||
"Google Gemini API", "OpenRouter audio route"
|
||||
]
|
||||
|
||||
|
||||
def test_openrouter_failure_falls_back_to_exact_mistral_two_audio_route(monkeypatch, tmp_path):
|
||||
monkeypatch.setenv("GEMINI_API_KEY", "invalid-direct-key")
|
||||
monkeypatch.setenv("OPENROUTER_API_KEY", "invalid-openrouter-key")
|
||||
monkeypatch.setenv("MISTRAL_API_KEY", "fake-mistral-key")
|
||||
monkeypatch.setattr(pipeline, "_resolve_gemini_model", lambda key: "gemini-fake")
|
||||
monkeypatch.setattr(pipeline.time, "sleep", lambda seconds: None)
|
||||
observed_mistral_body = {}
|
||||
mistral_calls = 0
|
||||
inner = json.dumps({
|
||||
dim: {"score": 4, "reason": "audible evidence"}
|
||||
for dim in pipeline.RUBRIC_DIMENSIONS
|
||||
}, ensure_ascii=False)
|
||||
|
||||
def _route(req, timeout=None):
|
||||
nonlocal mistral_calls
|
||||
import urllib.error
|
||||
if "googleapis" in req.full_url:
|
||||
raise urllib.error.HTTPError(
|
||||
req.full_url, 400, "bad key", {}, io.BytesIO(b"direct invalid")
|
||||
)
|
||||
if "openrouter" in req.full_url:
|
||||
raise urllib.error.HTTPError(
|
||||
req.full_url, 401, "bad key", {}, io.BytesIO(b"router invalid")
|
||||
)
|
||||
mistral_calls += 1
|
||||
if mistral_calls == 1:
|
||||
raise urllib.error.HTTPError(
|
||||
req.full_url, 500, "transient", {}, io.BytesIO(b"service unavailable")
|
||||
)
|
||||
observed_mistral_body.update(json.loads(req.data))
|
||||
return _FakeHTTPResp(json.dumps({
|
||||
"choices": [{"message": {"content": inner}}]
|
||||
}).encode())
|
||||
|
||||
monkeypatch.setattr("urllib.request.urlopen", _route)
|
||||
audio_bytes = b"\xff\xfb" + b"\x00" * 256
|
||||
reference_bytes = b"\xff\xfb" + b"\x01" * 256
|
||||
audio = tmp_path / "a.mp3"
|
||||
audio.write_bytes(audio_bytes)
|
||||
reference = tmp_path / "reference.mp3"
|
||||
reference.write_bytes(reference_bytes)
|
||||
|
||||
rub = pipeline.judge_gemini_audio("原文", "中性", str(audio), str(reference))
|
||||
|
||||
content = observed_mistral_body["messages"][0]["content"]
|
||||
assert [item["type"] for item in content] == [
|
||||
"text", "text", "input_audio", "text", "input_audio"
|
||||
]
|
||||
audio_chunks = [item for item in content if item["type"] == "input_audio"]
|
||||
assert len(audio_chunks) == 2
|
||||
assert audio_chunks[0]["input_audio"].startswith("data:audio/mpeg;base64,")
|
||||
assert audio_chunks[1]["input_audio"].startswith("data:audio/mpeg;base64,")
|
||||
assert rub.judge_model == "mistral/voxtral-small-latest"
|
||||
assert rub.evidence_mode == "direct-audio-with-reference"
|
||||
assert [attempt["provider"] for attempt in rub.provider_attempts] == [
|
||||
"Google Gemini API", "OpenRouter audio route", "Mistral Voxtral API"
|
||||
]
|
||||
assert [attempt["status"] for attempt in rub.provider_attempts] == [
|
||||
"unavailable", "unavailable", "ok"
|
||||
]
|
||||
assert rub.provider_attempts[-1]["attempts"] == 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
pytest.main([__file__, "-v"])
|
||||
@@ -0,0 +1,25 @@
|
||||
"""
|
||||
Test suite locking out TypeError in _resolve_gemini_model
|
||||
when API returns data with models: None or non-dict items.
|
||||
"""
|
||||
|
||||
import json
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from pipeline import _resolve_gemini_model, config
|
||||
|
||||
|
||||
def test_resolve_gemini_model_handles_null_models():
|
||||
"""
|
||||
Ensure _resolve_gemini_model returns default model without TypeError when models key is None.
|
||||
"""
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.read.return_value = json.dumps({"models": None}).encode("utf-8")
|
||||
mock_resp.__enter__ = MagicMock(return_value=mock_resp)
|
||||
mock_resp.__exit__ = MagicMock(return_value=None)
|
||||
|
||||
import urllib.request
|
||||
urllib.request.urlopen = MagicMock(return_value=mock_resp)
|
||||
|
||||
res = _resolve_gemini_model("dummy_key")
|
||||
assert res == config.GEMINI_MODEL_DEFAULT
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "7-6",
|
||||
"status": "incomplete",
|
||||
"generated_at_utc": "2026-07-30T04:26:35.560333+00:00",
|
||||
"run_dir": "chapter7/tts-quality-eval/validation/audio_fallback_probe_20260730",
|
||||
"git_commit": "f119cf510f3746cd5d071d0bce0051617f02f163",
|
||||
"command": "python demo.py --providers fishaudio --gemini --quick --output validation/audio_fallback_probe_20260730",
|
||||
"provider_receipt_count": 2,
|
||||
"status_reasons": [
|
||||
"Fish Audio returned two non-empty MP3 artifacts. The Google Gemini credential was rejected and the exact OpenRouter two-input_audio fallback returned HTTP 401, so no direct-audio judge record completed."
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/config.py",
|
||||
"bytes": 8642,
|
||||
"sha256": "891396771cd7beb32ab8e4b316f9a6f73683951e0327dcae6504d858ffd92ba1"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/demo.py",
|
||||
"bytes": 17225,
|
||||
"sha256": "c2da1f44b3ac3ec9e9b6648ee535e57c8fa757a14a45028f1e125ed9cb8f1938"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/pipeline.py",
|
||||
"bytes": 27511,
|
||||
"sha256": "db62ac8058561ae22b9a2dacd67aec42176f181a401e162009e1a3be7360ed54"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/audio_fallback_probe_20260730/fishaudio-s1-clone__num.mp3",
|
||||
"bytes": 99055,
|
||||
"sha256": "0bfb70e01370fe9f28ddbde6aebb9b23ec5cb5e312377edbdace44ad43a071a6"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/audio_fallback_probe_20260730/fishaudio-s1-clone__polyphone.mp3",
|
||||
"bytes": 95293,
|
||||
"sha256": "f90e5bc2f8f21f707697b5db949c1400b6fe899b3b64bd9e4e2c374bb79e0b77"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/audio_fallback_probe_20260730/results.json",
|
||||
"bytes": 4469,
|
||||
"sha256": "9e66a07e8817b8cfe261f65970d2dd46b4bd2fe118e9aa324feba849652d957f"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,105 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "7-6",
|
||||
"generated_at_utc": "2026-07-30T04:26:22.901148+00:00",
|
||||
"command_scope": {
|
||||
"providers": [
|
||||
"fishaudio"
|
||||
],
|
||||
"configurations": [
|
||||
{
|
||||
"name": "fishaudio-s1-clone",
|
||||
"provider": "fishaudio",
|
||||
"model": "s1",
|
||||
"voice": "6df3c1e14c9440e9ac978556536bf116",
|
||||
"speed": 1.0
|
||||
}
|
||||
],
|
||||
"corpus_ids": [
|
||||
"num",
|
||||
"polyphone"
|
||||
],
|
||||
"expected_records": 2,
|
||||
"direct_audio_judge": true,
|
||||
"optional_asr_enabled": false
|
||||
},
|
||||
"reference_audio": {
|
||||
"path": "../../../../chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
},
|
||||
"rubric_dimensions": [
|
||||
"准确性",
|
||||
"自然度",
|
||||
"情感表达",
|
||||
"音色一致性"
|
||||
],
|
||||
"completion": {
|
||||
"successful_records": 0,
|
||||
"direct_audio_four_dimension_records": 0,
|
||||
"expected_records": 2,
|
||||
"all_cells_complete": false,
|
||||
"multi_provider": false,
|
||||
"manuscript_core_complete": false
|
||||
},
|
||||
"records": [
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__num.mp3",
|
||||
"audio_sha256": "0bfb70e01370fe9f28ddbde6aebb9b23ec5cb5e312377edbdace44ad43a071a6",
|
||||
"audio_bytes": 99055,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__polyphone.mp3",
|
||||
"audio_sha256": "f90e5bc2f8f21f707697b5db949c1400b6fe899b3b64bd9e4e2c374bb79e0b77",
|
||||
"audio_bytes": 95293,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"n_ok": 0,
|
||||
"n": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "7-6",
|
||||
"status": "complete",
|
||||
"generated_at_utc": "2026-07-30T04:47:32.164712+00:00",
|
||||
"run_dir": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730",
|
||||
"git_commit": "f119cf510f3746cd5d071d0bce0051617f02f163",
|
||||
"command": "GEMINI_API_KEY= OPENROUTER_API_KEY= python demo.py --providers openai,fishaudio --gemini --limit 4 --output validation/mistral_multimodal_20260730",
|
||||
"status_reasons": [
|
||||
"Eight of eight cells completed across two real synthesis providers and four challenge texts. Every cell supplied a content-hashed candidate MP3 and the same content-hashed real reference MP3 to Mistral voxtral-small-latest.",
|
||||
"The judge returned all four manuscript rubric dimensions as integers in [1,5]. The structured completion gates recompute to manuscript_core_complete=true.",
|
||||
"Synthesis audio was retained from the preceding real provider run rather than regenerated under the currently exhausted OpenAI account. synthesis_source_results.json independently records the same eight audio hashes and shows those cells had reached the earlier multimodal-judge stage."
|
||||
],
|
||||
"provider_receipt_count": 8,
|
||||
"inputs": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/config.py",
|
||||
"bytes": 8642,
|
||||
"sha256": "891396771cd7beb32ab8e4b316f9a6f73683951e0327dcae6504d858ffd92ba1"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/demo.py",
|
||||
"bytes": 17662,
|
||||
"sha256": "75260a454a3425b749cc294aad87cd4751ef8a9c6882aae1dd577f6a121e38b1"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/pipeline.py",
|
||||
"bytes": 32401,
|
||||
"sha256": "0815da5ff74acd099a189a097a9a1e1aa02ade2e22ba230a2245b021374eb901"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/synthesis_source_results.json",
|
||||
"bytes": 7525,
|
||||
"sha256": "965ecf983521894175404106108d4c32ba8c9ff97a2ae5f740af679d04d204d8"
|
||||
},
|
||||
{
|
||||
"path": "chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"bytes": 73977,
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/tts1-alloy-1.0__num.mp3",
|
||||
"bytes": 132960,
|
||||
"sha256": "954eca94164d5af7d4a849def397e16a192d22a86168fab86f76ccd2443fb3ed"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/tts1-alloy-1.0__polyphone.mp3",
|
||||
"bytes": 130560,
|
||||
"sha256": "fa6e6ca29a3d17d33167c04c32b132d6aee2cb406d2acbf7bbeb13529af98abb"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/tts1-alloy-1.0__long.mp3",
|
||||
"bytes": 273600,
|
||||
"sha256": "c04786022e111c49f635eb361cb4cae3b786334291af86f7f36e69fff7909d1a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/tts1-alloy-1.0__emotion.mp3",
|
||||
"bytes": 116160,
|
||||
"sha256": "86215ce027003e1b0a721545b594ef0325745738a46bbf927a7843da3a15b514"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/fishaudio-s1-clone__num.mp3",
|
||||
"bytes": 101981,
|
||||
"sha256": "6e9717e58f422db10b8e9538aa3812de456a1dde4bd9028fde928c2eaac42ad3"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/fishaudio-s1-clone__polyphone.mp3",
|
||||
"bytes": 95293,
|
||||
"sha256": "6bff6109ec7682f85680d8b60ea58dd48fbe5b3eeda59fb326d2288d250c85b5"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/fishaudio-s1-clone__long.mp3",
|
||||
"bytes": 206052,
|
||||
"sha256": "89542f05312bac767d5151f7552d0c75065cc7ecdf739dcc180b3c92a67b21c5"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/fishaudio-s1-clone__emotion.mp3",
|
||||
"bytes": 104488,
|
||||
"sha256": "81608431645b08be6bf7171dbb250f391b1ba7f5714d535c0e2ce323d7f6ca29"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/mistral_multimodal_20260730/results.json",
|
||||
"bytes": 12890,
|
||||
"sha256": "9d5ae20c9fb9852882732bdc73d7dc78714f7ad5e0025574f90b8584b2841b86"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,384 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "7-6",
|
||||
"generated_at_utc": "2026-07-30T04:47:32.164712+00:00",
|
||||
"command_scope": {
|
||||
"providers": [
|
||||
"fishaudio",
|
||||
"openai"
|
||||
],
|
||||
"configurations": [
|
||||
{
|
||||
"name": "tts1-alloy-1.0",
|
||||
"provider": "openai",
|
||||
"model": "tts-1",
|
||||
"voice": "alloy",
|
||||
"speed": 1.0
|
||||
},
|
||||
{
|
||||
"name": "fishaudio-s1-clone",
|
||||
"provider": "fishaudio",
|
||||
"model": "s1",
|
||||
"voice": "6df3c1e14c9440e9ac978556536bf116",
|
||||
"speed": 1.0
|
||||
}
|
||||
],
|
||||
"corpus_ids": [
|
||||
"num",
|
||||
"polyphone",
|
||||
"long",
|
||||
"emotion"
|
||||
],
|
||||
"expected_records": 8,
|
||||
"direct_audio_judge": true,
|
||||
"optional_asr_enabled": false
|
||||
},
|
||||
"reference_audio": {
|
||||
"path": "../../../../chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
},
|
||||
"rubric_dimensions": [
|
||||
"准确性",
|
||||
"自然度",
|
||||
"情感表达",
|
||||
"音色一致性"
|
||||
],
|
||||
"completion": {
|
||||
"successful_records": 8,
|
||||
"direct_audio_four_dimension_records": 8,
|
||||
"expected_records": 8,
|
||||
"all_cells_complete": true,
|
||||
"multi_provider": true,
|
||||
"manuscript_core_complete": true
|
||||
},
|
||||
"records": [
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "openai",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "tts1-alloy-1.0__num.mp3",
|
||||
"audio_sha256": "954eca94164d5af7d4a849def397e16a192d22a86168fab86f76ccd2443fb3ed",
|
||||
"audio_bytes": 132960,
|
||||
"duration": 6.648,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 4.061371841155235,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 3,
|
||||
"音色一致性": 2
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "合成语音准确无误地朗读了原文，没有漏读、错读、添读、数字、专名和多音字的问题。",
|
||||
"自然度": "合成语音流畅度较好，但有一些不自然的停顿，特别是在数字和百分点的表达上。",
|
||||
"情感表达": "合成语音的语调和语速基本符合中性情感，但缺乏参考说话人语音中的自然感和情感表达。",
|
||||
"音色一致性": "合成语音的音色与参考说话人语音存在明显差异，音色不一致。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "openai",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "tts1-alloy-1.0__polyphone.mp3",
|
||||
"audio_sha256": "fa6e6ca29a3d17d33167c04c32b132d6aee2cb406d2acbf7bbeb13529af98abb",
|
||||
"audio_bytes": 130560,
|
||||
"duration": 6.528,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 4.28921568627451,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "合成语音与原文完全一致，没有漏读、错读、添读、数字、专名和多音字的问题。",
|
||||
"自然度": "合成语音流畅度较好，但有一些不自然的停顿和重音。",
|
||||
"情感表达": "语调和语速基本符合中性情感，但强调稍显不足。",
|
||||
"音色一致性": "合成语音的音色与参考说话人有一定差异，但总体还算接近。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "openai",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "tts1-alloy-1.0__long.mp3",
|
||||
"audio_sha256": "c04786022e111c49f635eb361cb4cae3b786334291af86f7f36e69fff7909d1a",
|
||||
"audio_bytes": 273600,
|
||||
"duration": 13.68,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 4.45906432748538,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "合成语音与原文完全一致，没有漏读、错读、添读、数字、专名和多音字的问题。",
|
||||
"自然度": "合成语音流畅度较好，但有一些不自然的停顿和重音。",
|
||||
"情感表达": "语调和语速基本符合中性情感，但强调不够明显。",
|
||||
"音色一致性": "合成语音的音色与参考说话人有一些差异，但总体还算接近。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "openai",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "tts1-alloy-1.0__emotion.mp3",
|
||||
"audio_sha256": "86215ce027003e1b0a721545b594ef0325745738a46bbf927a7843da3a15b514",
|
||||
"audio_bytes": 116160,
|
||||
"duration": 5.808,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 5.3374655647382925,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "没有漏读、错读、添读、数字、专名和多音字。",
|
||||
"自然度": "有一些机器感，但总体流畅度较好。",
|
||||
"情感表达": "语调和语速表现出兴奋，但强调不够明显。",
|
||||
"音色一致性": "音色与参考说话人有一些差异，但总体相似。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "fishaudio",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "fishaudio-s1-clone__num.mp3",
|
||||
"audio_sha256": "6e9717e58f422db10b8e9538aa3812de456a1dde4bd9028fde928c2eaac42ad3",
|
||||
"audio_bytes": 101981,
|
||||
"duration": 6.373813,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 4.236082859663439,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "合成语音准确无误地朗读了原文，没有漏读、错读、添读、数字、专名和多音字的问题。",
|
||||
"自然度": "合成语音流畅度较好，但有一些不自然的停顿和重音。",
|
||||
"情感表达": "合成语音的语调和语速基本符合中性情感，但强调不够明显。",
|
||||
"音色一致性": "合成语音的音色与参考说话人有一些差异，但总体还算相似。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "fishaudio",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "fishaudio-s1-clone__polyphone.mp3",
|
||||
"audio_sha256": "6bff6109ec7682f85680d8b60ea58dd48fbe5b3eeda59fb326d2288d250c85b5",
|
||||
"audio_bytes": 95293,
|
||||
"duration": 5.955813,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 4.701289311803443,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "合成语音与原文完全一致，没有漏读、错读、添读、数字、专名和多音字的问题。",
|
||||
"自然度": "合成语音流畅度较好，但有一些不自然的停顿和重音。",
|
||||
"情感表达": "语调和语速基本符合中性情感，但强调稍显不足。",
|
||||
"音色一致性": "合成语音的音色与参考说话人有一定差异，但总体还算接近。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "fishaudio",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "fishaudio-s1-clone__long.mp3",
|
||||
"audio_sha256": "89542f05312bac767d5151f7552d0c75065cc7ecdf739dcc180b3c92a67b21c5",
|
||||
"audio_bytes": 206052,
|
||||
"duration": 12.87825,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 4.7366684137984585,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "合成语音与原文完全一致，没有漏读、错读、添读、数字、专名和多音字的问题。",
|
||||
"自然度": "合成语音流畅度较好，但有一些不自然的停顿和重音。",
|
||||
"情感表达": "合成语音的语调和语速基本符合中性情感，但缺乏一些细微的情感变化。",
|
||||
"音色一致性": "合成语音的音色与参考说话人有一些差异，但总体上还算相似。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "fishaudio",
|
||||
"ok": true,
|
||||
"error": null,
|
||||
"audio_path": "fishaudio-s1-clone__emotion.mp3",
|
||||
"audio_sha256": "81608431645b08be6bf7171dbb250f391b1ba7f5714d535c0e2ce323d7f6ca29",
|
||||
"audio_bytes": 104488,
|
||||
"duration": 6.5305,
|
||||
"hypothesis": null,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"speed": 4.746956588316362,
|
||||
"scores": {
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
"reasons": {
|
||||
"准确性": "没有漏读、错读、添读、数字、专名和多音字。",
|
||||
"自然度": "有轻微的机器感，但总体流畅度较好。",
|
||||
"情感表达": "语调和语速表现出兴奋，但强调稍显不足。",
|
||||
"音色一致性": "音色与参考说话人有明显差异。"
|
||||
},
|
||||
"judge_model": "mistral/voxtral-small-latest",
|
||||
"evidence_mode": "direct-audio-with-reference",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Mistral Voxtral API",
|
||||
"model": "voxtral-small-latest",
|
||||
"status": "ok",
|
||||
"attempts": 1
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"n_ok": 4,
|
||||
"n": 4,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 4,
|
||||
"音色一致性": 3
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"n_ok": 4,
|
||||
"n": 4,
|
||||
"cer": null,
|
||||
"asr_accuracy": null,
|
||||
"准确性": 5,
|
||||
"自然度": 4,
|
||||
"情感表达": 3.75,
|
||||
"音色一致性": 2.75
|
||||
}
|
||||
]
|
||||
}
|
||||
+208
@@ -0,0 +1,208 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "7-6",
|
||||
"generated_at_utc": "2026-07-30T04:17:40.702999+00:00",
|
||||
"command_scope": {
|
||||
"providers": [
|
||||
"fishaudio",
|
||||
"openai"
|
||||
],
|
||||
"configurations": [
|
||||
{
|
||||
"name": "tts1-alloy-1.0",
|
||||
"provider": "openai",
|
||||
"model": "tts-1",
|
||||
"voice": "alloy",
|
||||
"speed": 1.0
|
||||
},
|
||||
{
|
||||
"name": "fishaudio-s1-clone",
|
||||
"provider": "fishaudio",
|
||||
"model": "s1",
|
||||
"voice": "6df3c1e14c9440e9ac978556536bf116",
|
||||
"speed": 1.0
|
||||
}
|
||||
],
|
||||
"corpus_ids": [
|
||||
"num",
|
||||
"polyphone",
|
||||
"long",
|
||||
"emotion",
|
||||
"sad",
|
||||
"question"
|
||||
],
|
||||
"expected_records": 12,
|
||||
"direct_audio_judge": true,
|
||||
"optional_asr_enabled": false
|
||||
},
|
||||
"reference_audio": {
|
||||
"path": "../../../chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
},
|
||||
"rubric_dimensions": [
|
||||
"准确性",
|
||||
"自然度",
|
||||
"情感表达",
|
||||
"音色一致性"
|
||||
],
|
||||
"completion": {
|
||||
"successful_records": 0,
|
||||
"direct_audio_four_dimension_records": 0,
|
||||
"expected_records": 12,
|
||||
"all_cells_complete": false,
|
||||
"multi_provider": true,
|
||||
"manuscript_core_complete": false
|
||||
},
|
||||
"records": [
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "tts1-alloy-1.0__num.mp3",
|
||||
"audio_sha256": "954eca94164d5af7d4a849def397e16a192d22a86168fab86f76ccd2443fb3ed",
|
||||
"audio_bytes": 132960,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "tts1-alloy-1.0__polyphone.mp3",
|
||||
"audio_sha256": "fa6e6ca29a3d17d33167c04c32b132d6aee2cb406d2acbf7bbeb13529af98abb",
|
||||
"audio_bytes": 130560,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "tts1-alloy-1.0__long.mp3",
|
||||
"audio_sha256": "c04786022e111c49f635eb361cb4cae3b786334291af86f7f36e69fff7909d1a",
|
||||
"audio_bytes": 273600,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "tts1-alloy-1.0__emotion.mp3",
|
||||
"audio_sha256": "86215ce027003e1b0a721545b594ef0325745738a46bbf927a7843da3a15b514",
|
||||
"audio_bytes": 116160,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "sad",
|
||||
"challenge": "悲伤情感/低语速低语调",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "question",
|
||||
"challenge": "对话文体/疑问句升调",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__num.mp3",
|
||||
"audio_sha256": "6e9717e58f422db10b8e9538aa3812de456a1dde4bd9028fde928c2eaac42ad3",
|
||||
"audio_bytes": 101981,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__polyphone.mp3",
|
||||
"audio_sha256": "6bff6109ec7682f85680d8b60ea58dd48fbe5b3eeda59fb326d2288d250c85b5",
|
||||
"audio_bytes": 95293,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__long.mp3",
|
||||
"audio_sha256": "89542f05312bac767d5151f7552d0c75065cc7ecdf739dcc180b3c92a67b21c5",
|
||||
"audio_bytes": 206052,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__emotion.mp3",
|
||||
"audio_sha256": "81608431645b08be6bf7171dbb250f391b1ba7f5714d535c0e2ce323d7f6ca29",
|
||||
"audio_bytes": 104488,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "sad",
|
||||
"challenge": "悲伤情感/低语速低语调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__sad.mp3",
|
||||
"audio_sha256": "bf16b438885b651681197f15ad969a447b8b5cf781a298600580af5bea049f4c",
|
||||
"audio_bytes": 87352,
|
||||
"failed_stage": "multimodal_judge"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "question",
|
||||
"challenge": "对话文体/疑问句升调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "RuntimeError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__question.mp3",
|
||||
"audio_sha256": "1d5f16e92d66e1855c57b264e3ec1ddaf9e09aa9cfedca38fb5dfd623f829ad3",
|
||||
"audio_bytes": 76067,
|
||||
"failed_stage": "multimodal_judge"
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"n_ok": 0,
|
||||
"n": 6
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"n_ok": 0,
|
||||
"n": 6
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "7-6",
|
||||
"status": "incomplete",
|
||||
"generated_at_utc": "2026-07-30T04:32:15.010689+00:00",
|
||||
"run_dir": "chapter7/tts-quality-eval/validation/real_multimodal_20260730",
|
||||
"git_commit": "f119cf510f3746cd5d071d0bce0051617f02f163",
|
||||
"command": "python demo.py --providers openai,fishaudio --gemini --output validation/real_multimodal_20260730",
|
||||
"provider_receipt_count": 6,
|
||||
"status_reasons": [
|
||||
"Six content-hashed Fish Audio MP3 artifacts were available and re-evaluated, but both the Google Gemini API and exact OpenRouter two-input_audio route rejected their configured credentials; zero direct-audio four-dimension judge cells completed.",
|
||||
"The OpenAI account returned insufficient_quota before synthesis for all six requested cross-provider cells."
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/config.py",
|
||||
"bytes": 8642,
|
||||
"sha256": "891396771cd7beb32ab8e4b316f9a6f73683951e0327dcae6504d858ffd92ba1"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/demo.py",
|
||||
"bytes": 17225,
|
||||
"sha256": "c2da1f44b3ac3ec9e9b6648ee535e57c8fa757a14a45028f1e125ed9cb8f1938"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/pipeline.py",
|
||||
"bytes": 27511,
|
||||
"sha256": "db62ac8058561ae22b9a2dacd67aec42176f181a401e162009e1a3be7360ed54"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/real_multimodal_20260730/fishaudio-s1-clone__emotion.mp3",
|
||||
"bytes": 102817,
|
||||
"sha256": "4eca80508f1d6f7a5b50b87de0e329e700097fc01a092e38e47fb70d7dc3675a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/real_multimodal_20260730/fishaudio-s1-clone__long.mp3",
|
||||
"bytes": 203127,
|
||||
"sha256": "d0947fc60450d4668e75376eb56a714d80a99e3148ee76219c59133d7364eb12"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/real_multimodal_20260730/fishaudio-s1-clone__num.mp3",
|
||||
"bytes": 99055,
|
||||
"sha256": "15936f94727239ab9aadbc3814d70375fad93c89efd9b8562605d62790860898"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/real_multimodal_20260730/fishaudio-s1-clone__polyphone.mp3",
|
||||
"bytes": 97801,
|
||||
"sha256": "4f8c45b00455a9dbda4b1feebbda9f09de50ec9458d367ea701798ec05972a88"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/real_multimodal_20260730/fishaudio-s1-clone__question.mp3",
|
||||
"bytes": 75231,
|
||||
"sha256": "c48dfadf2f1eaa6b01a924e1de5ebf728cfbfad9df14234a70629e0e5890800a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/real_multimodal_20260730/fishaudio-s1-clone__sad.mp3",
|
||||
"bytes": 80665,
|
||||
"sha256": "295161d6fc7a169fc28ebd3fd07e2acaac6fca2f0cc24db6a51c3dc431310cf9"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/tts-quality-eval/validation/real_multimodal_20260730/results.json",
|
||||
"bytes": 14278,
|
||||
"sha256": "be1b2ea6fc585eceeacc7ba0d8f2552b41bfb8a245419063c69aa538775ef46d"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,280 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "7-6",
|
||||
"generated_at_utc": "2026-07-30T04:32:04.726815+00:00",
|
||||
"command_scope": {
|
||||
"providers": [
|
||||
"fishaudio",
|
||||
"openai"
|
||||
],
|
||||
"configurations": [
|
||||
{
|
||||
"name": "tts1-alloy-1.0",
|
||||
"provider": "openai",
|
||||
"model": "tts-1",
|
||||
"voice": "alloy",
|
||||
"speed": 1.0
|
||||
},
|
||||
{
|
||||
"name": "fishaudio-s1-clone",
|
||||
"provider": "fishaudio",
|
||||
"model": "s1",
|
||||
"voice": "6df3c1e14c9440e9ac978556536bf116",
|
||||
"speed": 1.0
|
||||
}
|
||||
],
|
||||
"corpus_ids": [
|
||||
"num",
|
||||
"polyphone",
|
||||
"long",
|
||||
"emotion",
|
||||
"sad",
|
||||
"question"
|
||||
],
|
||||
"expected_records": 12,
|
||||
"direct_audio_judge": true,
|
||||
"optional_asr_enabled": false
|
||||
},
|
||||
"reference_audio": {
|
||||
"path": "../../../../chapter9/controllable-tts/reference_audio/neutral_normal_formal.mp3",
|
||||
"sha256": "2cf4c64bb8222399ab54c63cc41049c572ea5f1b819ffdcbc496014daae004c0"
|
||||
},
|
||||
"rubric_dimensions": [
|
||||
"准确性",
|
||||
"自然度",
|
||||
"情感表达",
|
||||
"音色一致性"
|
||||
],
|
||||
"completion": {
|
||||
"successful_records": 0,
|
||||
"direct_audio_four_dimension_records": 0,
|
||||
"expected_records": 12,
|
||||
"all_cells_complete": false,
|
||||
"multi_provider": true,
|
||||
"manuscript_core_complete": false
|
||||
},
|
||||
"records": [
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "sad",
|
||||
"challenge": "悲伤情感/低语速低语调",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"sample": "question",
|
||||
"challenge": "对话文体/疑问句升调",
|
||||
"provider": "openai",
|
||||
"ok": false,
|
||||
"error": "RateLimitError: Error code: 429 - {'error': {'message': 'You have no credits remaining. Add credits to continue using the API at https://platform.openai.com/settings/organization/billing/.', 'type': 'insufficient_quota', 'param': None, 'code': 'credit_balance_exhausted'}}",
|
||||
"failed_stage": "synthesis"
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "num",
|
||||
"challenge": "数字/百分比/日期",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__num.mp3",
|
||||
"audio_sha256": "15936f94727239ab9aadbc3814d70375fad93c89efd9b8562605d62790860898",
|
||||
"audio_bytes": 99055,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "polyphone",
|
||||
"challenge": "多音字（行/长/重/还）",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__polyphone.mp3",
|
||||
"audio_sha256": "4f8c45b00455a9dbda4b1feebbda9f09de50ec9458d367ea701798ec05972a88",
|
||||
"audio_bytes": 97801,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "long",
|
||||
"challenge": "长句/新闻文体",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__long.mp3",
|
||||
"audio_sha256": "d0947fc60450d4668e75376eb56a714d80a99e3148ee76219c59133d7364eb12",
|
||||
"audio_bytes": 203127,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "emotion",
|
||||
"challenge": "专有名词 + 感叹情感",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__emotion.mp3",
|
||||
"audio_sha256": "4eca80508f1d6f7a5b50b87de0e329e700097fc01a092e38e47fb70d7dc3675a",
|
||||
"audio_bytes": 102817,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "sad",
|
||||
"challenge": "悲伤情感/低语速低语调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__sad.mp3",
|
||||
"audio_sha256": "295161d6fc7a169fc28ebd3fd07e2acaac6fca2f0cc24db6a51c3dc431310cf9",
|
||||
"audio_bytes": 80665,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"sample": "question",
|
||||
"challenge": "对话文体/疑问句升调",
|
||||
"provider": "fishaudio",
|
||||
"ok": false,
|
||||
"error": "JudgeRouteError: OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}",
|
||||
"audio_path": "fishaudio-s1-clone__question.mp3",
|
||||
"audio_sha256": "c48dfadf2f1eaa6b01a924e1de5ebf728cfbfad9df14234a70629e0e5890800a",
|
||||
"audio_bytes": 75231,
|
||||
"failed_stage": "multimodal_judge",
|
||||
"judge_provider_attempts": [
|
||||
{
|
||||
"provider": "Google Gemini API",
|
||||
"model": "gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "Gemini HTTP 400: {\n \"error\": {\n \"code\": 400,\n \"message\": \"API key not valid. Please pass a valid API key.\",\n \"status\": \"INVALID_ARGUMENT\",\n \"details\": [\n {\n \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n \"reason\": \"API_KEY_INVALID\",\n \"domain\": \"googleapis.com\",\n \"metadata\": {\n \"service\": \"generativelanguage.googleapis.com\"\n }\n },\n {\n \"@type\": \"type.googleapis.com/google.rpc.LocalizedMessage\",\n \"locale\": \"en-US\",\n \"message\": \"API key not valid. Please pass a valid API key.\"\n }\n ]\n }\n}\n"
|
||||
},
|
||||
{
|
||||
"provider": "OpenRouter audio route",
|
||||
"model": "google/gemini-3.5-flash",
|
||||
"status": "unavailable",
|
||||
"error": "OpenRouter audio HTTP 401: {\"error\":{\"message\":\"User not found.\",\"code\":401}}"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"summary": [
|
||||
{
|
||||
"config": "tts1-alloy-1.0",
|
||||
"n_ok": 0,
|
||||
"n": 6
|
||||
},
|
||||
{
|
||||
"config": "fishaudio-s1-clone",
|
||||
"n_ok": 0,
|
||||
"n": 6
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user