ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
.env
|
||||
audio/
|
||||
__pycache__/
|
||||
*.pyc
|
||||
validation/scenarios/
|
||||
validation/noise_only.json
|
||||
@@ -0,0 +1,82 @@
|
||||
# 实验 6-4：Qwen2-Audio 递增前缀模拟流式感知
|
||||
|
||||
运行器、验证器与 canonical 证据目录均使用实验 6-4 的统一标识 `exp6-4-*`。
|
||||
|
||||
本项目实际运行 `Qwen/Qwen2-Audio-7B-Instruct`：每收到一个新块，就把 `[0:t]` 的完整累积音频再次送入 Qwen2-Audio，输出当前 transcript 和声学事件。它不是 Whisper 替代实现，也不会把这种全量重编码称作真流式。
|
||||
|
||||
对照组是传统 600ms 端点 VAD + 开源 Whisper。三类场景均被测量：正常对话、含 900ms 中途停顿的长句、混入粉红背景噪声的对话。证据记录每个前缀的模型原始输出、单块延迟、最终 CER、事件 token，以及 VAD 分段点、Whisper 推理时长和 CER。
|
||||
|
||||
## 安装
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 6 core environment
|
||||
uv sync --locked --python 3.12 --extra ch6
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch9]"
|
||||
|
||||
cd chapter6/streaming-speech
|
||||
|
||||
# Install this experiment's local audio/model runtime dependencies.
|
||||
python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
NVIDIA 路径使用原始 BF16 权重：
|
||||
|
||||
```bash
|
||||
python demo.py --model Qwen/Qwen2-Audio-7B-Instruct --device cuda ...
|
||||
```
|
||||
|
||||
Apple Silicon 可运行同一 Qwen2-Audio 架构的 4-bit MLX 量化权重（LLM 量化，音频编码器和 projector 保持 BF16）：
|
||||
|
||||
```bash
|
||||
python prepare_scenarios.py audio/sentence.wav validation/scenarios
|
||||
|
||||
python demo.py \
|
||||
--model mlx-community/Qwen2-Audio-7B-Instruct-4bit --device mlx \
|
||||
--chunk-seconds 2 --whisper-model tiny \
|
||||
--audio validation/scenarios/normal.wav --scenario normal \
|
||||
--reference '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。' \
|
||||
--audio validation/scenarios/long_pause.wav --scenario pause \
|
||||
--reference '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。' \
|
||||
--audio validation/scenarios/background_noise.wav --scenario noise \
|
||||
--reference '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。'
|
||||
```
|
||||
|
||||
结果写入 `validation/latest.json`。`--skip-whisper` 只用于单独调试 Qwen，不能完成书中的对照验收。原始 BF16 模型约 16.8GB；MLX 量化权重约 6.6GB。
|
||||
|
||||
保持上述科学设计不变并生成完整验收 manifest：
|
||||
|
||||
```bash
|
||||
python run_official_experiment.py --run-id exp6-4-qwen2audio-whisper-provenance-YYYYMMDD-vN
|
||||
```
|
||||
|
||||
官方 runner 在运行前后核对源码 hash，并绑定三类测试音频、原始源音频、Whisper
|
||||
checkpoint、Qwen2-Audio snapshot 的每个文件（包括 6.56GB 权重）、13 个原始前缀
|
||||
输出、运行日志和独立 acceptance 文件。
|
||||
|
||||
## 已验证结果
|
||||
|
||||
当前 canonical 记录是 [`validation/runs/exp6-4-qwen2audio-whisper-provenance-20260730-v3/manifest.json`](validation/runs/exp6-4-qwen2audio-whisper-provenance-20260730-v3/manifest.json)。
|
||||
2026-07-30 在 Apple Silicon 上严格复跑 `mlx-community/Qwen2-Audio-7B-Instruct-4bit`；8/8
|
||||
执行与溯源门禁通过，但正文结果只复现 2/6。13 次前缀推理实测 8.4–11.3s，不能据此声称
|
||||
100–200ms；传统路径也未在三类输入上全部落入 800–1100ms。900ms 停顿被 VAD 分为两段，
|
||||
但 Qwen 漏报 `<|silence|>`；强噪声样本检出 `<|noise|>`，同时误报 `<|cough|>` 与
|
||||
`<|laughter|>`。这些负结果与所有原始响应都保留在验收记录中。2026-07-29 的 `latest.json`
|
||||
和 `latest_v2.json` 作为历史运行保留，不再承担 canonical 选择职责。
|
||||
|
||||
```bash
|
||||
pytest -q
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
This is actual Qwen2-Audio growing-prefix inference, not a Whisper substitute. Every `[0:t]` prefix is fully re-encoded and compared with a real 600ms-VAD + open-source Whisper pipeline on normal, long-pause, and noisy speech. CUDA uses the original model; Apple Silicon can use the published 4-bit MLX conversion of the same Qwen2-Audio architecture. The canonical v3 manifest binds raw responses, sources, audio, the Whisper checkpoint, and every Qwen snapshot file. Execution passed while the manuscript result bundle did not: only 2/6 claims reproduced, with 8.4–11.3s prefix inference and retained acoustic-event errors.
|
||||
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run Qwen2-Audio growing-prefix perception against VAD + Whisper."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import os
|
||||
import platform
|
||||
import subprocess
|
||||
import unicodedata
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
from opencc import OpenCC
|
||||
from qwen2_streaming import Qwen2AudioRecognizer, growing_prefix, serialize
|
||||
from whisper_baseline import LocalWhisper, run_whisper_baseline, serialize as serialize_baseline
|
||||
|
||||
HERE = Path(__file__).parent
|
||||
|
||||
|
||||
T2S = OpenCC("t2s")
|
||||
|
||||
|
||||
def normalize_for_cer(text: str) -> str:
|
||||
"""Normalize width, case, Chinese script, whitespace, and punctuation."""
|
||||
text = T2S.convert(unicodedata.normalize("NFKC", text)).casefold()
|
||||
return "".join(char for char in text if unicodedata.category(char)[0] not in {"P", "S", "Z"})
|
||||
|
||||
|
||||
def cer(reference: str, hypothesis: str) -> float:
|
||||
reference, hypothesis = normalize_for_cer(reference), normalize_for_cer(hypothesis)
|
||||
if not reference:
|
||||
return 0.0 if not hypothesis else 1.0
|
||||
row = list(range(len(hypothesis) + 1))
|
||||
for i, left in enumerate(reference, 1):
|
||||
new = [i]
|
||||
for j, right in enumerate(hypothesis, 1):
|
||||
new.append(min(new[-1] + 1, row[j] + 1, row[j - 1] + (left != right)))
|
||||
row = new
|
||||
return row[-1] / len(reference)
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
def command_output(*args: str) -> str | None:
|
||||
try:
|
||||
return subprocess.check_output(args, text=True).strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def model_provenance(model_id: str) -> dict:
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
snapshot = Path(snapshot_download(model_id, local_files_only=True))
|
||||
files = []
|
||||
for path in sorted(item for item in snapshot.rglob("*") if item.is_file()):
|
||||
size = path.stat().st_size
|
||||
files.append({
|
||||
"path": str(path.relative_to(snapshot)),
|
||||
"size_bytes": size,
|
||||
"sha256": sha256(path) if size <= 10 * 1024 * 1024 else None,
|
||||
})
|
||||
return {
|
||||
"repository": model_id,
|
||||
"snapshot_revision": snapshot.name,
|
||||
"snapshot_path": str(snapshot),
|
||||
"total_bytes": sum(item["size_bytes"] for item in files),
|
||||
"files": files,
|
||||
"large_weight_hash_note": "Snapshot revision pins large files; files over 10 MiB are inventoried by path and size without rehashing.",
|
||||
}
|
||||
|
||||
|
||||
def host_provenance() -> dict:
|
||||
whisper_cache = Path.home() / ".cache" / "whisper" / "tiny.pt"
|
||||
return {
|
||||
"platform": platform.platform(),
|
||||
"machine": platform.machine(),
|
||||
"cpu": command_output("sysctl", "-n", "machdep.cpu.brand_string") or platform.processor(),
|
||||
"memory_bytes": int(command_output("sysctl", "-n", "hw.memsize") or 0),
|
||||
"python": platform.python_version(),
|
||||
"packages": {
|
||||
name: importlib.metadata.version(name)
|
||||
for name in ("mlx-audio", "openai-whisper", "librosa", "opencc-python-reimplemented")
|
||||
},
|
||||
"whisper_baseline": {
|
||||
"model": "tiny",
|
||||
"path": str(whisper_cache),
|
||||
"sha256": sha256(whisper_cache),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
EXPECTED_EVENTS = {
|
||||
"normal": [],
|
||||
"pause": ["<|silence|>"],
|
||||
"noise": ["<|noise|>"],
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description="Experiment 6-4: actual Qwen2-Audio growing-prefix inference")
|
||||
parser.add_argument("--audio", action="append", required=True, help="Audio path; repeat for normal/pause/noise cases")
|
||||
parser.add_argument("--reference", action="append", required=True, help="Reference transcript matching each --audio")
|
||||
parser.add_argument("--scenario", action="append", choices=["normal", "pause", "noise"], required=True)
|
||||
parser.add_argument("--chunk-seconds", type=float, default=1.0)
|
||||
parser.add_argument("--model", default="Qwen/Qwen2-Audio-7B-Instruct")
|
||||
parser.add_argument("--device", default="auto", choices=["auto", "cuda", "mps", "cpu", "mlx"])
|
||||
parser.add_argument("--skip-whisper", action="store_true")
|
||||
parser.add_argument("--whisper-model", default="small")
|
||||
parser.add_argument("--evidence", default=str(HERE / "validation" / "latest.json"))
|
||||
args = parser.parse_args()
|
||||
if not (len(args.audio) == len(args.reference) == len(args.scenario)):
|
||||
parser.error("--audio, --reference and --scenario counts must match")
|
||||
load_dotenv(HERE / ".env")
|
||||
recognizer = Qwen2AudioRecognizer(args.model, args.device)
|
||||
whisper = None if args.skip_whisper else LocalWhisper(args.whisper_model)
|
||||
cases = []
|
||||
for path, reference, scenario in zip(args.audio, args.reference, args.scenario):
|
||||
print(f"\n[{scenario}] {path}")
|
||||
prefixes = growing_prefix(
|
||||
recognizer, path, args.chunk_seconds,
|
||||
on_result=lambda r: print(f" {r.prefix_seconds:5.2f}s | {r.inference_seconds:6.2f}s | {r.transcript} {r.acoustic_events}"),
|
||||
)
|
||||
baseline = run_whisper_baseline(path, whisper) if whisper else None
|
||||
case = {
|
||||
"scenario": scenario,
|
||||
"audio": str(Path(path)),
|
||||
"reference": reference,
|
||||
"media": {
|
||||
"sha256": sha256(Path(path)),
|
||||
"expected_acoustic_events": EXPECTED_EVENTS[scenario],
|
||||
},
|
||||
"qwen2_audio": serialize(prefixes),
|
||||
"qwen2_final_cer": cer(reference, prefixes[-1].transcript),
|
||||
"whisper_vad": serialize_baseline(baseline) if baseline else None,
|
||||
"whisper_final_cer": cer(reference, baseline.transcript) if baseline else None,
|
||||
}
|
||||
cases.append(case)
|
||||
by_scenario = {case["scenario"]: case for case in cases}
|
||||
for case in cases:
|
||||
actual = set(case["qwen2_audio"][-1]["acoustic_events"])
|
||||
expected = set(case["media"]["expected_acoustic_events"])
|
||||
case["qwen2_event_evaluation"] = {
|
||||
"true_positive": sorted(actual & expected),
|
||||
"false_positive": sorted(actual - expected),
|
||||
"false_negative": sorted(expected - actual),
|
||||
"exact_match": actual == expected,
|
||||
}
|
||||
qwen_latencies = [prefix["inference_seconds"] for case in cases for prefix in case["qwen2_audio"]]
|
||||
normal_start = by_scenario["normal"]["whisper_vad"]["first_speech_start_seconds"] if by_scenario.get("normal", {}).get("whisper_vad") else None
|
||||
noise_start = by_scenario["noise"]["whisper_vad"]["first_speech_start_seconds"] if by_scenario.get("noise", {}).get("whisper_vad") else None
|
||||
pause_case = by_scenario.get("pause", {})
|
||||
noise_case = by_scenario.get("noise", {})
|
||||
result_claims = {
|
||||
"qwen_incremental_latency_100_to_200ms": all(0.1 <= value <= 0.2 for value in qwen_latencies),
|
||||
"traditional_post_endpoint_latency_800_to_1100ms": all(
|
||||
0.8 <= case["whisper_vad"]["post_endpoint_response_seconds"] <= 1.1
|
||||
for case in cases if case.get("whisper_vad")
|
||||
),
|
||||
"pause_split_into_two_segments": pause_case.get("whisper_vad", {}).get("segment_count") == 2,
|
||||
"pause_specific_two_to_zero_error": "零点" in normalize_for_cer(pause_case.get("whisper_vad", {}).get("transcript", "")),
|
||||
"noise_token_detected": "<|noise|>" in noise_case.get("qwen2_audio", [{}])[-1].get("acoustic_events", []),
|
||||
"noise_caused_earlier_vad_start": (
|
||||
normal_start is not None and noise_start is not None and noise_start + 0.1 < normal_start
|
||||
),
|
||||
}
|
||||
evidence = {
|
||||
"schema_version": 2,
|
||||
"experiment": "6-4",
|
||||
"timestamp_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"model": args.model,
|
||||
"device": recognizer.device,
|
||||
"method": "growing-prefix full re-encoding (not true streaming)",
|
||||
"parameters": {
|
||||
"chunk_seconds": args.chunk_seconds,
|
||||
"whisper_model": args.whisper_model,
|
||||
"vad_silence_ms": 600,
|
||||
"cer_normalization": "NFKC + Traditional-to-Simplified Chinese + casefold + remove punctuation/symbols/separators",
|
||||
},
|
||||
"provenance": {
|
||||
"host": host_provenance(),
|
||||
"qwen2_audio": model_provenance(args.model),
|
||||
},
|
||||
"cases": cases,
|
||||
"cost": {"paid_external_requests": 0, "total_usd": 0, "note": "Both models ran locally."},
|
||||
"acceptance": {
|
||||
"execution_gates": {
|
||||
"real_qwen2_audio": bool(cases) and all(case["qwen2_audio"] for case in cases),
|
||||
"growing_prefix_full_reencoding": True,
|
||||
"real_600ms_vad_whisper_baseline": all(case.get("whisper_vad") for case in cases),
|
||||
"normal_pause_noise_scenarios": set(by_scenario) == {"normal", "pause", "noise"},
|
||||
"corrected_vad_latency_accounting": all(
|
||||
abs(case["whisper_vad"]["post_speech_vad_delay_seconds"] - 0.6) <= 0.021
|
||||
for case in cases if case.get("whisper_vad")
|
||||
),
|
||||
"normalized_cer": True,
|
||||
"provenance_complete": True,
|
||||
},
|
||||
"execution_passed": True,
|
||||
"manuscript_result_claims": result_claims,
|
||||
"manuscript_results_reproduced": all(result_claims.values()),
|
||||
},
|
||||
}
|
||||
evidence["acceptance"]["execution_passed"] = all(evidence["acceptance"]["execution_gates"].values())
|
||||
output = Path(args.evidence)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
output.write_text(json.dumps(evidence, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(f"\nSanitized evidence: {output}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,5 @@
|
||||
# Optional Hugging Face token: raises download rate limits for Qwen2-Audio weights.
|
||||
HF_TOKEN=your_huggingface_token
|
||||
|
||||
# The runtime itself uses local Qwen2-Audio and local open-source Whisper;
|
||||
# no OpenAI/DashScope API key is required.
|
||||
@@ -0,0 +1,385 @@
|
||||
"""Duplex Interruption Manager for Real-Time Streaming Speech Systems.
|
||||
|
||||
Monitors real-time Voice Activity Detection (VAD) energy signals during active TTS audio playback,
|
||||
enabling instant audio stream cancellation upon user barge-in, dialogue context truncation,
|
||||
and re-planning trigger generation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, Dict, List, Optional, Union
|
||||
import numpy as np
|
||||
|
||||
|
||||
@dataclass
|
||||
class InterruptionEvent:
|
||||
"""Event payload generated when a user barge-in interrupts active TTS playback."""
|
||||
timestamp: float
|
||||
barge_in_id: int
|
||||
energy_level: float
|
||||
vad_threshold: float
|
||||
truncated_turns: int
|
||||
reason: str
|
||||
replan_triggered: bool
|
||||
cancelled_audio_bytes: int = 0
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
"""Convert interruption event to dictionary representation."""
|
||||
return {
|
||||
"timestamp": self.timestamp,
|
||||
"barge_in_id": self.barge_in_id,
|
||||
"energy_level": self.energy_level,
|
||||
"vad_threshold": self.vad_threshold,
|
||||
"truncated_turns": self.truncated_turns,
|
||||
"reason": self.reason,
|
||||
"replan_triggered": self.replan_triggered,
|
||||
"cancelled_audio_bytes": self.cancelled_audio_bytes,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class DialogueTurn:
|
||||
"""Represents a turn in the dialogue context."""
|
||||
role: str
|
||||
content: str
|
||||
status: str = "completed" # "completed", "interrupted", "pending"
|
||||
metadata: Dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
|
||||
class DuplexInterruptionManager:
|
||||
"""Manages real-time interruption (barge-in) detection and handling for duplex speech systems.
|
||||
|
||||
Monitors user audio input streams via VAD energy analysis while TTS audio is actively playing.
|
||||
If speech is detected during active TTS output, it instantly cancels playback, truncates
|
||||
the dialogue context to match what was actually delivered, and emits a re-planning trigger.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
vad_threshold: float = 0.02,
|
||||
consecutive_frames_required: int = 1,
|
||||
on_barge_in: Optional[Callable[[InterruptionEvent], None]] = None,
|
||||
on_replan: Optional[Callable[[Dict[str, Any]], None]] = None,
|
||||
) -> None:
|
||||
"""Initialize the DuplexInterruptionManager.
|
||||
|
||||
Args:
|
||||
vad_threshold: RMS energy threshold above which audio frame is treated as voice active.
|
||||
consecutive_frames_required: Number of consecutive active frames required to trigger barge-in.
|
||||
on_barge_in: Optional callback invoked when a barge-in event occurs.
|
||||
on_replan: Optional callback invoked when re-planning is triggered.
|
||||
"""
|
||||
self.vad_threshold = float(vad_threshold)
|
||||
self.consecutive_frames_required = max(1, int(consecutive_frames_required))
|
||||
self.on_barge_in = on_barge_in
|
||||
self.on_replan = on_replan
|
||||
|
||||
# Playback & state management
|
||||
self.is_playing: bool = False
|
||||
self._consecutive_active_frames: int = 0
|
||||
self.barge_in_count: int = 0
|
||||
self.dialogue_context: List[DialogueTurn] = []
|
||||
self.pending_audio_stream: List[bytes] = []
|
||||
self.last_interruption_event: Optional[InterruptionEvent] = None
|
||||
self.replan_triggers: List[Dict[str, Any]] = []
|
||||
|
||||
def start_playback(self, initial_audio_stream: Optional[List[bytes]] = None) -> None:
|
||||
"""Mark TTS playback as active and optionally register pending audio stream chunks."""
|
||||
self.is_playing = True
|
||||
self._consecutive_active_frames = 0
|
||||
if initial_audio_stream is not None:
|
||||
self.pending_audio_stream = list(initial_audio_stream)
|
||||
|
||||
def stop_playback(self) -> None:
|
||||
"""Mark TTS playback as inactive and clear pending audio stream."""
|
||||
self.is_playing = False
|
||||
self._consecutive_active_frames = 0
|
||||
self.pending_audio_stream.clear()
|
||||
|
||||
def calculate_energy(
|
||||
self,
|
||||
audio_data: Union[np.ndarray, bytes, bytearray, memoryview, List[float], List[int]],
|
||||
sample_format: Optional[str] = None,
|
||||
) -> float:
|
||||
"""Calculate Root Mean Square (RMS) energy level of an audio chunk.
|
||||
|
||||
Supports numpy arrays, raw bytes/bytearray/memoryview (16-bit PCM, uint8, or float32), or float/int lists.
|
||||
sample_format can be 'int16', 'uint8', 'float32', or None for auto detection.
|
||||
"""
|
||||
if audio_data is None:
|
||||
return 0.0
|
||||
|
||||
fmt = (sample_format or "").lower()
|
||||
if isinstance(audio_data, (bytes, bytearray, memoryview)):
|
||||
if len(audio_data) == 0:
|
||||
return 0.0
|
||||
if fmt in ("float32", "float"):
|
||||
arr = np.frombuffer(audio_data, dtype=np.float32)
|
||||
elif fmt in ("uint8", "u8"):
|
||||
arr = (np.frombuffer(audio_data, dtype=np.uint8).astype(np.float32) - 128.0) / 128.0
|
||||
elif fmt in ("int8", "i8"):
|
||||
arr = np.frombuffer(audio_data, dtype=np.int8).astype(np.float32) / 128.0
|
||||
elif fmt in ("int16", "i16"):
|
||||
arr = np.frombuffer(audio_data, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
else:
|
||||
if len(audio_data) % 2 != 0:
|
||||
arr = (np.frombuffer(audio_data, dtype=np.uint8).astype(np.float32) - 128.0) / 128.0
|
||||
else:
|
||||
arr = np.frombuffer(audio_data, dtype=np.int16).astype(np.float32) / 32768.0
|
||||
elif isinstance(audio_data, (list, tuple)):
|
||||
if len(audio_data) == 0:
|
||||
return 0.0
|
||||
raw_arr = np.array(audio_data)
|
||||
if np.issubdtype(raw_arr.dtype, np.integer):
|
||||
if raw_arr.dtype == np.uint8 or fmt in ("uint8", "u8"):
|
||||
arr = (raw_arr.astype(np.float32) - 128.0) / 128.0
|
||||
elif raw_arr.dtype == np.int8 or fmt in ("int8", "i8"):
|
||||
arr = raw_arr.astype(np.float32) / 128.0
|
||||
elif raw_arr.dtype == np.int16 or fmt in ("int16", "i16"):
|
||||
arr = raw_arr.astype(np.float32) / 32768.0
|
||||
else:
|
||||
max_abs = float(np.max(np.abs(raw_arr))) if raw_arr.size > 0 else 0.0
|
||||
if max_abs <= 128.0:
|
||||
scale = 128.0
|
||||
elif max_abs <= 32768.0:
|
||||
scale = 32768.0
|
||||
elif max_abs <= 2147483648.0:
|
||||
scale = 2147483648.0
|
||||
else:
|
||||
scale = float(np.iinfo(raw_arr.dtype).max)
|
||||
arr = raw_arr.astype(np.float32) / scale
|
||||
else:
|
||||
arr = raw_arr.astype(np.float32)
|
||||
# If values are in integer PCM range (>1.0), normalize to [-1, 1].
|
||||
# Use a fixed int16 scale rather than per-chunk max to preserve
|
||||
# relative volume across chunks.
|
||||
max_abs = float(np.max(np.abs(arr))) if arr.size > 0 else 0.0
|
||||
if max_abs > 1.0:
|
||||
if max_abs <= 128.0:
|
||||
arr = arr / 128.0
|
||||
elif max_abs <= 32768.0:
|
||||
arr = arr / 32768.0
|
||||
else:
|
||||
arr = arr / 2147483648.0
|
||||
elif isinstance(audio_data, np.ndarray):
|
||||
if audio_data.size == 0:
|
||||
return 0.0
|
||||
if np.issubdtype(audio_data.dtype, np.integer):
|
||||
if audio_data.dtype == np.uint8 or fmt in ("uint8", "u8"):
|
||||
arr = (audio_data.astype(np.float32) - 128.0) / 128.0
|
||||
elif audio_data.dtype == np.int8 or fmt in ("int8", "i8"):
|
||||
arr = audio_data.astype(np.float32) / 128.0
|
||||
elif audio_data.dtype == np.int16 or fmt in ("int16", "i16"):
|
||||
arr = audio_data.astype(np.float32) / 32768.0
|
||||
else:
|
||||
max_abs = float(np.max(np.abs(audio_data))) if audio_data.size > 0 else 0.0
|
||||
if max_abs <= 128.0:
|
||||
scale = 128.0
|
||||
elif max_abs <= 32768.0:
|
||||
scale = 32768.0
|
||||
elif max_abs <= 2147483648.0:
|
||||
scale = 2147483648.0
|
||||
else:
|
||||
scale = float(np.iinfo(audio_data.dtype).max)
|
||||
arr = audio_data.astype(np.float32) / scale
|
||||
else:
|
||||
arr = audio_data.astype(np.float32)
|
||||
max_abs = float(np.max(np.abs(arr))) if arr.size > 0 else 0.0
|
||||
if max_abs > 1.0:
|
||||
if max_abs <= 128.0:
|
||||
arr = arr / 128.0
|
||||
elif max_abs <= 32768.0:
|
||||
arr = arr / 32768.0
|
||||
else:
|
||||
arr = arr / 2147483648.0
|
||||
else:
|
||||
return 0.0
|
||||
|
||||
if arr.size == 0:
|
||||
return 0.0
|
||||
|
||||
rms = float(np.sqrt(np.mean(arr ** 2) + 1e-12))
|
||||
return rms
|
||||
|
||||
def is_voice_active(
|
||||
self,
|
||||
audio_data: Union[np.ndarray, bytes, bytearray, memoryview, List[float], List[int]],
|
||||
sample_format: Optional[str] = None,
|
||||
) -> bool:
|
||||
"""Check if incoming audio chunk exceeds the VAD energy threshold."""
|
||||
energy = self.calculate_energy(audio_data, sample_format=sample_format)
|
||||
return energy >= self.vad_threshold
|
||||
|
||||
def process_audio_chunk(
|
||||
self,
|
||||
audio_data: Union[np.ndarray, bytes, bytearray, memoryview, List[float], List[int]],
|
||||
sample_rate: int = 16000,
|
||||
sample_format: Optional[str] = None,
|
||||
) -> Dict[str, Any]:
|
||||
"""Process real-time incoming audio chunk from user.
|
||||
|
||||
Monitors VAD energy signal during active TTS audio playback.
|
||||
If VAD energy surpasses threshold while playing, triggers barge-in.
|
||||
|
||||
Returns:
|
||||
Dict containing VAD analysis results, playback status, and interruption info.
|
||||
"""
|
||||
energy = self.calculate_energy(audio_data, sample_format=sample_format)
|
||||
is_speech = energy >= self.vad_threshold
|
||||
|
||||
if not self.is_playing:
|
||||
self._consecutive_active_frames = 0
|
||||
return {
|
||||
"barge_in": False,
|
||||
"is_speech": is_speech,
|
||||
"consecutive_frames": 0,
|
||||
"energy": energy,
|
||||
"vad_threshold": self.vad_threshold,
|
||||
"is_playing": False,
|
||||
"message": "TTS playback inactive; audio processed normally.",
|
||||
}
|
||||
|
||||
if is_speech:
|
||||
self._consecutive_active_frames += 1
|
||||
if self._consecutive_active_frames >= self.consecutive_frames_required:
|
||||
current_consecutive = self._consecutive_active_frames
|
||||
# Trigger instant barge-in
|
||||
barge_in_result = self.handle_barge_in(
|
||||
reason="user_barge_in_detected",
|
||||
energy_level=energy,
|
||||
)
|
||||
barge_in_result["energy"] = energy
|
||||
barge_in_result["is_speech"] = True
|
||||
barge_in_result["consecutive_frames"] = current_consecutive
|
||||
barge_in_result["vad_threshold"] = self.vad_threshold
|
||||
barge_in_result["is_playing"] = False
|
||||
return barge_in_result
|
||||
else:
|
||||
self._consecutive_active_frames = 0
|
||||
|
||||
return {
|
||||
"barge_in": False,
|
||||
"is_speech": is_speech,
|
||||
"consecutive_frames": self._consecutive_active_frames,
|
||||
"energy": energy,
|
||||
"vad_threshold": self.vad_threshold,
|
||||
"is_playing": True,
|
||||
"message": (
|
||||
"Voice activity detected; awaiting consecutive frames."
|
||||
if is_speech
|
||||
else "No voice activity detected during TTS playback."
|
||||
),
|
||||
}
|
||||
|
||||
def handle_barge_in(
|
||||
self,
|
||||
truncated_length: Optional[int] = None,
|
||||
reason: str = "user_barge_in",
|
||||
energy_level: float = 0.0,
|
||||
) -> Dict[str, Any]:
|
||||
"""Handle instant audio stream cancellation, dialogue context truncation, and re-planning.
|
||||
|
||||
Entrypoint called upon barge-in detection or manual invocation.
|
||||
|
||||
Returns:
|
||||
Dict containing complete interruption event outcome details.
|
||||
"""
|
||||
# 1. Instant audio stream cancellation
|
||||
was_playing = self.is_playing
|
||||
cancelled_bytes = sum(len(b) for b in self.pending_audio_stream) if was_playing else 0
|
||||
|
||||
if not was_playing:
|
||||
return {
|
||||
"status": "ignored",
|
||||
"barge_in": False,
|
||||
"playback_cancelled": False,
|
||||
"cancelled_audio_bytes": 0,
|
||||
"context_truncated": False,
|
||||
"truncated_turns_count": 0,
|
||||
"replan_triggered": False,
|
||||
"replan_payload": None,
|
||||
"barge_in_count": self.barge_in_count,
|
||||
"event": None,
|
||||
}
|
||||
|
||||
self.stop_playback()
|
||||
self.barge_in_count += 1
|
||||
truncated_turns_count = 0
|
||||
if self.dialogue_context:
|
||||
last_turn = self.dialogue_context[-1]
|
||||
if last_turn.role in ("assistant", "system", "agent") and last_turn.status != "interrupted":
|
||||
last_turn.status = "interrupted"
|
||||
truncated_turns_count += 1
|
||||
if truncated_length is not None and truncated_length < len(last_turn.content):
|
||||
last_turn.content = last_turn.content[:truncated_length] + " [interrupted...]"
|
||||
else:
|
||||
last_turn.content = last_turn.content + " [interrupted]"
|
||||
|
||||
# 3. Re-planning trigger generation
|
||||
replan_payload = {
|
||||
"trigger": "barge_in",
|
||||
"barge_in_id": self.barge_in_count,
|
||||
"timestamp": time.time(),
|
||||
"reason": reason,
|
||||
"dialogue_state": [
|
||||
{"role": t.role, "content": t.content, "status": t.status}
|
||||
for t in self.dialogue_context
|
||||
],
|
||||
}
|
||||
self.replan_triggers.append(replan_payload)
|
||||
|
||||
# Build interruption event
|
||||
event = InterruptionEvent(
|
||||
timestamp=time.time(),
|
||||
barge_in_id=self.barge_in_count,
|
||||
energy_level=energy_level,
|
||||
vad_threshold=self.vad_threshold,
|
||||
truncated_turns=truncated_turns_count,
|
||||
reason=reason,
|
||||
replan_triggered=True,
|
||||
cancelled_audio_bytes=cancelled_bytes,
|
||||
)
|
||||
self.last_interruption_event = event
|
||||
|
||||
# Callbacks
|
||||
if self.on_barge_in is not None:
|
||||
self.on_barge_in(event)
|
||||
if self.on_replan is not None:
|
||||
self.on_replan(replan_payload)
|
||||
|
||||
return {
|
||||
"status": "interrupted",
|
||||
"barge_in": True,
|
||||
"playback_cancelled": was_playing,
|
||||
"cancelled_audio_bytes": cancelled_bytes,
|
||||
"context_truncated": truncated_turns_count > 0,
|
||||
"truncated_turns_count": truncated_turns_count,
|
||||
"replan_triggered": True,
|
||||
"replan_payload": replan_payload,
|
||||
"barge_in_count": self.barge_in_count,
|
||||
"event": event.to_dict(),
|
||||
}
|
||||
|
||||
def add_dialogue_turn(self, role: str, content: str, status: str = "completed") -> DialogueTurn:
|
||||
"""Add a dialogue turn to the current context."""
|
||||
turn = DialogueTurn(role=role, content=content, status=status)
|
||||
self.dialogue_context.append(turn)
|
||||
return turn
|
||||
|
||||
def get_dialogue_context(self) -> List[Dict[str, Any]]:
|
||||
"""Return formatted dialogue context."""
|
||||
return [
|
||||
{"role": t.role, "content": t.content, "status": t.status, "metadata": t.metadata}
|
||||
for t in self.dialogue_context
|
||||
]
|
||||
|
||||
def reset(self) -> None:
|
||||
"""Reset internal state, counters, and buffers."""
|
||||
self.stop_playback()
|
||||
self.barge_in_count = 0
|
||||
self.dialogue_context.clear()
|
||||
self.replan_triggers.clear()
|
||||
self.last_interruption_event = None
|
||||
self._consecutive_active_frames = 0
|
||||
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Create normal, long-pause and background-noise variants from one WAV."""
|
||||
|
||||
import argparse
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("source", type=Path)
|
||||
parser.add_argument("output_dir", type=Path)
|
||||
args = parser.parse_args()
|
||||
args.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
duration = float(subprocess.check_output([
|
||||
"ffprobe", "-v", "error", "-show_entries", "format=duration", "-of", "csv=p=0", str(args.source)
|
||||
], text=True).strip())
|
||||
split = duration * 0.55
|
||||
normal = args.output_dir / "normal.wav"
|
||||
pause = args.output_dir / "long_pause.wav"
|
||||
noise = args.output_dir / "background_noise.wav"
|
||||
|
||||
subprocess.run(["ffmpeg", "-y", "-loglevel", "error", "-i", str(args.source), "-ar", "16000", "-ac", "1", str(normal)], check=True)
|
||||
filter_pause = (
|
||||
f"[0:a]atrim=0:{split},asetpts=PTS-STARTPTS[a];"
|
||||
f"[0:a]atrim={split},asetpts=PTS-STARTPTS[b];"
|
||||
"anullsrc=r=16000:cl=mono:d=0.9[s];[a][s][b]concat=n=3:v=0:a=1[out]"
|
||||
)
|
||||
subprocess.run([
|
||||
"ffmpeg", "-y", "-loglevel", "error", "-i", str(normal), "-filter_complex", filter_pause,
|
||||
"-map", "[out]", str(pause)
|
||||
], check=True)
|
||||
subprocess.run([
|
||||
"ffmpeg", "-y", "-loglevel", "error", "-i", str(normal),
|
||||
"-f", "lavfi", "-i", f"anoisesrc=color=pink:amplitude=0.12:d={duration}",
|
||||
"-filter_complex", "[0:a][1:a]amix=inputs=2:duration=first:weights='1 1'[out]",
|
||||
"-map", "[out]", "-ar", "16000", "-ac", "1", str(noise)
|
||||
], check=True)
|
||||
print(normal)
|
||||
print(pause)
|
||||
print(noise)
|
||||
@@ -0,0 +1,187 @@
|
||||
"""Qwen2-Audio growing-prefix inference for Experiment 6-4."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import ast
|
||||
import re
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Callable
|
||||
|
||||
import librosa
|
||||
|
||||
|
||||
EVENT_ALIASES = {
|
||||
"noise": "<|noise|>",
|
||||
"background noise": "<|noise|>",
|
||||
"laughter": "<|laughter|>",
|
||||
"laugh": "<|laughter|>",
|
||||
"silence": "<|silence|>",
|
||||
"pause": "<|silence|>",
|
||||
"cough": "<|cough|>",
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class PrefixResult:
|
||||
prefix_seconds: float
|
||||
inference_seconds: float
|
||||
transcript: str
|
||||
acoustic_events: list[str]
|
||||
raw_response: str
|
||||
|
||||
|
||||
def parse_response(raw: str) -> tuple[str, list[str]]:
|
||||
"""Parse the model's requested JSON while retaining raw output for audit."""
|
||||
cleaned = re.sub(r"^```(?:json)?|```$", "", raw.strip()).strip()
|
||||
try:
|
||||
payload = json.loads(cleaned)
|
||||
except json.JSONDecodeError:
|
||||
match = re.search(r"\{.*\}", cleaned, re.DOTALL)
|
||||
payload = None
|
||||
if match:
|
||||
try:
|
||||
payload = json.loads(match.group(0))
|
||||
except json.JSONDecodeError:
|
||||
try:
|
||||
payload = ast.literal_eval(match.group(0))
|
||||
except (ValueError, SyntaxError):
|
||||
pass
|
||||
if not isinstance(payload, dict):
|
||||
payload = {"transcript": cleaned, "acoustic_events": []}
|
||||
transcript = str(payload.get("transcript") or "")
|
||||
events: list[str] = []
|
||||
raw_events = payload.get("acoustic_events")
|
||||
if isinstance(raw_events, str):
|
||||
event_list = [raw_events]
|
||||
elif isinstance(raw_events, (list, tuple, set)):
|
||||
event_list = list(raw_events)
|
||||
else:
|
||||
event_list = []
|
||||
for event in event_list:
|
||||
if event is None:
|
||||
continue
|
||||
text = str(event).strip()
|
||||
if not text:
|
||||
continue
|
||||
normalized = EVENT_ALIASES.get(text.lower(), text)
|
||||
if normalized and normalized not in events:
|
||||
events.append(normalized)
|
||||
# Qwen may place an event token next to the transcription rather than in JSON.
|
||||
for token in re.findall(r"<\|[^|]+\|>", raw):
|
||||
if token not in events:
|
||||
events.append(token)
|
||||
return transcript, events
|
||||
|
||||
|
||||
class Qwen2AudioRecognizer:
|
||||
"""Actual ``Qwen/Qwen2-Audio-7B-Instruct`` inference backend."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_id: str = "Qwen/Qwen2-Audio-7B-Instruct",
|
||||
device: str = "auto",
|
||||
max_new_tokens: int = 128,
|
||||
) -> None:
|
||||
self._mlx = device == "mlx" or model_id.startswith("mlx-community/")
|
||||
if self._mlx:
|
||||
from mlx_audio.stt.utils import load_model
|
||||
|
||||
self.device = "mlx"
|
||||
self.model_id = model_id
|
||||
self.max_new_tokens = max_new_tokens
|
||||
self.model = load_model(model_id)
|
||||
self.processor = None
|
||||
return
|
||||
import torch
|
||||
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration
|
||||
|
||||
if device == "auto":
|
||||
device = "cuda" if torch.cuda.is_available() else ("mps" if torch.backends.mps.is_available() else "cpu")
|
||||
self.device = device
|
||||
self.model_id = model_id
|
||||
self.max_new_tokens = max_new_tokens
|
||||
dtype = torch.float16 if device in ("cuda", "mps") else torch.float32
|
||||
self.processor = AutoProcessor.from_pretrained(model_id)
|
||||
self.model = Qwen2AudioForConditionalGeneration.from_pretrained(
|
||||
model_id,
|
||||
torch_dtype=dtype,
|
||||
low_cpu_mem_usage=True,
|
||||
).to(device)
|
||||
self.model.eval()
|
||||
|
||||
def transcribe_array(self, audio: Any, sample_rate: int) -> tuple[str, list[str], str]:
|
||||
instruction = (
|
||||
"Transcribe all speech heard so far. Also detect non-verbal acoustic events. "
|
||||
"Return JSON only with keys transcript (string) and acoustic_events (array). "
|
||||
"The acoustic_events array must contain only events actually audible in this clip; "
|
||||
"return an empty array when none are audible. Valid event names are noise, laughter, silence, cough. "
|
||||
"Never copy the list of valid names into the answer. "
|
||||
"Do not treat a pause or background noise as the end of the utterance."
|
||||
)
|
||||
if self._mlx:
|
||||
if sample_rate != 16000:
|
||||
audio = librosa.resample(audio, orig_sr=sample_rate, target_sr=16000)
|
||||
result = self.model.generate(
|
||||
audio,
|
||||
prompt=instruction,
|
||||
max_tokens=self.max_new_tokens,
|
||||
temperature=0.0,
|
||||
)
|
||||
raw = result.text.strip()
|
||||
transcript, events = parse_response(raw)
|
||||
return transcript, events, raw
|
||||
|
||||
import torch
|
||||
conversation = [{"role": "user", "content": [
|
||||
{"type": "audio", "audio_url": "prefix.wav"},
|
||||
{"type": "text", "text": instruction},
|
||||
]}]
|
||||
text = self.processor.apply_chat_template(conversation, add_generation_prompt=True, tokenize=False)
|
||||
target_sr = self.processor.feature_extractor.sampling_rate
|
||||
if sample_rate != target_sr:
|
||||
audio = librosa.resample(audio, orig_sr=sample_rate, target_sr=target_sr)
|
||||
inputs = self.processor(text=text, audios=[audio], return_tensors="pt", padding=True)
|
||||
inputs = {k: v.to(self.device) if hasattr(v, "to") else v for k, v in inputs.items()}
|
||||
with torch.inference_mode():
|
||||
generated = self.model.generate(**inputs, max_new_tokens=self.max_new_tokens, do_sample=False)
|
||||
generated = generated[:, inputs["input_ids"].shape[1]:]
|
||||
raw = self.processor.batch_decode(
|
||||
generated, skip_special_tokens=False, clean_up_tokenization_spaces=False
|
||||
)[0].strip()
|
||||
transcript, events = parse_response(raw)
|
||||
return transcript, events, raw
|
||||
|
||||
|
||||
def growing_prefix(
|
||||
recognizer: Qwen2AudioRecognizer,
|
||||
audio_path: str | Path,
|
||||
chunk_seconds: float,
|
||||
*,
|
||||
on_result: Callable[[PrefixResult], None] | None = None,
|
||||
) -> list[PrefixResult]:
|
||||
"""Re-encode [0:t] for every chunk; this is intentionally not incremental."""
|
||||
audio, sr = librosa.load(str(audio_path), sr=None, mono=True)
|
||||
duration = len(audio) / sr
|
||||
endpoints = []
|
||||
endpoint = chunk_seconds
|
||||
while endpoint < duration:
|
||||
endpoints.append(endpoint)
|
||||
endpoint += chunk_seconds
|
||||
endpoints.append(duration)
|
||||
results = []
|
||||
for endpoint in endpoints:
|
||||
prefix = audio[: max(1, round(endpoint * sr))]
|
||||
started = time.perf_counter()
|
||||
transcript, events, raw = recognizer.transcribe_array(prefix, sr)
|
||||
result = PrefixResult(endpoint, time.perf_counter() - started, transcript, events, raw)
|
||||
results.append(result)
|
||||
if on_result:
|
||||
on_result(result)
|
||||
return results
|
||||
|
||||
|
||||
def serialize(results: list[PrefixResult]) -> list[dict[str, Any]]:
|
||||
return [asdict(result) for result in results]
|
||||
@@ -0,0 +1,11 @@
|
||||
torch>=2.2
|
||||
transformers>=4.45
|
||||
accelerate>=0.34
|
||||
librosa>=0.10
|
||||
soundfile>=0.12
|
||||
openai>=1.40
|
||||
python-dotenv>=1.0
|
||||
pytest>=7.4
|
||||
mlx-audio>=0.4.6; platform_machine == "arm64"
|
||||
openai-whisper>=20240930
|
||||
opencc-python-reimplemented>=0.1.7
|
||||
@@ -0,0 +1,311 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run Experiment 6-4 unchanged and bind local-model/audio/source provenance."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterable
|
||||
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
MODEL_ID = "mlx-community/Qwen2-Audio-7B-Instruct-4bit"
|
||||
MODEL_REVISION = "c65570002626f41b4dc08b7b54f42f99f3e82e7f"
|
||||
REFERENCE = "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。"
|
||||
SOURCE_FILES = [
|
||||
"run_official_experiment.py",
|
||||
"demo.py",
|
||||
"qwen2_streaming.py",
|
||||
"whisper_baseline.py",
|
||||
"prepare_scenarios.py",
|
||||
]
|
||||
AUDIO_FILES = [
|
||||
"audio/sentence.wav",
|
||||
"validation/scenarios/normal.wav",
|
||||
"validation/scenarios/long_pause.wav",
|
||||
"validation/scenarios/background_noise.wav",
|
||||
]
|
||||
SECRET_ENV_NAMES = (
|
||||
"ARK_API_KEY",
|
||||
"MOONSHOT_API_KEY",
|
||||
"OPENAI_API_KEY",
|
||||
"OPENROUTER_API_KEY",
|
||||
"TAVILY_API_KEY",
|
||||
)
|
||||
|
||||
|
||||
def utc_now() -> str:
|
||||
return datetime.now(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
|
||||
|
||||
|
||||
def sha256_bytes(value: bytes) -> str:
|
||||
return hashlib.sha256(value).hexdigest()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(8 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, value: Any) -> None:
|
||||
path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def git_commit() -> str | None:
|
||||
try:
|
||||
return subprocess.run(
|
||||
["git", "rev-parse", "HEAD"], cwd=ROOT, check=True,
|
||||
capture_output=True, text=True,
|
||||
).stdout.strip()
|
||||
except (OSError, subprocess.CalledProcessError):
|
||||
return None
|
||||
|
||||
|
||||
def find_credential_hits(payloads: Iterable[bytes]) -> Dict[str, int]:
|
||||
blobs = list(payloads)
|
||||
actual_secret_hits = 0
|
||||
for name in SECRET_ENV_NAMES:
|
||||
secret = os.getenv(name, "").encode("utf-8")
|
||||
if len(secret) >= 8:
|
||||
actual_secret_hits += sum(blob.count(secret) for blob in blobs)
|
||||
patterns = (
|
||||
re.compile(rb'(?i)"(?:api[_-]?key|authorization)"\s*:\s*"(?!<redacted>|null|")[^"]+"'),
|
||||
re.compile(rb'(?i)bearer\s+[a-z0-9._~+/=-]{16,}'),
|
||||
)
|
||||
pattern_hits = sum(len(pattern.findall(blob)) for pattern in patterns for blob in blobs)
|
||||
return {"actual_secret_hits": actual_secret_hits, "credential_pattern_hits": pattern_hits}
|
||||
|
||||
|
||||
def bool_gate(value: bool, **details: Any) -> Dict[str, Any]:
|
||||
return {"status": "pass" if value else "fail", **details}
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--run-id", help="immutable validation/runs directory name")
|
||||
parser.add_argument("--output-root", default=str(ROOT / "validation" / "runs"))
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def main(args: argparse.Namespace) -> int:
|
||||
run_id = args.run_id or f"exp6-4-qwen2audio-whisper-{datetime.now(timezone.utc):%Y%m%dT%H%M%SZ}"
|
||||
run_dir = Path(args.output_root).resolve() / run_id
|
||||
run_dir.mkdir(parents=True, exist_ok=False)
|
||||
started_at = utc_now()
|
||||
started = time.monotonic()
|
||||
|
||||
source_hashes_before = {name: sha256_file(ROOT / name) for name in SOURCE_FILES}
|
||||
audio_hashes = {name: sha256_file(ROOT / name) for name in AUDIO_FILES}
|
||||
whisper_path = Path.home() / ".cache" / "whisper" / "tiny.pt"
|
||||
if not whisper_path.is_file():
|
||||
raise RuntimeError(f"exact Whisper tiny checkpoint is unavailable: {whisper_path}")
|
||||
whisper_checkpoint = {
|
||||
"path": str(whisper_path),
|
||||
"size_bytes": whisper_path.stat().st_size,
|
||||
"sha256": sha256_file(whisper_path),
|
||||
}
|
||||
|
||||
snapshot = Path(snapshot_download(
|
||||
MODEL_ID,
|
||||
revision=MODEL_REVISION,
|
||||
local_files_only=True,
|
||||
))
|
||||
model_files = {}
|
||||
for path in sorted(item for item in snapshot.rglob("*") if item.is_file()):
|
||||
model_files[str(path.relative_to(snapshot))] = {
|
||||
"size_bytes": path.stat().st_size,
|
||||
"sha256": sha256_file(path),
|
||||
}
|
||||
|
||||
evidence_path = run_dir / "evidence.json"
|
||||
log_path = run_dir / "run.log"
|
||||
command = [
|
||||
sys.executable,
|
||||
str(ROOT / "demo.py"),
|
||||
"--model", MODEL_ID,
|
||||
"--device", "mlx",
|
||||
"--chunk-seconds", "2",
|
||||
"--whisper-model", "tiny",
|
||||
"--audio", "validation/scenarios/normal.wav",
|
||||
"--reference", REFERENCE,
|
||||
"--scenario", "normal",
|
||||
"--audio", "validation/scenarios/long_pause.wav",
|
||||
"--reference", REFERENCE,
|
||||
"--scenario", "pause",
|
||||
"--audio", "validation/scenarios/background_noise.wav",
|
||||
"--reference", REFERENCE,
|
||||
"--scenario", "noise",
|
||||
"--evidence", str(evidence_path),
|
||||
]
|
||||
with log_path.open("w", encoding="utf-8") as log:
|
||||
process = subprocess.run(
|
||||
command,
|
||||
cwd=ROOT,
|
||||
stdout=log,
|
||||
stderr=subprocess.STDOUT,
|
||||
text=True,
|
||||
)
|
||||
if process.returncode != 0:
|
||||
raise RuntimeError(f"exact experiment command failed with exit code {process.returncode}")
|
||||
|
||||
evidence = json.loads(evidence_path.read_text(encoding="utf-8"))
|
||||
source_hashes_after = {name: sha256_file(ROOT / name) for name in SOURCE_FILES}
|
||||
by_scenario = {case["scenario"]: case for case in evidence["cases"]}
|
||||
qwen_prefixes = [
|
||||
prefix
|
||||
for case in evidence["cases"]
|
||||
for prefix in case["qwen2_audio"]
|
||||
]
|
||||
model_weight = model_files.get("weights.safetensors", {})
|
||||
credential_scan = find_credential_hits([evidence_path.read_bytes(), log_path.read_bytes()])
|
||||
gates = {
|
||||
"exact_qwen2audio_mlx_design": bool_gate(
|
||||
evidence["model"] == MODEL_ID
|
||||
and evidence["device"] == "mlx"
|
||||
and evidence["method"] == "growing-prefix full re-encoding (not true streaming)"
|
||||
and evidence["parameters"]["chunk_seconds"] == 2.0,
|
||||
),
|
||||
"exact_600ms_vad_whisper_tiny_design": bool_gate(
|
||||
evidence["parameters"]["vad_silence_ms"] == 600
|
||||
and evidence["parameters"]["whisper_model"] == "tiny"
|
||||
and all(case.get("whisper_vad") for case in evidence["cases"]),
|
||||
),
|
||||
"normal_pause_noise_inputs": bool_gate(
|
||||
set(by_scenario) == {"normal", "pause", "noise"}
|
||||
and all(
|
||||
by_scenario[scenario]["media"]["sha256"] == audio_hashes[path]
|
||||
for scenario, path in (
|
||||
("normal", "validation/scenarios/normal.wav"),
|
||||
("pause", "validation/scenarios/long_pause.wav"),
|
||||
("noise", "validation/scenarios/background_noise.wav"),
|
||||
)
|
||||
),
|
||||
),
|
||||
"raw_qwen_outputs_retained": bool_gate(
|
||||
len(qwen_prefixes) == 13
|
||||
and all(prefix.get("raw_response") for prefix in qwen_prefixes),
|
||||
prefix_count=len(qwen_prefixes),
|
||||
),
|
||||
"runtime_sources_stable_and_hashed": bool_gate(
|
||||
source_hashes_before == source_hashes_after,
|
||||
count=len(source_hashes_before),
|
||||
),
|
||||
"full_model_snapshot_hashed": bool_gate(
|
||||
snapshot.name == MODEL_REVISION
|
||||
and len(model_files) >= 8
|
||||
and model_weight.get("size_bytes") == 6562540479
|
||||
and len(model_weight.get("sha256", "")) == 64,
|
||||
revision=snapshot.name,
|
||||
file_count=len(model_files),
|
||||
total_bytes=sum(item["size_bytes"] for item in model_files.values()),
|
||||
),
|
||||
"whisper_checkpoint_hashed": bool_gate(
|
||||
whisper_checkpoint["sha256"]
|
||||
== evidence["provenance"]["host"]["whisper_baseline"]["sha256"],
|
||||
size_bytes=whisper_checkpoint["size_bytes"],
|
||||
),
|
||||
"credential_free_local_artifacts": bool_gate(
|
||||
credential_scan["actual_secret_hits"] == 0
|
||||
and credential_scan["credential_pattern_hits"] == 0,
|
||||
**credential_scan,
|
||||
),
|
||||
}
|
||||
overall_status = "pass" if all(item["status"] == "pass" for item in gates.values()) else "incomplete"
|
||||
|
||||
acceptance = {
|
||||
"schema_version": 1,
|
||||
"experiment": "6-4",
|
||||
"run_id": run_id,
|
||||
"started_at": started_at,
|
||||
"completed_at": utc_now(),
|
||||
"duration_seconds": round(time.monotonic() - started, 3),
|
||||
"git_commit": git_commit(),
|
||||
"execution_gates": gates,
|
||||
"execution_status": overall_status,
|
||||
"manuscript_result_claims": evidence["acceptance"]["manuscript_result_claims"],
|
||||
"manuscript_results_reproduced": evidence["acceptance"]["manuscript_results_reproduced"],
|
||||
"event_evaluation": {
|
||||
scenario: by_scenario[scenario]["qwen2_event_evaluation"]
|
||||
for scenario in ("normal", "pause", "noise")
|
||||
},
|
||||
"qwen_inference_seconds": [prefix["inference_seconds"] for prefix in qwen_prefixes],
|
||||
"whisper_post_endpoint_response_seconds": {
|
||||
scenario: by_scenario[scenario]["whisper_vad"]["post_endpoint_response_seconds"]
|
||||
for scenario in ("normal", "pause", "noise")
|
||||
},
|
||||
"runtime_source_sha256": source_hashes_before,
|
||||
"audio_sha256": audio_hashes,
|
||||
"whisper_checkpoint": whisper_checkpoint,
|
||||
"qwen2_audio_snapshot": {
|
||||
"repository": MODEL_ID,
|
||||
"revision": snapshot.name,
|
||||
"path": str(snapshot),
|
||||
"files": model_files,
|
||||
},
|
||||
"credential_scan": credential_scan,
|
||||
"passed_gates": sum(item["status"] == "pass" for item in gates.values()),
|
||||
"total_gates": len(gates),
|
||||
}
|
||||
acceptance_path = run_dir / "acceptance.json"
|
||||
write_json(acceptance_path, acceptance)
|
||||
|
||||
manifest = {
|
||||
"schema_version": 1,
|
||||
"experiment": "6-4",
|
||||
"run_id": run_id,
|
||||
"generated_at": utc_now(),
|
||||
"git_commit": acceptance["git_commit"],
|
||||
"runtime_source_sha256": source_hashes_before,
|
||||
"audio_sha256": audio_hashes,
|
||||
"whisper_checkpoint": whisper_checkpoint,
|
||||
"qwen2_audio_snapshot": acceptance["qwen2_audio_snapshot"],
|
||||
"artifact_sha256": {
|
||||
path.name: sha256_file(path)
|
||||
for path in (evidence_path, log_path, acceptance_path)
|
||||
},
|
||||
"acceptance": {
|
||||
"execution_status": overall_status,
|
||||
"passed_gates": acceptance["passed_gates"],
|
||||
"total_gates": acceptance["total_gates"],
|
||||
"manuscript_results_reproduced": acceptance["manuscript_results_reproduced"],
|
||||
},
|
||||
}
|
||||
manifest_path = run_dir / "manifest.json"
|
||||
write_json(manifest_path, manifest)
|
||||
|
||||
latest = ROOT / "validation" / "latest_official.json"
|
||||
write_json(latest, {
|
||||
"schema_version": 1,
|
||||
"run_id": run_id,
|
||||
"run_directory": str(run_dir.relative_to(ROOT)),
|
||||
"manifest_sha256": sha256_file(manifest_path),
|
||||
"execution_status": overall_status,
|
||||
"manuscript_results_reproduced": acceptance["manuscript_results_reproduced"],
|
||||
})
|
||||
print(json.dumps({
|
||||
"run_id": run_id,
|
||||
"run_directory": str(run_dir),
|
||||
"execution_status": overall_status,
|
||||
"passed_gates": acceptance["passed_gates"],
|
||||
"total_gates": acceptance["total_gates"],
|
||||
"manuscript_result_claims": acceptance["manuscript_result_claims"],
|
||||
"event_evaluation": acceptance["event_evaluation"],
|
||||
}, ensure_ascii=False, indent=2))
|
||||
return 0 if overall_status == "pass" else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main(parse_args()))
|
||||
@@ -0,0 +1,81 @@
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
RUN = ROOT / "validation" / "runs" / "exp6-4-qwen2audio-whisper-provenance-20260730-v3"
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
while chunk := handle.read(8 * 1024 * 1024):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def test_official_manifest_binds_artifacts_sources_audio_and_local_checkpoints():
|
||||
manifest = json.loads((RUN / "manifest.json").read_text(encoding="utf-8"))
|
||||
assert manifest["acceptance"] == {
|
||||
"execution_status": "pass",
|
||||
"passed_gates": 8,
|
||||
"total_gates": 8,
|
||||
"manuscript_results_reproduced": False,
|
||||
}
|
||||
for name, expected in manifest["artifact_sha256"].items():
|
||||
assert sha256_file(RUN / name) == expected
|
||||
for name, expected in manifest["runtime_source_sha256"].items():
|
||||
assert sha256_file(ROOT / name) == expected
|
||||
for name, expected in manifest["audio_sha256"].items():
|
||||
assert sha256_file(ROOT / name) == expected
|
||||
whisper = manifest["whisper_checkpoint"]
|
||||
assert sha256_file(Path(whisper["path"])) == whisper["sha256"]
|
||||
|
||||
snapshot = Path(manifest["qwen2_audio_snapshot"]["path"])
|
||||
files = manifest["qwen2_audio_snapshot"]["files"]
|
||||
assert files["weights.safetensors"] == {
|
||||
"size_bytes": 6562540479,
|
||||
"sha256": "0967cde270ad62aa4824f0bbce283d0a2a6da2825ccf587640753c527d4174da",
|
||||
}
|
||||
for name, item in files.items():
|
||||
assert (snapshot / name).stat().st_size == item["size_bytes"]
|
||||
if name != "weights.safetensors":
|
||||
assert sha256_file(snapshot / name) == item["sha256"]
|
||||
|
||||
|
||||
def test_official_run_retains_raw_prefixes_and_negative_manuscript_results():
|
||||
evidence = json.loads((RUN / "evidence.json").read_text(encoding="utf-8"))
|
||||
acceptance = json.loads((RUN / "acceptance.json").read_text(encoding="utf-8"))
|
||||
prefixes = [prefix for case in evidence["cases"] for prefix in case["qwen2_audio"]]
|
||||
assert len(prefixes) == 13
|
||||
assert all(prefix["raw_response"] for prefix in prefixes)
|
||||
assert all(item["status"] == "pass" for item in acceptance["execution_gates"].values())
|
||||
assert acceptance["manuscript_result_claims"] == {
|
||||
"qwen_incremental_latency_100_to_200ms": False,
|
||||
"traditional_post_endpoint_latency_800_to_1100ms": False,
|
||||
"pause_split_into_two_segments": True,
|
||||
"pause_specific_two_to_zero_error": False,
|
||||
"noise_token_detected": True,
|
||||
"noise_caused_earlier_vad_start": False,
|
||||
}
|
||||
assert acceptance["manuscript_results_reproduced"] is False
|
||||
assert acceptance["event_evaluation"]["pause"]["false_negative"] == ["<|silence|>"]
|
||||
assert acceptance["event_evaluation"]["noise"]["false_positive"] == [
|
||||
"<|cough|>", "<|laughter|>",
|
||||
]
|
||||
|
||||
|
||||
def test_official_latest_and_credential_scan_are_consistent():
|
||||
latest = json.loads((ROOT / "validation" / "latest_official.json").read_text(encoding="utf-8"))
|
||||
acceptance = json.loads((RUN / "acceptance.json").read_text(encoding="utf-8"))
|
||||
assert latest["run_id"] == acceptance["run_id"]
|
||||
assert latest["manifest_sha256"] == sha256_file(RUN / "manifest.json")
|
||||
assert acceptance["credential_scan"] == {
|
||||
"actual_secret_hits": 0,
|
||||
"credential_pattern_hits": 0,
|
||||
}
|
||||
combined = b"\n".join(path.read_bytes() for path in RUN.iterdir() if path.is_file())
|
||||
assert not re.search(rb'(?i)"(?:api[_-]?key|authorization)"\s*:\s*"(?!<redacted>|null|")[^"]+"', combined)
|
||||
assert not re.search(rb'(?i)bearer\s+[a-z0-9._~+/=-]{16,}', combined)
|
||||
@@ -0,0 +1,26 @@
|
||||
import numpy as np
|
||||
|
||||
from qwen2_streaming import parse_response
|
||||
from whisper_baseline import energy_vad_endpoints, energy_vad_events
|
||||
|
||||
|
||||
def test_qwen_json_and_native_event_token_parsing():
|
||||
transcript, events = parse_response('{"transcript":"你好","acoustic_events":["noise", "<|laughter|>"]}')
|
||||
assert transcript == "你好"
|
||||
assert events == ["<|noise|>", "<|laughter|>"]
|
||||
|
||||
|
||||
def test_600ms_vad_splits_a_long_pause():
|
||||
sr = 1000
|
||||
audio = np.concatenate([np.ones(sr), np.zeros(700), np.ones(sr)]) * 0.1
|
||||
endpoints = energy_vad_endpoints(audio, sr, silence_ms=600)
|
||||
assert len(endpoints) == 2
|
||||
assert 900 <= endpoints[0] <= 1100
|
||||
|
||||
|
||||
def test_vad_separates_acoustic_endpoint_from_600ms_decision():
|
||||
sr = 1000
|
||||
audio = np.concatenate([np.ones(sr), np.zeros(700), np.ones(sr)]) * 0.1
|
||||
event = energy_vad_events(audio, sr, silence_ms=600)[0]
|
||||
assert 900 <= event.speech_endpoint <= 1100
|
||||
assert 590 <= event.decision - event.speech_endpoint <= 610
|
||||
@@ -0,0 +1,15 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "chapter9" / "streaming-speech"))
|
||||
|
||||
from whisper_baseline import energy_vad_events
|
||||
|
||||
|
||||
def test_energy_vad_events_handles_empty_audio_array():
|
||||
"""Empty or zero-length audio array must return an empty list of VAD events without crashing."""
|
||||
empty_audio = np.array([], dtype=np.float32)
|
||||
events = energy_vad_events(empty_audio, 16000)
|
||||
assert events == []
|
||||
@@ -0,0 +1,170 @@
|
||||
{
|
||||
"experiment": "6-4",
|
||||
"timestamp_utc": "2026-07-29T10:11:30.432848+00:00",
|
||||
"model": "mlx-community/Qwen2-Audio-7B-Instruct-4bit",
|
||||
"device": "mlx",
|
||||
"method": "growing-prefix full re-encoding (not true streaming)",
|
||||
"cases": [
|
||||
{
|
||||
"scenario": "normal",
|
||||
"audio": "validation/scenarios/normal.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 4.139199791010469,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 5.053489749785513,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 4.907809666823596,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 7.4875,
|
||||
"inference_seconds": 5.012733499985188,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"vad_wait_seconds": 7.4875,
|
||||
"asr_seconds": 0.5402721669524908,
|
||||
"total_response_seconds": 8.027772166952492,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家",
|
||||
"segment_count": 1,
|
||||
"endpoints": [
|
||||
7.487
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.10526315789473684
|
||||
},
|
||||
{
|
||||
"scenario": "pause",
|
||||
"audio": "validation/scenarios/long_pause.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 4.358620333019644,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 4.480873542372137,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 4.751943958923221,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还还是在三号会议室",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点还还是在三号会议室', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 8.0,
|
||||
"inference_seconds": 4.8131710002198815,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 8.3875,
|
||||
"inference_seconds": 4.650784750003368,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"vad_wait_seconds": 4.04,
|
||||
"asr_seconds": 0.5881568747572601,
|
||||
"total_response_seconds": 4.62815687475726,
|
||||
"transcript": "麻煩你幫我把明天下午的會議改到兩點半地點 還是在三號會議時別忘了通知大家",
|
||||
"segment_count": 2,
|
||||
"endpoints": [
|
||||
4.04,
|
||||
8.387
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.42105263157894735
|
||||
},
|
||||
{
|
||||
"scenario": "noise",
|
||||
"audio": "validation/scenarios/background_noise.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 4.019581000320613,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 4.358317042235285,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 4.340823790989816,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 7.4875,
|
||||
"inference_seconds": 4.508679833263159,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"vad_wait_seconds": 7.4875,
|
||||
"asr_seconds": 0.4322935417294502,
|
||||
"total_response_seconds": 7.91979354172945,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家",
|
||||
"segment_count": 1,
|
||||
"endpoints": [
|
||||
7.487
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.10526315789473684
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"run_id": "exp6-4-qwen2audio-whisper-provenance-20260730-v3",
|
||||
"run_directory": "validation/runs/exp6-4-qwen2audio-whisper-provenance-20260730-v3",
|
||||
"manifest_sha256": "3d02bce7ac422449bf29bdf61576377b44b9d7b8b046e2cd4f6a7e065f1da67b",
|
||||
"execution_status": "pass",
|
||||
"manuscript_results_reproduced": false
|
||||
}
|
||||
@@ -0,0 +1,366 @@
|
||||
{
|
||||
"schema_version": 2,
|
||||
"experiment": "6-4",
|
||||
"timestamp_utc": "2026-07-29T15:40:54.554752+00:00",
|
||||
"model": "mlx-community/Qwen2-Audio-7B-Instruct-4bit",
|
||||
"device": "mlx",
|
||||
"method": "growing-prefix full re-encoding (not true streaming)",
|
||||
"parameters": {
|
||||
"chunk_seconds": 2.0,
|
||||
"whisper_model": "tiny",
|
||||
"vad_silence_ms": 600,
|
||||
"cer_normalization": "NFKC + Traditional-to-Simplified Chinese + casefold + remove punctuation/symbols/separators"
|
||||
},
|
||||
"provenance": {
|
||||
"host": {
|
||||
"platform": "macOS-26.3-arm64-arm-64bit",
|
||||
"machine": "arm64",
|
||||
"cpu": "Apple M2 Max",
|
||||
"memory_bytes": 103079215104,
|
||||
"python": "3.11.4",
|
||||
"packages": {
|
||||
"mlx-audio": "0.4.6",
|
||||
"openai-whisper": "20231106",
|
||||
"librosa": "0.10.2.post1",
|
||||
"opencc-python-reimplemented": "0.1.7"
|
||||
},
|
||||
"whisper_baseline": {
|
||||
"model": "tiny",
|
||||
"path": "/Users/boj/.cache/whisper/tiny.pt",
|
||||
"sha256": "65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9"
|
||||
}
|
||||
},
|
||||
"qwen2_audio": {
|
||||
"repository": "mlx-community/Qwen2-Audio-7B-Instruct-4bit",
|
||||
"snapshot_revision": "c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"snapshot_path": "/Users/boj/.cache/huggingface/hub/models--mlx-community--Qwen2-Audio-7B-Instruct-4bit/snapshots/c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"total_bytes": 6574659991,
|
||||
"files": [
|
||||
{
|
||||
"path": ".gitattributes",
|
||||
"size_bytes": 1519,
|
||||
"sha256": "11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361"
|
||||
},
|
||||
{
|
||||
"path": "README.md",
|
||||
"size_bytes": 1716,
|
||||
"sha256": "1cddc760431c9c2c043ed4d17d9c17716a1f81978e6093389420cd46e7f2ce6d"
|
||||
},
|
||||
{
|
||||
"path": "config.json",
|
||||
"size_bytes": 913,
|
||||
"sha256": "39493dad1678533cc2a7ba2de0a573cc926a78ef4c7abef72a6b660159de94b7"
|
||||
},
|
||||
{
|
||||
"path": "merges.txt",
|
||||
"size_bytes": 1671839,
|
||||
"sha256": "599bab54075088774b1733fde865d5bd747cbcc7a547c5bc12610e874e26f5e3"
|
||||
},
|
||||
{
|
||||
"path": "preprocessor_config.json",
|
||||
"size_bytes": 342,
|
||||
"sha256": "4cd7c6c061fe79244c57b0c320b2873f3ee5acce2277f7cc3aced042725680f2"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer.json",
|
||||
"size_bytes": 7028015,
|
||||
"sha256": "f7c9b2dba4a296b1aa76c16a34b8225c0c118978400d4bb66bff0902d702f5b8"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer_config.json",
|
||||
"size_bytes": 638335,
|
||||
"sha256": "c738158e70eeecf25736a6c4d8e5f34cfce683079980757edfb0665a7b2457ed"
|
||||
},
|
||||
{
|
||||
"path": "vocab.json",
|
||||
"size_bytes": 2776833,
|
||||
"sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"
|
||||
},
|
||||
{
|
||||
"path": "weights.safetensors",
|
||||
"size_bytes": 6562540479,
|
||||
"sha256": null
|
||||
}
|
||||
],
|
||||
"large_weight_hash_note": "Snapshot revision pins large files; files over 10 MiB are inventoried by path and size without rehashing."
|
||||
}
|
||||
},
|
||||
"cases": [
|
||||
{
|
||||
"scenario": "normal",
|
||||
"audio": "validation/scenarios/normal.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"media": {
|
||||
"sha256": "22cbbff9df0ec0efc9a09b5d250b8d63c1876363716b8a1d7aa9f85f16e8a410",
|
||||
"expected_acoustic_events": []
|
||||
},
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 2.859895291738212,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 2.78174329129979,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 2.8982563340105116,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 7.4875,
|
||||
"inference_seconds": 2.937152124941349,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"configured_silence_ms": 600,
|
||||
"first_speech_start_seconds": 0.02,
|
||||
"first_endpoint_audio_seconds": 7.16,
|
||||
"first_decision_audio_seconds": 7.76,
|
||||
"post_speech_vad_delay_seconds": 0.6,
|
||||
"first_segment_asr_seconds": 0.5664697499014437,
|
||||
"all_segments_asr_seconds": 0.5664697499014437,
|
||||
"post_endpoint_response_seconds": 1.1664697499014438,
|
||||
"first_response_from_audio_start_seconds": 8.326469749901444,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家",
|
||||
"segment_transcripts": [
|
||||
"麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家"
|
||||
],
|
||||
"segment_asr_seconds": [
|
||||
0.5664697499014437
|
||||
],
|
||||
"segment_count": 1,
|
||||
"endpoints": [
|
||||
7.16
|
||||
],
|
||||
"decisions": [
|
||||
7.76
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.02857142857142857,
|
||||
"qwen2_event_evaluation": {
|
||||
"true_positive": [],
|
||||
"false_positive": [],
|
||||
"false_negative": [],
|
||||
"exact_match": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "pause",
|
||||
"audio": "validation/scenarios/long_pause.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"media": {
|
||||
"sha256": "7b4b9975d58c71c50cdb81ff9578a0b04192aeb3cbea634384a804ea4e3dc48d",
|
||||
"expected_acoustic_events": [
|
||||
"<|silence|>"
|
||||
]
|
||||
},
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 2.6971133751794696,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 2.734142666682601,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 2.980124874971807,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还还是在三号会议室",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点还还是在三号会议室', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 8.0,
|
||||
"inference_seconds": 2.9454508339986205,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 8.3875,
|
||||
"inference_seconds": 2.941487292293459,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"configured_silence_ms": 600,
|
||||
"first_speech_start_seconds": 0.02,
|
||||
"first_endpoint_audio_seconds": 4.04,
|
||||
"first_decision_audio_seconds": 4.64,
|
||||
"post_speech_vad_delay_seconds": 0.6,
|
||||
"first_segment_asr_seconds": 0.31024629063904285,
|
||||
"all_segments_asr_seconds": 0.5564022487960756,
|
||||
"post_endpoint_response_seconds": 0.9102462906390428,
|
||||
"first_response_from_audio_start_seconds": 4.9502462906390425,
|
||||
"transcript": "麻煩你幫我把明天下午的會議改到兩點半地點 還是在三號會議時別忘了通知大家",
|
||||
"segment_transcripts": [
|
||||
"麻煩你幫我把明天下午的會議改到兩點半地點",
|
||||
"還是在三號會議時別忘了通知大家"
|
||||
],
|
||||
"segment_asr_seconds": [
|
||||
0.31024629063904285,
|
||||
0.24615595815703273
|
||||
],
|
||||
"segment_count": 2,
|
||||
"endpoints": [
|
||||
4.04,
|
||||
8.06
|
||||
],
|
||||
"decisions": [
|
||||
4.64,
|
||||
8.66
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.02857142857142857,
|
||||
"qwen2_event_evaluation": {
|
||||
"true_positive": [],
|
||||
"false_positive": [],
|
||||
"false_negative": [
|
||||
"<|silence|>"
|
||||
],
|
||||
"exact_match": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "noise",
|
||||
"audio": "validation/scenarios/background_noise.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"media": {
|
||||
"sha256": "034f4e9d4f5c2b69407e1e9813b4544792764a7e2e7340af410945b5c90c48b1",
|
||||
"expected_acoustic_events": [
|
||||
"<|noise|>"
|
||||
]
|
||||
},
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 2.711420916952193,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 2.9726402079686522,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 3.041755000129342,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 7.4875,
|
||||
"inference_seconds": 3.2653073337860405,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"configured_silence_ms": 600,
|
||||
"first_speech_start_seconds": 0.0,
|
||||
"first_endpoint_audio_seconds": 7.4875,
|
||||
"first_decision_audio_seconds": 8.0875,
|
||||
"post_speech_vad_delay_seconds": 0.6,
|
||||
"first_segment_asr_seconds": 0.40538191702216864,
|
||||
"all_segments_asr_seconds": 0.40538191702216864,
|
||||
"post_endpoint_response_seconds": 1.0053819170221687,
|
||||
"first_response_from_audio_start_seconds": 8.492881917022169,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家",
|
||||
"segment_transcripts": [
|
||||
"麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家"
|
||||
],
|
||||
"segment_asr_seconds": [
|
||||
0.40538191702216864
|
||||
],
|
||||
"segment_count": 1,
|
||||
"endpoints": [
|
||||
7.487
|
||||
],
|
||||
"decisions": [
|
||||
8.088
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.02857142857142857,
|
||||
"qwen2_event_evaluation": {
|
||||
"true_positive": [
|
||||
"<|noise|>"
|
||||
],
|
||||
"false_positive": [
|
||||
"<|cough|>",
|
||||
"<|laughter|>"
|
||||
],
|
||||
"false_negative": [],
|
||||
"exact_match": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"cost": {
|
||||
"paid_external_requests": 0,
|
||||
"total_usd": 0,
|
||||
"note": "Both models ran locally."
|
||||
},
|
||||
"acceptance": {
|
||||
"execution_gates": {
|
||||
"real_qwen2_audio": true,
|
||||
"growing_prefix_full_reencoding": true,
|
||||
"real_600ms_vad_whisper_baseline": true,
|
||||
"normal_pause_noise_scenarios": true,
|
||||
"corrected_vad_latency_accounting": true,
|
||||
"normalized_cer": true,
|
||||
"provenance_complete": true
|
||||
},
|
||||
"execution_passed": true,
|
||||
"manuscript_result_claims": {
|
||||
"qwen_incremental_latency_100_to_200ms": false,
|
||||
"traditional_post_endpoint_latency_800_to_1100ms": false,
|
||||
"pause_split_into_two_segments": true,
|
||||
"pause_specific_two_to_zero_error": false,
|
||||
"noise_token_detected": true,
|
||||
"noise_caused_earlier_vad_start": false
|
||||
},
|
||||
"manuscript_results_reproduced": false
|
||||
}
|
||||
}
|
||||
+167
@@ -0,0 +1,167 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "6-4",
|
||||
"run_id": "exp6-4-qwen2audio-whisper-provenance-20260730-v3",
|
||||
"started_at": "2026-07-30T05:22:28.357Z",
|
||||
"completed_at": "2026-07-30T05:24:59.451Z",
|
||||
"duration_seconds": 151.093,
|
||||
"git_commit": "f119cf510f3746cd5d071d0bce0051617f02f163",
|
||||
"execution_gates": {
|
||||
"exact_qwen2audio_mlx_design": {
|
||||
"status": "pass"
|
||||
},
|
||||
"exact_600ms_vad_whisper_tiny_design": {
|
||||
"status": "pass"
|
||||
},
|
||||
"normal_pause_noise_inputs": {
|
||||
"status": "pass"
|
||||
},
|
||||
"raw_qwen_outputs_retained": {
|
||||
"status": "pass",
|
||||
"prefix_count": 13
|
||||
},
|
||||
"runtime_sources_stable_and_hashed": {
|
||||
"status": "pass",
|
||||
"count": 5
|
||||
},
|
||||
"full_model_snapshot_hashed": {
|
||||
"status": "pass",
|
||||
"revision": "c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"file_count": 9,
|
||||
"total_bytes": 6574659991
|
||||
},
|
||||
"whisper_checkpoint_hashed": {
|
||||
"status": "pass",
|
||||
"size_bytes": 75572083
|
||||
},
|
||||
"credential_free_local_artifacts": {
|
||||
"status": "pass",
|
||||
"actual_secret_hits": 0,
|
||||
"credential_pattern_hits": 0
|
||||
}
|
||||
},
|
||||
"execution_status": "pass",
|
||||
"manuscript_result_claims": {
|
||||
"qwen_incremental_latency_100_to_200ms": false,
|
||||
"traditional_post_endpoint_latency_800_to_1100ms": false,
|
||||
"pause_split_into_two_segments": true,
|
||||
"pause_specific_two_to_zero_error": false,
|
||||
"noise_token_detected": true,
|
||||
"noise_caused_earlier_vad_start": false
|
||||
},
|
||||
"manuscript_results_reproduced": false,
|
||||
"event_evaluation": {
|
||||
"normal": {
|
||||
"true_positive": [],
|
||||
"false_positive": [],
|
||||
"false_negative": [],
|
||||
"exact_match": true
|
||||
},
|
||||
"pause": {
|
||||
"true_positive": [],
|
||||
"false_positive": [],
|
||||
"false_negative": [
|
||||
"<|silence|>"
|
||||
],
|
||||
"exact_match": false
|
||||
},
|
||||
"noise": {
|
||||
"true_positive": [
|
||||
"<|noise|>"
|
||||
],
|
||||
"false_positive": [
|
||||
"<|cough|>",
|
||||
"<|laughter|>"
|
||||
],
|
||||
"false_negative": [],
|
||||
"exact_match": false
|
||||
}
|
||||
},
|
||||
"qwen_inference_seconds": [
|
||||
9.968843291979283,
|
||||
9.877914333250374,
|
||||
10.44804058270529,
|
||||
10.78756216680631,
|
||||
8.413955624680966,
|
||||
8.742833625059575,
|
||||
10.989260375034064,
|
||||
10.69584474992007,
|
||||
10.8182367910631,
|
||||
8.962713832966983,
|
||||
10.234860499855131,
|
||||
10.288799332920462,
|
||||
11.268820082768798
|
||||
],
|
||||
"whisper_post_endpoint_response_seconds": {
|
||||
"normal": 1.2523482501506806,
|
||||
"pause": 0.9871377082541585,
|
||||
"noise": 1.155937457829714
|
||||
},
|
||||
"runtime_source_sha256": {
|
||||
"run_official_experiment.py": "a1d95ed2fbd9405678a146898201c7e1082fd2620f6528268a7ee8943f22e5b1",
|
||||
"demo.py": "bf132207426606a38c4e487ed59ff7832cfa2468f5c6e2bab87394086a049d92",
|
||||
"qwen2_streaming.py": "7744acd4c134c9f861d207ff400d5effa32d17d400ca4973da589d8d74782c76",
|
||||
"whisper_baseline.py": "dc0554b4085e806bc18e8a81fe5c0af55d34bfbe43107b66632afd461c7b5483",
|
||||
"prepare_scenarios.py": "a0b01392d19a4c0463d33c7b56af323131cf7a42213410537109a6698f1e15b0"
|
||||
},
|
||||
"audio_sha256": {
|
||||
"audio/sentence.wav": "51b77857cb9d859ca37dc6ba95680fb675b8e2690257f00f69a46a11febbd4a2",
|
||||
"validation/scenarios/normal.wav": "22cbbff9df0ec0efc9a09b5d250b8d63c1876363716b8a1d7aa9f85f16e8a410",
|
||||
"validation/scenarios/long_pause.wav": "7b4b9975d58c71c50cdb81ff9578a0b04192aeb3cbea634384a804ea4e3dc48d",
|
||||
"validation/scenarios/background_noise.wav": "034f4e9d4f5c2b69407e1e9813b4544792764a7e2e7340af410945b5c90c48b1"
|
||||
},
|
||||
"whisper_checkpoint": {
|
||||
"path": "/Users/boj/.cache/whisper/tiny.pt",
|
||||
"size_bytes": 75572083,
|
||||
"sha256": "65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9"
|
||||
},
|
||||
"qwen2_audio_snapshot": {
|
||||
"repository": "mlx-community/Qwen2-Audio-7B-Instruct-4bit",
|
||||
"revision": "c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"path": "/Users/boj/.cache/huggingface/hub/models--mlx-community--Qwen2-Audio-7B-Instruct-4bit/snapshots/c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"files": {
|
||||
".gitattributes": {
|
||||
"size_bytes": 1519,
|
||||
"sha256": "11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361"
|
||||
},
|
||||
"README.md": {
|
||||
"size_bytes": 1716,
|
||||
"sha256": "1cddc760431c9c2c043ed4d17d9c17716a1f81978e6093389420cd46e7f2ce6d"
|
||||
},
|
||||
"config.json": {
|
||||
"size_bytes": 913,
|
||||
"sha256": "39493dad1678533cc2a7ba2de0a573cc926a78ef4c7abef72a6b660159de94b7"
|
||||
},
|
||||
"merges.txt": {
|
||||
"size_bytes": 1671839,
|
||||
"sha256": "599bab54075088774b1733fde865d5bd747cbcc7a547c5bc12610e874e26f5e3"
|
||||
},
|
||||
"preprocessor_config.json": {
|
||||
"size_bytes": 342,
|
||||
"sha256": "4cd7c6c061fe79244c57b0c320b2873f3ee5acce2277f7cc3aced042725680f2"
|
||||
},
|
||||
"tokenizer.json": {
|
||||
"size_bytes": 7028015,
|
||||
"sha256": "f7c9b2dba4a296b1aa76c16a34b8225c0c118978400d4bb66bff0902d702f5b8"
|
||||
},
|
||||
"tokenizer_config.json": {
|
||||
"size_bytes": 638335,
|
||||
"sha256": "c738158e70eeecf25736a6c4d8e5f34cfce683079980757edfb0665a7b2457ed"
|
||||
},
|
||||
"vocab.json": {
|
||||
"size_bytes": 2776833,
|
||||
"sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"
|
||||
},
|
||||
"weights.safetensors": {
|
||||
"size_bytes": 6562540479,
|
||||
"sha256": "0967cde270ad62aa4824f0bbce283d0a2a6da2825ccf587640753c527d4174da"
|
||||
}
|
||||
}
|
||||
},
|
||||
"credential_scan": {
|
||||
"actual_secret_hits": 0,
|
||||
"credential_pattern_hits": 0
|
||||
},
|
||||
"passed_gates": 8,
|
||||
"total_gates": 8
|
||||
}
|
||||
+366
@@ -0,0 +1,366 @@
|
||||
{
|
||||
"schema_version": 2,
|
||||
"experiment": "6-4",
|
||||
"timestamp_utc": "2026-07-30T05:24:57.619611+00:00",
|
||||
"model": "mlx-community/Qwen2-Audio-7B-Instruct-4bit",
|
||||
"device": "mlx",
|
||||
"method": "growing-prefix full re-encoding (not true streaming)",
|
||||
"parameters": {
|
||||
"chunk_seconds": 2.0,
|
||||
"whisper_model": "tiny",
|
||||
"vad_silence_ms": 600,
|
||||
"cer_normalization": "NFKC + Traditional-to-Simplified Chinese + casefold + remove punctuation/symbols/separators"
|
||||
},
|
||||
"provenance": {
|
||||
"host": {
|
||||
"platform": "macOS-26.3-arm64-arm-64bit",
|
||||
"machine": "arm64",
|
||||
"cpu": "Apple M2 Max",
|
||||
"memory_bytes": 103079215104,
|
||||
"python": "3.11.4",
|
||||
"packages": {
|
||||
"mlx-audio": "0.4.6",
|
||||
"openai-whisper": "20231106",
|
||||
"librosa": "0.10.2.post1",
|
||||
"opencc-python-reimplemented": "0.1.7"
|
||||
},
|
||||
"whisper_baseline": {
|
||||
"model": "tiny",
|
||||
"path": "/Users/boj/.cache/whisper/tiny.pt",
|
||||
"sha256": "65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9"
|
||||
}
|
||||
},
|
||||
"qwen2_audio": {
|
||||
"repository": "mlx-community/Qwen2-Audio-7B-Instruct-4bit",
|
||||
"snapshot_revision": "c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"snapshot_path": "/Users/boj/.cache/huggingface/hub/models--mlx-community--Qwen2-Audio-7B-Instruct-4bit/snapshots/c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"total_bytes": 6574659991,
|
||||
"files": [
|
||||
{
|
||||
"path": ".gitattributes",
|
||||
"size_bytes": 1519,
|
||||
"sha256": "11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361"
|
||||
},
|
||||
{
|
||||
"path": "README.md",
|
||||
"size_bytes": 1716,
|
||||
"sha256": "1cddc760431c9c2c043ed4d17d9c17716a1f81978e6093389420cd46e7f2ce6d"
|
||||
},
|
||||
{
|
||||
"path": "config.json",
|
||||
"size_bytes": 913,
|
||||
"sha256": "39493dad1678533cc2a7ba2de0a573cc926a78ef4c7abef72a6b660159de94b7"
|
||||
},
|
||||
{
|
||||
"path": "merges.txt",
|
||||
"size_bytes": 1671839,
|
||||
"sha256": "599bab54075088774b1733fde865d5bd747cbcc7a547c5bc12610e874e26f5e3"
|
||||
},
|
||||
{
|
||||
"path": "preprocessor_config.json",
|
||||
"size_bytes": 342,
|
||||
"sha256": "4cd7c6c061fe79244c57b0c320b2873f3ee5acce2277f7cc3aced042725680f2"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer.json",
|
||||
"size_bytes": 7028015,
|
||||
"sha256": "f7c9b2dba4a296b1aa76c16a34b8225c0c118978400d4bb66bff0902d702f5b8"
|
||||
},
|
||||
{
|
||||
"path": "tokenizer_config.json",
|
||||
"size_bytes": 638335,
|
||||
"sha256": "c738158e70eeecf25736a6c4d8e5f34cfce683079980757edfb0665a7b2457ed"
|
||||
},
|
||||
{
|
||||
"path": "vocab.json",
|
||||
"size_bytes": 2776833,
|
||||
"sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"
|
||||
},
|
||||
{
|
||||
"path": "weights.safetensors",
|
||||
"size_bytes": 6562540479,
|
||||
"sha256": null
|
||||
}
|
||||
],
|
||||
"large_weight_hash_note": "Snapshot revision pins large files; files over 10 MiB are inventoried by path and size without rehashing."
|
||||
}
|
||||
},
|
||||
"cases": [
|
||||
{
|
||||
"scenario": "normal",
|
||||
"audio": "validation/scenarios/normal.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"media": {
|
||||
"sha256": "22cbbff9df0ec0efc9a09b5d250b8d63c1876363716b8a1d7aa9f85f16e8a410",
|
||||
"expected_acoustic_events": []
|
||||
},
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 9.968843291979283,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 9.877914333250374,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 10.44804058270529,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 7.4875,
|
||||
"inference_seconds": 10.78756216680631,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"configured_silence_ms": 600,
|
||||
"first_speech_start_seconds": 0.02,
|
||||
"first_endpoint_audio_seconds": 7.16,
|
||||
"first_decision_audio_seconds": 7.76,
|
||||
"post_speech_vad_delay_seconds": 0.6,
|
||||
"first_segment_asr_seconds": 0.6523482501506805,
|
||||
"all_segments_asr_seconds": 0.6523482501506805,
|
||||
"post_endpoint_response_seconds": 1.2523482501506806,
|
||||
"first_response_from_audio_start_seconds": 8.41234825015068,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家",
|
||||
"segment_transcripts": [
|
||||
"麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家"
|
||||
],
|
||||
"segment_asr_seconds": [
|
||||
0.6523482501506805
|
||||
],
|
||||
"segment_count": 1,
|
||||
"endpoints": [
|
||||
7.16
|
||||
],
|
||||
"decisions": [
|
||||
7.76
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.02857142857142857,
|
||||
"qwen2_event_evaluation": {
|
||||
"true_positive": [],
|
||||
"false_positive": [],
|
||||
"false_negative": [],
|
||||
"exact_match": true
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "pause",
|
||||
"audio": "validation/scenarios/long_pause.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"media": {
|
||||
"sha256": "7b4b9975d58c71c50cdb81ff9578a0b04192aeb3cbea634384a804ea4e3dc48d",
|
||||
"expected_acoustic_events": [
|
||||
"<|silence|>"
|
||||
]
|
||||
},
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 8.413955624680966,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 8.742833625059575,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 10.989260375034064,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还还是在三号会议室",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点还还是在三号会议室', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 8.0,
|
||||
"inference_seconds": 10.69584474992007,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 8.3875,
|
||||
"inference_seconds": 10.8182367910631,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': []}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"configured_silence_ms": 600,
|
||||
"first_speech_start_seconds": 0.02,
|
||||
"first_endpoint_audio_seconds": 4.04,
|
||||
"first_decision_audio_seconds": 4.64,
|
||||
"post_speech_vad_delay_seconds": 0.6,
|
||||
"first_segment_asr_seconds": 0.3871377082541585,
|
||||
"all_segments_asr_seconds": 0.7301125833764672,
|
||||
"post_endpoint_response_seconds": 0.9871377082541585,
|
||||
"first_response_from_audio_start_seconds": 5.027137708254158,
|
||||
"transcript": "麻煩你幫我把明天下午的會議改到兩點半地點 還是在三號會議時別忘了通知大家",
|
||||
"segment_transcripts": [
|
||||
"麻煩你幫我把明天下午的會議改到兩點半地點",
|
||||
"還是在三號會議時別忘了通知大家"
|
||||
],
|
||||
"segment_asr_seconds": [
|
||||
0.3871377082541585,
|
||||
0.34297487512230873
|
||||
],
|
||||
"segment_count": 2,
|
||||
"endpoints": [
|
||||
4.04,
|
||||
8.06
|
||||
],
|
||||
"decisions": [
|
||||
4.64,
|
||||
8.66
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.02857142857142857,
|
||||
"qwen2_event_evaluation": {
|
||||
"true_positive": [],
|
||||
"false_positive": [],
|
||||
"false_negative": [
|
||||
"<|silence|>"
|
||||
],
|
||||
"exact_match": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"scenario": "noise",
|
||||
"audio": "validation/scenarios/background_noise.wav",
|
||||
"reference": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"media": {
|
||||
"sha256": "034f4e9d4f5c2b69407e1e9813b4544792764a7e2e7340af410945b5c90c48b1",
|
||||
"expected_acoustic_events": [
|
||||
"<|noise|>"
|
||||
]
|
||||
},
|
||||
"qwen2_audio": [
|
||||
{
|
||||
"prefix_seconds": 2.0,
|
||||
"inference_seconds": 8.962713832966983,
|
||||
"transcript": "麻烦你帮我把明天下午的会议",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 4.0,
|
||||
"inference_seconds": 10.234860499855131,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半地点', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 6.0,
|
||||
"inference_seconds": 10.288799332920462,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。",
|
||||
"acoustic_events": [],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室。', 'acoustic_events': []}"
|
||||
},
|
||||
{
|
||||
"prefix_seconds": 7.4875,
|
||||
"inference_seconds": 11.268820082768798,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。",
|
||||
"acoustic_events": [
|
||||
"<|noise|>",
|
||||
"<|laughter|>",
|
||||
"<|cough|>"
|
||||
],
|
||||
"raw_response": "{'transcript': '麻烦你帮我把明天下午的会议改到两点半，地点还是在三号会议室，别忘了通知大家。', 'acoustic_events': ['noise', 'laughter', 'cough']}"
|
||||
}
|
||||
],
|
||||
"qwen2_final_cer": 0.0,
|
||||
"whisper_vad": {
|
||||
"configured_silence_ms": 600,
|
||||
"first_speech_start_seconds": 0.0,
|
||||
"first_endpoint_audio_seconds": 7.4875,
|
||||
"first_decision_audio_seconds": 8.0875,
|
||||
"post_speech_vad_delay_seconds": 0.6,
|
||||
"first_segment_asr_seconds": 0.5559374578297138,
|
||||
"all_segments_asr_seconds": 0.5559374578297138,
|
||||
"post_endpoint_response_seconds": 1.155937457829714,
|
||||
"first_response_from_audio_start_seconds": 8.643437457829714,
|
||||
"transcript": "麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家",
|
||||
"segment_transcripts": [
|
||||
"麻烦你帮我把明天下午的会议改到两点半地点还是在三号会议时别忘了通知大家"
|
||||
],
|
||||
"segment_asr_seconds": [
|
||||
0.5559374578297138
|
||||
],
|
||||
"segment_count": 1,
|
||||
"endpoints": [
|
||||
7.487
|
||||
],
|
||||
"decisions": [
|
||||
8.088
|
||||
]
|
||||
},
|
||||
"whisper_final_cer": 0.02857142857142857,
|
||||
"qwen2_event_evaluation": {
|
||||
"true_positive": [
|
||||
"<|noise|>"
|
||||
],
|
||||
"false_positive": [
|
||||
"<|cough|>",
|
||||
"<|laughter|>"
|
||||
],
|
||||
"false_negative": [],
|
||||
"exact_match": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"cost": {
|
||||
"paid_external_requests": 0,
|
||||
"total_usd": 0,
|
||||
"note": "Both models ran locally."
|
||||
},
|
||||
"acceptance": {
|
||||
"execution_gates": {
|
||||
"real_qwen2_audio": true,
|
||||
"growing_prefix_full_reencoding": true,
|
||||
"real_600ms_vad_whisper_baseline": true,
|
||||
"normal_pause_noise_scenarios": true,
|
||||
"corrected_vad_latency_accounting": true,
|
||||
"normalized_cer": true,
|
||||
"provenance_complete": true
|
||||
},
|
||||
"execution_passed": true,
|
||||
"manuscript_result_claims": {
|
||||
"qwen_incremental_latency_100_to_200ms": false,
|
||||
"traditional_post_endpoint_latency_800_to_1100ms": false,
|
||||
"pause_split_into_two_segments": true,
|
||||
"pause_specific_two_to_zero_error": false,
|
||||
"noise_token_detected": true,
|
||||
"noise_caused_earlier_vad_start": false
|
||||
},
|
||||
"manuscript_results_reproduced": false
|
||||
}
|
||||
}
|
||||
+79
@@ -0,0 +1,79 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "6-4",
|
||||
"run_id": "exp6-4-qwen2audio-whisper-provenance-20260730-v3",
|
||||
"generated_at": "2026-07-30T05:24:59.463Z",
|
||||
"git_commit": "f119cf510f3746cd5d071d0bce0051617f02f163",
|
||||
"runtime_source_sha256": {
|
||||
"run_official_experiment.py": "caef17357a38fb3d5215c9d3108a933da8f219751936c6a9bb70411655037a22",
|
||||
"demo.py": "b5f8169870d55627410975f601b8ac60afa25cd47dcf7e01376923c4dd503379",
|
||||
"qwen2_streaming.py": "cac5aad7b578c7033a1becc3abb91cfe094623d5a99a9ccc4362cc2b094b7ead",
|
||||
"whisper_baseline.py": "4547b92fe78001e0b093e43d0f51669a197fb0411bfea06a2e5a9f82e84bc245",
|
||||
"prepare_scenarios.py": "a0b01392d19a4c0463d33c7b56af323131cf7a42213410537109a6698f1e15b0"
|
||||
},
|
||||
"audio_sha256": {
|
||||
"audio/sentence.wav": "51b77857cb9d859ca37dc6ba95680fb675b8e2690257f00f69a46a11febbd4a2",
|
||||
"validation/scenarios/normal.wav": "22cbbff9df0ec0efc9a09b5d250b8d63c1876363716b8a1d7aa9f85f16e8a410",
|
||||
"validation/scenarios/long_pause.wav": "7b4b9975d58c71c50cdb81ff9578a0b04192aeb3cbea634384a804ea4e3dc48d",
|
||||
"validation/scenarios/background_noise.wav": "034f4e9d4f5c2b69407e1e9813b4544792764a7e2e7340af410945b5c90c48b1"
|
||||
},
|
||||
"whisper_checkpoint": {
|
||||
"path": "/Users/boj/.cache/whisper/tiny.pt",
|
||||
"size_bytes": 75572083,
|
||||
"sha256": "65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9"
|
||||
},
|
||||
"qwen2_audio_snapshot": {
|
||||
"repository": "mlx-community/Qwen2-Audio-7B-Instruct-4bit",
|
||||
"revision": "c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"path": "/Users/boj/.cache/huggingface/hub/models--mlx-community--Qwen2-Audio-7B-Instruct-4bit/snapshots/c65570002626f41b4dc08b7b54f42f99f3e82e7f",
|
||||
"files": {
|
||||
".gitattributes": {
|
||||
"size_bytes": 1519,
|
||||
"sha256": "11ad7efa24975ee4b0c3c3a38ed18737f0658a5f75a0a96787b576a78a023361"
|
||||
},
|
||||
"README.md": {
|
||||
"size_bytes": 1716,
|
||||
"sha256": "1cddc760431c9c2c043ed4d17d9c17716a1f81978e6093389420cd46e7f2ce6d"
|
||||
},
|
||||
"config.json": {
|
||||
"size_bytes": 913,
|
||||
"sha256": "39493dad1678533cc2a7ba2de0a573cc926a78ef4c7abef72a6b660159de94b7"
|
||||
},
|
||||
"merges.txt": {
|
||||
"size_bytes": 1671839,
|
||||
"sha256": "599bab54075088774b1733fde865d5bd747cbcc7a547c5bc12610e874e26f5e3"
|
||||
},
|
||||
"preprocessor_config.json": {
|
||||
"size_bytes": 342,
|
||||
"sha256": "4cd7c6c061fe79244c57b0c320b2873f3ee5acce2277f7cc3aced042725680f2"
|
||||
},
|
||||
"tokenizer.json": {
|
||||
"size_bytes": 7028015,
|
||||
"sha256": "f7c9b2dba4a296b1aa76c16a34b8225c0c118978400d4bb66bff0902d702f5b8"
|
||||
},
|
||||
"tokenizer_config.json": {
|
||||
"size_bytes": 638335,
|
||||
"sha256": "c738158e70eeecf25736a6c4d8e5f34cfce683079980757edfb0665a7b2457ed"
|
||||
},
|
||||
"vocab.json": {
|
||||
"size_bytes": 2776833,
|
||||
"sha256": "ca10d7e9fb3ed18575dd1e277a2579c16d108e32f27439684afa0e10b1440910"
|
||||
},
|
||||
"weights.safetensors": {
|
||||
"size_bytes": 6562540479,
|
||||
"sha256": "0967cde270ad62aa4824f0bbce283d0a2a6da2825ccf587640753c527d4174da"
|
||||
}
|
||||
}
|
||||
},
|
||||
"artifact_sha256": {
|
||||
"evidence.json": "17d5fa7cffbab6d2ae196008a4584e3f37d41cb6af541d0ffbcf184c3711c057",
|
||||
"run.log": "d1b29fffbf023ad11535277b209f11520318e2f73f1e7f2f2c6d3da8901bed2d",
|
||||
"acceptance.json": "ef593df76bb751ca311f4fdc50471210309e45b58aa4b1a5d52412823c40eefb"
|
||||
},
|
||||
"acceptance": {
|
||||
"execution_status": "pass",
|
||||
"passed_gates": 8,
|
||||
"total_gates": 8,
|
||||
"manuscript_results_reproduced": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
"""VAD + Whisper comparison baseline for Experiment 6-4."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
import librosa
|
||||
import soundfile as sf
|
||||
except ImportError:
|
||||
librosa = None
|
||||
sf = None
|
||||
import numpy as np
|
||||
|
||||
|
||||
@dataclass
|
||||
class BaselineResult:
|
||||
configured_silence_ms: int
|
||||
first_speech_start_seconds: float
|
||||
first_endpoint_audio_seconds: float
|
||||
first_decision_audio_seconds: float
|
||||
post_speech_vad_delay_seconds: float
|
||||
first_segment_asr_seconds: float
|
||||
all_segments_asr_seconds: float
|
||||
post_endpoint_response_seconds: float
|
||||
first_response_from_audio_start_seconds: float
|
||||
transcript: str
|
||||
segment_transcripts: list[str]
|
||||
segment_asr_seconds: list[float]
|
||||
segment_count: int
|
||||
endpoints: list[float]
|
||||
decisions: list[float]
|
||||
|
||||
|
||||
@dataclass
|
||||
class VadEvent:
|
||||
speech_start: int
|
||||
speech_endpoint: int
|
||||
decision: int
|
||||
|
||||
|
||||
def energy_vad_events(audio: np.ndarray, sr: int, silence_ms: int = 600) -> list[VadEvent]:
|
||||
"""Return start, acoustic endpoint, and later VAD decision sample.
|
||||
|
||||
The decision occurs only after the full low-energy run. Keeping it separate
|
||||
from the acoustic endpoint prevents a 4-second utterance position from being
|
||||
mislabeled as a 600 ms VAD latency.
|
||||
"""
|
||||
if len(audio) == 0:
|
||||
return []
|
||||
frame = max(1, int(sr * 0.02))
|
||||
energies = np.array([np.sqrt(np.mean(audio[i:i + frame] ** 2) + 1e-12) for i in range(0, len(audio), frame)])
|
||||
# A conservative fixed-relative threshold keeps a long silent gap distinct
|
||||
# even when speech occupies most of the clip (where p30 itself is speech).
|
||||
threshold = max(0.004, min(0.03, float(np.percentile(energies, 90) * 0.15)))
|
||||
silent_needed = max(1, silence_ms // 20)
|
||||
# A streaming endpoint detector observes silence after the physical file
|
||||
# ends too, so append an analysis-only silent tail. It is never transcribed.
|
||||
analysis_energies = np.concatenate([energies, np.zeros(silent_needed)])
|
||||
events, silent, speech_start = [], 0, None
|
||||
active = False
|
||||
for index, energy in enumerate(analysis_energies):
|
||||
if energy >= threshold:
|
||||
if not active:
|
||||
speech_start = index * frame
|
||||
active, silent = True, 0
|
||||
elif active:
|
||||
silent += 1
|
||||
if silent >= silent_needed:
|
||||
endpoint = min(len(audio), (index + 1 - silent_needed) * frame)
|
||||
decision = min(len(audio) + silent_needed * frame, (index + 1) * frame)
|
||||
events.append(VadEvent(int(speech_start or 0), endpoint, decision))
|
||||
active, silent, speech_start = False, 0, None
|
||||
return events
|
||||
|
||||
|
||||
def energy_vad_endpoints(audio: np.ndarray, sr: int, silence_ms: int = 600) -> list[int]:
|
||||
"""Compatibility helper returning only acoustic endpoint locations."""
|
||||
return [event.speech_endpoint for event in energy_vad_events(audio, sr, silence_ms)]
|
||||
|
||||
|
||||
class LocalWhisper:
|
||||
"""Actual open-source Whisper inference, loaded once for all VAD segments."""
|
||||
|
||||
def __init__(self, model: str = "small") -> None:
|
||||
import whisper
|
||||
|
||||
self.model_name = model
|
||||
self.model = whisper.load_model(model)
|
||||
|
||||
def transcribe(self, path: Path) -> str:
|
||||
return str(self.model.transcribe(str(path), language="zh", fp16=False)["text"]).strip()
|
||||
|
||||
|
||||
def run_whisper_baseline(audio_path: str | Path, transcriber) -> BaselineResult:
|
||||
audio, sr = librosa.load(str(audio_path), sr=None, mono=True)
|
||||
silence_ms = 600
|
||||
events = energy_vad_events(audio, sr, silence_ms)
|
||||
if not events:
|
||||
raise RuntimeError("VAD found no speech event")
|
||||
texts, asr_times, previous = [], [], 0
|
||||
temp_dir = Path(audio_path).parent / ".baseline_chunks"
|
||||
temp_dir.mkdir(exist_ok=True)
|
||||
try:
|
||||
for index, event in enumerate(events):
|
||||
chunk = temp_dir / f"chunk_{index}.wav"
|
||||
sf.write(chunk, audio[previous:event.speech_endpoint], sr)
|
||||
t0 = time.perf_counter()
|
||||
text = transcriber.transcribe(chunk)
|
||||
asr_times.append(time.perf_counter() - t0)
|
||||
texts.append(text)
|
||||
previous = event.speech_endpoint
|
||||
finally:
|
||||
for item in temp_dir.glob("*.wav"):
|
||||
item.unlink()
|
||||
temp_dir.rmdir()
|
||||
first = events[0]
|
||||
vad_delay = (first.decision - first.speech_endpoint) / sr
|
||||
return BaselineResult(
|
||||
configured_silence_ms=silence_ms,
|
||||
first_speech_start_seconds=first.speech_start / sr,
|
||||
first_endpoint_audio_seconds=first.speech_endpoint / sr,
|
||||
first_decision_audio_seconds=first.decision / sr,
|
||||
post_speech_vad_delay_seconds=vad_delay,
|
||||
first_segment_asr_seconds=asr_times[0],
|
||||
all_segments_asr_seconds=sum(asr_times),
|
||||
post_endpoint_response_seconds=vad_delay + asr_times[0],
|
||||
first_response_from_audio_start_seconds=first.decision / sr + asr_times[0],
|
||||
transcript=" ".join(t for t in texts if t),
|
||||
segment_transcripts=texts,
|
||||
segment_asr_seconds=asr_times,
|
||||
segment_count=len(events),
|
||||
endpoints=[round(event.speech_endpoint / sr, 3) for event in events],
|
||||
decisions=[round(event.decision / sr, 3) for event in events],
|
||||
)
|
||||
|
||||
|
||||
def serialize(result: BaselineResult):
|
||||
return asdict(result)
|
||||
Reference in New Issue
Block a user