ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,682 @@
|
||||
## English
|
||||
|
||||
# Live Voice Chat Demo
|
||||
|
||||
A real-time voice chat demo featuring speech-to-text, AI conversation, and text-to-speech capabilities. The application supports multiple AI service providers and provides a seamless conversational experience with minimal latency.
|
||||
|
||||
> This is the companion code for **实验 6-3「构建传统语音 Agent」** in 《深入理解 AI Agent》第 6 章. It implements the **cascaded** voice pipeline (VAD → ASR → LLM → TTS) discussed there: the frontend captures the microphone and streams audio over a WebSocket; the backend runs Silero VAD to detect end-of-speech (~500 ms of silence), then routes the utterance through pluggable ASR, LLM, and TTS providers and streams synthesized audio back for playback.
|
||||
|
||||
## Code map
|
||||
|
||||
- **Run first:** `node backend/check-setup.js`, then the browser demo with a single utterance.
|
||||
- **Start here:** backend/server.js owns the WebSocket session and media loop.
|
||||
- **Core behavior:** backend/utils/vad.js → speechToText.js → provider LLM → TTS; frontend audioWorklet.js supplies chunks.
|
||||
- **State / protocol:** WebSocket message/audio events and the per-utterance provider result.
|
||||
- **Verifier:** backend tests plus the validation evidence; record actual media/model hashes and latency.
|
||||
- **Experiment variable:** VAD endpointing, provider combination and streaming versus buffered response.
|
||||
- **Skip on first pass:** Next.js styling and provider-specific credential plumbing.
|
||||
|
||||
## Features
|
||||
|
||||
- 🎤 Real-time voice input with Voice Activity Detection (VAD)
|
||||
- 🤖 AI-powered conversations with **multiple provider support**
|
||||
- 🔊 Text-to-speech synthesis
|
||||
- ⚡ Low-latency audio streaming
|
||||
- 📊 Real-time latency monitoring and logging
|
||||
- 🎯 WebSocket-based communication
|
||||
- 🔧 **Flexible provider selection** for ASR, LLM, and TTS services
|
||||
|
||||
## Supported AI Providers
|
||||
|
||||
### ASR (Automatic Speech Recognition)
|
||||
- **OpenAI Whisper**: High accuracy, excellent language support
|
||||
- **SenseVoice** (via Siliconflow): Low latency, cost-effective, auto language detection
|
||||
|
||||
### LLM (Large Language Model)
|
||||
- **OpenAI GPT-4o**: Excellent reasoning, balanced performance
|
||||
- **OpenRouter GPT-4o**: No geographic restrictions, unified interface
|
||||
- **OpenRouter Gemini**: Fast response, optimized for real-time chat
|
||||
- **ARK Doubao**: Low latency in China, optimized for Chinese language
|
||||
|
||||
### TTS (Text-to-Speech)
|
||||
- **CosyVoice2** (via Siliconflow): Natural voice synthesis, multiple system voices
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
The system consists of a frontend-backend architecture with real-time audio processing and **pluggable provider architecture**:
|
||||
|
||||
### Frontend (Next.js)
|
||||
- **Audio Capture**: Uses Web Audio API to capture microphone input
|
||||
- **Audio Processing**: Client-side audio processing and streaming to backend
|
||||
- **WebSocket Communication**: Sends audio stream to backend and receives responses
|
||||
- **Audio Playback**: Plays back TTS audio responses from the backend
|
||||
|
||||
### Backend (Node.js)
|
||||
- **WebSocket Server**: Handles real-time audio streaming and client connections
|
||||
- **Voice Activity Detection**: Server-side Silero VAD processing to detect speech boundaries with high accuracy
|
||||
- **Multi-Provider Support**: Flexible ASR, LLM, and TTS provider integration
|
||||
- **Provider Factories**: Dynamic provider creation and switching capabilities
|
||||
|
||||
### Data Flow
|
||||
```
|
||||
User Speech → WebSocket → Backend VAD → Multi-Provider STT → Multi-Provider LLM → TTS → Audio Response
|
||||
```
|
||||
|
||||
### Ports
|
||||
|
||||
| Component | Port | Notes |
|
||||
|-----------|------|-------|
|
||||
| Backend (WebSocket server) | **8848** | Set by `LISTEN_PORT` in `backend/config.js`. The frontend connects to `ws://localhost:8848`. |
|
||||
| Frontend (Next.js dev server) | **3000** | Open http://localhost:3000 in the browser. |
|
||||
|
||||
The frontend learns the backend port from the `WEBSOCKET_PORT` environment variable (see `frontend/.env.example`). It must match the backend's `LISTEN_PORT`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Node.js (v16 or higher)
|
||||
- npm or yarn
|
||||
- **FFmpeg** - Required for audio processing and format conversion
|
||||
- **Google Chrome** (recommended) - Best performance and compatibility for real-time audio
|
||||
- Not recommended: Safari, Edge, or other browsers due to WebAudio API limitations
|
||||
- **API keys** from the supported providers (see Configuration section)
|
||||
|
||||
### Installing FFmpeg
|
||||
|
||||
#### macOS (using Homebrew)
|
||||
```bash
|
||||
brew install ffmpeg
|
||||
```
|
||||
|
||||
#### Ubuntu/Debian
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
#### Windows
|
||||
- Download from https://ffmpeg.org/download.html
|
||||
- Or use Chocolatey: `choco install ffmpeg`
|
||||
- Make sure `ffmpeg` is in your PATH
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
/backend
|
||||
- server.js: Main WebSocket server with provider integration
|
||||
- config.js: Multi-provider configuration settings
|
||||
- utils/
|
||||
- providers/
|
||||
- asrProviders.js: ASR provider implementations (OpenAI, Siliconflow)
|
||||
- llmProviders.js: LLM provider implementations (OpenAI, OpenRouter, ARK)
|
||||
- vad.js: Voice Activity Detection implementation
|
||||
- speechToText.js: Provider-aware STT service
|
||||
- textProcessor.js: Text preprocessing utilities
|
||||
- tests/
|
||||
- provider-tests.js: Comprehensive provider testing
|
||||
- run-tests.js: Test runner with environment validation
|
||||
- utils/providers/: Provider configuration (ASR / LLM / TTS)
|
||||
- package.json: Backend dependencies and scripts
|
||||
```
|
||||
|
||||
```
|
||||
/frontend
|
||||
- pages/: Next.js pages
|
||||
- index.tsx: Main application interface
|
||||
- components/: Reusable UI components
|
||||
- public/: Static assets
|
||||
- audioWorklet.js: Audio processing and VAD implementation
|
||||
- next.config.js: Next.js configuration
|
||||
- tailwind.config.js: Tailwind CSS settings
|
||||
- package.json: Frontend dependencies and scripts
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
1. Clone the repository
|
||||
2. Install backend dependencies:
|
||||
```bash
|
||||
cd backend && npm install
|
||||
```
|
||||
3. Install frontend dependencies:
|
||||
```bash
|
||||
cd frontend && npm install
|
||||
```
|
||||
4. Download the Silero VAD model (already included in this repo at `backend/models/silero_vad.onnx`; only needed if missing):
|
||||
```bash
|
||||
cd backend/models
|
||||
wget https://huggingface.co/deepghs/silero-vad-onnx/resolve/main/silero_vad.onnx
|
||||
```
|
||||
5. Configure the frontend's WebSocket port (defaults to 8848 if omitted):
|
||||
```bash
|
||||
cd frontend && cp .env.example .env # sets WEBSOCKET_PORT=8848 to match the backend
|
||||
```
|
||||
|
||||
After installing, verify your environment (Node version, FFmpeg, VAD model, provider keys) without needing a microphone or browser:
|
||||
|
||||
```bash
|
||||
cd backend && npm run check # or: node check-setup.js
|
||||
```
|
||||
|
||||
This prints which prerequisites are satisfied and which selected providers have their API keys set. It exits non-zero only if a hard prerequisite (Node < 16, missing FFmpeg, or missing VAD model) is absent.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Provider-Based Configuration
|
||||
|
||||
The system now supports **multiple AI service providers** for maximum flexibility. You can mix and match different providers for ASR, LLM, and TTS services.
|
||||
|
||||
### 1. Environment Variables Setup
|
||||
|
||||
Set up your API keys as environment variables:
|
||||
|
||||
```bash
|
||||
# Required for OpenAI services
|
||||
export OPENAI_API_KEY="your-openai-api-key"
|
||||
|
||||
# Required for OpenRouter services
|
||||
export OPENROUTER_API_KEY="your-openrouter-api-key"
|
||||
|
||||
# Required for ARK (Doubao) services
|
||||
export ARK_API_KEY="your-ark-api-key"
|
||||
|
||||
# Required for Siliconflow services (ASR and TTS)
|
||||
export SILICONFLOW_API_KEY="your-siliconflow-api-key"
|
||||
|
||||
# For future use
|
||||
export ANTHROPIC_API_KEY="your-anthropic-api-key"
|
||||
```
|
||||
|
||||
### 2. Provider Selection
|
||||
|
||||
1. This repo already ships a ready-to-edit `backend/config.js`. If it is missing (e.g. a fresh checkout that ignores it), copy the example first:
|
||||
```bash
|
||||
cp backend/config.js.example backend/config.js
|
||||
```
|
||||
|
||||
2. Edit `backend/config.js` to select your preferred providers:
|
||||
```javascript
|
||||
const config = {
|
||||
// Provider Selection - Choose your preferred providers
|
||||
ASR_PROVIDER: 'siliconflow', // 'openai' (whisper-1) or 'siliconflow' (SenseVoice)
|
||||
LLM_PROVIDER: 'openrouter', // 'openrouter' (gpt-5.6-luna, default), 'openai', 'openrouter-gemini', 'ark'
|
||||
TTS_PROVIDER: 'siliconflow', // 'siliconflow' (CosyVoice2)
|
||||
|
||||
// API Keys (loaded from environment variables)
|
||||
OPENAI_API_KEY: process.env.OPENAI_API_KEY,
|
||||
OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY,
|
||||
ARK_API_KEY: process.env.ARK_API_KEY,
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY,
|
||||
|
||||
// ... other configuration options
|
||||
};
|
||||
```
|
||||
|
||||
### 3. Recommended Provider Combinations
|
||||
|
||||
#### Default / Recommended (works anywhere with an OpenRouter key)
|
||||
```javascript
|
||||
ASR_PROVIDER: 'siliconflow', // SenseVoice
|
||||
LLM_PROVIDER: 'openrouter', // openai/gpt-5.6-luna via OpenRouter (avoids gpt-5.6* org verification)
|
||||
TTS_PROVIDER: 'siliconflow', // CosyVoice2
|
||||
```
|
||||
|
||||
#### For Real-time Performance (Low Latency in China)
|
||||
```javascript
|
||||
ASR_PROVIDER: 'siliconflow', // SenseVoice
|
||||
LLM_PROVIDER: 'ark', // Doubao (fast in China); or 'openrouter' for gpt-5.6-luna
|
||||
TTS_PROVIDER: 'siliconflow', // CosyVoice2
|
||||
```
|
||||
|
||||
#### For Best Accuracy
|
||||
```javascript
|
||||
ASR_PROVIDER: 'openai', // Accurate Whisper
|
||||
LLM_PROVIDER: 'openrouter', // openai/gpt-5.6-luna via OpenRouter
|
||||
TTS_PROVIDER: 'siliconflow' // CosyVoice2
|
||||
```
|
||||
|
||||
### 4. API Key Requirements
|
||||
|
||||
You only need the API keys for the providers you plan to use:
|
||||
|
||||
| Provider | ASR | LLM | TTS | Required API Key |
|
||||
|----------|-----|-----|-----|------------------|
|
||||
| OpenAI | ✅ Whisper | ✅ gpt-5.6-luna | ❌ | `OPENAI_API_KEY` |
|
||||
| OpenRouter | ❌ | ✅ gpt-5.6-luna, Gemini | ❌ | `OPENROUTER_API_KEY` |
|
||||
| ARK (Doubao) | ❌ | ✅ Doubao | ❌ | `ARK_API_KEY` |
|
||||
| Siliconflow | ✅ SenseVoice | ❌ | ✅ CosyVoice2 | `SILICONFLOW_API_KEY` |
|
||||
|
||||
### 5. Configuration Validation
|
||||
|
||||
The system includes comprehensive validation and testing tools:
|
||||
|
||||
```bash
|
||||
# Test all configured providers
|
||||
npm run test:providers
|
||||
|
||||
# Run the full test suite with environment validation
|
||||
node run-tests.js
|
||||
```
|
||||
|
||||
### Legacy Configuration Support
|
||||
|
||||
The system maintains backward compatibility with the previous hardcoded configuration format, but using the new provider selection is strongly recommended for better flexibility.
|
||||
|
||||
## Usage
|
||||
|
||||
1. **Set up your API keys** (see Configuration section)
|
||||
|
||||
2. **Configure your preferred providers** in `backend/config.js`
|
||||
|
||||
3. (Optional) **Verify your setup**: `cd backend && npm run check`
|
||||
|
||||
4. Start the backend server (WebSocket server on port **8848**):
|
||||
```bash
|
||||
cd backend && npm start
|
||||
```
|
||||
You should see `Server is running on 0.0.0.0:8848`.
|
||||
|
||||
5. Start the frontend development server (on port **3000**):
|
||||
```bash
|
||||
cd frontend && npm run dev
|
||||
```
|
||||
|
||||
6. Open http://localhost:3000 in your browser (Chrome recommended)
|
||||
|
||||
7. Click "Start Recording" and grant microphone permission to begin a conversation
|
||||
|
||||
**Expected behavior**: after you finish speaking, the backend detects ~500 ms of silence (VAD), transcribes your speech (ASR), streams an LLM reply, and synthesizes it back as audio (TTS) that plays automatically. The on-screen log panel shows per-stage latency (WebSocket RTT, transcription, LLM, TTS). If you start speaking again while the assistant is talking, playback is interrupted.
|
||||
|
||||
## Testing
|
||||
|
||||
### Provider Testing
|
||||
|
||||
Test individual providers and all combinations:
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
|
||||
# Test all providers with your API keys
|
||||
node run-tests.js
|
||||
|
||||
# Test specific providers only
|
||||
npm run test:providers
|
||||
|
||||
# Install test dependencies if needed
|
||||
npm install
|
||||
```
|
||||
|
||||
The test suite will automatically skip providers for which you don't have API keys configured.
|
||||
|
||||
### Test Coverage
|
||||
|
||||
- ✅ ASR provider functionality (OpenAI Whisper, SenseVoice)
|
||||
- ✅ LLM provider functionality (OpenAI, OpenRouter GPT-4o, OpenRouter Gemini, ARK Doubao)
|
||||
- ✅ TTS provider functionality (CosyVoice2 via Siliconflow)
|
||||
- ✅ All provider combinations (8 ASR+LLM combinations)
|
||||
- ✅ Dynamic provider switching
|
||||
- ✅ Error handling and fallback mechanisms
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
1. **Missing API Keys**: Ensure required environment variables are set
|
||||
2. **FFmpeg Not Found**: Ensure FFmpeg is installed and available in your system PATH
|
||||
- Test with: `ffmpeg -version`
|
||||
- If not found, refer to the FFmpeg installation instructions above
|
||||
3. **Network Issues**: Check connectivity to API endpoints
|
||||
4. **Rate Limiting**: Consider switching providers or implementing retry logic
|
||||
5. **Geographic Restrictions**: Use OpenRouter for global access
|
||||
6. **ONNX Runtime Issues**: The backend uses ONNX Runtime for voice activity detection
|
||||
- Usually resolved by the `onnxruntime-node` package automatically
|
||||
- On some systems, you may need additional system libraries
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
- **Low Latency**: Use Siliconflow ASR + OpenRouter Gemini
|
||||
- **High Accuracy**: Use OpenAI ASR + OpenAI LLM
|
||||
- **China Deployment**: Use Siliconflow ASR + ARK LLM
|
||||
|
||||
For provider configuration, see [`backend/config.js.example`](backend/config.js.example) and the provider implementations under [`backend/utils/providers/`](backend/utils/providers).
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
# 实时语音聊天演示
|
||||
|
||||
一个具备语音转文本、AI 对话和文本转语音能力的实时语音聊天演示。该应用支持多家 AI 服务提供商，以极低延迟提供流畅的对话体验。
|
||||
|
||||
> 这是《深入理解 AI Agent》第 6 章 **实验 6-3「构建传统语音 Agent」**的配套代码。它实现了书中讨论的**级联式**语音流水线（VAD → ASR → LLM → TTS）：前端采集麦克风音频并通过 WebSocket 以流的形式传输；后端运行 Silero VAD，通过约 500 ms 的静音检测语音结束，随后将话语依次交给可插拔的 ASR、LLM 和 TTS 提供商，并将合成音频流式返回播放。
|
||||
|
||||
## 功能特性
|
||||
|
||||
- 🎤 采用语音活动检测（VAD）的实时语音输入
|
||||
- 🤖 支持**多家提供商**的 AI 对话
|
||||
- 🔊 文本转语音合成
|
||||
- ⚡ 低延迟音频流
|
||||
- 📊 实时延迟监控与日志记录
|
||||
- 🎯 基于 WebSocket 的通信
|
||||
- 🔧 可灵活选择 ASR、LLM 和 TTS 服务提供商
|
||||
|
||||
## 支持的 AI 提供商
|
||||
|
||||
### ASR（自动语音识别）
|
||||
- **OpenAI Whisper**：准确率高，语言支持出色
|
||||
- **SenseVoice**（通过 Siliconflow）：低延迟、经济实惠、自动检测语言
|
||||
|
||||
### LLM（大语言模型）
|
||||
- **OpenAI GPT-4o**：推理能力出色、性能均衡
|
||||
- **OpenRouter GPT-4o**：无地域限制、统一接口
|
||||
- **OpenRouter Gemini**：响应迅速，针对实时聊天优化
|
||||
- **ARK Doubao**：在中国低延迟，针对中文优化
|
||||
|
||||
### TTS（文本转语音）
|
||||
- **CosyVoice2**（通过 Siliconflow）：自然语音合成，提供多种系统音色
|
||||
|
||||
## 架构概览
|
||||
|
||||
系统采用前后端架构，具备实时音频处理和**可插拔的提供商架构**：
|
||||
|
||||
### 前端（Next.js）
|
||||
- **音频采集**：使用 Web Audio API 采集麦克风输入
|
||||
- **音频处理**：在客户端处理音频并将其流式传输至后端
|
||||
- **WebSocket 通信**：向后端发送音频流并接收响应
|
||||
- **音频播放**：播放后端返回的 TTS 音频响应
|
||||
|
||||
### 后端（Node.js）
|
||||
- **WebSocket 服务器**：处理实时音频流和客户端连接
|
||||
- **语音活动检测**：在服务端运行 Silero VAD，以高准确率检测语音边界
|
||||
- **多提供商支持**：灵活集成 ASR、LLM 和 TTS 提供商
|
||||
- **提供商工厂**：支持动态创建和切换提供商
|
||||
|
||||
### 数据流
|
||||
```text
|
||||
用户语音 → WebSocket → 后端 VAD → 多提供商 STT → 多提供商 LLM → TTS → 音频响应
|
||||
```
|
||||
|
||||
### 端口
|
||||
|
||||
| 组件 | 端口 | 说明 |
|
||||
|-----------|------|-------|
|
||||
| 后端（WebSocket 服务器） | **8848** | 由 `backend/config.js` 中的 `LISTEN_PORT` 设置。前端连接到 `ws://localhost:8848`。 |
|
||||
| 前端（Next.js 开发服务器） | **3000** | 在浏览器中打开 http://localhost:3000。 |
|
||||
|
||||
前端从 `WEBSOCKET_PORT` 环境变量获取后端端口（参见 `frontend/.env.example`）。它必须与后端的 `LISTEN_PORT` 一致。
|
||||
|
||||
## 前置条件
|
||||
|
||||
- Node.js（v16 或更高版本）
|
||||
- npm 或 yarn
|
||||
- **FFmpeg**——音频处理和格式转换所必需
|
||||
- **Google Chrome**（推荐）——实时音频的性能和兼容性最佳
|
||||
- 不推荐：Safari、Edge 或其他浏览器，因为 WebAudio API 存在限制
|
||||
- 支持的提供商所需的 **API key**（参见“配置”一节）
|
||||
|
||||
### 安装 FFmpeg
|
||||
|
||||
#### macOS（使用 Homebrew）
|
||||
```bash
|
||||
brew install ffmpeg
|
||||
```
|
||||
|
||||
#### Ubuntu/Debian
|
||||
```bash
|
||||
sudo apt update
|
||||
sudo apt install ffmpeg
|
||||
```
|
||||
|
||||
#### Windows
|
||||
- 从 https://ffmpeg.org/download.html 下载
|
||||
- 或使用 Chocolatey：`choco install ffmpeg`
|
||||
- 确保 `ffmpeg` 位于 PATH 中
|
||||
|
||||
## 项目结构
|
||||
|
||||
```text
|
||||
/backend
|
||||
- server.js: 集成提供商的主 WebSocket 服务器
|
||||
- config.js: 多提供商配置设置
|
||||
- utils/
|
||||
- providers/
|
||||
- asrProviders.js: ASR 提供商实现（OpenAI、Siliconflow）
|
||||
- llmProviders.js: LLM 提供商实现（OpenAI、OpenRouter、ARK）
|
||||
- vad.js: 语音活动检测实现
|
||||
- speechToText.js: 感知提供商的 STT 服务
|
||||
- textProcessor.js: 文本预处理工具
|
||||
- tests/
|
||||
- provider-tests.js: 完整的提供商测试
|
||||
- run-tests.js: 带环境校验的测试运行器
|
||||
- utils/providers/: 提供商配置（ASR / LLM / TTS）
|
||||
- package.json: 后端依赖和脚本
|
||||
```
|
||||
|
||||
```text
|
||||
/frontend
|
||||
- pages/: Next.js 页面
|
||||
- index.tsx: 主应用界面
|
||||
- components/: 可复用 UI 组件
|
||||
- public/: 静态资源
|
||||
- audioWorklet.js: 音频处理与 VAD 实现
|
||||
- next.config.js: Next.js 配置
|
||||
- tailwind.config.js: Tailwind CSS 设置
|
||||
- package.json: 前端依赖和脚本
|
||||
```
|
||||
|
||||
## 安装
|
||||
|
||||
1. 克隆仓库
|
||||
2. 安装后端依赖：
|
||||
```bash
|
||||
cd backend && npm install
|
||||
```
|
||||
3. 安装前端依赖：
|
||||
```bash
|
||||
cd frontend && npm install
|
||||
```
|
||||
4. 下载 Silero VAD 模型（本仓库已在 `backend/models/silero_vad.onnx` 包含该文件；仅在文件缺失时需要）：
|
||||
```bash
|
||||
cd backend/models
|
||||
wget https://huggingface.co/deepghs/silero-vad-onnx/resolve/main/silero_vad.onnx
|
||||
```
|
||||
5. 配置前端的 WebSocket 端口（省略时默认为 8848）：
|
||||
```bash
|
||||
cd frontend && cp .env.example .env # 将 WEBSOCKET_PORT=8848 设为与后端一致
|
||||
```
|
||||
|
||||
安装完成后，无需麦克风或浏览器即可检查环境（Node 版本、FFmpeg、VAD 模型和提供商 key）：
|
||||
|
||||
```bash
|
||||
cd backend && npm run check # 或：node check-setup.js
|
||||
```
|
||||
|
||||
该命令会打印哪些前置条件已经满足，以及所选提供商是否已设置 API key。只有在缺少硬性前置条件（Node < 16、缺少 FFmpeg 或缺少 VAD 模型）时才会以非零状态退出。
|
||||
|
||||
## 配置
|
||||
|
||||
### 基于提供商的配置
|
||||
|
||||
系统现在支持**多家 AI 服务提供商**，以获得最大的灵活性。ASR、LLM 和 TTS 服务可以自由混合搭配不同提供商。
|
||||
|
||||
### 1. 设置环境变量
|
||||
|
||||
将 API key 设置为环境变量：
|
||||
|
||||
```bash
|
||||
# OpenAI 服务所必需
|
||||
export OPENAI_API_KEY="your-openai-api-key"
|
||||
|
||||
# OpenRouter 服务所必需
|
||||
export OPENROUTER_API_KEY="your-openrouter-api-key"
|
||||
|
||||
# ARK（Doubao）服务所必需
|
||||
export ARK_API_KEY="your-ark-api-key"
|
||||
|
||||
# Siliconflow 服务（ASR 和 TTS）所必需
|
||||
export SILICONFLOW_API_KEY="your-siliconflow-api-key"
|
||||
|
||||
# 留作将来使用
|
||||
export ANTHROPIC_API_KEY="your-anthropic-api-key"
|
||||
```
|
||||
|
||||
### 2. 选择提供商
|
||||
|
||||
1. 本仓库已经提供可直接编辑的 `backend/config.js`。如果该文件缺失（例如全新检出时被忽略），请先复制示例：
|
||||
```bash
|
||||
cp backend/config.js.example backend/config.js
|
||||
```
|
||||
|
||||
2. 编辑 `backend/config.js`，选择偏好的提供商：
|
||||
```javascript
|
||||
const config = {
|
||||
// 提供商选择——选择偏好的提供商
|
||||
ASR_PROVIDER: 'siliconflow', // 'openai'（whisper-1）或 'siliconflow'（SenseVoice）
|
||||
LLM_PROVIDER: 'openrouter', // 'openrouter'（gpt-5.6-luna，默认）、'openai'、'openrouter-gemini'、'ark'
|
||||
TTS_PROVIDER: 'siliconflow', // 'siliconflow'（CosyVoice2）
|
||||
|
||||
// API Key（从环境变量加载）
|
||||
OPENAI_API_KEY: process.env.OPENAI_API_KEY,
|
||||
OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY,
|
||||
ARK_API_KEY: process.env.ARK_API_KEY,
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY,
|
||||
|
||||
// ……其他配置选项
|
||||
};
|
||||
```
|
||||
|
||||
### 3. 推荐的提供商组合
|
||||
|
||||
#### 默认 / 推荐（只要有 OpenRouter key 即可在任何地方使用）
|
||||
```javascript
|
||||
ASR_PROVIDER: 'siliconflow', // SenseVoice
|
||||
LLM_PROVIDER: 'openrouter', // 通过 OpenRouter 使用 openai/gpt-5.6-luna（避免 gpt-5.6* 组织验证）
|
||||
TTS_PROVIDER: 'siliconflow', // CosyVoice2
|
||||
```
|
||||
|
||||
#### 实时性能优先（在中国低延迟）
|
||||
```javascript
|
||||
ASR_PROVIDER: 'siliconflow', // SenseVoice
|
||||
LLM_PROVIDER: 'ark', // Doubao（在中国速度快）；也可用 'openrouter' 运行 gpt-5.6-luna
|
||||
TTS_PROVIDER: 'siliconflow', // CosyVoice2
|
||||
```
|
||||
|
||||
#### 准确率优先
|
||||
```javascript
|
||||
ASR_PROVIDER: 'openai', // 高准确率的 Whisper
|
||||
LLM_PROVIDER: 'openrouter', // 通过 OpenRouter 使用 openai/gpt-5.6-luna
|
||||
TTS_PROVIDER: 'siliconflow' // CosyVoice2
|
||||
```
|
||||
|
||||
### 4. API Key 要求
|
||||
|
||||
只需配置计划使用的提供商所需的 API key：
|
||||
|
||||
| 提供商 | ASR | LLM | TTS | 所需 API Key |
|
||||
|----------|-----|-----|-----|------------------|
|
||||
| OpenAI | ✅ Whisper | ✅ gpt-5.6-luna | ❌ | `OPENAI_API_KEY` |
|
||||
| OpenRouter | ❌ | ✅ gpt-5.6-luna、Gemini | ❌ | `OPENROUTER_API_KEY` |
|
||||
| ARK（Doubao） | ❌ | ✅ Doubao | ❌ | `ARK_API_KEY` |
|
||||
| Siliconflow | ✅ SenseVoice | ❌ | ✅ CosyVoice2 | `SILICONFLOW_API_KEY` |
|
||||
|
||||
### 5. 配置校验
|
||||
|
||||
系统包含完整的校验与测试工具：
|
||||
|
||||
```bash
|
||||
# 测试所有已配置的提供商
|
||||
npm run test:providers
|
||||
|
||||
# 运行带环境校验的完整测试套件
|
||||
node run-tests.js
|
||||
```
|
||||
|
||||
### 旧版配置支持
|
||||
|
||||
系统继续向后兼容先前的硬编码配置格式，但强烈建议使用新的提供商选择机制，以获得更好的灵活性。
|
||||
|
||||
## 使用方法
|
||||
|
||||
1. **设置 API key**（参见“配置”一节）
|
||||
|
||||
2. 在 `backend/config.js` 中**配置偏好的提供商**
|
||||
|
||||
3. （可选）**验证配置**：`cd backend && npm run check`
|
||||
|
||||
4. 启动后端服务器（WebSocket 服务器使用端口 **8848**）：
|
||||
```bash
|
||||
cd backend && npm start
|
||||
```
|
||||
此时应看到 `Server is running on 0.0.0.0:8848`。
|
||||
|
||||
5. 启动前端开发服务器（使用端口 **3000**）：
|
||||
```bash
|
||||
cd frontend && npm run dev
|
||||
```
|
||||
此时应看到 `Server is running on 0.0.0.0:3000`。
|
||||
|
||||
6. 在浏览器中打开 http://localhost:3000（推荐 Chrome）
|
||||
|
||||
7. 点击“Start Recording”并授予麦克风权限，开始对话
|
||||
|
||||
**预期行为**：说话结束后，后端检测约 500 ms 的静音（VAD），转录语音（ASR），以流的形式生成 LLM 回复，再将其合成为自动播放的音频（TTS）。屏幕日志面板会显示各阶段延迟（WebSocket RTT、转录、LLM、TTS）。如果在助手说话时再次开口，播放会被打断。
|
||||
|
||||
## 测试
|
||||
|
||||
### 提供商测试
|
||||
|
||||
测试各个提供商和所有组合：
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
|
||||
# 使用 API key 测试所有提供商
|
||||
node run-tests.js
|
||||
|
||||
# 仅测试指定提供商
|
||||
npm run test:providers
|
||||
|
||||
# 如有需要，安装测试依赖
|
||||
npm install
|
||||
```
|
||||
|
||||
测试套件会自动跳过未配置 API key 的提供商。
|
||||
|
||||
### 测试覆盖范围
|
||||
|
||||
- ✅ ASR 提供商功能（OpenAI Whisper、SenseVoice）
|
||||
- ✅ LLM 提供商功能（OpenAI、OpenRouter GPT-4o、OpenRouter Gemini、ARK Doubao）
|
||||
- ✅ TTS 提供商功能（通过 Siliconflow 使用 CosyVoice2）
|
||||
- ✅ 所有提供商组合（8 种 ASR+LLM 组合）
|
||||
- ✅ 动态切换提供商
|
||||
- ✅ 错误处理和回退机制
|
||||
|
||||
## 故障排查
|
||||
|
||||
### 常见问题
|
||||
|
||||
1. **缺少 API Key**：确保已经设置所需的环境变量
|
||||
2. **找不到 FFmpeg**：确保 FFmpeg 已安装且位于系统 PATH 中
|
||||
- 使用 `ffmpeg -version` 测试
|
||||
- 如果找不到，请参阅上面的 FFmpeg 安装说明
|
||||
3. **网络问题**：检查与 API 端点的连通性
|
||||
4. **速率限制**：考虑切换提供商或实现重试逻辑
|
||||
5. **地域限制**：使用 OpenRouter 获得全球访问能力
|
||||
6. **ONNX Runtime 问题**：后端使用 ONNX Runtime 进行语音活动检测
|
||||
- 通常会由 `onnxruntime-node` 包自动解决
|
||||
- 在某些系统上，可能需要额外的系统库
|
||||
|
||||
### 性能优化
|
||||
|
||||
- **低延迟**：使用 Siliconflow ASR + OpenRouter Gemini
|
||||
- **高准确率**：使用 OpenAI ASR + OpenAI LLM
|
||||
- **中国部署**：使用 Siliconflow ASR + ARK LLM
|
||||
|
||||
提供商配置请参见 [`backend/config.js.example`](backend/config.js.example)，实现代码位于 [`backend/utils/providers/`](backend/utils/providers)。
|
||||
|
||||
## 许可证
|
||||
|
||||
MIT
|
||||
@@ -0,0 +1,111 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* check-setup.js — Headless environment verifier for the Live Voice Chat demo (实验 6-3).
|
||||
*
|
||||
* Runs WITHOUT any API keys, microphone, or browser. It only checks that the
|
||||
* local prerequisites for the cascaded VAD -> ASR -> LLM -> TTS pipeline are in
|
||||
* place, and reports which providers you have credentials for, so a reader can
|
||||
* confirm their setup before opening the browser UI.
|
||||
*
|
||||
* Usage:
|
||||
* node check-setup.js
|
||||
*
|
||||
* Exit code 0 means the backend can start; non-zero means a hard prerequisite
|
||||
* (Node version, VAD model, or a loadable config) is missing.
|
||||
*/
|
||||
|
||||
const { execFileSync } = require('child_process');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
let hardFailures = 0;
|
||||
let warnings = 0;
|
||||
|
||||
const ok = (msg) => console.log(` ✅ ${msg}`);
|
||||
const warn = (msg) => { console.log(` ⚠️ ${msg}`); warnings++; };
|
||||
const fail = (msg) => { console.log(` ❌ ${msg}`); hardFailures++; };
|
||||
|
||||
console.log('🔍 Live Voice Chat — setup check (实验 6-3)');
|
||||
console.log('='.repeat(60));
|
||||
|
||||
// 1. Node version (README requires v16+)
|
||||
console.log('\nNode.js runtime:');
|
||||
const major = parseInt(process.versions.node.split('.')[0], 10);
|
||||
if (major >= 16) {
|
||||
ok(`Node ${process.version} (>= v16 required)`);
|
||||
} else {
|
||||
fail(`Node ${process.version} is too old; v16 or higher is required`);
|
||||
}
|
||||
|
||||
// 2. FFmpeg (used by server.js to convert incoming audio for ASR)
|
||||
console.log('\nFFmpeg (audio format conversion):');
|
||||
try {
|
||||
const out = execFileSync('ffmpeg', ['-version'], { encoding: 'utf8' });
|
||||
ok(`ffmpeg found: ${out.split('\n')[0]}`);
|
||||
} catch (e) {
|
||||
fail('ffmpeg not found on PATH. Install it (e.g. `brew install ffmpeg`) — see README.');
|
||||
}
|
||||
|
||||
// 3. Silero VAD model file
|
||||
console.log('\nSilero VAD model:');
|
||||
const modelPath = path.join(__dirname, 'models', 'silero_vad.onnx');
|
||||
if (fs.existsSync(modelPath)) {
|
||||
const kb = (fs.statSync(modelPath).size / 1024).toFixed(0);
|
||||
ok(`silero_vad.onnx present (${kb} KB)`);
|
||||
} else {
|
||||
fail(`Missing ${modelPath}. Download it — see README "Download the Silero VAD model".`);
|
||||
}
|
||||
|
||||
// 4. Config loads, and report selected providers + key availability
|
||||
console.log('\nConfiguration & providers:');
|
||||
let config;
|
||||
try {
|
||||
config = require('./config');
|
||||
ok('config.js loaded');
|
||||
} catch (e) {
|
||||
fail(`config.js failed to load: ${e.message}`);
|
||||
}
|
||||
|
||||
if (config) {
|
||||
// Map each selected provider to the env var its credentials come from.
|
||||
const keyFor = {
|
||||
asr: (config.ASR_PROVIDERS[config.ASR_PROVIDER] || {}).apiKey,
|
||||
llm: (config.LLM_PROVIDERS[config.LLM_PROVIDER] || {}).apiKey,
|
||||
tts: (config.TTS_PROVIDERS[config.TTS_PROVIDER] || {}).apiKey,
|
||||
};
|
||||
|
||||
const stageLine = (stage, provider, envVar) => {
|
||||
if (!provider || !envVar) {
|
||||
fail(`${stage.toUpperCase()}: provider "${provider}" is not defined in config.js`);
|
||||
return;
|
||||
}
|
||||
const val = process.env[envVar];
|
||||
const placeholder = !val || /your-.*-api-key-here/.test(val);
|
||||
if (placeholder) {
|
||||
warn(`${stage.toUpperCase()}: provider "${provider}" selected, but ${envVar} is not set`);
|
||||
} else {
|
||||
ok(`${stage.toUpperCase()}: provider "${provider}" ready (${envVar} set)`);
|
||||
}
|
||||
};
|
||||
|
||||
stageLine('asr', config.ASR_PROVIDER, keyFor.asr);
|
||||
stageLine('llm', config.LLM_PROVIDER, keyFor.llm);
|
||||
stageLine('tts', config.TTS_PROVIDER, keyFor.tts);
|
||||
|
||||
console.log('\nServer will listen on:');
|
||||
ok(`ws://${config.LISTEN_HOST}:${config.LISTEN_PORT} (WebSocket) — frontend must use WEBSOCKET_PORT=${config.LISTEN_PORT}`);
|
||||
}
|
||||
|
||||
// Summary
|
||||
console.log('\n' + '='.repeat(60));
|
||||
if (hardFailures === 0 && warnings === 0) {
|
||||
console.log('✅ Setup looks good. Start the backend with: npm start');
|
||||
} else if (hardFailures === 0) {
|
||||
console.log(`⚠️ Prerequisites OK, but ${warnings} provider key(s) missing.`);
|
||||
console.log(' The backend will start; set the missing API key(s) before recording.');
|
||||
} else {
|
||||
console.log(`❌ ${hardFailures} hard prerequisite(s) missing — fix these before running the backend.`);
|
||||
}
|
||||
|
||||
process.exit(hardFailures === 0 ? 0 : 1);
|
||||
@@ -0,0 +1,99 @@
|
||||
const config = {
|
||||
// API Keys
|
||||
OPENAI_API_KEY: process.env.OPENAI_API_KEY || 'your-openai-api-key-here',
|
||||
OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY || 'your-openrouter-api-key-here',
|
||||
ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY || 'your-anthropic-api-key-here',
|
||||
ARK_API_KEY: process.env.ARK_API_KEY || 'your-ark-api-key-here',
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY || 'your-siliconflow-api-key-here',
|
||||
FISH_API_KEY: process.env.FISH_API_KEY || 'your-fish-api-key-here',
|
||||
|
||||
// Provider Selection
|
||||
ASR_PROVIDER: 'siliconflow', // 'openai' (whisper-1) or 'siliconflow' (SenseVoice)
|
||||
LLM_PROVIDER: 'openrouter', // 'openrouter' (gpt-5.6-luna, default), 'openai', 'openrouter-gemini', 'ark'
|
||||
TTS_PROVIDER: 'siliconflow', // 'siliconflow' (CosyVoice2, keep current)
|
||||
|
||||
// ASR Configuration
|
||||
ASR_PROVIDERS: {
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/audio/transcriptions',
|
||||
model: 'whisper-1',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
siliconflow: {
|
||||
apiUrl: 'https://api.siliconflow.cn/v1/audio/transcriptions',
|
||||
model: 'FunAudioLLM/SenseVoiceSmall',
|
||||
apiKey: 'SILICONFLOW_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
// LLM Configuration
|
||||
LLM_PROVIDERS: {
|
||||
// OpenRouter with a current cheap flagship chat model (default / recommended:
|
||||
// gpt-5.6* on OpenAI direct needs org verification, OpenRouter avoids that step).
|
||||
openrouter: {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'openai/gpt-5.6-luna',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/chat/completions',
|
||||
model: 'gpt-5.6-luna',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
'openrouter-gpt': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'openai/gpt-5.6-luna',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
'openrouter-gemini': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'google/gemini-3.5-flash',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
ark: {
|
||||
apiUrl: 'https://ark.cn-beijing.volces.com/api/v3/chat/completions',
|
||||
model: 'doubao-seed-1-6-flash-250615',
|
||||
apiKey: 'ARK_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
// TTS Configuration (keep current)
|
||||
TTS_PROVIDERS: {
|
||||
siliconflow: {
|
||||
apiUrl: 'https://api.siliconflow.cn/v1/audio/speech',
|
||||
model: 'FunAudioLLM/CosyVoice2-0.5B',
|
||||
voice: 'FunAudioLLM/CosyVoice2-0.5B:diana',
|
||||
apiKey: 'SILICONFLOW_API_KEY'
|
||||
},
|
||||
fish: {
|
||||
model: 's1',
|
||||
apiKey: 'FISH_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
// Legacy support (will be deprecated)
|
||||
LLM_MODEL: 'gpt-5.6-luna',
|
||||
LLM_API_URL: 'https://api.openai.com/v1/chat/completions',
|
||||
STT_API_URL: 'https://api.openai.com/v1/audio/transcriptions',
|
||||
STT_MODEL: 'whisper-1',
|
||||
TTS_API_URL: 'https://api.siliconflow.cn/v1/audio/speech',
|
||||
|
||||
// Common Configuration
|
||||
VISION_MAX_TOKENS: 4096,
|
||||
|
||||
// Silero VAD Configuration
|
||||
VAD_THRESHOLD: 0.5, // Speech probability threshold for Silero VAD (0.0 to 1.0)
|
||||
VAD_FRAME_LENGTH: 512, // Frame length for VAD analysis (samples)
|
||||
VAD_MIN_SPEECH_DURATION: 250, // Minimum speech duration in ms
|
||||
VAD_MAX_SILENCE_DURATION: 500, // Maximum silence duration before ending speech in ms
|
||||
AUDIO_SAMPLE_RATE: 16000, // Sample rate for audio processing (required for Silero VAD)
|
||||
AUDIO_CHUNK_SIZE: 4096, // Audio chunk size for processing
|
||||
|
||||
// Server Configuration
|
||||
LISTEN_PORT: 8848,
|
||||
LISTEN_HOST: '0.0.0.0',
|
||||
SYSTEM_PROMPT: 'You are a helpful AI assistant.',
|
||||
CANCEL_PLAYBACK_TIME_THRESHOLD: 3000,
|
||||
};
|
||||
|
||||
module.exports = config;
|
||||
@@ -0,0 +1,94 @@
|
||||
const config = {
|
||||
// API Keys
|
||||
OPENAI_API_KEY: process.env.OPENAI_API_KEY || 'your-openai-api-key-here',
|
||||
OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY || 'your-openrouter-api-key-here',
|
||||
ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY || 'your-anthropic-api-key-here',
|
||||
ARK_API_KEY: process.env.ARK_API_KEY || 'your-ark-api-key-here',
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY || 'your-siliconflow-api-key-here',
|
||||
|
||||
// Provider Selection
|
||||
ASR_PROVIDER: 'siliconflow', // 'openai' (whisper-1) or 'siliconflow' (SenseVoice)
|
||||
LLM_PROVIDER: 'openrouter', // 'openrouter' (gpt-5.6-luna, default), 'openai', 'openrouter-gemini', 'ark'
|
||||
TTS_PROVIDER: 'siliconflow', // 'siliconflow' (CosyVoice2, keep current)
|
||||
|
||||
// ASR Configuration
|
||||
ASR_PROVIDERS: {
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/audio/transcriptions',
|
||||
model: 'whisper-1',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
siliconflow: {
|
||||
apiUrl: 'https://api.siliconflow.cn/v1/audio/transcriptions',
|
||||
model: 'FunAudioLLM/SenseVoiceSmall',
|
||||
apiKey: 'SILICONFLOW_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
// LLM Configuration
|
||||
LLM_PROVIDERS: {
|
||||
// OpenRouter with a current cheap flagship chat model (default / recommended:
|
||||
// gpt-5.6* on OpenAI direct needs org verification, OpenRouter avoids that step).
|
||||
openrouter: {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'openai/gpt-5.6-luna',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/chat/completions',
|
||||
model: 'gpt-5.6-luna',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
'openrouter-gpt4o': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'openai/gpt-4o',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
'openrouter-gemini': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'google/gemini-2.5-flash',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
ark: {
|
||||
apiUrl: 'https://ark.cn-beijing.volces.com/api/v3/chat/completions',
|
||||
model: 'doubao-seed-1-6-flash-250615',
|
||||
apiKey: 'ARK_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
// TTS Configuration (keep current)
|
||||
TTS_PROVIDERS: {
|
||||
siliconflow: {
|
||||
apiUrl: 'https://api.siliconflow.cn/v1/audio/speech',
|
||||
model: 'FunAudioLLM/CosyVoice2-0.5B',
|
||||
voice: 'FunAudioLLM/CosyVoice2-0.5B:diana',
|
||||
apiKey: 'SILICONFLOW_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
// Legacy support (will be deprecated)
|
||||
LLM_MODEL: 'gpt-5.6-luna',
|
||||
LLM_API_URL: 'https://api.openai.com/v1/chat/completions',
|
||||
STT_API_URL: 'https://api.openai.com/v1/audio/transcriptions',
|
||||
STT_MODEL: 'whisper-1',
|
||||
TTS_API_URL: 'https://api.siliconflow.cn/v1/audio/speech',
|
||||
|
||||
// Common Configuration
|
||||
VISION_MAX_TOKENS: 4096,
|
||||
|
||||
// Silero VAD Configuration
|
||||
VAD_THRESHOLD: 0.5, // Speech probability threshold for Silero VAD (0.0 to 1.0)
|
||||
VAD_FRAME_LENGTH: 512, // Frame length for VAD analysis (samples)
|
||||
VAD_MIN_SPEECH_DURATION: 250, // Minimum speech duration in ms
|
||||
VAD_MAX_SILENCE_DURATION: 500, // Maximum silence duration before ending speech in ms
|
||||
AUDIO_SAMPLE_RATE: 16000, // Sample rate for audio processing (required for Silero VAD)
|
||||
AUDIO_CHUNK_SIZE: 4096, // Audio chunk size for processing
|
||||
|
||||
// Server Configuration
|
||||
LISTEN_PORT: 8848,
|
||||
LISTEN_HOST: '0.0.0.0',
|
||||
SYSTEM_PROMPT: 'You are a helpful AI assistant.',
|
||||
CANCEL_PLAYBACK_TIME_THRESHOLD: 3000,
|
||||
};
|
||||
|
||||
module.exports = config;
|
||||
@@ -0,0 +1,90 @@
|
||||
const crypto = require('crypto');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
function sha256(filePath) {
|
||||
return crypto.createHash('sha256').update(fs.readFileSync(filePath)).digest('hex');
|
||||
}
|
||||
|
||||
function isRealStage(stage) {
|
||||
return Boolean(
|
||||
stage &&
|
||||
stage.execution === 'real' &&
|
||||
stage.mock !== true &&
|
||||
stage.probe_only !== true &&
|
||||
stage.fallback_used !== true
|
||||
);
|
||||
}
|
||||
|
||||
function validateExperimentEvidence(evidence, evidenceDir) {
|
||||
const source = evidence?.source_media || {};
|
||||
const vad = evidence?.stages?.vad || {};
|
||||
const asr = evidence?.stages?.asr || {};
|
||||
const llm = evidence?.stages?.llm || {};
|
||||
const tts = evidence?.stages?.tts || {};
|
||||
|
||||
const sourcePath = source.path ? path.resolve(evidenceDir, source.path) : '';
|
||||
const segmentPath = vad.segment_path ? path.resolve(evidenceDir, vad.segment_path) : '';
|
||||
const outputPath = tts.output_path ? path.resolve(evidenceDir, tts.output_path) : '';
|
||||
const sourceAuthentic = Boolean(
|
||||
source.capture_method === 'browser_microphone_over_websocket' &&
|
||||
sourcePath && fs.existsSync(sourcePath) &&
|
||||
source.sha256 === sha256(sourcePath) &&
|
||||
source.sample_rate_hz === 16000 &&
|
||||
source.channels === 1 &&
|
||||
source.bits_per_sample === 16
|
||||
);
|
||||
const segmentAuthentic = Boolean(
|
||||
segmentPath && fs.existsSync(segmentPath) && vad.segment_sha256 === sha256(segmentPath)
|
||||
);
|
||||
const outputAuthentic = Boolean(
|
||||
outputPath && fs.existsSync(outputPath) && tts.output_sha256 === sha256(outputPath) &&
|
||||
Number(tts.output_bytes) > 1000 && Number(tts.output_duration_seconds) > 0
|
||||
);
|
||||
|
||||
const gates = {
|
||||
schema_and_scope: evidence?.schema_version === 1 && evidence?.experiment === '6-3',
|
||||
real_websocket_microphone_media: sourceAuthentic,
|
||||
real_silero_vad_endpoint: Boolean(
|
||||
isRealStage(vad) &&
|
||||
vad.implementation === 'Silero VAD ONNX' &&
|
||||
vad.model_sha256 &&
|
||||
vad.endpoint_detected === true &&
|
||||
vad.forced_endpoint === false &&
|
||||
Number(vad.max_silence_ms) === 500 &&
|
||||
Number(vad.observed_trailing_silence_ms) >= 500 &&
|
||||
segmentAuthentic
|
||||
),
|
||||
real_asr: Boolean(
|
||||
isRealStage(asr) && asr.inference_completed === true &&
|
||||
asr.provider && asr.model && String(asr.transcript || '').trim()
|
||||
),
|
||||
real_streaming_llm: Boolean(
|
||||
isRealStage(llm) && llm.api_request_completed === true && llm.streamed === true &&
|
||||
llm.provider && llm.model && Number(llm.first_token_seconds) > 0 &&
|
||||
String(llm.response || '').trim()
|
||||
),
|
||||
real_tts_media: Boolean(
|
||||
isRealStage(tts) && tts.api_request_completed === true &&
|
||||
tts.provider && tts.model && Number(tts.first_audio_byte_seconds) > 0 && outputAuthentic
|
||||
),
|
||||
measured_stage_latencies: ['vad', 'asr', 'llm', 'tts'].every(
|
||||
name => Number(evidence?.stages?.[name]?.latency_seconds) > 0
|
||||
),
|
||||
provenance_complete: Boolean(
|
||||
evidence?.provenance?.host?.platform &&
|
||||
evidence?.provenance?.host?.architecture &&
|
||||
evidence?.provenance?.runtime?.node &&
|
||||
evidence?.provenance?.runtime?.onnxruntime_node &&
|
||||
source.original_sha256 === source.sha256
|
||||
),
|
||||
no_mock_probe_or_fallback: [vad, asr, llm, tts].every(isRealStage),
|
||||
};
|
||||
return {
|
||||
gates,
|
||||
passed: Object.values(gates).every(Boolean),
|
||||
statement: 'Passing proves one saved real microphone turn completed Silero VAD -> real ASR -> real LLM -> real TTS. It does not benchmark concurrency or production load.',
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = { sha256, validateExperimentEvidence };
|
||||
+2231
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "livechat-backend",
|
||||
"version": "1.0.0",
|
||||
"main": "server.js",
|
||||
"scripts": {
|
||||
"test": "mocha tests/*.js --timeout 120000",
|
||||
"test:providers": "mocha tests/provider-tests.js --timeout 120000",
|
||||
"test:asr": "mocha tests/test-asr-providers.js --timeout 120000",
|
||||
"test:llm": "mocha tests/test-llm-providers.js --timeout 120000",
|
||||
"test:tts": "mocha tests/test-tts-providers.js --timeout 120000",
|
||||
"check": "node check-setup.js",
|
||||
"start": "node server.js"
|
||||
},
|
||||
"keywords": [],
|
||||
"author": "",
|
||||
"license": "ISC",
|
||||
"description": "",
|
||||
"dependencies": {
|
||||
"axios": "^1.7.5",
|
||||
"emoji-regex": "^10.4.0",
|
||||
"express": "^4.21.1",
|
||||
"fluent-ffmpeg": "^2.1.3",
|
||||
"form-data": "^4.0.0",
|
||||
"franc": "^6.2.0",
|
||||
"node-wav": "^0.0.2",
|
||||
"onnxruntime-node": "^1.22.0-rev",
|
||||
"wav": "^1.0.2",
|
||||
"ws": "^8.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"mocha": "^10.2.0"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Test runner for provider tests
|
||||
* This script sets up the environment and runs comprehensive tests for all provider combinations
|
||||
*
|
||||
* Usage:
|
||||
* node run-tests.js
|
||||
*
|
||||
* Environment variables required:
|
||||
* - OPENROUTER_API_KEY: OpenRouter API key
|
||||
* - ANTHROPIC_API_KEY: Anthropic API key
|
||||
* - ARK_API_KEY: ARK (Doubao) API key
|
||||
* - SILICONFLOW_API_KEY: Siliconflow API key
|
||||
* - OPENAI_API_KEY: OpenAI API key (optional if using others)
|
||||
*/
|
||||
|
||||
const { spawn } = require('child_process');
|
||||
const path = require('path');
|
||||
|
||||
console.log('🧪 Starting Provider Tests for Live Audio Backend');
|
||||
console.log('=' .repeat(60));
|
||||
|
||||
// Check environment variables
|
||||
const requiredEnvVars = {
|
||||
'OPENROUTER_API_KEY': 'OpenRouter API key',
|
||||
'ANTHROPIC_API_KEY': 'Anthropic API key',
|
||||
'ARK_API_KEY': 'ARK (Doubao) API key',
|
||||
'SILICONFLOW_API_KEY': 'Siliconflow API key'
|
||||
};
|
||||
|
||||
const missingKeys = [];
|
||||
const availableKeys = [];
|
||||
|
||||
Object.entries(requiredEnvVars).forEach(([key, description]) => {
|
||||
if (process.env[key]) {
|
||||
availableKeys.push(`✅ ${description}`);
|
||||
} else {
|
||||
missingKeys.push(`❌ ${description} (${key})`);
|
||||
}
|
||||
});
|
||||
|
||||
console.log('\n🔑 API Key Status:');
|
||||
availableKeys.forEach(key => console.log(` ${key}`));
|
||||
missingKeys.forEach(key => console.log(` ${key}`));
|
||||
|
||||
if (process.env.OPENAI_API_KEY) {
|
||||
console.log(` ✅ OpenAI API key (optional)`);
|
||||
}
|
||||
|
||||
console.log('\n📋 Test Plan:');
|
||||
console.log(' 1. ASR Provider Tests (OpenAI Whisper, SenseVoice)');
|
||||
console.log(' 2. LLM Provider Tests (OpenAI, OpenRouter GPT-4o, OpenRouter Gemini, ARK Doubao)');
|
||||
console.log(' 3. Integration Tests (All ASR+LLM combinations)');
|
||||
console.log(' 4. Provider Switching Tests');
|
||||
|
||||
console.log('\n🚀 Running Tests...\n');
|
||||
|
||||
// Run the tests
|
||||
const testProcess = spawn('npm', ['run', 'test:providers'], {
|
||||
stdio: 'inherit',
|
||||
cwd: __dirname,
|
||||
env: process.env
|
||||
});
|
||||
|
||||
testProcess.on('close', (code) => {
|
||||
console.log('\n' + '='.repeat(60));
|
||||
if (code === 0) {
|
||||
console.log('✅ All tests completed successfully!');
|
||||
console.log('\n📊 Test Summary:');
|
||||
console.log(' - Provider creation and configuration ✓');
|
||||
console.log(' - ASR transcription functionality ✓');
|
||||
console.log(' - LLM chat completion functionality ✓');
|
||||
console.log(' - Provider integration ✓');
|
||||
console.log(' - Dynamic provider switching ✓');
|
||||
} else {
|
||||
console.log(`❌ Tests failed with exit code ${code}`);
|
||||
console.log('\n🔧 Troubleshooting:');
|
||||
console.log(' 1. Ensure all required API keys are set as environment variables');
|
||||
console.log(' 2. Check network connectivity to API endpoints');
|
||||
console.log(' 3. Verify API key permissions and quotas');
|
||||
console.log(' 4. Check the test output above for specific error details');
|
||||
}
|
||||
|
||||
process.exit(code);
|
||||
});
|
||||
|
||||
testProcess.on('error', (error) => {
|
||||
console.error('❌ Failed to start test process:', error);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,441 @@
|
||||
#!/usr/bin/env node
|
||||
const axios = require('axios');
|
||||
const crypto = require('crypto');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
const { execFileSync } = require('child_process');
|
||||
|
||||
const config = require('./config');
|
||||
const VoiceActivityDetector = require('./utils/vad');
|
||||
const { ASRProviderFactory } = require('./utils/providers/asrProviders');
|
||||
const { LLMProviderFactory } = require('./utils/providers/llmProviders');
|
||||
const { sha256, validateExperimentEvidence } = require('./experiment_validation');
|
||||
|
||||
function arg(name, fallback) {
|
||||
const index = process.argv.indexOf(name);
|
||||
return index >= 0 ? process.argv[index + 1] : fallback;
|
||||
}
|
||||
|
||||
function requireCredential(name) {
|
||||
const value = process.env[name];
|
||||
if (!value || value.startsWith('your-')) throw new Error(`Required real credential is not set: ${name}`);
|
||||
return value;
|
||||
}
|
||||
|
||||
function redact(value) {
|
||||
let text = String(value || '');
|
||||
for (const [name, secret] of Object.entries(process.env)) {
|
||||
if ((name.includes('KEY') || name.includes('TOKEN')) && secret) text = text.split(secret).join('[REDACTED]');
|
||||
}
|
||||
return text.replace(/\b(?:sk|ak)-[A-Za-z0-9_-]{12,}\b/g, '[REDACTED]').slice(0, 3000);
|
||||
}
|
||||
|
||||
function parsePcmWav(filePath) {
|
||||
const raw = fs.readFileSync(filePath);
|
||||
if (raw.length < 44 || raw.subarray(0, 4).toString() !== 'RIFF' || raw.subarray(8, 12).toString() !== 'WAVE') {
|
||||
throw new Error('Input must be a PCM WAV file');
|
||||
}
|
||||
const format = raw.readUInt16LE(20);
|
||||
const channels = raw.readUInt16LE(22);
|
||||
const sampleRate = raw.readUInt32LE(24);
|
||||
const bits = raw.readUInt16LE(34);
|
||||
if (format !== 1 || channels !== 1 || sampleRate !== 16000 || bits !== 16) {
|
||||
throw new Error(`Expected PCM 16kHz mono 16-bit WAV; got format=${format}, channels=${channels}, sampleRate=${sampleRate}, bits=${bits}`);
|
||||
}
|
||||
return { raw, pcm: raw.subarray(44), channels, sampleRate, bits };
|
||||
}
|
||||
|
||||
function wavBuffer(pcm, sampleRate = 16000) {
|
||||
const header = Buffer.alloc(44);
|
||||
header.write('RIFF', 0); header.writeUInt32LE(36 + pcm.length, 4); header.write('WAVE', 8);
|
||||
header.write('fmt ', 12); header.writeUInt32LE(16, 16); header.writeUInt16LE(1, 20);
|
||||
header.writeUInt16LE(1, 22); header.writeUInt32LE(sampleRate, 24);
|
||||
header.writeUInt32LE(sampleRate * 2, 28); header.writeUInt16LE(2, 32); header.writeUInt16LE(16, 34);
|
||||
header.write('data', 36); header.writeUInt32LE(pcm.length, 40);
|
||||
return Buffer.concat([header, pcm]);
|
||||
}
|
||||
|
||||
function mediaProbe(filePath) {
|
||||
const output = execFileSync('ffprobe', [
|
||||
'-v', 'error', '-show_entries', 'format=duration,size,format_name', '-of', 'json', filePath,
|
||||
], { encoding: 'utf8' });
|
||||
const format = JSON.parse(output).format || {};
|
||||
return {
|
||||
duration_seconds: Number(format.duration),
|
||||
size_bytes: Number(format.size),
|
||||
format_name: format.format_name,
|
||||
};
|
||||
}
|
||||
|
||||
function commandVersion(command, args) {
|
||||
try {
|
||||
return execFileSync(command, args, { encoding: 'utf8' }).split('\n')[0].trim();
|
||||
} catch (_) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function resolveWhisperPython(requested) {
|
||||
if (requested) return requested;
|
||||
const launcher = execFileSync('which', ['whisper'], { encoding: 'utf8' }).trim();
|
||||
const firstLine = fs.readFileSync(launcher, 'utf8').split('\n')[0];
|
||||
if (!firstLine.startsWith('#!')) throw new Error(`Cannot resolve Python from Whisper launcher: ${launcher}`);
|
||||
return firstLine.slice(2).trim();
|
||||
}
|
||||
|
||||
function buildProvenance() {
|
||||
const dependencies = require('./package.json').dependencies || {};
|
||||
return {
|
||||
host: {
|
||||
platform: os.platform(),
|
||||
release: os.release(),
|
||||
architecture: os.arch(),
|
||||
cpu_model: os.cpus()[0]?.model || 'unknown',
|
||||
logical_cpu_count: os.cpus().length,
|
||||
total_memory_bytes: os.totalmem(),
|
||||
},
|
||||
runtime: {
|
||||
node: process.version,
|
||||
onnxruntime_node: dependencies['onnxruntime-node'] || null,
|
||||
axios: dependencies.axios || null,
|
||||
ffprobe: commandVersion('ffprobe', ['-version']),
|
||||
},
|
||||
timing_clock: 'process.hrtime.bigint monotonic clock',
|
||||
};
|
||||
}
|
||||
|
||||
const sleep = ms => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
async function runVad(pcm, outputDir) {
|
||||
const detector = new VoiceActivityDetector({ maxSilenceDuration: 500 });
|
||||
await detector.initializationPromise;
|
||||
const started = process.hrtime.bigint();
|
||||
const events = [];
|
||||
const frameBytes = 512 * 2;
|
||||
try {
|
||||
for (let offset = 0; offset + frameBytes <= pcm.length; offset += frameBytes) {
|
||||
const frame = pcm.subarray(offset, offset + frameBytes);
|
||||
events.push(...await detector.processAudioChunk(frame));
|
||||
await sleep(32);
|
||||
}
|
||||
} finally {
|
||||
await detector.cleanup();
|
||||
}
|
||||
const endpoints = events.filter(event => event.type === 'speech_end');
|
||||
const ended = endpoints.sort((left, right) => right.audioData.length - left.audioData.length)[0];
|
||||
if (!ended) throw new Error('Silero did not observe a non-forced endpoint after 500 ms silence');
|
||||
const segmentPath = path.join(outputDir, 'vad_segment.wav');
|
||||
fs.writeFileSync(segmentPath, wavBuffer(ended.audioData));
|
||||
return {
|
||||
execution: 'real', mock: false, probe_only: false, fallback_used: false,
|
||||
implementation: 'Silero VAD ONNX',
|
||||
model: 'models/silero_vad.onnx',
|
||||
model_sha256: sha256(path.join(__dirname, 'models', 'silero_vad.onnx')),
|
||||
threshold: 0.5,
|
||||
max_silence_ms: 500,
|
||||
endpoint_detected: true,
|
||||
detected_endpoint_count: endpoints.length,
|
||||
selected_endpoint: 'longest detected speech segment',
|
||||
forced_endpoint: false,
|
||||
observed_trailing_silence_ms: ended.observedSilenceDuration,
|
||||
speech_duration_ms: ended.duration,
|
||||
latency_seconds: Number(process.hrtime.bigint() - started) / 1e9,
|
||||
segment_path: 'vad_segment.wav',
|
||||
segment_sha256: sha256(segmentPath),
|
||||
segment_bytes: fs.statSync(segmentPath).size,
|
||||
};
|
||||
}
|
||||
|
||||
function runLocalWhisper(segmentPath, requestedPython, modelName) {
|
||||
const python = resolveWhisperPython(requestedPython);
|
||||
const script = [
|
||||
'import hashlib, json, pathlib, sys, time',
|
||||
'import torch, whisper',
|
||||
'audio, model_name = sys.argv[1], sys.argv[2]',
|
||||
'cache = pathlib.Path.home() / ".cache" / "whisper" / (model_name + ".pt")',
|
||||
'started = time.perf_counter()',
|
||||
'model = whisper.load_model(model_name)',
|
||||
'loaded = time.perf_counter()',
|
||||
'result = model.transcribe(audio, fp16=False, verbose=False)',
|
||||
'finished = time.perf_counter()',
|
||||
'payload = {"text": str(result.get("text") or "").strip(), "language": result.get("language") or "unknown",',
|
||||
' "model_load_seconds": loaded-started, "inference_seconds": finished-loaded,',
|
||||
' "python": sys.version.split()[0], "torch": torch.__version__, "whisper": getattr(whisper, "__version__", "unknown"),',
|
||||
' "model_path": str(cache), "model_sha256": hashlib.sha256(cache.read_bytes()).hexdigest() if cache.exists() else None}',
|
||||
'print("EXPERIMENT_JSON=" + json.dumps(payload, ensure_ascii=False))',
|
||||
].join('\n');
|
||||
const started = process.hrtime.bigint();
|
||||
const output = execFileSync(python, ['-c', script, segmentPath, modelName], {
|
||||
encoding: 'utf8', maxBuffer: 10 * 1024 * 1024,
|
||||
});
|
||||
const marker = output.split('\n').find(line => line.startsWith('EXPERIMENT_JSON='));
|
||||
if (!marker) throw new Error('Local Whisper returned no structured result');
|
||||
const result = JSON.parse(marker.slice('EXPERIMENT_JSON='.length));
|
||||
if (!result.text) throw new Error('Local Whisper returned an empty transcript');
|
||||
return {
|
||||
execution: 'real', mock: false, probe_only: false, fallback_used: false,
|
||||
provider: 'local-openai-whisper', model: `whisper-${modelName}`,
|
||||
inference_completed: true, api_request_completed: false, external_request: false,
|
||||
latency_seconds: Number(process.hrtime.bigint() - started) / 1e9,
|
||||
transcript: result.text, language: result.language,
|
||||
runtime: { python: result.python, torch: result.torch, openai_whisper: result.whisper },
|
||||
model_path: result.model_path, model_sha256: result.model_sha256,
|
||||
model_load_seconds: result.model_load_seconds,
|
||||
model_inference_seconds: result.inference_seconds,
|
||||
provider_reported_cost_usd: 0,
|
||||
cost_note: 'Local open-source Whisper inference; no external ASR charge.',
|
||||
};
|
||||
}
|
||||
|
||||
async function runAsr(segmentPath, providerName, evidenceDir, options = {}) {
|
||||
if (providerName === 'local-whisper') {
|
||||
return runLocalWhisper(segmentPath, options.whisperPython, options.whisperModel || 'tiny');
|
||||
}
|
||||
const providerConfig = config.ASR_PROVIDERS[providerName];
|
||||
if (!providerConfig) throw new Error(`Unknown ASR provider: ${providerName}`);
|
||||
requireCredential(providerConfig.apiKey);
|
||||
const provider = ASRProviderFactory.createProvider(providerName, config, config);
|
||||
const pcm = parsePcmWav(segmentPath).pcm;
|
||||
const tempDir = path.join(evidenceDir, '.asr-temp');
|
||||
fs.mkdirSync(tempDir, { recursive: true });
|
||||
const started = process.hrtime.bigint();
|
||||
const result = await provider.transcribe(pcm, tempDir, {});
|
||||
fs.rmSync(tempDir, { recursive: true, force: true });
|
||||
if (!result.success || !String(result.text || '').trim()) throw new Error(`Real ASR failed: ${result.error || 'empty transcript'}`);
|
||||
return {
|
||||
execution: 'real', mock: false, probe_only: false, fallback_used: false,
|
||||
provider: providerName,
|
||||
model: providerConfig.model,
|
||||
inference_completed: true, api_request_completed: true, external_request: true,
|
||||
latency_seconds: Number(process.hrtime.bigint() - started) / 1e9,
|
||||
transcript: String(result.text).trim(),
|
||||
language: result.language || 'unknown',
|
||||
provider_request_id: result.requestId || null,
|
||||
provider_response_model: result.responseModel || providerConfig.model,
|
||||
billed_input_audio_seconds: mediaProbe(segmentPath).duration_seconds,
|
||||
provider_reported_cost_usd: null,
|
||||
cost_note: 'The transcription response did not expose a monetary charge; consult the provider billing ledger.',
|
||||
};
|
||||
}
|
||||
|
||||
async function runLlm(transcript, providerName, requestedModel) {
|
||||
if (!config.LLM_PROVIDERS[providerName]) throw new Error(`Unknown LLM provider: ${providerName}`);
|
||||
const providerConfig = { ...config.LLM_PROVIDERS[providerName] };
|
||||
requireCredential(providerConfig.apiKey);
|
||||
if (requestedModel) providerConfig.model = requestedModel;
|
||||
const localConfig = { ...config, LLM_PROVIDERS: { ...config.LLM_PROVIDERS, [providerName]: providerConfig } };
|
||||
const provider = LLMProviderFactory.createProvider(providerName, localConfig, localConfig);
|
||||
const started = process.hrtime.bigint();
|
||||
const result = await provider.createChatCompletion([
|
||||
{ role: 'system', content: 'Reply to the user in the same language using one short, natural sentence suitable for speech.' },
|
||||
{ role: 'user', content: transcript },
|
||||
], { max_tokens: 80, temperature: 0, stream_options: { include_usage: true } });
|
||||
if (!result.success) throw new Error(`Real LLM failed: ${result.error}`);
|
||||
let buffer = '', response = '', firstToken = null, usage = null, finishReason = null;
|
||||
for await (const chunk of result.response.data) {
|
||||
buffer += chunk.toString();
|
||||
const lines = buffer.split('\n'); buffer = lines.pop() || '';
|
||||
for (const line of lines) {
|
||||
if (!line.startsWith('data: ') || line.trim() === 'data: [DONE]') continue;
|
||||
const data = JSON.parse(line.slice(6));
|
||||
const piece = data.choices?.[0]?.delta?.content || '';
|
||||
if (piece && firstToken === null) firstToken = Number(process.hrtime.bigint() - started) / 1e9;
|
||||
response += piece;
|
||||
if (data.usage) usage = data.usage;
|
||||
if (data.choices?.[0]?.finish_reason) finishReason = data.choices[0].finish_reason;
|
||||
}
|
||||
}
|
||||
if (!response.trim() || firstToken === null) throw new Error('Real LLM stream returned no text');
|
||||
return {
|
||||
execution: 'real', mock: false, probe_only: false, fallback_used: false,
|
||||
provider: providerName, model: providerConfig.model, streamed: true,
|
||||
api_request_completed: true, external_request: true, first_token_seconds: firstToken,
|
||||
latency_seconds: Number(process.hrtime.bigint() - started) / 1e9,
|
||||
response: response.trim(),
|
||||
provider_request_id: result.response.headers?.['x-request-id'] || result.response.headers?.['request-id'] || null,
|
||||
finish_reason: finishReason,
|
||||
usage,
|
||||
provider_reported_cost_usd: usage?.cost != null && Number.isFinite(Number(usage.cost)) ? Number(usage.cost) : null,
|
||||
cost_note: usage?.cost == null ? 'The streamed response did not expose a monetary charge; consult the provider billing ledger.' : null,
|
||||
};
|
||||
}
|
||||
|
||||
function resolveFishReferenceId(requested) {
|
||||
if (requested) return { id: requested, source: 'command line or FISH_TTS_REFERENCE_ID' };
|
||||
const manifestPath = path.resolve(__dirname, '..', '..', 'controllable-tts', 'reference_audio', 'manifest.json');
|
||||
if (!fs.existsSync(manifestPath)) {
|
||||
throw new Error('Fish TTS requires FISH_TTS_REFERENCE_ID or the authorized Experiment 6-6 reference manifest');
|
||||
}
|
||||
const manifest = JSON.parse(fs.readFileSync(manifestPath, 'utf8'));
|
||||
if (!manifest.source_reference_id) throw new Error('Fish reference manifest has no source_reference_id');
|
||||
return { id: manifest.source_reference_id, source: path.relative(__dirname, manifestPath) };
|
||||
}
|
||||
|
||||
function runFishTts(text, outputDir, options = {}) {
|
||||
requireCredential('FISH_API_KEY');
|
||||
const python = options.fishPython || resolveWhisperPython();
|
||||
const reference = resolveFishReferenceId(options.fishReferenceId);
|
||||
const outputPath = path.join(outputDir, 'assistant_response.mp3');
|
||||
const script = [
|
||||
'import json, os, pathlib, sys, time',
|
||||
'from fish_audio_sdk import Session, TTSRequest',
|
||||
'text, reference_id, output = sys.argv[1], sys.argv[2], pathlib.Path(sys.argv[3])',
|
||||
'started = time.perf_counter(); first = None; chunks = []',
|
||||
'for chunk in Session(os.environ["FISH_API_KEY"]).tts(TTSRequest(text=text, reference_id=reference_id, format="mp3"), backend="s1"):',
|
||||
' if first is None: first = time.perf_counter() - started',
|
||||
' chunks.append(chunk)',
|
||||
'output.write_bytes(b"".join(chunks)); finished = time.perf_counter() - started',
|
||||
'print("EXPERIMENT_JSON=" + json.dumps({"first_byte_seconds": first, "latency_seconds": finished, "bytes": output.stat().st_size}))',
|
||||
].join('\n');
|
||||
const output = execFileSync(python, ['-c', script, text, reference.id, outputPath], {
|
||||
encoding: 'utf8', maxBuffer: 10 * 1024 * 1024,
|
||||
});
|
||||
const marker = output.split('\n').find(line => line.startsWith('EXPERIMENT_JSON='));
|
||||
if (!marker) throw new Error('Fish Audio returned no structured result');
|
||||
const result = JSON.parse(marker.slice('EXPERIMENT_JSON='.length));
|
||||
const probe = mediaProbe(outputPath);
|
||||
return {
|
||||
execution: 'real', mock: false, probe_only: false, fallback_used: false,
|
||||
provider: 'fish', model: 's1', voice: 'authorized zero-shot reference',
|
||||
api_request_completed: true, external_request: true,
|
||||
first_audio_byte_seconds: result.first_byte_seconds,
|
||||
latency_seconds: result.latency_seconds,
|
||||
output_path: 'assistant_response.mp3', output_sha256: sha256(outputPath),
|
||||
output_bytes: result.bytes, output_duration_seconds: probe.duration_seconds,
|
||||
output_format: probe.format_name,
|
||||
reference_id_sha256: crypto.createHash('sha256').update(reference.id).digest('hex'),
|
||||
reference_id_source: reference.source,
|
||||
billed_input_characters: [...text].length,
|
||||
provider_reported_cost_usd: null,
|
||||
cost_note: 'The Fish Audio SDK response did not expose a monetary charge; consult the provider billing ledger.',
|
||||
};
|
||||
}
|
||||
|
||||
async function runTts(text, providerName, outputDir, options = {}) {
|
||||
if (providerName === 'fish') return runFishTts(text, outputDir, options);
|
||||
if (providerName !== 'siliconflow') throw new Error(`Unknown TTS provider: ${providerName}`);
|
||||
const providerConfig = config.TTS_PROVIDERS[providerName];
|
||||
const apiKey = requireCredential(providerConfig.apiKey);
|
||||
const started = process.hrtime.bigint();
|
||||
const response = await axios.post(providerConfig.apiUrl, {
|
||||
model: providerConfig.model, input: text, voice: providerConfig.voice,
|
||||
response_format: 'mp3', sample_rate: 32000, stream: true, speed: 1, gain: 0,
|
||||
}, {
|
||||
headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
|
||||
responseType: 'stream', timeout: 120000,
|
||||
});
|
||||
let firstByte = null;
|
||||
const chunks = [];
|
||||
for await (const chunk of response.data) {
|
||||
if (firstByte === null) firstByte = Number(process.hrtime.bigint() - started) / 1e9;
|
||||
chunks.push(chunk);
|
||||
}
|
||||
const audio = Buffer.concat(chunks);
|
||||
const outputPath = path.join(outputDir, 'assistant_response.mp3');
|
||||
fs.writeFileSync(outputPath, audio);
|
||||
const probe = mediaProbe(outputPath);
|
||||
return {
|
||||
execution: 'real', mock: false, probe_only: false, fallback_used: false,
|
||||
provider: providerName, model: providerConfig.model, voice: providerConfig.voice,
|
||||
api_request_completed: response.status >= 200 && response.status < 300, external_request: true,
|
||||
first_audio_byte_seconds: firstByte,
|
||||
latency_seconds: Number(process.hrtime.bigint() - started) / 1e9,
|
||||
output_path: 'assistant_response.mp3', output_sha256: sha256(outputPath),
|
||||
output_bytes: audio.length, output_duration_seconds: probe.duration_seconds,
|
||||
output_format: probe.format_name,
|
||||
provider_request_id: response.headers?.['x-request-id'] || response.headers?.['request-id'] || null,
|
||||
billed_input_characters: [...text].length,
|
||||
provider_reported_cost_usd: null,
|
||||
cost_note: 'The speech response did not expose a monetary charge; consult the provider billing ledger.',
|
||||
};
|
||||
}
|
||||
|
||||
function renderReport(e) {
|
||||
const a = e.acceptance;
|
||||
return `# Experiment 6-3 real traditional-voice validation\n\n` +
|
||||
`- Run ID: \`${e.run_id}\`\n- Complete: **${e.experiment_complete}**\n` +
|
||||
`- Source: \`${e.source_media.path}\` (${e.source_media.duration_seconds.toFixed(3)} s, saved browser microphone/WebSocket capture)\n` +
|
||||
`- VAD: Silero ONNX, 500 ms silence, non-forced endpoint = ${e.stages.vad.endpoint_detected}\n` +
|
||||
`- ASR: ${e.stages.asr.provider} / ${e.stages.asr.model}, ${e.stages.asr.latency_seconds.toFixed(3)} s\n` +
|
||||
`- Transcript: ${e.stages.asr.transcript}\n` +
|
||||
`- LLM: ${e.stages.llm.provider} / ${e.stages.llm.model}, TTFT ${e.stages.llm.first_token_seconds.toFixed(3)} s, total ${e.stages.llm.latency_seconds.toFixed(3)} s\n` +
|
||||
`- Response: ${e.stages.llm.response}\n` +
|
||||
`- TTS: ${e.stages.tts.provider} / ${e.stages.tts.model}, first byte ${e.stages.tts.first_audio_byte_seconds.toFixed(3)} s, total ${e.stages.tts.latency_seconds.toFixed(3)} s\n` +
|
||||
`- Post-endpoint time to first audio byte: ${e.latency.post_endpoint_to_first_audio_byte_seconds.toFixed(3)} s\n\n` +
|
||||
`## Strict gates\n\n${Object.entries(a.gates).map(([k,v]) => `- ${k}: **${v}**`).join('\n')}\n\n` +
|
||||
`${a.statement}\n`;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const input = path.resolve(arg('--input', path.join(__dirname, 'recordings', 'recording_2025-02-06T14-51-38-205Z.wav')));
|
||||
const outputDir = path.resolve(arg('--output-dir', path.join(__dirname, 'validation', `real_pipeline_${new Date().toISOString().slice(0,10).replaceAll('-', '')}`)));
|
||||
const asrProvider = arg('--asr-provider', 'local-whisper');
|
||||
const whisperPython = arg('--whisper-python', process.env.WHISPER_PYTHON);
|
||||
const whisperModel = arg('--whisper-model', 'tiny');
|
||||
const llmProvider = arg('--llm-provider', 'openrouter-gemini');
|
||||
const llmModel = arg('--llm-model', config.LLM_PROVIDERS[llmProvider]?.model);
|
||||
const ttsProvider = arg('--tts-provider', 'fish');
|
||||
const fishPython = arg('--fish-python', process.env.FISH_PYTHON);
|
||||
const fishReferenceId = arg('--fish-reference-id', process.env.FISH_TTS_REFERENCE_ID);
|
||||
if (fs.existsSync(outputDir)) throw new Error(`Output directory already exists: ${outputDir}`);
|
||||
fs.mkdirSync(outputDir, { recursive: true });
|
||||
const wav = parsePcmWav(input);
|
||||
const sourceCopy = path.join(outputDir, 'microphone_input.wav');
|
||||
fs.copyFileSync(input, sourceCopy);
|
||||
const probe = mediaProbe(sourceCopy);
|
||||
const evidence = {
|
||||
schema_version: 1, experiment: '6-3', run_id: `exp6-3-${new Date().toISOString().replace(/[-:.]/g, '')}`,
|
||||
generated_at_utc: new Date().toISOString(), credentials_persisted: false,
|
||||
provenance: buildProvenance(),
|
||||
source_media: {
|
||||
path: 'microphone_input.wav', capture_method: 'browser_microphone_over_websocket',
|
||||
original_repository_path: path.relative(path.join(__dirname, '..'), input),
|
||||
sha256: sha256(sourceCopy), size_bytes: fs.statSync(sourceCopy).size,
|
||||
original_sha256: sha256(input),
|
||||
duration_seconds: probe.duration_seconds, sample_rate_hz: wav.sampleRate,
|
||||
channels: wav.channels, bits_per_sample: wav.bits,
|
||||
provenance_note: 'Existing real microphone capture saved by live-audio/backend/server.js; replayed through the production Silero class for reproducible validation.',
|
||||
}, stages: {}, experiment_complete: false,
|
||||
};
|
||||
try {
|
||||
evidence.stages.vad = await runVad(wav.pcm, outputDir);
|
||||
evidence.stages.asr = await runAsr(
|
||||
path.join(outputDir, evidence.stages.vad.segment_path), asrProvider, outputDir,
|
||||
{ whisperPython, whisperModel },
|
||||
);
|
||||
evidence.stages.llm = await runLlm(evidence.stages.asr.transcript, llmProvider, llmModel);
|
||||
evidence.stages.tts = await runTts(
|
||||
evidence.stages.llm.response, ttsProvider, outputDir,
|
||||
{ fishPython, fishReferenceId },
|
||||
);
|
||||
evidence.latency = {
|
||||
post_endpoint_to_first_audio_byte_seconds:
|
||||
evidence.stages.asr.latency_seconds + evidence.stages.llm.first_token_seconds + evidence.stages.tts.first_audio_byte_seconds,
|
||||
complete_serial_pipeline_seconds: ['vad','asr','llm','tts'].reduce((n,k) => n + evidence.stages[k].latency_seconds, 0),
|
||||
measurement_clock: 'process.hrtime.bigint monotonic clock',
|
||||
};
|
||||
evidence.cost = {
|
||||
paid_external_requests: [evidence.stages.asr, evidence.stages.llm, evidence.stages.tts]
|
||||
.filter(stage => stage.external_request === true).length,
|
||||
provider_reported_total_usd: [evidence.stages.asr, evidence.stages.llm, evidence.stages.tts]
|
||||
.map(stage => stage.provider_reported_cost_usd)
|
||||
.filter(value => Number.isFinite(value))
|
||||
.reduce((sum, value) => sum + value, 0),
|
||||
complete: [evidence.stages.asr, evidence.stages.llm, evidence.stages.tts]
|
||||
.every(stage => Number.isFinite(stage.provider_reported_cost_usd)),
|
||||
note: 'A zero total is not a zero-cost claim when complete=false; some providers omit per-request charges.',
|
||||
};
|
||||
evidence.acceptance = validateExperimentEvidence(evidence, outputDir);
|
||||
evidence.experiment_complete = evidence.acceptance.passed;
|
||||
} catch (error) {
|
||||
evidence.error = redact(error?.stack || error);
|
||||
evidence.acceptance = validateExperimentEvidence(evidence, outputDir);
|
||||
}
|
||||
fs.writeFileSync(path.join(outputDir, 'evidence.json'), JSON.stringify(evidence, null, 2) + '\n');
|
||||
if (evidence.experiment_complete) fs.writeFileSync(path.join(outputDir, 'report.md'), renderReport(evidence));
|
||||
console.log(`Evidence: ${path.join(outputDir, 'evidence.json')}`);
|
||||
if (!evidence.experiment_complete) throw new Error(evidence.error || 'Strict acceptance gates failed');
|
||||
}
|
||||
|
||||
main().catch(error => { console.error(redact(error.message)); process.exit(1); });
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,83 @@
|
||||
const axios = require('axios');
|
||||
const config = require('./config');
|
||||
|
||||
async function testLLMEndpoint() {
|
||||
console.log('Testing LLM endpoint...');
|
||||
console.log(`URL: ${config.LLM_API_URL}`);
|
||||
console.log(`Model: ${config.LLM_MODEL}`);
|
||||
|
||||
const testPrompt = "Tell me a short joke.";
|
||||
|
||||
try {
|
||||
const response = await axios.post(
|
||||
config.LLM_API_URL,
|
||||
{
|
||||
model: config.LLM_MODEL,
|
||||
messages: [
|
||||
{ role: "user", content: testPrompt }
|
||||
],
|
||||
stream: true,
|
||||
max_tokens: 100
|
||||
},
|
||||
{
|
||||
headers: {
|
||||
'Authorization': `Bearer ${config.OPENAI_API_KEY}`,
|
||||
'Content-Type': 'application/json',
|
||||
},
|
||||
responseType: 'stream'
|
||||
}
|
||||
);
|
||||
|
||||
console.log('Connection established successfully');
|
||||
|
||||
let fullResponse = '';
|
||||
|
||||
response.data.on('data', chunk => {
|
||||
const lines = chunk.toString().split('\n');
|
||||
for (const line of lines) {
|
||||
if (line.trim() === '') continue;
|
||||
if (line.trim() === 'data: [DONE]') continue;
|
||||
if (!line.startsWith('data: ')) continue;
|
||||
|
||||
try {
|
||||
const jsonData = JSON.parse(line.replace('data: ', ''));
|
||||
const content = jsonData.choices[0]?.delta?.content || '';
|
||||
fullResponse += content;
|
||||
process.stdout.write(content); // Stream the response
|
||||
} catch (e) {
|
||||
console.error('Error parsing chunk:', e);
|
||||
console.error('Raw chunk:', line);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
response.data.on('end', () => {
|
||||
console.log('\n\nFull response received:', fullResponse);
|
||||
console.log('\nTest completed successfully');
|
||||
});
|
||||
|
||||
response.data.on('error', (error) => {
|
||||
console.error('Stream error:', error);
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error('\nError testing LLM endpoint:', error.message);
|
||||
if (error.response) {
|
||||
console.error('Response status:', error.response.status);
|
||||
console.error('Response headers:', JSON.stringify(error.response.headers, null, 2));
|
||||
}
|
||||
|
||||
// Log safe error properties
|
||||
const safeError = {
|
||||
message: error.message,
|
||||
name: error.name,
|
||||
stack: error.stack,
|
||||
code: error.code,
|
||||
status: error.status
|
||||
};
|
||||
console.error('\nError details:', JSON.stringify(safeError, null, 2));
|
||||
}
|
||||
}
|
||||
|
||||
// Run the test
|
||||
testLLMEndpoint();
|
||||
@@ -0,0 +1,365 @@
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
// Import provider factories
|
||||
const { ASRProviderFactory } = require('../utils/providers/asrProviders');
|
||||
const { LLMProviderFactory } = require('../utils/providers/llmProviders');
|
||||
|
||||
// Import services
|
||||
const SpeechToTextService = require('../utils/speechToText');
|
||||
|
||||
// Mock configuration for testing
|
||||
const testConfig = {
|
||||
// API Keys from environment variables
|
||||
OPENAI_API_KEY: process.env.OPENAI_API_KEY,
|
||||
OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY,
|
||||
ANTHROPIC_API_KEY: process.env.ANTHROPIC_API_KEY,
|
||||
ARK_API_KEY: process.env.ARK_API_KEY,
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY,
|
||||
|
||||
// Provider configurations
|
||||
ASR_PROVIDERS: {
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/audio/transcriptions',
|
||||
model: 'whisper-1',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
siliconflow: {
|
||||
apiUrl: 'https://api.siliconflow.cn/v1/audio/transcriptions',
|
||||
model: 'FunAudioLLM/SenseVoiceSmall',
|
||||
apiKey: 'SILICONFLOW_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
LLM_PROVIDERS: {
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/chat/completions',
|
||||
model: 'gpt-5.6-luna',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
'openrouter-gpt': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'openai/gpt-5.6-luna',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
'openrouter-gemini': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'google/gemini-3.5-flash',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
ark: {
|
||||
apiUrl: 'https://ark.cn-beijing.volces.com/api/v3/chat/completions',
|
||||
model: 'doubao-seed-1-6-flash-250615',
|
||||
apiKey: 'ARK_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
// Audio configuration
|
||||
AUDIO_SAMPLE_RATE: 16000,
|
||||
VISION_MAX_TOKENS: 4096
|
||||
};
|
||||
|
||||
/**
|
||||
* Create a sample audio buffer for testing
|
||||
* This creates a simple sine wave audio buffer
|
||||
*/
|
||||
function createTestAudioBuffer(durationSeconds = 2, sampleRate = 16000) {
|
||||
const numSamples = durationSeconds * sampleRate;
|
||||
const buffer = Buffer.alloc(numSamples * 2); // 16-bit = 2 bytes per sample
|
||||
|
||||
// Generate a simple sine wave at 440Hz
|
||||
const frequency = 440;
|
||||
for (let i = 0; i < numSamples; i++) {
|
||||
const sample = Math.sin(2 * Math.PI * frequency * i / sampleRate) * 0.5;
|
||||
const intSample = Math.round(sample * 32767);
|
||||
buffer.writeInt16LE(intSample, i * 2);
|
||||
}
|
||||
|
||||
return buffer;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test suite for ASR providers
|
||||
*/
|
||||
describe('ASR Provider Tests', function() {
|
||||
this.timeout(60000); // 60 second timeout for API calls
|
||||
|
||||
const asrProviders = ['openai', 'siliconflow'];
|
||||
|
||||
asrProviders.forEach(providerName => {
|
||||
describe(`${providerName} ASR Provider`, function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
// Skip test if API key is not available
|
||||
const apiKeyName = testConfig.ASR_PROVIDERS[providerName].apiKey;
|
||||
if (!testConfig[apiKeyName]) {
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = ASRProviderFactory.createProvider(providerName, testConfig, testConfig);
|
||||
} catch (error) {
|
||||
console.error(`Failed to create ${providerName} provider:`, error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should be created successfully', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert(provider.config, 'Provider should have config');
|
||||
assert(provider.apiKey, 'Provider should have API key');
|
||||
});
|
||||
|
||||
it('should transcribe test audio', async function() {
|
||||
const audioBuffer = createTestAudioBuffer(2); // 2 seconds of audio
|
||||
const tempDir = path.join(__dirname, '../temp');
|
||||
|
||||
// Ensure temp directory exists
|
||||
if (!fs.existsSync(tempDir)) {
|
||||
fs.mkdirSync(tempDir, { recursive: true });
|
||||
}
|
||||
|
||||
const result = await provider.transcribe(audioBuffer, tempDir);
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert(typeof result.text === 'string', 'Should have text field');
|
||||
assert(result.provider === providerName, 'Should have correct provider name');
|
||||
|
||||
console.log(`${providerName} ASR Result:`, {
|
||||
success: result.success,
|
||||
text: result.text.substring(0, 100) + (result.text.length > 100 ? '...' : ''),
|
||||
language: result.language,
|
||||
provider: result.provider
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('SpeechToTextService Integration', function() {
|
||||
asrProviders.forEach(providerName => {
|
||||
it(`should work with ${providerName} provider`, async function() {
|
||||
// Skip test if API key is not available
|
||||
const apiKeyName = testConfig.ASR_PROVIDERS[providerName].apiKey;
|
||||
if (!testConfig[apiKeyName]) {
|
||||
this.skip();
|
||||
}
|
||||
|
||||
// Override config for this test
|
||||
const originalConfig = require('../config');
|
||||
Object.assign(originalConfig, testConfig);
|
||||
originalConfig.ASR_PROVIDER = providerName;
|
||||
|
||||
const sttService = new SpeechToTextService();
|
||||
const audioBuffer = createTestAudioBuffer(2);
|
||||
|
||||
const result = await sttService.transcribeAudio(audioBuffer);
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert(typeof result.text === 'string', 'Should have text field');
|
||||
|
||||
console.log(`STT Service with ${providerName}:`, {
|
||||
success: result.success,
|
||||
text: result.text.substring(0, 100) + (result.text.length > 100 ? '...' : ''),
|
||||
provider: result.provider
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Test suite for LLM providers
|
||||
*/
|
||||
describe('LLM Provider Tests', function() {
|
||||
this.timeout(60000); // 60 second timeout for API calls
|
||||
|
||||
const llmProviders = ['openai', 'openrouter-gpt', 'openrouter-gemini', 'ark'];
|
||||
const testMessages = [
|
||||
{ role: 'system', content: 'You are a helpful AI assistant.' },
|
||||
{ role: 'user', content: 'Hello! Please respond with a short greeting.' }
|
||||
];
|
||||
|
||||
llmProviders.forEach(providerName => {
|
||||
describe(`${providerName} LLM Provider`, function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
// Skip test if API key is not available
|
||||
const apiKeyName = testConfig.LLM_PROVIDERS[providerName].apiKey;
|
||||
if (!testConfig[apiKeyName]) {
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = LLMProviderFactory.createProvider(providerName, testConfig, testConfig);
|
||||
} catch (error) {
|
||||
console.error(`Failed to create ${providerName} provider:`, error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should be created successfully', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert(provider.config, 'Provider should have config');
|
||||
assert(provider.apiKey, 'Provider should have API key');
|
||||
});
|
||||
|
||||
it('should generate chat completion', async function() {
|
||||
const result = await provider.createChatCompletion(testMessages, {
|
||||
max_tokens: 100
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert(result.provider === providerName.split('-')[0], 'Should have correct provider name');
|
||||
|
||||
if (result.success) {
|
||||
assert(result.response, 'Should have response object');
|
||||
console.log(`${providerName} LLM Result: Success`);
|
||||
} else {
|
||||
console.log(`${providerName} LLM Result:`, result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should stream chat completion', async function() {
|
||||
const result = await provider.createChatCompletion(testMessages, {
|
||||
max_tokens: 50,
|
||||
stream: true
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
|
||||
if (result.success) {
|
||||
assert(result.response, 'Should have response object');
|
||||
assert(result.response.data, 'Should have data stream');
|
||||
|
||||
// Test streaming by collecting some data
|
||||
let receivedData = false;
|
||||
const timeout = setTimeout(() => {
|
||||
if (!receivedData) {
|
||||
console.log(`${providerName}: No data received within timeout`);
|
||||
}
|
||||
}, 10000);
|
||||
|
||||
result.response.data.on('data', (chunk) => {
|
||||
receivedData = true;
|
||||
clearTimeout(timeout);
|
||||
console.log(`${providerName} LLM Streaming: Received data chunk`);
|
||||
});
|
||||
|
||||
result.response.data.on('error', (error) => {
|
||||
clearTimeout(timeout);
|
||||
console.error(`${providerName} LLM Streaming Error:`, error.message);
|
||||
});
|
||||
|
||||
} else {
|
||||
console.log(`${providerName} LLM Streaming Result:`, result.error);
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Integration tests for all provider combinations
|
||||
*/
|
||||
describe('Provider Integration Tests', function() {
|
||||
this.timeout(120000); // 2 minute timeout for integration tests
|
||||
|
||||
const testCombinations = [
|
||||
{ asr: 'openai', llm: 'openai', description: 'OpenAI ASR + OpenAI LLM' },
|
||||
{ asr: 'openai', llm: 'openrouter-gpt', description: 'OpenAI ASR + OpenRouter GPT' },
|
||||
{ asr: 'openai', llm: 'openrouter-gemini', description: 'OpenAI ASR + OpenRouter Gemini' },
|
||||
{ asr: 'openai', llm: 'ark', description: 'OpenAI ASR + ARK Doubao' },
|
||||
{ asr: 'siliconflow', llm: 'openai', description: 'SenseVoice ASR + OpenAI LLM' },
|
||||
{ asr: 'siliconflow', llm: 'openrouter-gpt', description: 'SenseVoice ASR + OpenRouter GPT' },
|
||||
{ asr: 'siliconflow', llm: 'openrouter-gemini', description: 'SenseVoice ASR + OpenRouter Gemini' },
|
||||
{ asr: 'siliconflow', llm: 'ark', description: 'SenseVoice ASR + ARK Doubao' }
|
||||
];
|
||||
|
||||
testCombinations.forEach(combination => {
|
||||
it(`should work with ${combination.description}`, async function() {
|
||||
// Check if API keys are available
|
||||
const asrApiKey = testConfig.ASR_PROVIDERS[combination.asr].apiKey;
|
||||
const llmApiKey = testConfig.LLM_PROVIDERS[combination.llm].apiKey;
|
||||
|
||||
if (!testConfig[asrApiKey] || !testConfig[llmApiKey]) {
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
// Create providers
|
||||
const asrProvider = ASRProviderFactory.createProvider(combination.asr, testConfig, testConfig);
|
||||
const llmProvider = LLMProviderFactory.createProvider(combination.llm, testConfig, testConfig);
|
||||
|
||||
// Test ASR
|
||||
const audioBuffer = createTestAudioBuffer(2);
|
||||
const tempDir = path.join(__dirname, '../temp');
|
||||
if (!fs.existsSync(tempDir)) {
|
||||
fs.mkdirSync(tempDir, { recursive: true });
|
||||
}
|
||||
|
||||
const asrResult = await asrProvider.transcribe(audioBuffer, tempDir);
|
||||
assert(asrResult.success !== undefined, 'ASR should return result');
|
||||
|
||||
// Test LLM with a simple message
|
||||
const messages = [
|
||||
{ role: 'system', content: 'You are a helpful assistant.' },
|
||||
{ role: 'user', content: 'Say hello in one word.' }
|
||||
];
|
||||
|
||||
const llmResult = await llmProvider.createChatCompletion(messages, {
|
||||
max_tokens: 10
|
||||
});
|
||||
|
||||
assert(llmResult.success !== undefined, 'LLM should return result');
|
||||
|
||||
console.log(`Integration Test - ${combination.description}:`, {
|
||||
asr: { success: asrResult.success, provider: asrResult.provider },
|
||||
llm: { success: llmResult.success, provider: llmResult.provider }
|
||||
});
|
||||
|
||||
} catch (error) {
|
||||
console.error(`Integration test failed for ${combination.description}:`, error.message);
|
||||
throw error;
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* Provider switching tests
|
||||
*/
|
||||
describe('Provider Switching Tests', function() {
|
||||
it('should switch ASR providers dynamically', function() {
|
||||
const originalConfig = require('../config');
|
||||
Object.assign(originalConfig, testConfig);
|
||||
|
||||
const sttService = new SpeechToTextService();
|
||||
const originalProvider = sttService.getProviderInfo();
|
||||
|
||||
// Try switching to different provider
|
||||
const availableProviders = ['openai', 'siliconflow'];
|
||||
const newProvider = availableProviders.find(p => p !== originalProvider.provider);
|
||||
|
||||
if (newProvider && testConfig[testConfig.ASR_PROVIDERS[newProvider].apiKey]) {
|
||||
sttService.switchProvider(newProvider);
|
||||
const newProviderInfo = sttService.getProviderInfo();
|
||||
|
||||
assert(newProviderInfo.provider !== originalProvider.provider, 'Provider should change');
|
||||
console.log('ASR Provider switched from', originalProvider.provider, 'to', newProviderInfo.provider);
|
||||
} else {
|
||||
console.log('Skipping ASR provider switching test - insufficient API keys');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// Export test configuration for use in other test files
|
||||
module.exports = {
|
||||
testConfig,
|
||||
createTestAudioBuffer
|
||||
};
|
||||
@@ -0,0 +1,307 @@
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
// Import ASR provider factory
|
||||
const { ASRProviderFactory } = require('../utils/providers/asrProviders');
|
||||
|
||||
// Test configuration
|
||||
const testConfig = {
|
||||
OPENAI_API_KEY: process.env.OPENAI_API_KEY,
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY,
|
||||
|
||||
ASR_PROVIDERS: {
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/audio/transcriptions',
|
||||
model: 'whisper-1',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
siliconflow: {
|
||||
apiUrl: 'https://api.siliconflow.cn/v1/audio/transcriptions',
|
||||
model: 'FunAudioLLM/SenseVoiceSmall',
|
||||
apiKey: 'SILICONFLOW_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
AUDIO_SAMPLE_RATE: 16000
|
||||
};
|
||||
|
||||
/**
|
||||
* Create a test audio buffer with spoken content
|
||||
* This creates a simple sine wave that simulates audio content
|
||||
*/
|
||||
function createTestAudioBuffer(durationSeconds = 3, sampleRate = 16000) {
|
||||
const numSamples = durationSeconds * sampleRate;
|
||||
const buffer = Buffer.alloc(numSamples * 2); // 16-bit = 2 bytes per sample
|
||||
|
||||
// Generate a more complex wave pattern to simulate speech
|
||||
for (let i = 0; i < numSamples; i++) {
|
||||
// Mix multiple frequencies to simulate speech-like content
|
||||
const t = i / sampleRate;
|
||||
const sample =
|
||||
0.3 * Math.sin(2 * Math.PI * 300 * t) + // Base frequency
|
||||
0.2 * Math.sin(2 * Math.PI * 600 * t) + // Harmonic
|
||||
0.1 * Math.sin(2 * Math.PI * 1200 * t) + // Higher harmonic
|
||||
0.05 * (Math.random() - 0.5); // Noise for realism
|
||||
|
||||
const intSample = Math.round(sample * 16000); // Scale to 16-bit range
|
||||
buffer.writeInt16LE(intSample, i * 2);
|
||||
}
|
||||
|
||||
return buffer;
|
||||
}
|
||||
|
||||
/**
|
||||
* ASR Provider Individual Tests
|
||||
*/
|
||||
describe('ASR Providers - Individual Testing', function() {
|
||||
this.timeout(120000); // 2 minute timeout for API calls
|
||||
|
||||
const tempDir = path.join(__dirname, '../temp');
|
||||
|
||||
before(function() {
|
||||
// Ensure temp directory exists
|
||||
if (!fs.existsSync(tempDir)) {
|
||||
fs.mkdirSync(tempDir, { recursive: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe('OpenAI Whisper ASR Provider', function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
if (!testConfig.OPENAI_API_KEY) {
|
||||
console.log('⚠️ Skipping OpenAI ASR tests - OPENAI_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = ASRProviderFactory.createProvider('openai', testConfig, testConfig);
|
||||
console.log('✅ OpenAI ASR Provider created successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Failed to create OpenAI ASR provider:', error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should initialize with correct configuration', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert.strictEqual(provider.config.model, 'whisper-1', 'Should use whisper-1 model');
|
||||
assert.strictEqual(provider.config.apiUrl, 'https://api.openai.com/v1/audio/transcriptions', 'Should use correct API URL');
|
||||
assert(provider.apiKey, 'Should have API key');
|
||||
console.log('📋 OpenAI ASR Config:', {
|
||||
model: provider.config.model,
|
||||
apiUrl: provider.config.apiUrl
|
||||
});
|
||||
});
|
||||
|
||||
it('should transcribe test audio successfully', async function() {
|
||||
const audioBuffer = createTestAudioBuffer(3); // 3 seconds of test audio
|
||||
console.log('🎵 Testing OpenAI ASR with', audioBuffer.length, 'bytes of audio data');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.transcribe(audioBuffer, tempDir);
|
||||
const duration = Date.now() - startTime;
|
||||
|
||||
console.log('📊 OpenAI ASR Results:', {
|
||||
success: result.success,
|
||||
transcriptionTime: `${duration}ms`,
|
||||
textLength: result.text ? result.text.length : 0,
|
||||
language: result.language,
|
||||
confidence: result.confidence,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert(typeof result.text === 'string', 'Should have text field');
|
||||
assert.strictEqual(result.provider, 'openai', 'Should identify as openai provider');
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ OpenAI ASR transcription successful');
|
||||
if (result.text.length > 0) {
|
||||
console.log('📝 Transcribed text preview:', result.text.substring(0, 100) + (result.text.length > 100 ? '...' : ''));
|
||||
}
|
||||
} else {
|
||||
console.log('❌ OpenAI ASR transcription failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle empty audio gracefully', async function() {
|
||||
const emptyBuffer = Buffer.alloc(0);
|
||||
const result = await provider.transcribe(emptyBuffer, tempDir);
|
||||
|
||||
console.log('🔍 OpenAI ASR Empty Audio Test:', {
|
||||
success: result.success,
|
||||
error: result.error,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
// Empty audio should typically fail or return empty text
|
||||
if (!result.success) {
|
||||
assert(result.error, 'Should have error message for empty audio');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('SenseVoice (Siliconflow) ASR Provider', function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
if (!testConfig.SILICONFLOW_API_KEY) {
|
||||
console.log('⚠️ Skipping Siliconflow ASR tests - SILICONFLOW_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = ASRProviderFactory.createProvider('siliconflow', testConfig, testConfig);
|
||||
console.log('✅ Siliconflow ASR Provider created successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Failed to create Siliconflow ASR provider:', error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should initialize with correct configuration', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert.strictEqual(provider.config.model, 'FunAudioLLM/SenseVoiceSmall', 'Should use SenseVoiceSmall model');
|
||||
assert.strictEqual(provider.config.apiUrl, 'https://api.siliconflow.cn/v1/audio/transcriptions', 'Should use correct API URL');
|
||||
assert(provider.apiKey, 'Should have API key');
|
||||
console.log('📋 Siliconflow ASR Config:', {
|
||||
model: provider.config.model,
|
||||
apiUrl: provider.config.apiUrl
|
||||
});
|
||||
});
|
||||
|
||||
it('should transcribe test audio successfully', async function() {
|
||||
const audioBuffer = createTestAudioBuffer(3); // 3 seconds of test audio
|
||||
console.log('🎵 Testing Siliconflow ASR with', audioBuffer.length, 'bytes of audio data');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.transcribe(audioBuffer, tempDir);
|
||||
const duration = Date.now() - startTime;
|
||||
|
||||
console.log('📊 Siliconflow ASR Results:', {
|
||||
success: result.success,
|
||||
transcriptionTime: `${duration}ms`,
|
||||
textLength: result.text ? result.text.length : 0,
|
||||
language: result.language,
|
||||
confidence: result.confidence,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert(typeof result.text === 'string', 'Should have text field');
|
||||
assert.strictEqual(result.provider, 'siliconflow', 'Should identify as siliconflow provider');
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ Siliconflow ASR transcription successful');
|
||||
if (result.text.length > 0) {
|
||||
console.log('📝 Transcribed text preview:', result.text.substring(0, 100) + (result.text.length > 100 ? '...' : ''));
|
||||
}
|
||||
} else {
|
||||
console.log('❌ Siliconflow ASR transcription failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle different audio lengths', async function() {
|
||||
const shortAudio = createTestAudioBuffer(1); // 1 second
|
||||
const longAudio = createTestAudioBuffer(5); // 5 seconds
|
||||
|
||||
console.log('🎵 Testing Siliconflow ASR with different audio lengths');
|
||||
|
||||
const shortResult = await provider.transcribe(shortAudio, tempDir);
|
||||
const longResult = await provider.transcribe(longAudio, tempDir);
|
||||
|
||||
console.log('📊 Audio Length Tests:', {
|
||||
short: { success: shortResult.success, textLength: shortResult.text.length },
|
||||
long: { success: longResult.success, textLength: longResult.text.length }
|
||||
});
|
||||
|
||||
assert(shortResult, 'Should handle short audio');
|
||||
assert(longResult, 'Should handle long audio');
|
||||
assert(typeof shortResult.success === 'boolean', 'Short audio should have success field');
|
||||
assert(typeof longResult.success === 'boolean', 'Long audio should have success field');
|
||||
});
|
||||
});
|
||||
|
||||
describe('ASR Provider Comparison', function() {
|
||||
it('should compare provider performance', async function() {
|
||||
const availableProviders = [];
|
||||
|
||||
if (testConfig.OPENAI_API_KEY) {
|
||||
availableProviders.push('openai');
|
||||
}
|
||||
if (testConfig.SILICONFLOW_API_KEY) {
|
||||
availableProviders.push('siliconflow');
|
||||
}
|
||||
|
||||
if (availableProviders.length < 2) {
|
||||
console.log('⚠️ Skipping provider comparison - need at least 2 providers');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
const audioBuffer = createTestAudioBuffer(3);
|
||||
const results = {};
|
||||
|
||||
console.log('🏁 Comparing ASR provider performance...');
|
||||
|
||||
for (const providerName of availableProviders) {
|
||||
const provider = ASRProviderFactory.createProvider(providerName, testConfig, testConfig);
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.transcribe(audioBuffer, tempDir);
|
||||
const duration = Date.now() - startTime;
|
||||
|
||||
results[providerName] = {
|
||||
success: result.success,
|
||||
duration: duration,
|
||||
textLength: result.text ? result.text.length : 0,
|
||||
error: result.error
|
||||
};
|
||||
}
|
||||
|
||||
console.log('📊 ASR Provider Performance Comparison:');
|
||||
Object.entries(results).forEach(([provider, result]) => {
|
||||
console.log(` ${provider}:`, {
|
||||
success: result.success ? '✅' : '❌',
|
||||
time: `${result.duration}ms`,
|
||||
textLength: result.textLength,
|
||||
error: result.error || 'none'
|
||||
});
|
||||
});
|
||||
|
||||
// Find fastest successful provider
|
||||
const successfulProviders = Object.entries(results).filter(([_, result]) => result.success);
|
||||
if (successfulProviders.length > 0) {
|
||||
const fastest = successfulProviders.reduce((prev, curr) =>
|
||||
prev[1].duration < curr[1].duration ? prev : curr
|
||||
);
|
||||
console.log(`🏆 Fastest provider: ${fastest[0]} (${fastest[1].duration}ms)`);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
after(function() {
|
||||
// Cleanup temp files
|
||||
try {
|
||||
const files = fs.readdirSync(tempDir);
|
||||
files.forEach(file => {
|
||||
if (file.startsWith('audio_')) {
|
||||
fs.unlinkSync(path.join(tempDir, file));
|
||||
}
|
||||
});
|
||||
console.log('🧹 Cleaned up temporary audio files');
|
||||
} catch (error) {
|
||||
console.warn('⚠️ Failed to cleanup temp files:', error.message);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
module.exports = {
|
||||
testConfig,
|
||||
createTestAudioBuffer
|
||||
};
|
||||
@@ -0,0 +1,58 @@
|
||||
const assert = require('assert');
|
||||
const fs = require('fs');
|
||||
const os = require('os');
|
||||
const path = require('path');
|
||||
|
||||
const { sha256, validateExperimentEvidence } = require('../experiment_validation');
|
||||
|
||||
function fixture(dir) {
|
||||
fs.writeFileSync(path.join(dir, 'input.wav'), Buffer.from('real-input'));
|
||||
fs.writeFileSync(path.join(dir, 'segment.wav'), Buffer.from('real-segment'));
|
||||
fs.writeFileSync(path.join(dir, 'output.mp3'), Buffer.alloc(2000, 1));
|
||||
const real = { execution: 'real', mock: false, probe_only: false, fallback_used: false, latency_seconds: 0.1 };
|
||||
return {
|
||||
schema_version: 1, experiment: '6-3',
|
||||
provenance: { host: { platform: 'test', architecture: 'test' }, runtime: { node: 'test', onnxruntime_node: 'test' } },
|
||||
source_media: { path: 'input.wav', capture_method: 'browser_microphone_over_websocket', sha256: sha256(path.join(dir, 'input.wav')), original_sha256: sha256(path.join(dir, 'input.wav')), sample_rate_hz: 16000, channels: 1, bits_per_sample: 16 },
|
||||
stages: {
|
||||
vad: { ...real, implementation: 'Silero VAD ONNX', model_sha256: 'abc', endpoint_detected: true, forced_endpoint: false, max_silence_ms: 500, observed_trailing_silence_ms: 512, segment_path: 'segment.wav', segment_sha256: sha256(path.join(dir, 'segment.wav')) },
|
||||
asr: { ...real, provider: 'openai', model: 'whisper-1', inference_completed: true, api_request_completed: true, transcript: 'hello' },
|
||||
llm: { ...real, provider: 'openai', model: 'gpt-real', api_request_completed: true, streamed: true, first_token_seconds: 0.1, response: 'Hi.' },
|
||||
tts: { ...real, provider: 'siliconflow', model: 'CosyVoice', api_request_completed: true, first_audio_byte_seconds: 0.1, output_path: 'output.mp3', output_sha256: sha256(path.join(dir, 'output.mp3')), output_bytes: 2000, output_duration_seconds: 1 },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe('Experiment 6-3 strict evidence gates', () => {
|
||||
let dir;
|
||||
beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'exp6-3-')); });
|
||||
afterEach(() => fs.rmSync(dir, { recursive: true, force: true }));
|
||||
|
||||
it('accepts direct real media and all four real stages', () => {
|
||||
assert.strictEqual(validateExperimentEvidence(fixture(dir), dir).passed, true);
|
||||
});
|
||||
|
||||
for (const field of ['mock', 'probe_only', 'fallback_used']) {
|
||||
it(`rejects a stage marked ${field}`, () => {
|
||||
const evidence = fixture(dir); evidence.stages.asr[field] = true;
|
||||
assert.strictEqual(validateExperimentEvidence(evidence, dir).passed, false);
|
||||
});
|
||||
}
|
||||
|
||||
it('rejects a forced VAD flush and missing 500ms silence', () => {
|
||||
const evidence = fixture(dir);
|
||||
evidence.stages.vad.forced_endpoint = true;
|
||||
evidence.stages.vad.observed_trailing_silence_ms = 100;
|
||||
assert.strictEqual(validateExperimentEvidence(evidence, dir).passed, false);
|
||||
});
|
||||
|
||||
it('rejects README claims without hashed output media', () => {
|
||||
const evidence = fixture(dir); evidence.stages.tts.output_sha256 = 'claimed-only';
|
||||
assert.strictEqual(validateExperimentEvidence(evidence, dir).passed, false);
|
||||
});
|
||||
|
||||
it('rejects evidence without reproducibility provenance', () => {
|
||||
const evidence = fixture(dir); delete evidence.provenance.runtime.onnxruntime_node;
|
||||
assert.strictEqual(validateExperimentEvidence(evidence, dir).passed, false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,560 @@
|
||||
const assert = require('assert');
|
||||
|
||||
// Import LLM provider factory
|
||||
const { LLMProviderFactory } = require('../utils/providers/llmProviders');
|
||||
|
||||
// Test configuration
|
||||
const testConfig = {
|
||||
OPENAI_API_KEY: process.env.OPENAI_API_KEY,
|
||||
OPENROUTER_API_KEY: process.env.OPENROUTER_API_KEY,
|
||||
ARK_API_KEY: process.env.ARK_API_KEY,
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY,
|
||||
|
||||
LLM_PROVIDERS: {
|
||||
openai: {
|
||||
apiUrl: 'https://api.openai.com/v1/chat/completions',
|
||||
model: 'gpt-5.6-luna',
|
||||
apiKey: 'OPENAI_API_KEY'
|
||||
},
|
||||
'openrouter-gpt': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'openai/gpt-5.6-luna',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
'openrouter-gemini': {
|
||||
apiUrl: 'https://openrouter.ai/api/v1/chat/completions',
|
||||
model: 'google/gemini-3.5-flash',
|
||||
apiKey: 'OPENROUTER_API_KEY'
|
||||
},
|
||||
ark: {
|
||||
apiUrl: 'https://ark.cn-beijing.volces.com/api/v3/chat/completions',
|
||||
model: 'doubao-seed-1-6-flash-250615',
|
||||
apiKey: 'ARK_API_KEY'
|
||||
}
|
||||
},
|
||||
|
||||
VISION_MAX_TOKENS: 4096
|
||||
};
|
||||
|
||||
// Test messages for different scenarios
|
||||
const testMessages = {
|
||||
simple: [
|
||||
{ role: 'system', content: 'You are a helpful AI assistant.' },
|
||||
{ role: 'user', content: 'Hello! Please respond with exactly the word "SUCCESS" to confirm you are working.' }
|
||||
],
|
||||
conversation: [
|
||||
{ role: 'system', content: 'You are a conversational AI assistant.' },
|
||||
{ role: 'user', content: 'What is the capital of France?' },
|
||||
{ role: 'assistant', content: 'The capital of France is Paris.' },
|
||||
{ role: 'user', content: 'What is its population?' }
|
||||
],
|
||||
creative: [
|
||||
{ role: 'system', content: 'You are a creative writing assistant.' },
|
||||
{ role: 'user', content: 'Write a very short story about a robot learning to paint. Keep it under 50 words.' }
|
||||
]
|
||||
};
|
||||
|
||||
/**
|
||||
* Collect streaming response data
|
||||
*/
|
||||
async function collectStreamingResponse(response, timeout = 30000) {
|
||||
return new Promise((resolve, reject) => {
|
||||
let buffer = '';
|
||||
let accumulatedContent = '';
|
||||
let firstTokenTime = null;
|
||||
let tokenCount = 0;
|
||||
|
||||
const timeoutHandle = setTimeout(() => {
|
||||
reject(new Error('Streaming response timeout'));
|
||||
}, timeout);
|
||||
|
||||
response.data.on('data', (chunk) => {
|
||||
try {
|
||||
if (!firstTokenTime) {
|
||||
firstTokenTime = Date.now();
|
||||
}
|
||||
|
||||
buffer += chunk.toString();
|
||||
const lines = buffer.split('\n');
|
||||
// Keep the last line if it's incomplete
|
||||
buffer = lines.pop() || '';
|
||||
|
||||
for (const line of lines) {
|
||||
const trimmedLine = line.trim();
|
||||
if (!trimmedLine || trimmedLine === '[DONE]') continue;
|
||||
if (!trimmedLine.startsWith('data: ')) continue;
|
||||
|
||||
try {
|
||||
const jsonData = JSON.parse(trimmedLine.replace('data: ', ''));
|
||||
const content = jsonData.choices[0]?.delta?.content || '';
|
||||
if (content) {
|
||||
accumulatedContent += content;
|
||||
tokenCount++;
|
||||
}
|
||||
} catch (parseError) {
|
||||
// Skip malformed JSON
|
||||
continue;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
reject(error);
|
||||
}
|
||||
});
|
||||
|
||||
response.data.on('end', () => {
|
||||
clearTimeout(timeoutHandle);
|
||||
resolve({
|
||||
content: accumulatedContent,
|
||||
tokenCount: tokenCount,
|
||||
firstTokenTime: firstTokenTime,
|
||||
success: true
|
||||
});
|
||||
});
|
||||
|
||||
response.data.on('error', (error) => {
|
||||
clearTimeout(timeoutHandle);
|
||||
reject(error);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* LLM Provider Individual Tests
|
||||
*/
|
||||
describe('LLM Providers - Individual Testing', function() {
|
||||
this.timeout(120000); // 2 minute timeout for API calls
|
||||
|
||||
describe('OpenAI GPT-4o LLM Provider', function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
if (!testConfig.OPENAI_API_KEY) {
|
||||
console.log('⚠️ Skipping OpenAI LLM tests - OPENAI_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = LLMProviderFactory.createProvider('openai', testConfig, testConfig);
|
||||
console.log('✅ OpenAI LLM Provider created successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Failed to create OpenAI LLM provider:', error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should initialize with correct configuration', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert.strictEqual(provider.config.model, 'gpt-5.6-luna', 'Should use gpt-5.6-luna model');
|
||||
assert.strictEqual(provider.config.apiUrl, 'https://api.openai.com/v1/chat/completions', 'Should use correct API URL');
|
||||
assert(provider.apiKey, 'Should have API key');
|
||||
console.log('📋 OpenAI LLM Config:', {
|
||||
model: provider.config.model,
|
||||
apiUrl: provider.config.apiUrl
|
||||
});
|
||||
});
|
||||
|
||||
it('should generate simple chat completion', async function() {
|
||||
console.log('🤖 Testing OpenAI LLM with simple message');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.createChatCompletion(testMessages.simple, {
|
||||
max_tokens: 100
|
||||
});
|
||||
const responseTime = Date.now() - startTime;
|
||||
|
||||
console.log('📊 OpenAI LLM Simple Test:', {
|
||||
success: result.success,
|
||||
responseTime: `${responseTime}ms`,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert.strictEqual(result.provider, 'openai', 'Should identify as openai provider');
|
||||
|
||||
if (result.success) {
|
||||
assert(result.response, 'Should have response object');
|
||||
console.log('✅ OpenAI LLM simple completion successful');
|
||||
} else {
|
||||
console.log('❌ OpenAI LLM simple completion failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle streaming chat completion', async function() {
|
||||
console.log('🌊 Testing OpenAI LLM streaming');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.createChatCompletion(testMessages.simple, {
|
||||
max_tokens: 50,
|
||||
stream: true
|
||||
});
|
||||
|
||||
if (!result.success) {
|
||||
console.log('❌ OpenAI LLM streaming setup failed:', result.error);
|
||||
assert(false, 'Streaming setup should succeed');
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const streamResult = await collectStreamingResponse(result.response);
|
||||
const totalTime = Date.now() - startTime;
|
||||
const timeToFirstToken = streamResult.firstTokenTime ? streamResult.firstTokenTime - startTime : 0;
|
||||
|
||||
console.log('📊 OpenAI LLM Streaming Results:', {
|
||||
success: streamResult.success,
|
||||
totalTime: `${totalTime}ms`,
|
||||
timeToFirstToken: `${timeToFirstToken}ms`,
|
||||
tokenCount: streamResult.tokenCount,
|
||||
contentLength: streamResult.content.length
|
||||
});
|
||||
|
||||
assert(streamResult.success, 'Streaming should be successful');
|
||||
assert(streamResult.content.length > 0, 'Should receive content');
|
||||
assert(streamResult.tokenCount > 0, 'Should receive tokens');
|
||||
|
||||
if (streamResult.content.length > 0) {
|
||||
console.log('📝 Generated content preview:', streamResult.content.substring(0, 100) + (streamResult.content.length > 100 ? '...' : ''));
|
||||
}
|
||||
|
||||
console.log('✅ OpenAI LLM streaming successful');
|
||||
} catch (streamError) {
|
||||
console.error('❌ OpenAI LLM streaming failed:', streamError.message);
|
||||
assert(false, 'Streaming should not fail: ' + streamError.message);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle conversation context', async function() {
|
||||
console.log('💬 Testing OpenAI LLM with conversation context');
|
||||
|
||||
const result = await provider.createChatCompletion(testMessages.conversation, {
|
||||
max_tokens: 100
|
||||
});
|
||||
|
||||
console.log('📊 OpenAI LLM Conversation Test:', {
|
||||
success: result.success,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ OpenAI LLM conversation handling successful');
|
||||
} else {
|
||||
console.log('❌ OpenAI LLM conversation handling failed:', result.error);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('OpenRouter GPT LLM Provider', function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
if (!testConfig.OPENROUTER_API_KEY) {
|
||||
console.log('⚠️ Skipping OpenRouter GPT tests - OPENROUTER_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = LLMProviderFactory.createProvider('openrouter-gpt', testConfig, testConfig);
|
||||
console.log('✅ OpenRouter GPT Provider created successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Failed to create OpenRouter GPT provider:', error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should initialize with correct configuration', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert.strictEqual(provider.config.model, 'openai/gpt-5.6-luna', 'Should use openai/gpt-5.6-luna model');
|
||||
assert.strictEqual(provider.config.apiUrl, 'https://openrouter.ai/api/v1/chat/completions', 'Should use OpenRouter API URL');
|
||||
assert(provider.apiKey, 'Should have API key');
|
||||
console.log('📋 OpenRouter GPT Config:', {
|
||||
model: provider.config.model,
|
||||
apiUrl: provider.config.apiUrl
|
||||
});
|
||||
});
|
||||
|
||||
it('should generate chat completion via OpenRouter', async function() {
|
||||
console.log('🤖 Testing OpenRouter GPT');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.createChatCompletion(testMessages.simple, {
|
||||
max_tokens: 100
|
||||
});
|
||||
const responseTime = Date.now() - startTime;
|
||||
|
||||
console.log('📊 OpenRouter GPT Test:', {
|
||||
success: result.success,
|
||||
responseTime: `${responseTime}ms`,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert.strictEqual(result.provider, 'openrouter', 'Should identify as openrouter provider');
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ OpenRouter GPT completion successful');
|
||||
} else {
|
||||
console.log('❌ OpenRouter GPT completion failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle streaming via OpenRouter', async function() {
|
||||
console.log('🌊 Testing OpenRouter GPT streaming');
|
||||
|
||||
const result = await provider.createChatCompletion(testMessages.simple, {
|
||||
max_tokens: 50,
|
||||
stream: true
|
||||
});
|
||||
|
||||
if (!result.success) {
|
||||
console.log('❌ OpenRouter GPT streaming setup failed:', result.error);
|
||||
return; // Don't fail the test, just log
|
||||
}
|
||||
|
||||
try {
|
||||
const streamResult = await collectStreamingResponse(result.response);
|
||||
|
||||
console.log('📊 OpenRouter GPT Streaming:', {
|
||||
success: streamResult.success,
|
||||
tokenCount: streamResult.tokenCount,
|
||||
contentLength: streamResult.content.length
|
||||
});
|
||||
|
||||
if (streamResult.success) {
|
||||
console.log('✅ OpenRouter GPT streaming successful');
|
||||
}
|
||||
} catch (streamError) {
|
||||
console.log('❌ OpenRouter GPT streaming error:', streamError.message);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('OpenRouter Gemini LLM Provider', function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
if (!testConfig.OPENROUTER_API_KEY) {
|
||||
console.log('⚠️ Skipping OpenRouter Gemini tests - OPENROUTER_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = LLMProviderFactory.createProvider('openrouter-gemini', testConfig, testConfig);
|
||||
console.log('✅ OpenRouter Gemini Provider created successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Failed to create OpenRouter Gemini provider:', error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should initialize with correct configuration', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert.strictEqual(provider.config.model, 'google/gemini-3.5-flash', 'Should use gemini-3.5-flash model');
|
||||
assert.strictEqual(provider.config.apiUrl, 'https://openrouter.ai/api/v1/chat/completions', 'Should use OpenRouter API URL');
|
||||
assert(provider.apiKey, 'Should have API key');
|
||||
console.log('📋 OpenRouter Gemini Config:', {
|
||||
model: provider.config.model,
|
||||
apiUrl: provider.config.apiUrl
|
||||
});
|
||||
});
|
||||
|
||||
it('should generate chat completion with Gemini', async function() {
|
||||
console.log('🤖 Testing OpenRouter Gemini');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.createChatCompletion(testMessages.simple, {
|
||||
max_tokens: 100
|
||||
});
|
||||
const responseTime = Date.now() - startTime;
|
||||
|
||||
console.log('📊 OpenRouter Gemini Test:', {
|
||||
success: result.success,
|
||||
responseTime: `${responseTime}ms`,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert.strictEqual(result.provider, 'openrouter', 'Should identify as openrouter provider');
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ OpenRouter Gemini completion successful');
|
||||
} else {
|
||||
console.log('❌ OpenRouter Gemini completion failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle creative tasks with Gemini', async function() {
|
||||
console.log('🎨 Testing OpenRouter Gemini creative task');
|
||||
|
||||
const result = await provider.createChatCompletion(testMessages.creative, {
|
||||
max_tokens: 150
|
||||
});
|
||||
|
||||
console.log('📊 OpenRouter Gemini Creative Test:', {
|
||||
success: result.success,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ OpenRouter Gemini creative task successful');
|
||||
} else {
|
||||
console.log('❌ OpenRouter Gemini creative task failed:', result.error);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('ARK Doubao LLM Provider', function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
if (!testConfig.ARK_API_KEY) {
|
||||
console.log('⚠️ Skipping ARK Doubao tests - ARK_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
provider = LLMProviderFactory.createProvider('ark', testConfig, testConfig);
|
||||
console.log('✅ ARK Doubao Provider created successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Failed to create ARK Doubao provider:', error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should initialize with correct configuration', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert.strictEqual(provider.config.model, 'doubao-seed-1-6-flash-250615', 'Should use doubao model');
|
||||
assert.strictEqual(provider.config.apiUrl, 'https://ark.cn-beijing.volces.com/api/v3/chat/completions', 'Should use ARK API URL');
|
||||
assert(provider.apiKey, 'Should have API key');
|
||||
console.log('📋 ARK Doubao Config:', {
|
||||
model: provider.config.model,
|
||||
apiUrl: provider.config.apiUrl
|
||||
});
|
||||
});
|
||||
|
||||
it('should generate chat completion with Doubao', async function() {
|
||||
console.log('🤖 Testing ARK Doubao');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.createChatCompletion(testMessages.simple, {
|
||||
max_tokens: 100
|
||||
});
|
||||
const responseTime = Date.now() - startTime;
|
||||
|
||||
console.log('📊 ARK Doubao Test:', {
|
||||
success: result.success,
|
||||
responseTime: `${responseTime}ms`,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert.strictEqual(result.provider, 'ark', 'Should identify as ark provider');
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ ARK Doubao completion successful');
|
||||
} else {
|
||||
console.log('❌ ARK Doubao completion failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle Chinese language tasks', async function() {
|
||||
console.log('🇨🇳 Testing ARK Doubao with Chinese');
|
||||
|
||||
const chineseMessages = [
|
||||
{ role: 'system', content: '你是一个有用的AI助手。' },
|
||||
{ role: 'user', content: '请用一句话介绍北京。' }
|
||||
];
|
||||
|
||||
const result = await provider.createChatCompletion(chineseMessages, {
|
||||
max_tokens: 100
|
||||
});
|
||||
|
||||
console.log('📊 ARK Doubao Chinese Test:', {
|
||||
success: result.success,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
if (result.success) {
|
||||
console.log('✅ ARK Doubao Chinese handling successful');
|
||||
} else {
|
||||
console.log('❌ ARK Doubao Chinese handling failed:', result.error);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('LLM Provider Performance Comparison', function() {
|
||||
it('should compare provider response times', async function() {
|
||||
const availableProviders = [];
|
||||
|
||||
if (testConfig.OPENAI_API_KEY) availableProviders.push('openai');
|
||||
if (testConfig.OPENROUTER_API_KEY) {
|
||||
availableProviders.push('openrouter-gpt', 'openrouter-gemini');
|
||||
}
|
||||
if (testConfig.ARK_API_KEY) availableProviders.push('ark');
|
||||
|
||||
if (availableProviders.length < 2) {
|
||||
console.log('⚠️ Skipping provider comparison - need at least 2 providers');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
const results = {};
|
||||
console.log('🏁 Comparing LLM provider performance...');
|
||||
|
||||
for (const providerName of availableProviders) {
|
||||
try {
|
||||
const provider = LLMProviderFactory.createProvider(providerName, testConfig, testConfig);
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.createChatCompletion(testMessages.simple, {
|
||||
max_tokens: 50
|
||||
});
|
||||
const duration = Date.now() - startTime;
|
||||
|
||||
results[providerName] = {
|
||||
success: result.success,
|
||||
duration: duration,
|
||||
error: result.error
|
||||
};
|
||||
} catch (error) {
|
||||
results[providerName] = {
|
||||
success: false,
|
||||
duration: 0,
|
||||
error: error.message
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
console.log('📊 LLM Provider Performance Comparison:');
|
||||
Object.entries(results).forEach(([provider, result]) => {
|
||||
console.log(` ${provider}:`, {
|
||||
success: result.success ? '✅' : '❌',
|
||||
time: `${result.duration}ms`,
|
||||
error: result.error || 'none'
|
||||
});
|
||||
});
|
||||
|
||||
// Find fastest successful provider
|
||||
const successfulProviders = Object.entries(results).filter(([_, result]) => result.success);
|
||||
if (successfulProviders.length > 0) {
|
||||
const fastest = successfulProviders.reduce((prev, curr) =>
|
||||
prev[1].duration < curr[1].duration ? prev : curr
|
||||
);
|
||||
console.log(`🏆 Fastest LLM provider: ${fastest[0]} (${fastest[1].duration}ms)`);
|
||||
}
|
||||
|
||||
// At least one provider should work
|
||||
assert(successfulProviders.length > 0, 'At least one LLM provider should be successful');
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
module.exports = {
|
||||
testConfig,
|
||||
testMessages,
|
||||
collectStreamingResponse
|
||||
};
|
||||
@@ -0,0 +1,424 @@
|
||||
const assert = require('assert');
|
||||
const axios = require('axios');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
// Test configuration
|
||||
const testConfig = {
|
||||
SILICONFLOW_API_KEY: process.env.SILICONFLOW_API_KEY,
|
||||
|
||||
TTS_PROVIDERS: {
|
||||
siliconflow: {
|
||||
apiUrl: 'https://api.siliconflow.cn/v1/audio/speech',
|
||||
model: 'FunAudioLLM/CosyVoice2-0.5B',
|
||||
voice: 'FunAudioLLM/CosyVoice2-0.5B:diana',
|
||||
apiKey: 'SILICONFLOW_API_KEY'
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// Test texts for different scenarios
|
||||
const testTexts = {
|
||||
simple: 'Hello, this is a simple text-to-speech test.',
|
||||
multilingual: 'Hello world. 你好世界. こんにちは世界.',
|
||||
punctuation: 'Testing punctuation: question? exclamation! comma, period.',
|
||||
numbers: 'The year is 2025, and the time is 12:34 PM.',
|
||||
long: 'This is a longer text to test the text-to-speech synthesis capability. It contains multiple sentences and should demonstrate the natural flow of speech generation. The quality and naturalness of the audio output will be evaluated.'
|
||||
};
|
||||
|
||||
/**
|
||||
* TTS Provider for CosyVoice2 via Siliconflow
|
||||
*/
|
||||
class SiliconflowTTSProvider {
|
||||
constructor(config, apiKey) {
|
||||
this.config = config;
|
||||
this.apiKey = apiKey;
|
||||
}
|
||||
|
||||
async synthesize(text, options = {}) {
|
||||
try {
|
||||
const response = await axios({
|
||||
method: 'post',
|
||||
url: this.config.apiUrl,
|
||||
data: {
|
||||
model: this.config.model,
|
||||
input: text,
|
||||
voice: options.voice || this.config.voice,
|
||||
response_format: options.format || 'mp3',
|
||||
sample_rate: options.sampleRate || 32000,
|
||||
stream: options.stream || false,
|
||||
speed: options.speed || 1.0,
|
||||
gain: options.gain || 0
|
||||
},
|
||||
headers: {
|
||||
'Authorization': `Bearer ${this.apiKey}`,
|
||||
'Content-Type': 'application/json'
|
||||
},
|
||||
responseType: 'arraybuffer',
|
||||
timeout: 60000 // 60 second timeout
|
||||
});
|
||||
|
||||
const result = {
|
||||
success: true,
|
||||
audioData: Buffer.from(response.data),
|
||||
format: options.format || 'mp3',
|
||||
sampleRate: options.sampleRate || 32000,
|
||||
provider: 'siliconflow',
|
||||
timestamp: Date.now()
|
||||
};
|
||||
|
||||
return result;
|
||||
|
||||
} catch (error) {
|
||||
console.error('Siliconflow TTS error:', error.response?.data || error.message);
|
||||
|
||||
return {
|
||||
success: false,
|
||||
audioData: null,
|
||||
error: error.response?.data?.error?.message || error.message,
|
||||
provider: 'siliconflow',
|
||||
timestamp: Date.now()
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* TTS Provider Individual Tests
|
||||
*/
|
||||
describe('TTS Providers - Individual Testing', function() {
|
||||
this.timeout(120000); // 2 minute timeout for API calls
|
||||
|
||||
const tempDir = path.join(__dirname, '../temp');
|
||||
|
||||
before(function() {
|
||||
// Ensure temp directory exists
|
||||
if (!fs.existsSync(tempDir)) {
|
||||
fs.mkdirSync(tempDir, { recursive: true });
|
||||
}
|
||||
});
|
||||
|
||||
describe('CosyVoice2 (Siliconflow) TTS Provider', function() {
|
||||
let provider;
|
||||
|
||||
before(function() {
|
||||
if (!testConfig.SILICONFLOW_API_KEY) {
|
||||
console.log('⚠️ Skipping Siliconflow TTS tests - SILICONFLOW_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
try {
|
||||
const providerConfig = testConfig.TTS_PROVIDERS.siliconflow;
|
||||
provider = new SiliconflowTTSProvider(providerConfig, testConfig.SILICONFLOW_API_KEY);
|
||||
console.log('✅ Siliconflow TTS Provider created successfully');
|
||||
} catch (error) {
|
||||
console.error('❌ Failed to create Siliconflow TTS provider:', error);
|
||||
this.skip();
|
||||
}
|
||||
});
|
||||
|
||||
it('should initialize with correct configuration', function() {
|
||||
assert(provider, 'Provider should be created');
|
||||
assert.strictEqual(provider.config.model, 'FunAudioLLM/CosyVoice2-0.5B', 'Should use CosyVoice2 model');
|
||||
assert.strictEqual(provider.config.voice, 'FunAudioLLM/CosyVoice2-0.5B:diana', 'Should use diana voice');
|
||||
assert.strictEqual(provider.config.apiUrl, 'https://api.siliconflow.cn/v1/audio/speech', 'Should use correct API URL');
|
||||
assert(provider.apiKey, 'Should have API key');
|
||||
console.log('📋 Siliconflow TTS Config:', {
|
||||
model: provider.config.model,
|
||||
voice: provider.config.voice,
|
||||
apiUrl: provider.config.apiUrl
|
||||
});
|
||||
});
|
||||
|
||||
it('should synthesize simple text to speech', async function() {
|
||||
console.log('🎵 Testing Siliconflow TTS with simple text');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.synthesize(testTexts.simple);
|
||||
const synthesisTime = Date.now() - startTime;
|
||||
|
||||
console.log('📊 Siliconflow TTS Simple Test:', {
|
||||
success: result.success,
|
||||
synthesisTime: `${synthesisTime}ms`,
|
||||
audioSize: result.audioData ? `${result.audioData.length} bytes` : 'none',
|
||||
format: result.format,
|
||||
sampleRate: result.sampleRate,
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
assert.strictEqual(result.provider, 'siliconflow', 'Should identify as siliconflow provider');
|
||||
|
||||
if (result.success) {
|
||||
assert(result.audioData, 'Should have audio data');
|
||||
assert(result.audioData.length > 0, 'Audio data should not be empty');
|
||||
assert.strictEqual(result.format, 'mp3', 'Should return MP3 format');
|
||||
|
||||
// Save test audio file
|
||||
const testAudioPath = path.join(tempDir, `tts_simple_${Date.now()}.mp3`);
|
||||
fs.writeFileSync(testAudioPath, result.audioData);
|
||||
console.log('💾 Saved test audio to:', testAudioPath);
|
||||
console.log('✅ Siliconflow TTS simple synthesis successful');
|
||||
} else {
|
||||
console.log('❌ Siliconflow TTS simple synthesis failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle multilingual text', async function() {
|
||||
console.log('🌍 Testing Siliconflow TTS with multilingual text');
|
||||
|
||||
const result = await provider.synthesize(testTexts.multilingual);
|
||||
|
||||
console.log('📊 Siliconflow TTS Multilingual Test:', {
|
||||
success: result.success,
|
||||
audioSize: result.audioData ? `${result.audioData.length} bytes` : 'none',
|
||||
provider: result.provider
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
|
||||
if (result.success) {
|
||||
assert(result.audioData, 'Should have audio data for multilingual text');
|
||||
assert(result.audioData.length > 0, 'Multilingual audio data should not be empty');
|
||||
|
||||
// Save multilingual test audio
|
||||
const testAudioPath = path.join(tempDir, `tts_multilingual_${Date.now()}.mp3`);
|
||||
fs.writeFileSync(testAudioPath, result.audioData);
|
||||
console.log('💾 Saved multilingual audio to:', testAudioPath);
|
||||
console.log('✅ Siliconflow TTS multilingual synthesis successful');
|
||||
} else {
|
||||
console.log('❌ Siliconflow TTS multilingual synthesis failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle different audio formats and settings', async function() {
|
||||
console.log('⚙️ Testing Siliconflow TTS with different settings');
|
||||
|
||||
const settings = [
|
||||
{ format: 'mp3', sampleRate: 24000, speed: 1.0 },
|
||||
{ format: 'mp3', sampleRate: 32000, speed: 1.2 },
|
||||
{ format: 'mp3', sampleRate: 16000, speed: 0.8 }
|
||||
];
|
||||
|
||||
for (const setting of settings) {
|
||||
const result = await provider.synthesize(testTexts.simple, setting);
|
||||
|
||||
console.log(`📊 TTS Settings Test (${setting.format}, ${setting.sampleRate}Hz, ${setting.speed}x):`, {
|
||||
success: result.success,
|
||||
audioSize: result.audioData ? `${result.audioData.length} bytes` : 'none',
|
||||
error: result.error || 'none'
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
|
||||
if (result.success) {
|
||||
assert(result.audioData, 'Should have audio data');
|
||||
assert(result.audioData.length > 0, 'Audio data should not be empty');
|
||||
|
||||
// Save test audio with settings info
|
||||
const filename = `tts_settings_${setting.sampleRate}hz_${setting.speed}x_${Date.now()}.mp3`;
|
||||
const testAudioPath = path.join(tempDir, filename);
|
||||
fs.writeFileSync(testAudioPath, result.audioData);
|
||||
console.log(`💾 Saved settings test audio to: ${testAudioPath}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('✅ Siliconflow TTS settings variation tests completed');
|
||||
});
|
||||
|
||||
it('should handle punctuation and numbers correctly', async function() {
|
||||
console.log('🔢 Testing Siliconflow TTS with punctuation and numbers');
|
||||
|
||||
const punctuationResult = await provider.synthesize(testTexts.punctuation);
|
||||
const numbersResult = await provider.synthesize(testTexts.numbers);
|
||||
|
||||
console.log('📊 Siliconflow TTS Punctuation Test:', {
|
||||
success: punctuationResult.success,
|
||||
audioSize: punctuationResult.audioData ? `${punctuationResult.audioData.length} bytes` : 'none'
|
||||
});
|
||||
|
||||
console.log('📊 Siliconflow TTS Numbers Test:', {
|
||||
success: numbersResult.success,
|
||||
audioSize: numbersResult.audioData ? `${numbersResult.audioData.length} bytes` : 'none'
|
||||
});
|
||||
|
||||
assert(punctuationResult, 'Should handle punctuation');
|
||||
assert(numbersResult, 'Should handle numbers');
|
||||
|
||||
if (punctuationResult.success && numbersResult.success) {
|
||||
console.log('✅ Siliconflow TTS punctuation and numbers handling successful');
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle longer text synthesis', async function() {
|
||||
console.log('📝 Testing Siliconflow TTS with longer text');
|
||||
|
||||
const startTime = Date.now();
|
||||
const result = await provider.synthesize(testTexts.long);
|
||||
const synthesisTime = Date.now() - startTime;
|
||||
|
||||
console.log('📊 Siliconflow TTS Long Text Test:', {
|
||||
success: result.success,
|
||||
synthesisTime: `${synthesisTime}ms`,
|
||||
audioSize: result.audioData ? `${result.audioData.length} bytes` : 'none',
|
||||
textLength: testTexts.long.length
|
||||
});
|
||||
|
||||
assert(result, 'Should return a result');
|
||||
assert(typeof result.success === 'boolean', 'Should have success field');
|
||||
|
||||
if (result.success) {
|
||||
assert(result.audioData, 'Should have audio data for long text');
|
||||
assert(result.audioData.length > 0, 'Long text audio data should not be empty');
|
||||
|
||||
// Long text should produce more audio data
|
||||
const expectedMinSize = 50000; // Roughly 50KB minimum for longer text
|
||||
if (result.audioData.length > expectedMinSize) {
|
||||
console.log('✅ Long text produced appropriate amount of audio data');
|
||||
}
|
||||
|
||||
// Save long text audio
|
||||
const testAudioPath = path.join(tempDir, `tts_long_${Date.now()}.mp3`);
|
||||
fs.writeFileSync(testAudioPath, result.audioData);
|
||||
console.log('💾 Saved long text audio to:', testAudioPath);
|
||||
console.log('✅ Siliconflow TTS long text synthesis successful');
|
||||
} else {
|
||||
console.log('❌ Siliconflow TTS long text synthesis failed:', result.error);
|
||||
}
|
||||
});
|
||||
|
||||
it('should handle empty or invalid text gracefully', async function() {
|
||||
console.log('🔍 Testing Siliconflow TTS with edge cases');
|
||||
|
||||
const emptyResult = await provider.synthesize('');
|
||||
const spaceResult = await provider.synthesize(' ');
|
||||
const specialResult = await provider.synthesize('!@#$%^&*()');
|
||||
|
||||
console.log('📊 Siliconflow TTS Edge Cases:', {
|
||||
empty: { success: emptyResult.success, error: emptyResult.error || 'none' },
|
||||
spaces: { success: spaceResult.success, error: spaceResult.error || 'none' },
|
||||
special: { success: specialResult.success, error: specialResult.error || 'none' }
|
||||
});
|
||||
|
||||
// These should either work or fail gracefully
|
||||
assert(emptyResult, 'Should handle empty string');
|
||||
assert(spaceResult, 'Should handle whitespace');
|
||||
assert(specialResult, 'Should handle special characters');
|
||||
|
||||
console.log('✅ Siliconflow TTS edge cases handled');
|
||||
});
|
||||
|
||||
it('should measure synthesis performance metrics', async function() {
|
||||
console.log('🏁 Testing Siliconflow TTS performance metrics');
|
||||
|
||||
const testText = testTexts.simple;
|
||||
const iterations = 3;
|
||||
const results = [];
|
||||
|
||||
for (let i = 0; i < iterations; i++) {
|
||||
const startTime = Date.now();
|
||||
const result = await provider.synthesize(testText);
|
||||
const duration = Date.now() - startTime;
|
||||
|
||||
if (result.success) {
|
||||
results.push({
|
||||
duration: duration,
|
||||
audioSize: result.audioData.length,
|
||||
success: true
|
||||
});
|
||||
} else {
|
||||
results.push({
|
||||
duration: duration,
|
||||
audioSize: 0,
|
||||
success: false,
|
||||
error: result.error
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const successfulResults = results.filter(r => r.success);
|
||||
|
||||
if (successfulResults.length > 0) {
|
||||
const avgDuration = successfulResults.reduce((sum, r) => sum + r.duration, 0) / successfulResults.length;
|
||||
const avgAudioSize = successfulResults.reduce((sum, r) => sum + r.audioSize, 0) / successfulResults.length;
|
||||
const minDuration = Math.min(...successfulResults.map(r => r.duration));
|
||||
const maxDuration = Math.max(...successfulResults.map(r => r.duration));
|
||||
|
||||
console.log('📊 Siliconflow TTS Performance Metrics:', {
|
||||
successRate: `${successfulResults.length}/${iterations}`,
|
||||
avgSynthesisTime: `${Math.round(avgDuration)}ms`,
|
||||
minSynthesisTime: `${minDuration}ms`,
|
||||
maxSynthesisTime: `${maxDuration}ms`,
|
||||
avgAudioSize: `${Math.round(avgAudioSize)} bytes`,
|
||||
textLength: testText.length
|
||||
});
|
||||
|
||||
// Performance expectations
|
||||
assert(avgDuration < 10000, 'Average synthesis time should be under 10 seconds');
|
||||
assert(avgAudioSize > 1000, 'Should produce reasonable amount of audio data');
|
||||
|
||||
console.log('✅ Siliconflow TTS performance metrics acceptable');
|
||||
} else {
|
||||
console.log('❌ No successful TTS synthesis results for performance testing');
|
||||
}
|
||||
|
||||
assert(successfulResults.length > 0, 'At least one synthesis attempt should succeed');
|
||||
});
|
||||
});
|
||||
|
||||
describe('TTS Provider Integration', function() {
|
||||
it('should integrate with live audio system', function() {
|
||||
if (!testConfig.SILICONFLOW_API_KEY) {
|
||||
console.log('⚠️ Skipping TTS integration test - SILICONFLOW_API_KEY not found');
|
||||
this.skip();
|
||||
}
|
||||
|
||||
// Simulate the TTS configuration that would be used in the live system
|
||||
const liveConfig = {
|
||||
TTS_API_URL: 'https://api.siliconflow.cn/v1/audio/speech',
|
||||
TTS_PROVIDERS: testConfig.TTS_PROVIDERS,
|
||||
TTS_PROVIDER: 'siliconflow',
|
||||
SILICONFLOW_API_KEY: testConfig.SILICONFLOW_API_KEY
|
||||
};
|
||||
|
||||
assert(liveConfig.TTS_API_URL, 'Should have TTS API URL');
|
||||
assert(liveConfig.TTS_PROVIDERS.siliconflow, 'Should have Siliconflow TTS provider config');
|
||||
assert(liveConfig.SILICONFLOW_API_KEY, 'Should have Siliconflow API key');
|
||||
|
||||
console.log('📋 Live System TTS Integration Config:', {
|
||||
provider: liveConfig.TTS_PROVIDER,
|
||||
model: liveConfig.TTS_PROVIDERS.siliconflow.model,
|
||||
voice: liveConfig.TTS_PROVIDERS.siliconflow.voice,
|
||||
hasApiKey: !!liveConfig.SILICONFLOW_API_KEY
|
||||
});
|
||||
|
||||
console.log('✅ TTS provider integration configuration valid');
|
||||
});
|
||||
});
|
||||
|
||||
after(function() {
|
||||
// Cleanup temp audio files
|
||||
try {
|
||||
const files = fs.readdirSync(tempDir);
|
||||
let cleanedCount = 0;
|
||||
files.forEach(file => {
|
||||
if (file.startsWith('tts_') && file.endsWith('.mp3')) {
|
||||
fs.unlinkSync(path.join(tempDir, file));
|
||||
cleanedCount++;
|
||||
}
|
||||
});
|
||||
console.log(`🧹 Cleaned up ${cleanedCount} temporary TTS audio files`);
|
||||
} catch (error) {
|
||||
console.warn('⚠️ Failed to cleanup temp TTS files:', error.message);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
module.exports = {
|
||||
testConfig,
|
||||
testTexts,
|
||||
SiliconflowTTSProvider
|
||||
};
|
||||
@@ -0,0 +1,276 @@
|
||||
const axios = require('axios');
|
||||
const FormData = require('form-data');
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
/**
|
||||
* Base ASR Provider class
|
||||
*/
|
||||
class BaseASRProvider {
|
||||
constructor(config, apiKey) {
|
||||
this.config = config;
|
||||
this.apiKey = apiKey;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert raw audio buffer to WAV format
|
||||
* @param {Buffer} audioBuffer - Raw PCM audio data
|
||||
* @param {Object} options - Audio format options
|
||||
* @returns {Buffer} WAV formatted audio data
|
||||
*/
|
||||
createWavBuffer(audioBuffer, options = {}) {
|
||||
const sampleRate = options.sampleRate || 16000;
|
||||
const channels = options.channels || 1;
|
||||
const bitsPerSample = options.bitsPerSample || 16;
|
||||
|
||||
const byteRate = sampleRate * channels * bitsPerSample / 8;
|
||||
const blockAlign = channels * bitsPerSample / 8;
|
||||
const dataSize = audioBuffer.length;
|
||||
const fileSize = 36 + dataSize;
|
||||
|
||||
const header = Buffer.alloc(44);
|
||||
|
||||
// RIFF header
|
||||
header.write('RIFF', 0);
|
||||
header.writeUInt32LE(fileSize, 4);
|
||||
header.write('WAVE', 8);
|
||||
|
||||
// fmt chunk
|
||||
header.write('fmt ', 12);
|
||||
header.writeUInt32LE(16, 16); // PCM format chunk size
|
||||
header.writeUInt16LE(1, 20); // PCM format
|
||||
header.writeUInt16LE(channels, 22);
|
||||
header.writeUInt32LE(sampleRate, 24);
|
||||
header.writeUInt32LE(byteRate, 28);
|
||||
header.writeUInt16LE(blockAlign, 32);
|
||||
header.writeUInt16LE(bitsPerSample, 34);
|
||||
|
||||
// data chunk
|
||||
header.write('data', 36);
|
||||
header.writeUInt32LE(dataSize, 40);
|
||||
|
||||
return Buffer.concat([header, audioBuffer]);
|
||||
}
|
||||
|
||||
async transcribe(audioBuffer, options = {}) {
|
||||
throw new Error('transcribe method must be implemented by subclass');
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* OpenAI Whisper ASR Provider
|
||||
*/
|
||||
class OpenAIASRProvider extends BaseASRProvider {
|
||||
async transcribe(audioBuffer, tempDir, options = {}) {
|
||||
try {
|
||||
// Create WAV buffer
|
||||
const wavBuffer = this.createWavBuffer(audioBuffer, {
|
||||
sampleRate: 16000,
|
||||
channels: 1,
|
||||
bitsPerSample: 16
|
||||
});
|
||||
|
||||
// Create temporary file
|
||||
const tempFileName = `audio_${Date.now()}_${Math.random().toString(36).substring(2)}.wav`;
|
||||
const tempFilePath = path.join(tempDir, tempFileName);
|
||||
|
||||
// Write audio to temporary file
|
||||
fs.writeFileSync(tempFilePath, wavBuffer);
|
||||
|
||||
try {
|
||||
// Create form data
|
||||
const formData = new FormData();
|
||||
formData.append('file', fs.createReadStream(tempFilePath));
|
||||
formData.append('model', this.config.model);
|
||||
formData.append('response_format', 'json');
|
||||
|
||||
if (options.language) {
|
||||
formData.append('language', options.language);
|
||||
}
|
||||
|
||||
if (options.prompt) {
|
||||
formData.append('prompt', options.prompt);
|
||||
}
|
||||
|
||||
// Make API request
|
||||
const response = await axios({
|
||||
method: 'post',
|
||||
url: this.config.apiUrl,
|
||||
data: formData,
|
||||
headers: {
|
||||
'Authorization': `Bearer ${this.apiKey}`,
|
||||
...formData.getHeaders()
|
||||
},
|
||||
timeout: 30000 // 30 second timeout
|
||||
});
|
||||
|
||||
const result = {
|
||||
success: true,
|
||||
text: response.data.text || '',
|
||||
language: response.data.language || 'unknown',
|
||||
duration: response.data.duration || 0,
|
||||
confidence: response.data.confidence || 1.0,
|
||||
requestId: response.headers?.['x-request-id'] || response.headers?.['request-id'] || null,
|
||||
responseModel: response.data.model || this.config.model,
|
||||
timestamp: Date.now(),
|
||||
provider: 'openai'
|
||||
};
|
||||
|
||||
console.log('OpenAI ASR Result:', {
|
||||
text: result.text,
|
||||
language: result.language,
|
||||
duration: result.duration
|
||||
});
|
||||
|
||||
return result;
|
||||
|
||||
} finally {
|
||||
// Clean up temporary file
|
||||
try {
|
||||
fs.unlinkSync(tempFilePath);
|
||||
} catch (cleanupError) {
|
||||
console.warn('Failed to cleanup temp file:', cleanupError.message);
|
||||
}
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('OpenAI ASR error:', error.response?.data || error.message);
|
||||
|
||||
return {
|
||||
success: false,
|
||||
text: '',
|
||||
error: error.response?.data?.error?.message || error.message,
|
||||
timestamp: Date.now(),
|
||||
provider: 'openai'
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* SenseVoice via Siliconflow ASR Provider
|
||||
*/
|
||||
class SiliconflowASRProvider extends BaseASRProvider {
|
||||
async transcribe(audioBuffer, tempDir, options = {}) {
|
||||
try {
|
||||
// Create WAV buffer
|
||||
const wavBuffer = this.createWavBuffer(audioBuffer, {
|
||||
sampleRate: 16000,
|
||||
channels: 1,
|
||||
bitsPerSample: 16
|
||||
});
|
||||
|
||||
// Create temporary file
|
||||
const tempFileName = `audio_${Date.now()}_${Math.random().toString(36).substring(2)}.wav`;
|
||||
const tempFilePath = path.join(tempDir, tempFileName);
|
||||
|
||||
// Write audio to temporary file
|
||||
fs.writeFileSync(tempFilePath, wavBuffer);
|
||||
|
||||
try {
|
||||
// Create form data
|
||||
const formData = new FormData();
|
||||
formData.append('file', fs.createReadStream(tempFilePath));
|
||||
formData.append('model', this.config.model);
|
||||
formData.append('response_format', 'json');
|
||||
|
||||
// SenseVoice specific parameters
|
||||
if (options.language) {
|
||||
formData.append('language', options.language);
|
||||
} else {
|
||||
// SenseVoice supports auto language detection
|
||||
formData.append('language', 'auto');
|
||||
}
|
||||
|
||||
if (options.prompt) {
|
||||
formData.append('prompt', options.prompt);
|
||||
}
|
||||
|
||||
// Make API request
|
||||
const response = await axios({
|
||||
method: 'post',
|
||||
url: this.config.apiUrl,
|
||||
data: formData,
|
||||
headers: {
|
||||
'Authorization': `Bearer ${this.apiKey}`,
|
||||
...formData.getHeaders()
|
||||
},
|
||||
timeout: 30000 // 30 second timeout
|
||||
});
|
||||
|
||||
const result = {
|
||||
success: true,
|
||||
text: response.data.text || '',
|
||||
language: response.data.language || 'unknown',
|
||||
duration: response.data.duration || 0,
|
||||
confidence: response.data.confidence || 1.0,
|
||||
requestId: response.headers?.['x-request-id'] || response.headers?.['request-id'] || null,
|
||||
responseModel: response.data.model || this.config.model,
|
||||
timestamp: Date.now(),
|
||||
provider: 'siliconflow'
|
||||
};
|
||||
|
||||
console.log('SenseVoice ASR Result:', {
|
||||
text: result.text,
|
||||
language: result.language,
|
||||
duration: result.duration
|
||||
});
|
||||
|
||||
return result;
|
||||
|
||||
} finally {
|
||||
// Clean up temporary file
|
||||
try {
|
||||
fs.unlinkSync(tempFilePath);
|
||||
} catch (cleanupError) {
|
||||
console.warn('Failed to cleanup temp file:', cleanupError.message);
|
||||
}
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('SenseVoice ASR error:', error.response?.data || error.message);
|
||||
|
||||
return {
|
||||
success: false,
|
||||
text: '',
|
||||
error: error.response?.data?.error?.message || error.message,
|
||||
timestamp: Date.now(),
|
||||
provider: 'siliconflow'
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* ASR Provider Factory
|
||||
*/
|
||||
class ASRProviderFactory {
|
||||
static createProvider(providerName, config, globalConfig) {
|
||||
const providerConfig = config.ASR_PROVIDERS[providerName];
|
||||
if (!providerConfig) {
|
||||
throw new Error(`ASR provider ${providerName} not found in configuration`);
|
||||
}
|
||||
|
||||
// Get API key from global config
|
||||
const apiKey = globalConfig[providerConfig.apiKey];
|
||||
if (!apiKey) {
|
||||
throw new Error(`API key ${providerConfig.apiKey} not found in configuration`);
|
||||
}
|
||||
|
||||
switch (providerName) {
|
||||
case 'openai':
|
||||
return new OpenAIASRProvider(providerConfig, apiKey);
|
||||
case 'siliconflow':
|
||||
return new SiliconflowASRProvider(providerConfig, apiKey);
|
||||
default:
|
||||
throw new Error(`Unsupported ASR provider: ${providerName}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
BaseASRProvider,
|
||||
OpenAIASRProvider,
|
||||
SiliconflowASRProvider,
|
||||
ASRProviderFactory
|
||||
};
|
||||
@@ -0,0 +1,248 @@
|
||||
const axios = require('axios');
|
||||
|
||||
function providerErrorMessage(error) {
|
||||
const data = error.response?.data;
|
||||
if (data && typeof data === 'object' && !data.readable) {
|
||||
return data.error?.message || data.message || error.message;
|
||||
}
|
||||
return error.response?.status
|
||||
? `HTTP ${error.response.status}: ${error.response.statusText || error.message}`
|
||||
: error.message;
|
||||
}
|
||||
|
||||
/**
|
||||
* Base LLM Provider class
|
||||
*/
|
||||
class BaseLLMProvider {
|
||||
constructor(config, apiKey) {
|
||||
this.config = config;
|
||||
this.apiKey = apiKey;
|
||||
}
|
||||
|
||||
async createChatCompletion(messages, options = {}) {
|
||||
throw new Error('createChatCompletion method must be implemented by subclass');
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepare headers for API request
|
||||
* @returns {Object} Headers object
|
||||
*/
|
||||
getHeaders() {
|
||||
return {
|
||||
'Authorization': `Bearer ${this.apiKey}`,
|
||||
'Content-Type': 'application/json'
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Prepare request payload
|
||||
* @param {Array} messages - Chat messages
|
||||
* @param {Object} options - Additional options
|
||||
* @returns {Object} Request payload
|
||||
*/
|
||||
getRequestPayload(messages, options = {}) {
|
||||
return {
|
||||
model: this.config.model,
|
||||
messages: messages,
|
||||
stream: true,
|
||||
max_tokens: options.max_tokens || 4096,
|
||||
...options
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* OpenAI LLM Provider
|
||||
*/
|
||||
class OpenAILLMProvider extends BaseLLMProvider {
|
||||
async createChatCompletion(messages, options = {}) {
|
||||
try {
|
||||
const payload = this.getRequestPayload(messages, options);
|
||||
const headers = this.getHeaders();
|
||||
|
||||
console.log('OpenAI LLM Request:', {
|
||||
model: payload.model,
|
||||
messagesCount: messages.length,
|
||||
stream: payload.stream
|
||||
});
|
||||
|
||||
const response = await axios.post(
|
||||
this.config.apiUrl,
|
||||
payload,
|
||||
{
|
||||
headers: headers,
|
||||
cancelToken: options.cancelToken,
|
||||
responseType: 'stream'
|
||||
}
|
||||
);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
response: response,
|
||||
provider: 'openai'
|
||||
};
|
||||
|
||||
} catch (error) {
|
||||
const message = providerErrorMessage(error);
|
||||
console.error('OpenAI LLM error:', message);
|
||||
return {
|
||||
success: false,
|
||||
error: message,
|
||||
provider: 'openai'
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* OpenRouter LLM Provider (supports both GPT-4o and Gemini)
|
||||
*/
|
||||
class OpenRouterLLMProvider extends BaseLLMProvider {
|
||||
getHeaders() {
|
||||
return {
|
||||
'Authorization': `Bearer ${this.apiKey}`,
|
||||
'Content-Type': 'application/json',
|
||||
'HTTP-Referer': 'https://live-audio-chat.local',
|
||||
'X-Title': 'Live Audio Chat'
|
||||
};
|
||||
}
|
||||
|
||||
async createChatCompletion(messages, options = {}) {
|
||||
try {
|
||||
const payload = this.getRequestPayload(messages, options);
|
||||
const headers = this.getHeaders();
|
||||
|
||||
console.log('OpenRouter LLM Request:', {
|
||||
model: payload.model,
|
||||
messagesCount: messages.length,
|
||||
stream: payload.stream
|
||||
});
|
||||
|
||||
const response = await axios.post(
|
||||
this.config.apiUrl,
|
||||
payload,
|
||||
{
|
||||
headers: headers,
|
||||
cancelToken: options.cancelToken,
|
||||
responseType: 'stream'
|
||||
}
|
||||
);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
response: response,
|
||||
provider: 'openrouter'
|
||||
};
|
||||
|
||||
} catch (error) {
|
||||
const message = providerErrorMessage(error);
|
||||
console.error('OpenRouter LLM error:', message);
|
||||
return {
|
||||
success: false,
|
||||
error: message,
|
||||
provider: 'openrouter'
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* ARK (Doubao) LLM Provider
|
||||
*/
|
||||
class ARKLLMProvider extends BaseLLMProvider {
|
||||
getHeaders() {
|
||||
return {
|
||||
'Authorization': `Bearer ${this.apiKey}`,
|
||||
'Content-Type': 'application/json'
|
||||
};
|
||||
}
|
||||
|
||||
getRequestPayload(messages, options = {}) {
|
||||
// ARK API format is similar to OpenAI but may have slight differences
|
||||
return {
|
||||
model: this.config.model,
|
||||
messages: messages,
|
||||
stream: true,
|
||||
max_tokens: options.max_tokens || 4096,
|
||||
temperature: options.temperature || 0.7,
|
||||
...options
|
||||
};
|
||||
}
|
||||
|
||||
async createChatCompletion(messages, options = {}) {
|
||||
try {
|
||||
const payload = this.getRequestPayload(messages, options);
|
||||
const headers = this.getHeaders();
|
||||
|
||||
console.log('ARK (Doubao) LLM Request:', {
|
||||
model: payload.model,
|
||||
messagesCount: messages.length,
|
||||
stream: payload.stream
|
||||
});
|
||||
|
||||
const response = await axios.post(
|
||||
this.config.apiUrl,
|
||||
payload,
|
||||
{
|
||||
headers: headers,
|
||||
cancelToken: options.cancelToken,
|
||||
responseType: 'stream'
|
||||
}
|
||||
);
|
||||
|
||||
return {
|
||||
success: true,
|
||||
response: response,
|
||||
provider: 'ark'
|
||||
};
|
||||
|
||||
} catch (error) {
|
||||
const message = providerErrorMessage(error);
|
||||
console.error('ARK (Doubao) LLM error:', message);
|
||||
return {
|
||||
success: false,
|
||||
error: message,
|
||||
provider: 'ark'
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* LLM Provider Factory
|
||||
*/
|
||||
class LLMProviderFactory {
|
||||
static createProvider(providerName, config, globalConfig) {
|
||||
const providerConfig = config.LLM_PROVIDERS[providerName];
|
||||
if (!providerConfig) {
|
||||
throw new Error(`LLM provider ${providerName} not found in configuration`);
|
||||
}
|
||||
|
||||
// Get API key from global config
|
||||
const apiKey = globalConfig[providerConfig.apiKey];
|
||||
if (!apiKey) {
|
||||
throw new Error(`API key ${providerConfig.apiKey} not found in configuration`);
|
||||
}
|
||||
|
||||
switch (providerName) {
|
||||
case 'openai':
|
||||
return new OpenAILLMProvider(providerConfig, apiKey);
|
||||
case 'openrouter':
|
||||
case 'openrouter-gpt':
|
||||
case 'openrouter-gemini':
|
||||
return new OpenRouterLLMProvider(providerConfig, apiKey);
|
||||
case 'ark':
|
||||
return new ARKLLMProvider(providerConfig, apiKey);
|
||||
default:
|
||||
throw new Error(`Unsupported LLM provider: ${providerName}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
BaseLLMProvider,
|
||||
OpenAILLMProvider,
|
||||
OpenRouterLLMProvider,
|
||||
ARKLLMProvider,
|
||||
LLMProviderFactory
|
||||
};
|
||||
@@ -0,0 +1,147 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const config = require('../config');
|
||||
const { ASRProviderFactory } = require('./providers/asrProviders');
|
||||
|
||||
class SpeechToTextService {
|
||||
constructor() {
|
||||
this.tempDir = path.join(__dirname, '../temp');
|
||||
this.ensureTempDirectory();
|
||||
|
||||
// Initialize ASR provider based on configuration
|
||||
this.initializeProvider();
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize ASR provider based on configuration
|
||||
*/
|
||||
initializeProvider() {
|
||||
try {
|
||||
const providerName = config.ASR_PROVIDER || 'openai';
|
||||
this.asrProvider = ASRProviderFactory.createProvider(providerName, config, config);
|
||||
console.log(`ASR Provider initialized: ${providerName}`);
|
||||
} catch (error) {
|
||||
console.error('Failed to initialize ASR provider:', error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Switch ASR provider dynamically
|
||||
* @param {string} providerName - Provider name to switch to
|
||||
*/
|
||||
switchProvider(providerName) {
|
||||
try {
|
||||
this.asrProvider = ASRProviderFactory.createProvider(providerName, config, config);
|
||||
console.log(`ASR Provider switched to: ${providerName}`);
|
||||
} catch (error) {
|
||||
console.error('Failed to switch ASR provider:', error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure temp directory exists
|
||||
*/
|
||||
ensureTempDirectory() {
|
||||
if (!fs.existsSync(this.tempDir)) {
|
||||
fs.mkdirSync(this.tempDir, { recursive: true });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Transcribe audio using the configured ASR provider
|
||||
* @param {Buffer} audioBuffer - Raw audio data
|
||||
* @param {Object} options - Transcription options
|
||||
* @returns {Promise<Object>} Transcription result
|
||||
*/
|
||||
async transcribeAudio(audioBuffer, options = {}) {
|
||||
if (!this.asrProvider) {
|
||||
throw new Error('ASR provider not initialized');
|
||||
}
|
||||
|
||||
try {
|
||||
const result = await this.asrProvider.transcribe(audioBuffer, this.tempDir, options);
|
||||
return result;
|
||||
} catch (error) {
|
||||
console.error('Transcription error:', error);
|
||||
return {
|
||||
success: false,
|
||||
text: '',
|
||||
error: error.message,
|
||||
timestamp: Date.now(),
|
||||
provider: this.asrProvider.config ? 'unknown' : 'uninitialized'
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Clean up old temporary files
|
||||
*/
|
||||
cleanupTempFiles() {
|
||||
try {
|
||||
const files = fs.readdirSync(this.tempDir);
|
||||
const now = Date.now();
|
||||
const maxAge = 10 * 60 * 1000; // 10 minutes
|
||||
|
||||
files.forEach(file => {
|
||||
const filePath = path.join(this.tempDir, file);
|
||||
const stats = fs.statSync(filePath);
|
||||
|
||||
if (now - stats.mtime.getTime() > maxAge) {
|
||||
try {
|
||||
fs.unlinkSync(filePath);
|
||||
console.log('Cleaned up old temp file:', file);
|
||||
} catch (error) {
|
||||
console.warn('Failed to cleanup old temp file:', file, error.message);
|
||||
}
|
||||
}
|
||||
});
|
||||
} catch (error) {
|
||||
console.warn('Failed to cleanup temp directory:', error.message);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if audio buffer has sufficient content for transcription
|
||||
* @param {Buffer} audioBuffer - Audio buffer to check
|
||||
* @returns {boolean} True if buffer has sufficient content
|
||||
*/
|
||||
hasSufficientAudio(audioBuffer) {
|
||||
if (!audioBuffer || audioBuffer.length === 0) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Check minimum duration (at least 0.1 seconds of audio)
|
||||
const minSamples = config.AUDIO_SAMPLE_RATE * 0.1; // 0.1 seconds
|
||||
const minBytes = minSamples * 2; // 16-bit = 2 bytes per sample
|
||||
|
||||
if (audioBuffer.length < minBytes) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// Since we're using Silero VAD for speech detection, we trust its decision
|
||||
// and only check for minimum duration. The energy check is redundant and
|
||||
// can reject valid speech that Silero VAD correctly identified.
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get current provider information
|
||||
* @returns {Object} Current provider information
|
||||
*/
|
||||
getProviderInfo() {
|
||||
if (!this.asrProvider) {
|
||||
return { provider: 'none', status: 'not initialized' };
|
||||
}
|
||||
|
||||
return {
|
||||
provider: this.asrProvider.constructor.name,
|
||||
model: this.asrProvider.config.model,
|
||||
apiUrl: this.asrProvider.config.apiUrl,
|
||||
status: 'ready'
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = SpeechToTextService;
|
||||
@@ -0,0 +1,12 @@
|
||||
const assert = require('assert');
|
||||
const Module = require('module');
|
||||
const orig = Module.prototype.require;
|
||||
Module.prototype.require = function (id) {
|
||||
if (id === 'emoji-regex') return () => /(?!)/g;
|
||||
return orig.apply(this, arguments);
|
||||
};
|
||||
const { markdownToText } = require('./textProcessor.js');
|
||||
const out = markdownToText('Call get_user_id then save_to_db');
|
||||
assert.ok(out.includes('get_user_id'));
|
||||
assert.ok(out.includes('save_to_db'));
|
||||
console.log('ok');
|
||||
@@ -0,0 +1,13 @@
|
||||
const assert = require('assert');
|
||||
const Module = require('module');
|
||||
const orig = Module.prototype.require;
|
||||
Module.prototype.require = function (id) {
|
||||
if (id === 'emoji-regex') return () => /(?!)/g;
|
||||
return orig.apply(this, arguments);
|
||||
};
|
||||
const { numberToWords } = require('./textProcessor.js');
|
||||
assert.strictEqual(numberToWords(0), 'zero');
|
||||
assert.ok(!numberToWords(1e12).includes('undefined'));
|
||||
assert.ok(numberToWords(1e12).includes('trillion'));
|
||||
assert.strictEqual(numberToWords(Infinity), 'Infinity');
|
||||
console.log('ok');
|
||||
@@ -0,0 +1,195 @@
|
||||
const emojiRegex = require('emoji-regex');
|
||||
|
||||
// Remove emoji from the sentence
|
||||
function removeEmoji(sentence) {
|
||||
const regex = emojiRegex();
|
||||
return sentence.replace(regex, ' ').trim();
|
||||
}
|
||||
|
||||
// Convert markdown to plain text
|
||||
function markdownToText(markdown) {
|
||||
let text = markdown;
|
||||
|
||||
// Remove links, keeping only the link text
|
||||
text = text.replace(/\[([^\]]+)\]\([^\)]+\)/g, '$1');
|
||||
|
||||
// Remove headers
|
||||
text = text.replace(/^#+\s*/gm, '');
|
||||
|
||||
// Remove bold and italic markers
|
||||
// Keep snake_case identifiers; only strip markdown __bold__ markers.
|
||||
text = text.replace(/\*\*/g, '').replace(/\*/g, '')
|
||||
.replace(/__/g, '');
|
||||
|
||||
// Remove blockquotes
|
||||
text = text.replace(/^>\s*/gm, '');
|
||||
|
||||
// Remove horizontal rules
|
||||
text = text.replace(/[-*_]{3,}/g, '');
|
||||
|
||||
// Remove list markers
|
||||
text = text.replace(/^[-*+]\s*/gm, '');
|
||||
|
||||
// Remove code block markers
|
||||
text = text.replace(/```/g, '');
|
||||
|
||||
return text.trim();
|
||||
}
|
||||
|
||||
// Convert numbers to words
|
||||
function numberToWords(num) {
|
||||
const ones = ['', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine'];
|
||||
const tens = ['', '', 'twenty', 'thirty', 'forty', 'fifty', 'sixty', 'seventy', 'eighty', 'ninety'];
|
||||
const teens = ['ten', 'eleven', 'twelve', 'thirteen', 'fourteen', 'fifteen', 'sixteen', 'seventeen', 'eighteen', 'nineteen'];
|
||||
const scales = ['', 'thousand', 'million', 'billion', 'trillion', 'quadrillion'];
|
||||
|
||||
if (num === 0) return 'zero';
|
||||
if (!Number.isFinite(num)) return String(num);
|
||||
|
||||
function convertGroup(n) {
|
||||
let result = '';
|
||||
|
||||
if (n >= 100) {
|
||||
result += ones[Math.floor(n / 100)] + ' hundred ';
|
||||
n %= 100;
|
||||
}
|
||||
|
||||
if (n >= 20) {
|
||||
result += tens[Math.floor(n / 10)] + ' ';
|
||||
n %= 10;
|
||||
if (n > 0) {
|
||||
result += ones[n] + ' ';
|
||||
}
|
||||
} else if (n >= 10) {
|
||||
result += teens[n - 10] + ' ';
|
||||
} else if (n > 0) {
|
||||
result += ones[n] + ' ';
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
|
||||
let result = '';
|
||||
let groupIndex = 0;
|
||||
|
||||
while (num > 0) {
|
||||
const group = num % 1000;
|
||||
if (group !== 0) {
|
||||
const scale = scales[groupIndex] || '';
|
||||
result = convertGroup(group) + scale + ' ' + result;
|
||||
}
|
||||
num = Math.floor(num / 1000);
|
||||
groupIndex++;
|
||||
}
|
||||
|
||||
return result.trim();
|
||||
}
|
||||
|
||||
// Pronounce special characters
|
||||
function pronounceSpecialCharacters(text, isCodeBlock = false) {
|
||||
const specialCharMap = {
|
||||
'@': 'at',
|
||||
'#': 'hash',
|
||||
'$': 'dollar',
|
||||
'%': 'percent',
|
||||
'^': 'caret',
|
||||
'&': 'ampersand',
|
||||
'*': 'asterisk',
|
||||
'_': 'underscore',
|
||||
'=': 'equals',
|
||||
'+': 'plus',
|
||||
'[': 'left square bracket',
|
||||
']': 'right square bracket',
|
||||
'{': 'left curly brace',
|
||||
'}': 'right curly brace',
|
||||
'|': 'vertical bar',
|
||||
'\\': 'backslash',
|
||||
'<': 'less than',
|
||||
'>': 'greater than',
|
||||
'/': 'slash',
|
||||
'`': 'backtick',
|
||||
'~': 'tilde',
|
||||
};
|
||||
|
||||
const punctuationMap = {
|
||||
'!': 'exclamation',
|
||||
'.': 'dot',
|
||||
',': 'comma',
|
||||
'?': 'question mark',
|
||||
';': 'semicolon',
|
||||
':': 'colon',
|
||||
'"': 'double quote',
|
||||
"'": 'single quote',
|
||||
'-': 'minus',
|
||||
'(': 'left parenthesis',
|
||||
')': 'right parenthesis',
|
||||
};
|
||||
|
||||
let processedText = text;
|
||||
|
||||
// Replace special characters
|
||||
Object.entries(specialCharMap).forEach(([char, pronunciation]) => {
|
||||
processedText = processedText.replace(new RegExp(char.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'), 'g'), ` ${pronunciation} `);
|
||||
});
|
||||
|
||||
// Replace punctuation if in code block
|
||||
if (isCodeBlock) {
|
||||
Object.entries(punctuationMap).forEach(([char, pronunciation]) => {
|
||||
processedText = processedText.replace(new RegExp(char.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'), 'g'), ` ${pronunciation} `);
|
||||
});
|
||||
}
|
||||
|
||||
return processedText;
|
||||
}
|
||||
|
||||
// Pronounce numbers in text
|
||||
function pronounceNumbers(text, language) {
|
||||
if (language.startsWith('zh')) return text;
|
||||
|
||||
// Only consume the '.' when it is a real decimal point (digits follow).
|
||||
// The old /(\d+\.?\d*)/ also matched "42." at the end of a sentence, which
|
||||
// ate the period and emitted a dangling "point".
|
||||
return text.replace(/\d+(?:\.\d+)?/g, match => {
|
||||
const num = parseFloat(match);
|
||||
if (isNaN(num)) return match;
|
||||
|
||||
if (match.includes('.')) {
|
||||
const [integer, decimal] = match.split('.');
|
||||
return `${numberToWords(parseInt(integer))} point ${decimal.split('').map(d => numberToWords(parseInt(d))).join(' ')}`;
|
||||
}
|
||||
return numberToWords(parseInt(match));
|
||||
});
|
||||
}
|
||||
|
||||
// Remove emotional indicators
|
||||
function removeEmotions(text) {
|
||||
return text.replace(/\*[a-zA-Z0-9 -]*\*/g, '').trim();
|
||||
}
|
||||
|
||||
// Process code blocks
|
||||
function pronounceCodeBlock(text) {
|
||||
return text.replace(/`([^`\n]+)`|```(?:[\s\S]*?)```/g, (match) => {
|
||||
const content = match.startsWith('```')
|
||||
? match.slice(3, -3)
|
||||
: match.slice(1, -1);
|
||||
return pronounceSpecialCharacters(content, true);
|
||||
});
|
||||
}
|
||||
|
||||
// Main preprocessing function
|
||||
function preprocessSentence(sentence, language = 'en') {
|
||||
let processed = sentence;
|
||||
processed = pronounceCodeBlock(processed);
|
||||
processed = markdownToText(processed);
|
||||
processed = pronounceNumbers(processed, language);
|
||||
processed = removeEmotions(processed);
|
||||
processed = pronounceSpecialCharacters(processed);
|
||||
processed = removeEmoji(processed);
|
||||
return processed.trim();
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
preprocessSentence,
|
||||
numberToWords,
|
||||
markdownToText,
|
||||
};
|
||||
@@ -0,0 +1,306 @@
|
||||
const config = require('../config');
|
||||
const ort = require('onnxruntime-node');
|
||||
const path = require('path');
|
||||
const fs = require('fs');
|
||||
|
||||
class VoiceActivityDetector {
|
||||
constructor(options = {}) {
|
||||
this.threshold = options.threshold || config.VAD_THRESHOLD || 0.5;
|
||||
this.frameLength = options.frameLength || config.VAD_FRAME_LENGTH || 512;
|
||||
this.minSpeechDuration = options.minSpeechDuration || config.VAD_MIN_SPEECH_DURATION || 250;
|
||||
this.maxSilenceDuration = options.maxSilenceDuration || config.VAD_MAX_SILENCE_DURATION || 500;
|
||||
this.sampleRate = options.sampleRate || config.AUDIO_SAMPLE_RATE || 16000;
|
||||
|
||||
// Silero VAD specific parameters
|
||||
this.sileroFrameLength = 512; // Fixed frame length for Silero VAD
|
||||
this.sileroSampleRate = 16000; // Fixed sample rate for Silero VAD
|
||||
|
||||
// State variables
|
||||
this.isSpeaking = false;
|
||||
this.speechStartTime = null;
|
||||
this.lastSpeechTime = null;
|
||||
this.audioBuffer = Buffer.alloc(0);
|
||||
this.speechBuffer = Buffer.alloc(0);
|
||||
|
||||
// ONNX Runtime session
|
||||
this.session = null;
|
||||
this.isInitialized = false;
|
||||
this.initializationPromise = null;
|
||||
|
||||
// Silero VAD state
|
||||
this.state = null;
|
||||
this.sr = null;
|
||||
|
||||
// Initialize the Silero VAD
|
||||
this.initializationPromise = this.initializeSileroVAD();
|
||||
}
|
||||
|
||||
/**
|
||||
* Initialize the Silero VAD ONNX model
|
||||
*/
|
||||
async initializeSileroVAD() {
|
||||
try {
|
||||
console.log('Initializing Silero VAD...');
|
||||
|
||||
// Load the ONNX model
|
||||
const modelPath = path.join(__dirname, '../models/silero_vad.onnx');
|
||||
|
||||
if (!fs.existsSync(modelPath)) {
|
||||
throw new Error(`Silero VAD model not found at ${modelPath}`);
|
||||
}
|
||||
|
||||
// Create ONNX Runtime session
|
||||
this.session = await ort.InferenceSession.create(modelPath, {
|
||||
executionProviders: ['cpu'],
|
||||
graphOptimizationLevel: 'all',
|
||||
enableMemPattern: true
|
||||
});
|
||||
|
||||
// Initialize state tensors for Silero VAD
|
||||
this.resetSileroState();
|
||||
|
||||
this.isInitialized = true;
|
||||
console.log('Silero VAD initialized successfully');
|
||||
|
||||
} catch (error) {
|
||||
console.error('Failed to initialize Silero VAD:', error);
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset Silero VAD state tensors
|
||||
*/
|
||||
resetSileroState() {
|
||||
// Initialize state tensor for Silero VAD
|
||||
// The state tensor combines h and c LSTM hidden states
|
||||
this.state = new ort.Tensor('float32', new Float32Array(2 * 1 * 128).fill(0.0), [2, 1, 128]);
|
||||
this.sr = new ort.Tensor('int64', new BigInt64Array([BigInt(this.sileroSampleRate)]), [1]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert Buffer to Float32Array for Silero VAD
|
||||
* @param {Buffer} buffer - PCM 16-bit audio buffer
|
||||
* @returns {Float32Array} Normalized audio samples
|
||||
*/
|
||||
bufferToFloat32Array(buffer) {
|
||||
const samples = new Float32Array(buffer.length / 2);
|
||||
for (let i = 0; i < samples.length; i++) {
|
||||
// Convert 16-bit PCM to normalized float (-1 to 1)
|
||||
const sample = buffer.readInt16LE(i * 2);
|
||||
samples[i] = sample / 32768.0;
|
||||
}
|
||||
return samples;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run Silero VAD inference on audio chunk
|
||||
* @param {Float32Array} audioSamples - Normalized audio samples
|
||||
* @returns {Promise<number>} Speech probability (0-1)
|
||||
*/
|
||||
async runSileroVAD(audioSamples) {
|
||||
if (!this.session || !this.isInitialized) {
|
||||
throw new Error('Silero VAD not initialized');
|
||||
}
|
||||
|
||||
try {
|
||||
// Create input tensor
|
||||
const inputTensor = new ort.Tensor('float32', audioSamples, [1, audioSamples.length]);
|
||||
|
||||
// Run inference
|
||||
const feeds = {
|
||||
input: inputTensor,
|
||||
state: this.state,
|
||||
sr: this.sr
|
||||
};
|
||||
|
||||
const results = await this.session.run(feeds);
|
||||
|
||||
// Update state tensor for next inference
|
||||
this.state = results.stateN;
|
||||
|
||||
// Get speech probability
|
||||
const speechProb = results.output.data[0];
|
||||
|
||||
return speechProb;
|
||||
|
||||
} catch (error) {
|
||||
console.error('Error running Silero VAD inference:', error);
|
||||
return 0.0; // Return silence probability on error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Process audio chunk and detect voice activity using Silero VAD
|
||||
* @param {Buffer} audioChunk - Raw audio data
|
||||
* @returns {Promise<Array>} VAD result with detection status and audio data
|
||||
*/
|
||||
async processAudioChunk(audioChunk) {
|
||||
// Wait for initialization if not complete
|
||||
if (!this.isInitialized) {
|
||||
await this.initializationPromise;
|
||||
}
|
||||
|
||||
const currentTime = Date.now();
|
||||
const results = [];
|
||||
|
||||
// Add to buffer
|
||||
this.audioBuffer = Buffer.concat([this.audioBuffer, audioChunk]);
|
||||
|
||||
// Process audio in chunks suitable for Silero VAD (512 samples)
|
||||
const chunkSize = this.sileroFrameLength * 2; // 16-bit = 2 bytes per sample
|
||||
|
||||
while (this.audioBuffer.length >= chunkSize) {
|
||||
const chunk = this.audioBuffer.slice(0, chunkSize);
|
||||
this.audioBuffer = this.audioBuffer.slice(chunkSize);
|
||||
|
||||
try {
|
||||
// Convert to Float32Array for Silero VAD
|
||||
const audioSamples = this.bufferToFloat32Array(chunk);
|
||||
|
||||
// Run Silero VAD inference
|
||||
const speechProb = await this.runSileroVAD(audioSamples);
|
||||
|
||||
// Check if speech is detected
|
||||
const isVoiceActive = speechProb > this.threshold;
|
||||
|
||||
if (isVoiceActive) {
|
||||
if (!this.isSpeaking) {
|
||||
// Speech started
|
||||
this.isSpeaking = true;
|
||||
this.speechStartTime = currentTime;
|
||||
this.speechBuffer = Buffer.alloc(0);
|
||||
console.log('Silero VAD: Speech started (prob:', speechProb.toFixed(3), ')');
|
||||
// Emit the rising edge so the server can tell the client to barge
|
||||
// in. Without this the client's 'speech_start' handler is dead and
|
||||
// the assistant keeps talking over the user.
|
||||
results.push({ type: 'speech_start', timestamp: currentTime });
|
||||
}
|
||||
|
||||
this.lastSpeechTime = currentTime;
|
||||
this.speechBuffer = Buffer.concat([this.speechBuffer, chunk]);
|
||||
|
||||
} else if (this.isSpeaking) {
|
||||
// Check if silence duration exceeds threshold
|
||||
const silenceDuration = currentTime - this.lastSpeechTime;
|
||||
|
||||
if (silenceDuration > this.maxSilenceDuration) {
|
||||
// Speech ended
|
||||
const speechDuration = currentTime - this.speechStartTime;
|
||||
|
||||
if (speechDuration >= this.minSpeechDuration) {
|
||||
console.log('Silero VAD: Speech ended', {
|
||||
duration: speechDuration,
|
||||
bufferSize: this.speechBuffer.length
|
||||
});
|
||||
|
||||
results.push({
|
||||
type: 'speech_end',
|
||||
audioData: this.speechBuffer,
|
||||
duration: speechDuration,
|
||||
observedSilenceDuration: silenceDuration,
|
||||
endpointThreshold: this.maxSilenceDuration,
|
||||
timestamp: currentTime
|
||||
});
|
||||
}
|
||||
|
||||
this.isSpeaking = false;
|
||||
this.speechStartTime = null;
|
||||
this.lastSpeechTime = null;
|
||||
this.speechBuffer = Buffer.alloc(0);
|
||||
} else {
|
||||
// Still in speech, add chunk to buffer
|
||||
this.speechBuffer = Buffer.concat([this.speechBuffer, chunk]);
|
||||
}
|
||||
}
|
||||
|
||||
} catch (error) {
|
||||
console.error('Error processing audio chunk with Silero VAD:', error);
|
||||
// Continue processing other chunks
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Force end current speech session
|
||||
* @returns {Promise<Object|null>} Final speech data if available
|
||||
*/
|
||||
async forceEndSpeech() {
|
||||
if (this.isSpeaking && this.speechBuffer.length > 0) {
|
||||
const currentTime = Date.now();
|
||||
const speechDuration = currentTime - this.speechStartTime;
|
||||
|
||||
if (speechDuration >= this.minSpeechDuration) {
|
||||
const result = {
|
||||
type: 'speech_end',
|
||||
audioData: this.speechBuffer,
|
||||
duration: speechDuration,
|
||||
timestamp: currentTime
|
||||
};
|
||||
|
||||
this.isSpeaking = false;
|
||||
this.speechStartTime = null;
|
||||
this.lastSpeechTime = null;
|
||||
this.speechBuffer = Buffer.alloc(0);
|
||||
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
this.reset();
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Reset VAD state
|
||||
*/
|
||||
reset() {
|
||||
this.isSpeaking = false;
|
||||
this.speechStartTime = null;
|
||||
this.lastSpeechTime = null;
|
||||
this.audioBuffer = Buffer.alloc(0);
|
||||
this.speechBuffer = Buffer.alloc(0);
|
||||
|
||||
// Reset Silero VAD state
|
||||
if (this.isInitialized) {
|
||||
this.resetSileroState();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Get current VAD state
|
||||
* @returns {Object} Current state information
|
||||
*/
|
||||
getState() {
|
||||
return {
|
||||
isSpeaking: this.isSpeaking,
|
||||
speechDuration: this.speechStartTime ? Date.now() - this.speechStartTime : 0,
|
||||
bufferSize: this.speechBuffer.length,
|
||||
threshold: this.threshold,
|
||||
isInitialized: this.isInitialized
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Cleanup resources
|
||||
*/
|
||||
async cleanup() {
|
||||
if (this.session) {
|
||||
try {
|
||||
await this.session.release();
|
||||
} catch (error) {
|
||||
console.error('Error releasing ONNX session:', error);
|
||||
}
|
||||
this.session = null;
|
||||
}
|
||||
|
||||
this.state = null;
|
||||
this.sr = null;
|
||||
this.isInitialized = false;
|
||||
this.reset();
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = VoiceActivityDetector;
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "6-3",
|
||||
"run_id": "exp6-3-20260729T150939685Z",
|
||||
"generated_at_utc": "2026-07-29T15:09:39.685Z",
|
||||
"credentials_persisted": false,
|
||||
"source_media": {
|
||||
"path": "microphone_input.wav",
|
||||
"capture_method": "browser_microphone_over_websocket",
|
||||
"original_repository_path": "backend/recordings/recording_2025-02-06T14-51-38-205Z.wav",
|
||||
"sha256": "e69d77415ffa39e1a2e00b99e6dbedce675faa9b4c2b703a256585861c6d2085",
|
||||
"size_bytes": 161836,
|
||||
"duration_seconds": 5.056,
|
||||
"sample_rate_hz": 16000,
|
||||
"channels": 1,
|
||||
"bits_per_sample": 16,
|
||||
"provenance_note": "Existing real microphone capture saved by live-audio/backend/server.js; replayed through the production Silero class for reproducible validation."
|
||||
},
|
||||
"stages": {
|
||||
"vad": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"implementation": "Silero VAD ONNX",
|
||||
"model": "models/silero_vad.onnx",
|
||||
"model_sha256": "2623a2953f6ff3d2c1e61740c6cdb7168133479b267dfef114a4a3cc5bdd788f",
|
||||
"threshold": 0.5,
|
||||
"max_silence_ms": 500,
|
||||
"endpoint_detected": true,
|
||||
"forced_endpoint": false,
|
||||
"observed_trailing_silence_ms": 526,
|
||||
"speech_duration_ms": 692,
|
||||
"latency_seconds": 5.234941667,
|
||||
"segment_path": "vad_segment.wav",
|
||||
"segment_sha256": "068bab529128c19aa1b9d042c947434cca71c62043c55bdf6eb235e973991bfc",
|
||||
"segment_bytes": 21548
|
||||
}
|
||||
},
|
||||
"experiment_complete": false,
|
||||
"error": "Error: Real ASR failed: You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.\n at runAsr (/Users/boj/book/ai-agent-book/chapter6/live-audio/backend/run_real_validation.js:120:67)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async main (/Users/boj/book/ai-agent-book/chapter6/live-audio/backend/run_real_validation.js:244:27)",
|
||||
"acceptance": {
|
||||
"gates": {
|
||||
"schema_and_scope": true,
|
||||
"real_websocket_microphone_media": true,
|
||||
"real_silero_vad_endpoint": true,
|
||||
"real_asr": false,
|
||||
"real_streaming_llm": false,
|
||||
"real_tts_media": false,
|
||||
"measured_stage_latencies": false,
|
||||
"no_mock_probe_or_fallback": false
|
||||
},
|
||||
"passed": false,
|
||||
"statement": "Passing proves one saved real microphone turn completed Silero VAD -> real ASR -> real LLM -> real TTS. It does not benchmark concurrency or production load."
|
||||
}
|
||||
}
|
||||
+164
@@ -0,0 +1,164 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "6-3",
|
||||
"run_id": "exp6-3-20260729T153334474Z",
|
||||
"generated_at_utc": "2026-07-29T15:33:34.475Z",
|
||||
"credentials_persisted": false,
|
||||
"provenance": {
|
||||
"host": {
|
||||
"platform": "darwin",
|
||||
"release": "25.3.0",
|
||||
"architecture": "arm64",
|
||||
"cpu_model": "Apple M2 Max",
|
||||
"logical_cpu_count": 12,
|
||||
"total_memory_bytes": 103079215104
|
||||
},
|
||||
"runtime": {
|
||||
"node": "v25.6.0",
|
||||
"onnxruntime_node": "^1.22.0-rev",
|
||||
"axios": "^1.7.5",
|
||||
"ffprobe": "ffprobe version 8.0.1 Copyright (c) 2007-2025 the FFmpeg developers"
|
||||
},
|
||||
"timing_clock": "process.hrtime.bigint monotonic clock"
|
||||
},
|
||||
"source_media": {
|
||||
"path": "microphone_input.wav",
|
||||
"capture_method": "browser_microphone_over_websocket",
|
||||
"original_repository_path": "backend/recordings/recording_2025-02-06T14-51-38-205Z.wav",
|
||||
"sha256": "e69d77415ffa39e1a2e00b99e6dbedce675faa9b4c2b703a256585861c6d2085",
|
||||
"size_bytes": 161836,
|
||||
"original_sha256": "e69d77415ffa39e1a2e00b99e6dbedce675faa9b4c2b703a256585861c6d2085",
|
||||
"duration_seconds": 5.056,
|
||||
"sample_rate_hz": 16000,
|
||||
"channels": 1,
|
||||
"bits_per_sample": 16,
|
||||
"provenance_note": "Existing real microphone capture saved by live-audio/backend/server.js; replayed through the production Silero class for reproducible validation."
|
||||
},
|
||||
"stages": {
|
||||
"vad": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"implementation": "Silero VAD ONNX",
|
||||
"model": "models/silero_vad.onnx",
|
||||
"model_sha256": "2623a2953f6ff3d2c1e61740c6cdb7168133479b267dfef114a4a3cc5bdd788f",
|
||||
"threshold": 0.5,
|
||||
"max_silence_ms": 500,
|
||||
"endpoint_detected": true,
|
||||
"detected_endpoint_count": 2,
|
||||
"selected_endpoint": "longest detected speech segment",
|
||||
"forced_endpoint": false,
|
||||
"observed_trailing_silence_ms": 509,
|
||||
"speech_duration_ms": 1534,
|
||||
"latency_seconds": 5.393088166,
|
||||
"segment_path": "vad_segment.wav",
|
||||
"segment_sha256": "61d43291d32fdbce3a32a1809c7368e12a5bf98f799f28a31e08848b8e7dda06",
|
||||
"segment_bytes": 46124
|
||||
},
|
||||
"asr": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"provider": "local-openai-whisper",
|
||||
"model": "whisper-tiny",
|
||||
"inference_completed": true,
|
||||
"api_request_completed": false,
|
||||
"external_request": false,
|
||||
"latency_seconds": 2.710318042,
|
||||
"transcript": "是男朋友叫这样的",
|
||||
"language": "zh",
|
||||
"runtime": {
|
||||
"python": "3.11.4",
|
||||
"torch": "2.7.0",
|
||||
"openai_whisper": "20231106"
|
||||
},
|
||||
"model_path": "/Users/boj/.cache/whisper/tiny.pt",
|
||||
"model_sha256": "65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9",
|
||||
"model_load_seconds": 0.34045145800337195,
|
||||
"model_inference_seconds": 1.3047717502340674,
|
||||
"provider_reported_cost_usd": 0,
|
||||
"cost_note": "Local open-source Whisper inference; no external ASR charge."
|
||||
},
|
||||
"llm": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"provider": "ark",
|
||||
"model": "doubao-seed-1-6-flash-250615",
|
||||
"streamed": true,
|
||||
"api_request_completed": true,
|
||||
"external_request": true,
|
||||
"first_token_seconds": 0.749332709,
|
||||
"latency_seconds": 0.791579917,
|
||||
"response": "那是男朋友叫这样的呀",
|
||||
"provider_request_id": "021785339222980840fd37cfe82fdd108ca4a9d2a652575aa05c0",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"completion_tokens": 30,
|
||||
"prompt_tokens": 83,
|
||||
"total_tokens": 113,
|
||||
"prompt_tokens_details": {
|
||||
"cached_tokens": 0
|
||||
},
|
||||
"completion_tokens_details": {
|
||||
"reasoning_tokens": 23
|
||||
}
|
||||
},
|
||||
"provider_reported_cost_usd": null,
|
||||
"cost_note": "The streamed response did not expose a monetary charge; consult the provider billing ledger."
|
||||
},
|
||||
"tts": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"provider": "fish",
|
||||
"model": "s1",
|
||||
"voice": "authorized zero-shot reference",
|
||||
"api_request_completed": true,
|
||||
"external_request": true,
|
||||
"first_audio_byte_seconds": 2.8189757498912513,
|
||||
"latency_seconds": 3.2793434159830213,
|
||||
"output_path": "assistant_response.mp3",
|
||||
"output_sha256": "2ac0b766ac8afaafb6c6bd09e66a134eb6cc46188aa62edc403ddde32fdf92c8",
|
||||
"output_bytes": 30928,
|
||||
"output_duration_seconds": 1.933,
|
||||
"output_format": "mp3",
|
||||
"reference_id_sha256": "bb2ec197d6276bfad81bbdbcec4473040861e6bc20caf53cf4d1f8184a7416be",
|
||||
"reference_id_source": "../../controllable-tts/reference_audio/manifest.json",
|
||||
"billed_input_characters": 10,
|
||||
"provider_reported_cost_usd": null,
|
||||
"cost_note": "The Fish Audio SDK response did not expose a monetary charge; consult the provider billing ledger."
|
||||
}
|
||||
},
|
||||
"experiment_complete": true,
|
||||
"latency": {
|
||||
"post_endpoint_to_first_audio_byte_seconds": 6.278626500891251,
|
||||
"complete_serial_pipeline_seconds": 12.17432954098302,
|
||||
"measurement_clock": "process.hrtime.bigint monotonic clock"
|
||||
},
|
||||
"cost": {
|
||||
"paid_external_requests": 2,
|
||||
"provider_reported_total_usd": 0,
|
||||
"complete": false,
|
||||
"note": "A zero total is not a zero-cost claim when complete=false; some providers omit per-request charges."
|
||||
},
|
||||
"acceptance": {
|
||||
"gates": {
|
||||
"schema_and_scope": true,
|
||||
"real_websocket_microphone_media": true,
|
||||
"real_silero_vad_endpoint": true,
|
||||
"real_asr": true,
|
||||
"real_streaming_llm": true,
|
||||
"real_tts_media": true,
|
||||
"measured_stage_latencies": true,
|
||||
"provenance_complete": true,
|
||||
"no_mock_probe_or_fallback": true
|
||||
},
|
||||
"passed": true,
|
||||
"statement": "Passing proves one saved real microphone turn completed Silero VAD -> real ASR -> real LLM -> real TTS. It does not benchmark concurrency or production load."
|
||||
}
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
# Experiment 6-3 real traditional-voice validation
|
||||
|
||||
- Run ID: `exp6-3-20260729T153334474Z`
|
||||
- Complete: **true**
|
||||
- Source: `microphone_input.wav` (5.056 s, saved browser microphone/WebSocket capture)
|
||||
- VAD: Silero ONNX, 500 ms silence, non-forced endpoint = true
|
||||
- ASR: local-openai-whisper / whisper-tiny, 2.710 s
|
||||
- Transcript: 是男朋友叫这样的
|
||||
- LLM: ark / doubao-seed-1-6-flash-250615, TTFT 0.749 s, total 0.792 s
|
||||
- Response: 那是男朋友叫这样的呀
|
||||
- TTS: fish / s1, first byte 2.819 s, total 3.279 s
|
||||
- Post-endpoint time to first audio byte: 6.279 s
|
||||
|
||||
## Strict gates
|
||||
|
||||
- schema_and_scope: **true**
|
||||
- real_websocket_microphone_media: **true**
|
||||
- real_silero_vad_endpoint: **true**
|
||||
- real_asr: **true**
|
||||
- real_streaming_llm: **true**
|
||||
- real_tts_media: **true**
|
||||
- measured_stage_latencies: **true**
|
||||
- provenance_complete: **true**
|
||||
- no_mock_probe_or_fallback: **true**
|
||||
|
||||
Passing proves one saved real microphone turn completed Silero VAD -> real ASR -> real LLM -> real TTS. It does not benchmark concurrency or production load.
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "6-3",
|
||||
"run_id": "exp6-3-20260729T153226031Z",
|
||||
"generated_at_utc": "2026-07-29T15:32:26.031Z",
|
||||
"credentials_persisted": false,
|
||||
"provenance": {
|
||||
"host": {
|
||||
"platform": "darwin",
|
||||
"release": "25.3.0",
|
||||
"architecture": "arm64",
|
||||
"cpu_model": "Apple M2 Max",
|
||||
"logical_cpu_count": 12,
|
||||
"total_memory_bytes": 103079215104
|
||||
},
|
||||
"runtime": {
|
||||
"node": "v25.6.0",
|
||||
"onnxruntime_node": "^1.22.0-rev",
|
||||
"axios": "^1.7.5",
|
||||
"ffprobe": "ffprobe version 8.0.1 Copyright (c) 2007-2025 the FFmpeg developers"
|
||||
},
|
||||
"timing_clock": "process.hrtime.bigint monotonic clock"
|
||||
},
|
||||
"source_media": {
|
||||
"path": "microphone_input.wav",
|
||||
"capture_method": "browser_microphone_over_websocket",
|
||||
"original_repository_path": "backend/recordings/recording_2025-02-06T14-51-38-205Z.wav",
|
||||
"sha256": "e69d77415ffa39e1a2e00b99e6dbedce675faa9b4c2b703a256585861c6d2085",
|
||||
"size_bytes": 161836,
|
||||
"original_sha256": "e69d77415ffa39e1a2e00b99e6dbedce675faa9b4c2b703a256585861c6d2085",
|
||||
"duration_seconds": 5.056,
|
||||
"sample_rate_hz": 16000,
|
||||
"channels": 1,
|
||||
"bits_per_sample": 16,
|
||||
"provenance_note": "Existing real microphone capture saved by live-audio/backend/server.js; replayed through the production Silero class for reproducible validation."
|
||||
},
|
||||
"stages": {
|
||||
"vad": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"implementation": "Silero VAD ONNX",
|
||||
"model": "models/silero_vad.onnx",
|
||||
"model_sha256": "2623a2953f6ff3d2c1e61740c6cdb7168133479b267dfef114a4a3cc5bdd788f",
|
||||
"threshold": 0.5,
|
||||
"max_silence_ms": 500,
|
||||
"endpoint_detected": true,
|
||||
"detected_endpoint_count": 2,
|
||||
"selected_endpoint": "longest detected speech segment",
|
||||
"forced_endpoint": false,
|
||||
"observed_trailing_silence_ms": 511,
|
||||
"speech_duration_ms": 1531,
|
||||
"latency_seconds": 5.362510583,
|
||||
"segment_path": "vad_segment.wav",
|
||||
"segment_sha256": "61d43291d32fdbce3a32a1809c7368e12a5bf98f799f28a31e08848b8e7dda06",
|
||||
"segment_bytes": 46124
|
||||
},
|
||||
"asr": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"provider": "local-openai-whisper",
|
||||
"model": "whisper-tiny",
|
||||
"inference_completed": true,
|
||||
"api_request_completed": false,
|
||||
"external_request": false,
|
||||
"latency_seconds": 3.981504083,
|
||||
"transcript": "倒是男生就要战争了",
|
||||
"language": "zh",
|
||||
"runtime": {
|
||||
"python": "3.11.4",
|
||||
"torch": "2.7.0",
|
||||
"openai_whisper": "20231106"
|
||||
},
|
||||
"model_path": "/Users/boj/.cache/whisper/tiny.pt",
|
||||
"model_sha256": "65147644a518d12f04e32d6f3b26facc3f8dd46e5390956a9424a650c0ce22b9",
|
||||
"model_load_seconds": 0.34700404200702906,
|
||||
"model_inference_seconds": 2.5749531253241003,
|
||||
"provider_reported_cost_usd": 0,
|
||||
"cost_note": "Local open-source Whisper inference; no external ASR charge."
|
||||
}
|
||||
},
|
||||
"experiment_complete": false,
|
||||
"error": "Error: Real LLM failed: Request failed with status code 401\n at runLlm (/Users/boj/book/ai-agent-book/chapter6/live-audio/backend/run_real_validation.js:234:30)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async main (/Users/boj/book/ai-agent-book/chapter6/live-audio/backend/run_real_validation.js:407:27)",
|
||||
"acceptance": {
|
||||
"gates": {
|
||||
"schema_and_scope": true,
|
||||
"real_websocket_microphone_media": true,
|
||||
"real_silero_vad_endpoint": true,
|
||||
"real_asr": true,
|
||||
"real_streaming_llm": false,
|
||||
"real_tts_media": false,
|
||||
"measured_stage_latencies": false,
|
||||
"provenance_complete": true,
|
||||
"no_mock_probe_or_fallback": false
|
||||
},
|
||||
"passed": false,
|
||||
"statement": "Passing proves one saved real microphone turn completed Silero VAD -> real ASR -> real LLM -> real TTS. It does not benchmark concurrency or production load."
|
||||
}
|
||||
}
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "6-3",
|
||||
"run_id": "exp6-3-20260729T152935363Z",
|
||||
"generated_at_utc": "2026-07-29T15:29:35.363Z",
|
||||
"credentials_persisted": false,
|
||||
"provenance": {
|
||||
"host": {
|
||||
"platform": "darwin",
|
||||
"release": "25.3.0",
|
||||
"architecture": "arm64",
|
||||
"cpu_model": "Apple M2 Max",
|
||||
"logical_cpu_count": 12,
|
||||
"total_memory_bytes": 103079215104
|
||||
},
|
||||
"runtime": {
|
||||
"node": "v25.6.0",
|
||||
"onnxruntime_node": "^1.22.0-rev",
|
||||
"axios": "^1.7.5",
|
||||
"ffprobe": "ffprobe version 8.0.1 Copyright (c) 2007-2025 the FFmpeg developers"
|
||||
},
|
||||
"timing_clock": "process.hrtime.bigint monotonic clock"
|
||||
},
|
||||
"source_media": {
|
||||
"path": "microphone_input.wav",
|
||||
"capture_method": "browser_microphone_over_websocket",
|
||||
"original_repository_path": "backend/recordings/recording_2025-02-06T14-51-38-205Z.wav",
|
||||
"sha256": "e69d77415ffa39e1a2e00b99e6dbedce675faa9b4c2b703a256585861c6d2085",
|
||||
"size_bytes": 161836,
|
||||
"original_sha256": "e69d77415ffa39e1a2e00b99e6dbedce675faa9b4c2b703a256585861c6d2085",
|
||||
"duration_seconds": 5.056,
|
||||
"sample_rate_hz": 16000,
|
||||
"channels": 1,
|
||||
"bits_per_sample": 16,
|
||||
"provenance_note": "Existing real microphone capture saved by live-audio/backend/server.js; replayed through the production Silero class for reproducible validation."
|
||||
},
|
||||
"stages": {
|
||||
"vad": {
|
||||
"execution": "real",
|
||||
"mock": false,
|
||||
"probe_only": false,
|
||||
"fallback_used": false,
|
||||
"implementation": "Silero VAD ONNX",
|
||||
"model": "models/silero_vad.onnx",
|
||||
"model_sha256": "2623a2953f6ff3d2c1e61740c6cdb7168133479b267dfef114a4a3cc5bdd788f",
|
||||
"threshold": 0.5,
|
||||
"max_silence_ms": 500,
|
||||
"endpoint_detected": true,
|
||||
"detected_endpoint_count": 2,
|
||||
"selected_endpoint": "longest detected speech segment",
|
||||
"forced_endpoint": false,
|
||||
"observed_trailing_silence_ms": 509,
|
||||
"speech_duration_ms": 1532,
|
||||
"latency_seconds": 5.396249041,
|
||||
"segment_path": "vad_segment.wav",
|
||||
"segment_sha256": "61d43291d32fdbce3a32a1809c7368e12a5bf98f799f28a31e08848b8e7dda06",
|
||||
"segment_bytes": 46124
|
||||
}
|
||||
},
|
||||
"experiment_complete": false,
|
||||
"error": "Error: Real ASR failed: Request failed with status code 401\n at runAsr (/Users/boj/book/ai-agent-book/chapter6/live-audio/backend/run_real_validation.js:153:67)\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async main (/Users/boj/book/ai-agent-book/chapter6/live-audio/backend/run_real_validation.js:295:27)",
|
||||
"acceptance": {
|
||||
"gates": {
|
||||
"schema_and_scope": true,
|
||||
"real_websocket_microphone_media": true,
|
||||
"real_silero_vad_endpoint": true,
|
||||
"real_asr": false,
|
||||
"real_streaming_llm": false,
|
||||
"real_tts_media": false,
|
||||
"measured_stage_latencies": false,
|
||||
"provenance_complete": true,
|
||||
"no_mock_probe_or_fallback": false
|
||||
},
|
||||
"passed": false,
|
||||
"statement": "Passing proves one saved real microphone turn completed Silero VAD -> real ASR -> real LLM -> real TTS. It does not benchmark concurrency or production load."
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# Frontend environment for the Live Voice Chat demo.
|
||||
# Copy this file to `.env` (which is gitignored) before running `npm run dev`.
|
||||
#
|
||||
# WEBSOCKET_PORT must match the backend's LISTEN_PORT in backend/config.js
|
||||
# (default 8848). The frontend connects to ws://localhost:${WEBSOCKET_PORT}.
|
||||
WEBSOCKET_PORT=8848
|
||||
@@ -0,0 +1,56 @@
|
||||
import * as React from "react"
|
||||
import { Slot } from "@radix-ui/react-slot"
|
||||
import { cva, type VariantProps } from "class-variance-authority"
|
||||
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
const buttonVariants = cva(
|
||||
"inline-flex items-center justify-center rounded-md text-sm font-medium ring-offset-background transition-colors focus-visible:outline-none focus-visible:ring-2 focus-visible:ring-ring focus-visible:ring-offset-2 disabled:pointer-events-none disabled:opacity-50",
|
||||
{
|
||||
variants: {
|
||||
variant: {
|
||||
default: "bg-primary text-primary-foreground hover:bg-primary/90",
|
||||
destructive:
|
||||
"bg-destructive text-destructive-foreground hover:bg-destructive/90",
|
||||
outline:
|
||||
"border border-input bg-background hover:bg-accent hover:text-accent-foreground",
|
||||
secondary:
|
||||
"bg-secondary text-secondary-foreground hover:bg-secondary/80",
|
||||
ghost: "hover:bg-accent hover:text-accent-foreground",
|
||||
link: "text-primary underline-offset-4 hover:underline",
|
||||
},
|
||||
size: {
|
||||
default: "h-10 px-4 py-2",
|
||||
sm: "h-9 rounded-md px-3",
|
||||
lg: "h-11 rounded-md px-8",
|
||||
icon: "h-10 w-10",
|
||||
},
|
||||
},
|
||||
defaultVariants: {
|
||||
variant: "default",
|
||||
size: "default",
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
export interface ButtonProps
|
||||
extends React.ButtonHTMLAttributes<HTMLButtonElement>,
|
||||
VariantProps<typeof buttonVariants> {
|
||||
asChild?: boolean
|
||||
}
|
||||
|
||||
const Button = React.forwardRef<HTMLButtonElement, ButtonProps>(
|
||||
({ className, variant, size, asChild = false, ...props }, ref) => {
|
||||
const Comp = asChild ? Slot : "button"
|
||||
return (
|
||||
<Comp
|
||||
className={cn(buttonVariants({ variant, size, className }))}
|
||||
ref={ref}
|
||||
{...props}
|
||||
/>
|
||||
)
|
||||
}
|
||||
)
|
||||
Button.displayName = "Button"
|
||||
|
||||
export { Button, buttonVariants }
|
||||
@@ -0,0 +1,79 @@
|
||||
import * as React from "react"
|
||||
|
||||
import { cn } from "@/lib/utils"
|
||||
|
||||
const Card = React.forwardRef<
|
||||
HTMLDivElement,
|
||||
React.HTMLAttributes<HTMLDivElement>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<div
|
||||
ref={ref}
|
||||
className={cn(
|
||||
"rounded-lg border bg-card text-card-foreground shadow-sm",
|
||||
className
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
Card.displayName = "Card"
|
||||
|
||||
const CardHeader = React.forwardRef<
|
||||
HTMLDivElement,
|
||||
React.HTMLAttributes<HTMLDivElement>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<div
|
||||
ref={ref}
|
||||
className={cn("flex flex-col space-y-1.5 p-6", className)}
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
CardHeader.displayName = "CardHeader"
|
||||
|
||||
const CardTitle = React.forwardRef<
|
||||
HTMLParagraphElement,
|
||||
React.HTMLAttributes<HTMLHeadingElement>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<h3
|
||||
ref={ref}
|
||||
className={cn(
|
||||
"text-2xl font-semibold leading-none tracking-tight",
|
||||
className
|
||||
)}
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
CardTitle.displayName = "CardTitle"
|
||||
|
||||
const CardDescription = React.forwardRef<
|
||||
HTMLParagraphElement,
|
||||
React.HTMLAttributes<HTMLParagraphElement>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<p
|
||||
ref={ref}
|
||||
className={cn("text-sm text-muted-foreground", className)}
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
CardDescription.displayName = "CardDescription"
|
||||
|
||||
const CardContent = React.forwardRef<
|
||||
HTMLDivElement,
|
||||
React.HTMLAttributes<HTMLDivElement>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<div ref={ref} className={cn("p-6 pt-0", className)} {...props} />
|
||||
))
|
||||
CardContent.displayName = "CardContent"
|
||||
|
||||
const CardFooter = React.forwardRef<
|
||||
HTMLDivElement,
|
||||
React.HTMLAttributes<HTMLDivElement>
|
||||
>(({ className, ...props }, ref) => (
|
||||
<div
|
||||
ref={ref}
|
||||
className={cn("flex items-center p-6 pt-0", className)}
|
||||
{...props}
|
||||
/>
|
||||
))
|
||||
CardFooter.displayName = "CardFooter"
|
||||
|
||||
export { Card, CardHeader, CardFooter, CardTitle, CardDescription, CardContent }
|
||||
@@ -0,0 +1,6 @@
|
||||
import { type ClassValue, clsx } from "clsx"
|
||||
import { twMerge } from "tailwind-merge"
|
||||
|
||||
export function cn(...inputs: ClassValue[]) {
|
||||
return twMerge(clsx(inputs))
|
||||
}
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
/// <reference types="next" />
|
||||
/// <reference types="next/image-types/global" />
|
||||
|
||||
// NOTE: This file should not be edited
|
||||
// see https://nextjs.org/docs/basic-features/typescript for more information.
|
||||
@@ -0,0 +1,24 @@
|
||||
const path = require('path');
|
||||
|
||||
/** @type {import('next').NextConfig} */
|
||||
const nextConfig = {
|
||||
reactStrictMode: true,
|
||||
output: 'export',
|
||||
env: {
|
||||
WEBSOCKET_PORT: process.env.WEBSOCKET_PORT || '8848',
|
||||
IS_PRODUCTION: process.env.NODE_ENV === 'production',
|
||||
},
|
||||
// Since we're using static export, we need to disable image optimization
|
||||
images: {
|
||||
unoptimized: true,
|
||||
},
|
||||
webpack: (config) => {
|
||||
config.resolve.alias = {
|
||||
...config.resolve.alias,
|
||||
'@': path.resolve(__dirname),
|
||||
};
|
||||
return config;
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = nextConfig
|
||||
+7553
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"name": "livechat-frontend",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"scripts": {
|
||||
"dev": "next dev",
|
||||
"build": "cross-env NODE_ENV=production next build",
|
||||
"start": "cross-env NODE_ENV=production next start",
|
||||
"lint": "next lint"
|
||||
},
|
||||
"dependencies": {
|
||||
"@radix-ui/react-slot": "^1.0.2",
|
||||
"@types/node": "18.15.11",
|
||||
"@types/react": "18.0.33",
|
||||
"@types/react-dom": "18.0.11",
|
||||
"class-variance-authority": "^0.6.0",
|
||||
"clsx": "^1.2.1",
|
||||
"lucide-react": "^0.130.1",
|
||||
"next": "13.3.0",
|
||||
"react": "18.2.0",
|
||||
"react-dom": "18.2.0",
|
||||
"react-markdown": "^9.0.1",
|
||||
"react-syntax-highlighter": "^15.6.1",
|
||||
"remark-gfm": "^4.0.0",
|
||||
"tailwind-merge": "^1.10.0",
|
||||
"tailwindcss-animate": "^1.0.5",
|
||||
"typescript": "5.0.4",
|
||||
"web-audio-resampler": "^1.0.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@tailwindcss/typography": "^0.5.15",
|
||||
"autoprefixer": "^10.4.14",
|
||||
"cross-env": "^7.0.3",
|
||||
"eslint": "9.30.1",
|
||||
"eslint-config-next": "15.3.5",
|
||||
"postcss": "^8.4.21",
|
||||
"tailwindcss": "^3.3.1"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
import type { AppProps } from 'next/app'
|
||||
import '../styles/globals.css'
|
||||
|
||||
export default function App({ Component, pageProps }: AppProps) {
|
||||
return <Component {...pageProps} />
|
||||
}
|
||||
@@ -0,0 +1,751 @@
|
||||
import React, { useState, useRef, useEffect } from 'react';
|
||||
import { Button } from '../components/ui/button';
|
||||
import { Card, CardContent, CardHeader, CardTitle } from '../components/ui/card';
|
||||
import { Mic, MicOff, Trash2 } from 'lucide-react';
|
||||
import ReactMarkdown from 'react-markdown';
|
||||
import remarkGfm from 'remark-gfm';
|
||||
import { Prism as SyntaxHighlighter } from 'react-syntax-highlighter';
|
||||
import { oneDark } from 'react-syntax-highlighter/dist/cjs/styles/prism';
|
||||
|
||||
interface LogEntry {
|
||||
timestamp: number;
|
||||
message: string;
|
||||
type: 'info' | 'error' | 'latency' | 'llm';
|
||||
}
|
||||
|
||||
interface ChatMessage {
|
||||
role: 'user' | 'assistant' | 'transcript';
|
||||
content: string;
|
||||
isFinal?: boolean;
|
||||
}
|
||||
|
||||
interface TabButtonProps {
|
||||
active: boolean;
|
||||
onClick: () => void;
|
||||
children: React.ReactNode;
|
||||
}
|
||||
|
||||
const CodeBlock = ({ node, inline, className, children, ...props }) => {
|
||||
const match = /language-(\w+)/.exec(className || '');
|
||||
const lang = match ? match[1] : '';
|
||||
|
||||
if (!inline && lang) {
|
||||
return (
|
||||
<SyntaxHighlighter
|
||||
language={lang}
|
||||
style={oneDark}
|
||||
customStyle={{
|
||||
margin: '0.5em 0',
|
||||
borderRadius: '0.375rem',
|
||||
fontSize: '0.875rem',
|
||||
}}
|
||||
{...props}
|
||||
>
|
||||
{String(children).replace(/\n$/, '')}
|
||||
</SyntaxHighlighter>
|
||||
);
|
||||
}
|
||||
|
||||
return <code className={className} {...props}>{children}</code>;
|
||||
};
|
||||
|
||||
const TabButton: React.FC<TabButtonProps> = ({ active, onClick, children }) => (
|
||||
<button
|
||||
onClick={onClick}
|
||||
className={`flex-1 py-2 text-sm font-medium border-b-2 ${
|
||||
active
|
||||
? 'border-blue-500 text-blue-600'
|
||||
: 'border-transparent text-gray-500 hover:text-gray-700 hover:border-gray-300'
|
||||
}`}
|
||||
>
|
||||
{children}
|
||||
</button>
|
||||
);
|
||||
|
||||
export default function Home() {
|
||||
const [isRecording, setIsRecording] = useState(false);
|
||||
const [logs, setLogs] = useState<LogEntry[]>([]);
|
||||
const websocketRef = useRef<WebSocket | null>(null);
|
||||
const audioContextRef = useRef<AudioContext | null>(null);
|
||||
const workletNodeRef = useRef<AudioWorkletNode | null>(null);
|
||||
const sourceNodeRef = useRef<MediaStreamAudioSourceNode | null>(null);
|
||||
const streamRef = useRef<MediaStream | null>(null);
|
||||
const echoNodeRef = useRef<AudioWorkletNode | null>(null);
|
||||
const audioFormatRef = useRef<any>(null);
|
||||
const speechEndTimeRef = useRef<number | null>(null);
|
||||
const llmStartTimeRef = useRef<number | null>(null);
|
||||
const vadEndTimeRef = useRef<number | null>(null);
|
||||
const hasPlaybackLatencyRef = useRef<boolean>(false);
|
||||
const vadStartTimeRef = useRef<number | null>(null);
|
||||
const audioQueueRef = useRef<Float32Array[]>([]);
|
||||
const pingIntervalRef = useRef<NodeJS.Timeout | null>(null);
|
||||
const [chatHistory, setChatHistory] = useState<ChatMessage[]>([]);
|
||||
const playbackStartTimeRef = useRef<number | null>(null);
|
||||
const [finalTranscripts, setFinalTranscripts] = useState<string>('');
|
||||
// Add this ref to track the current valid message ID
|
||||
const currentValidMessageIdRef = useRef<string | null>(null);
|
||||
// Add this ref to track if audio is currently playing
|
||||
const isPlayingRef = useRef<boolean>(false);
|
||||
// Add these refs after other refs
|
||||
const chatHistoryRef = useRef<HTMLDivElement>(null);
|
||||
const logsRef = useRef<HTMLDivElement>(null);
|
||||
const shouldAutoScrollChatRef = useRef(true);
|
||||
const shouldAutoScrollLogsRef = useRef(true);
|
||||
const [activeTab, setActiveTab] = useState<'chat' | 'logs'>('chat');
|
||||
|
||||
const addLog = (message: string, type: LogEntry['type'] = 'info') => {
|
||||
setLogs(prev => [...prev, {
|
||||
timestamp: Date.now(),
|
||||
message,
|
||||
type
|
||||
}]);
|
||||
};
|
||||
|
||||
const setupWebSocket = () => {
|
||||
if (websocketRef.current?.readyState === WebSocket.OPEN) return;
|
||||
|
||||
// Determine if we're running on localhost
|
||||
const isLocalhost = window.location.hostname === 'localhost' || window.location.hostname === '127.0.0.1';
|
||||
|
||||
let wsUrl;
|
||||
if (isLocalhost) {
|
||||
// For localhost development
|
||||
wsUrl = `ws://localhost:${process.env.WEBSOCKET_PORT}`;
|
||||
} else {
|
||||
// For production deployment
|
||||
const protocol = window.location.protocol === 'https:' ? 'wss:' : 'ws:';
|
||||
wsUrl = `${protocol}//${window.location.host}/ws`;
|
||||
}
|
||||
|
||||
console.log('Connecting to WebSocket:', wsUrl);
|
||||
websocketRef.current = new WebSocket(wsUrl);
|
||||
websocketRef.current.binaryType = 'arraybuffer';
|
||||
|
||||
websocketRef.current.onopen = () => {
|
||||
// Start sending pings when connection opens
|
||||
if (pingIntervalRef.current) {
|
||||
clearInterval(pingIntervalRef.current);
|
||||
}
|
||||
pingIntervalRef.current = setInterval(() => {
|
||||
if (websocketRef.current?.readyState === WebSocket.OPEN) {
|
||||
websocketRef.current.send(JSON.stringify({
|
||||
type: 'ping',
|
||||
timestamp: Date.now()
|
||||
}));
|
||||
}
|
||||
}, 1000);
|
||||
};
|
||||
|
||||
websocketRef.current.onmessage = async (event) => {
|
||||
try {
|
||||
if (event.data instanceof ArrayBuffer) {
|
||||
const now = Date.now();
|
||||
|
||||
if (audioFormatRef.current) {
|
||||
const int16Array = new Int16Array(event.data);
|
||||
const float32Array = new Float32Array(int16Array.length);
|
||||
|
||||
for (let i = 0; i < int16Array.length; i++) {
|
||||
float32Array[i] = int16Array[i] / 32768.0;
|
||||
}
|
||||
|
||||
if (echoNodeRef.current) {
|
||||
// Set playback start time when first audio chunk is received
|
||||
if (!playbackStartTimeRef.current) {
|
||||
playbackStartTimeRef.current = Date.now();
|
||||
}
|
||||
|
||||
echoNodeRef.current.port.postMessage({ type: 'unmute' });
|
||||
|
||||
if (vadEndTimeRef.current && !hasPlaybackLatencyRef.current) {
|
||||
const latency = now - vadEndTimeRef.current;
|
||||
const serverVadLatency = now - (speechEndTimeRef.current || vadEndTimeRef.current);
|
||||
addLog(`First audio playback latency - Browser VAD: ${latency}ms`, 'latency');
|
||||
addLog(`First audio playback latency - Server VAD: ${serverVadLatency}ms`, 'latency');
|
||||
hasPlaybackLatencyRef.current = true;
|
||||
}
|
||||
echoNodeRef.current.port.postMessage(float32Array);
|
||||
}
|
||||
}
|
||||
} else {
|
||||
const jsonData = JSON.parse(event.data);
|
||||
const now = Date.now();
|
||||
|
||||
// Add ping handling
|
||||
if (jsonData.type === 'pong') {
|
||||
const latency = now - jsonData.timestamp;
|
||||
addLog(`WebSocket latency: ${latency}ms`, 'latency');
|
||||
return;
|
||||
}
|
||||
|
||||
switch (jsonData.type) {
|
||||
case 'speech_end':
|
||||
speechEndTimeRef.current = now;
|
||||
if (vadEndTimeRef.current) {
|
||||
const vadToServerLatency = now - vadEndTimeRef.current;
|
||||
addLog(`[Server] Speech end detected (${vadToServerLatency}ms after frontend VAD)`, 'latency');
|
||||
} else {
|
||||
// Use server's speech end as VAD end time if frontend didn't detect it
|
||||
vadEndTimeRef.current = now;
|
||||
hasPlaybackLatencyRef.current = false;
|
||||
addLog('[Server] Speech end detected (using as VAD endpoint)', 'latency');
|
||||
}
|
||||
break;
|
||||
|
||||
case 'speech_start':
|
||||
// Handle interrupt
|
||||
handleInterrupt();
|
||||
// Clear audio queue and stop playback
|
||||
if (echoNodeRef.current) {
|
||||
echoNodeRef.current.port.postMessage({ type: 'clear' });
|
||||
}
|
||||
audioQueueRef.current = [];
|
||||
|
||||
if (vadStartTimeRef.current) {
|
||||
const vadToServerLatency = now - vadStartTimeRef.current;
|
||||
addLog(`[Server] Speech start detected (${vadToServerLatency}ms after frontend VAD)`, 'latency');
|
||||
} else {
|
||||
vadStartTimeRef.current = now;
|
||||
addLog('[Server] Speech start detected (using as VAD start point)', 'latency');
|
||||
}
|
||||
break;
|
||||
|
||||
case 'llm_start':
|
||||
llmStartTimeRef.current = now;
|
||||
if (vadEndTimeRef.current) {
|
||||
const browserVadLatency = now - vadEndTimeRef.current;
|
||||
const serverVadLatency = now - (speechEndTimeRef.current || vadEndTimeRef.current);
|
||||
addLog(`Transcribe latency - Browser VAD: ${browserVadLatency}ms`, 'latency');
|
||||
addLog(`Transcribe latency - Server VAD: ${serverVadLatency}ms`, 'latency');
|
||||
}
|
||||
break;
|
||||
|
||||
case 'llm_first_token':
|
||||
if (llmStartTimeRef.current) {
|
||||
const latency = now - llmStartTimeRef.current;
|
||||
addLog(`LLM time to first token: ${latency}ms`, 'latency');
|
||||
}
|
||||
break;
|
||||
|
||||
case 'llm_first_sentence':
|
||||
if (vadEndTimeRef.current) {
|
||||
const llmLatency = now - llmStartTimeRef.current;
|
||||
const browserVadLatency = now - vadEndTimeRef.current;
|
||||
const serverVadLatency = now - (speechEndTimeRef.current || vadEndTimeRef.current);
|
||||
addLog(`LLM first sentence latency - LLM: ${llmLatency}ms`, 'latency');
|
||||
addLog(`LLM first sentence latency - Browser VAD: ${browserVadLatency}ms`, 'latency');
|
||||
addLog(`LLM first sentence latency - Server VAD: ${serverVadLatency}ms`, 'latency');
|
||||
}
|
||||
break;
|
||||
|
||||
case 'tts_complete':
|
||||
if (vadEndTimeRef.current) {
|
||||
addLog(`TTS synthesis time: ${jsonData.synthesisTime}ms`, 'latency');
|
||||
}
|
||||
break;
|
||||
|
||||
case 'audio_start':
|
||||
// Reset playback start time when new audio stream starts
|
||||
playbackStartTimeRef.current = null;
|
||||
if (echoNodeRef.current) {
|
||||
echoNodeRef.current.port.postMessage({ type: 'unmute' });
|
||||
}
|
||||
addLog(`Audio format: ${JSON.stringify(jsonData.format)}`);
|
||||
audioFormatRef.current = jsonData.format;
|
||||
break;
|
||||
|
||||
case 'audio_end':
|
||||
addLog('Audio streaming completed');
|
||||
break;
|
||||
|
||||
case 'transcript':
|
||||
if (jsonData.messageId) {
|
||||
// Update the current valid message ID when receiving a transcript
|
||||
currentValidMessageIdRef.current = jsonData.messageId;
|
||||
}
|
||||
if (jsonData.isFinal) {
|
||||
addLog(`[ASR] Final transcript: "${jsonData.text}"`);
|
||||
} else {
|
||||
addLog(`[ASR] Interim transcript: "${jsonData.text}"`);
|
||||
}
|
||||
break;
|
||||
|
||||
case 'error':
|
||||
addLog(`Error: ${jsonData.message}`, 'error');
|
||||
break;
|
||||
|
||||
case 'llm_sentence':
|
||||
if (jsonData.messageId === currentValidMessageIdRef.current) {
|
||||
addLog(`LLM: "${jsonData.text}"`, 'llm');
|
||||
} else {
|
||||
addLog(`Ignored LLM response for recalled message: "${jsonData.text}"`, 'info');
|
||||
}
|
||||
break;
|
||||
|
||||
case 'websocket_latency':
|
||||
addLog(`WebSocket RTT: ${jsonData.roundTripTime}ms`, 'latency');
|
||||
break;
|
||||
|
||||
case 'chat_history_delta':
|
||||
setChatHistory(prev => {
|
||||
// Remove messages from startIndex onwards and insert new messages
|
||||
return [
|
||||
...prev.slice(0, jsonData.startIndex),
|
||||
...jsonData.messages
|
||||
];
|
||||
});
|
||||
break;
|
||||
|
||||
case 'debug_info':
|
||||
addLog(jsonData.message, 'info');
|
||||
break;
|
||||
|
||||
case 'vad_status':
|
||||
// Silently ignore VAD status messages to reduce log noise
|
||||
break;
|
||||
|
||||
default:
|
||||
addLog(`Received message: ${JSON.stringify(jsonData)}`);
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
addLog(`Error processing message: ${error}`, 'error');
|
||||
}
|
||||
};
|
||||
|
||||
websocketRef.current.onclose = () => {
|
||||
addLog('WebSocket closed');
|
||||
websocketRef.current = null;
|
||||
// A server restart or network blip fires onclose mid-session. Without
|
||||
// tearing down, the button stays "Stop Recording", the mic keeps
|
||||
// capturing, the ping interval keeps firing, and the audio worklet
|
||||
// silently discards every captured buffer (it checks
|
||||
// websocketRef.current?.readyState, now null) -- a "zombie recording".
|
||||
// websocketRef is already null here, so stopRecording won't re-close the
|
||||
// socket or re-enter onclose.
|
||||
stopRecording();
|
||||
};
|
||||
|
||||
websocketRef.current.onerror = (error) => {
|
||||
addLog(`WebSocket error: ${error}`, 'error');
|
||||
};
|
||||
};
|
||||
|
||||
const startRecording = async () => {
|
||||
try {
|
||||
// Setup WebSocket first
|
||||
setupWebSocket();
|
||||
|
||||
// Check if AudioContext and AudioWorklet are supported
|
||||
if (!window.AudioContext && !(window as any).webkitAudioContext) {
|
||||
throw new Error('AudioContext is not supported in this browser');
|
||||
}
|
||||
|
||||
const AudioContextClass = window.AudioContext || (window as any).webkitAudioContext;
|
||||
|
||||
// Create AudioContext with specific sample rate
|
||||
audioContextRef.current = new AudioContextClass({
|
||||
sampleRate: 16000
|
||||
});
|
||||
|
||||
// Check if audioWorklet is supported
|
||||
if (!audioContextRef.current.audioWorklet) {
|
||||
throw new Error('AudioWorklet is not supported in this browser. Please use a modern browser like Chrome or Firefox.');
|
||||
}
|
||||
|
||||
// Resume the audio context first (needed for some browsers)
|
||||
if (audioContextRef.current.state === 'suspended') {
|
||||
await audioContextRef.current.resume();
|
||||
}
|
||||
|
||||
console.log('Audio context created with sample rate:', audioContextRef.current.sampleRate);
|
||||
|
||||
try {
|
||||
// Add the audio worklet module with the full URL path
|
||||
const workletUrl = new URL('/audioWorklet.js', window.location.origin).href;
|
||||
console.log('Loading audio worklet from:', workletUrl);
|
||||
|
||||
// Add a timeout to the worklet loading
|
||||
const workletLoadPromise = audioContextRef.current.audioWorklet.addModule(workletUrl);
|
||||
const timeoutPromise = new Promise((_, reject) => {
|
||||
setTimeout(() => reject(new Error('Audio worklet load timeout')), 5000);
|
||||
});
|
||||
|
||||
await Promise.race([workletLoadPromise, timeoutPromise]);
|
||||
console.log('Audio worklet loaded successfully');
|
||||
|
||||
} catch (workletError) {
|
||||
console.error('Error loading audio worklet:', workletError);
|
||||
throw new Error(`Failed to load audio worklet module: ${workletError.message}`);
|
||||
}
|
||||
|
||||
// Check if getUserMedia is supported
|
||||
if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
|
||||
throw new Error('getUserMedia is not supported in this browser');
|
||||
}
|
||||
|
||||
// Get user media after worklet is loaded
|
||||
try {
|
||||
streamRef.current = await navigator.mediaDevices.getUserMedia({
|
||||
audio: {
|
||||
channelCount: 1,
|
||||
sampleRate: 16000,
|
||||
echoCancellation: true,
|
||||
noiseSuppression: true,
|
||||
autoGainControl: true
|
||||
}
|
||||
});
|
||||
} catch (mediaError) {
|
||||
console.error('Error accessing microphone:', mediaError);
|
||||
throw new Error(`Microphone access failed: ${mediaError.message}`);
|
||||
}
|
||||
|
||||
try {
|
||||
sourceNodeRef.current = audioContextRef.current.createMediaStreamSource(streamRef.current);
|
||||
workletNodeRef.current = new AudioWorkletNode(audioContextRef.current, 'audio-processor');
|
||||
echoNodeRef.current = new AudioWorkletNode(audioContextRef.current, 'echo-processor');
|
||||
|
||||
sourceNodeRef.current.connect(workletNodeRef.current);
|
||||
echoNodeRef.current.connect(audioContextRef.current.destination);
|
||||
|
||||
console.log('Audio nodes connected successfully');
|
||||
} catch (nodeError) {
|
||||
console.error('Error setting up audio nodes:', nodeError);
|
||||
throw new Error(`Audio node setup failed: ${nodeError.message}`);
|
||||
}
|
||||
|
||||
workletNodeRef.current.port.onmessage = (event) => {
|
||||
if (event.data instanceof ArrayBuffer) {
|
||||
if (websocketRef.current?.readyState === WebSocket.OPEN) {
|
||||
websocketRef.current.send(event.data);
|
||||
}
|
||||
} else if (event.data.type === 'vad') {
|
||||
if (event.data.status === 'speech_end') {
|
||||
vadEndTimeRef.current = Date.now();
|
||||
hasPlaybackLatencyRef.current = false;
|
||||
addLog('[Frontend VAD] End of speech detected');
|
||||
} else if (event.data.status === 'speech_start') {
|
||||
vadStartTimeRef.current = Date.now();
|
||||
addLog('[Frontend VAD] Start of speech detected');
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// Add message handler for the echo node
|
||||
echoNodeRef.current.port.onmessage = (event) => {
|
||||
if (event.data.type === 'queue_empty' && isPlayingRef.current) {
|
||||
isPlayingRef.current = false;
|
||||
|
||||
// Reset playback state
|
||||
playbackStartTimeRef.current = null;
|
||||
hasPlaybackLatencyRef.current = false;
|
||||
addLog('Audio playback completed');
|
||||
}
|
||||
};
|
||||
|
||||
setIsRecording(true);
|
||||
} catch (error) {
|
||||
console.error('Error in startRecording:', error);
|
||||
addLog(`Error: ${error.message}`, 'error');
|
||||
// Clean up any partially initialized resources
|
||||
stopRecording();
|
||||
}
|
||||
};
|
||||
|
||||
const stopRecording = () => {
|
||||
if (pingIntervalRef.current) {
|
||||
clearInterval(pingIntervalRef.current);
|
||||
pingIntervalRef.current = null;
|
||||
}
|
||||
hasPlaybackLatencyRef.current = false;
|
||||
if (workletNodeRef.current) {
|
||||
workletNodeRef.current.disconnect();
|
||||
}
|
||||
if (echoNodeRef.current) {
|
||||
echoNodeRef.current.disconnect();
|
||||
}
|
||||
if (sourceNodeRef.current) {
|
||||
sourceNodeRef.current.disconnect();
|
||||
}
|
||||
if (streamRef.current) {
|
||||
streamRef.current.getTracks().forEach(track => track.stop());
|
||||
}
|
||||
if (audioContextRef.current) {
|
||||
// Guard against closing an already-closed context (stopRecording is now
|
||||
// idempotent because onclose also calls it).
|
||||
if (audioContextRef.current.state !== 'closed') {
|
||||
audioContextRef.current.close();
|
||||
}
|
||||
audioContextRef.current = null;
|
||||
}
|
||||
if (websocketRef.current) {
|
||||
websocketRef.current.close();
|
||||
websocketRef.current = null;
|
||||
}
|
||||
setIsRecording(false);
|
||||
};
|
||||
|
||||
const clearLogs = () => {
|
||||
setLogs([]);
|
||||
};
|
||||
|
||||
// Add this helper function before the return statement
|
||||
const formatLatencyLog = (message: string) => {
|
||||
// Check if it's a latency message with ":" or "-"
|
||||
const splitChar = message.includes(':') ? ':' : message.includes('-') ? '-' : null;
|
||||
if (!splitChar) return message;
|
||||
|
||||
const [label, values] = message.split(splitChar);
|
||||
return (
|
||||
<div className="grid grid-cols-[1fr,auto] gap-2">
|
||||
<span>{label}{splitChar}</span>
|
||||
<span className="font-mono">{values}</span>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
// Update the handleInterrupt function
|
||||
const handleInterrupt = () => {
|
||||
// Stop current audio playback and clear queue
|
||||
if (echoNodeRef.current) {
|
||||
echoNodeRef.current.port.postMessage({ type: 'clear' });
|
||||
echoNodeRef.current.port.postMessage({ type: 'mute' });
|
||||
echoNodeRef.current.disconnect();
|
||||
echoNodeRef.current.connect(audioContextRef.current!.destination);
|
||||
}
|
||||
audioQueueRef.current = [];
|
||||
|
||||
// Reset playback state
|
||||
playbackStartTimeRef.current = null;
|
||||
hasPlaybackLatencyRef.current = false;
|
||||
isPlayingRef.current = false;
|
||||
};
|
||||
|
||||
// Add scroll event handlers
|
||||
const handleChatScroll = () => {
|
||||
if (!chatHistoryRef.current) return;
|
||||
const { scrollTop, scrollHeight, clientHeight } = chatHistoryRef.current;
|
||||
// Consider "at bottom" if within 100 pixels of the bottom
|
||||
shouldAutoScrollChatRef.current = scrollHeight - (scrollTop + clientHeight) < 100;
|
||||
};
|
||||
|
||||
const handleLogsScroll = () => {
|
||||
if (!logsRef.current) return;
|
||||
const { scrollTop, scrollHeight, clientHeight } = logsRef.current;
|
||||
// Consider "at bottom" if within 100 pixels of the bottom
|
||||
shouldAutoScrollLogsRef.current = scrollHeight - (scrollTop + clientHeight) < 100;
|
||||
};
|
||||
|
||||
// Add scroll to bottom functions
|
||||
const scrollChatToBottom = () => {
|
||||
if (chatHistoryRef.current && shouldAutoScrollChatRef.current) {
|
||||
chatHistoryRef.current.scrollTop = chatHistoryRef.current.scrollHeight;
|
||||
}
|
||||
};
|
||||
|
||||
const scrollLogsToBottom = () => {
|
||||
if (logsRef.current && shouldAutoScrollLogsRef.current) {
|
||||
logsRef.current.scrollTop = logsRef.current.scrollHeight;
|
||||
}
|
||||
};
|
||||
|
||||
// Update useEffect to scroll when chat history or logs change
|
||||
useEffect(() => {
|
||||
scrollChatToBottom();
|
||||
}, [chatHistory]);
|
||||
|
||||
useEffect(() => {
|
||||
scrollLogsToBottom();
|
||||
}, [logs]);
|
||||
|
||||
// Tear down the mic stream, AudioContext, WebSocket and ping interval if the
|
||||
// component unmounts mid-recording (client-side nav / fast refresh), instead
|
||||
// of leaving the microphone active and the ping interval firing.
|
||||
useEffect(() => {
|
||||
return () => {
|
||||
stopRecording();
|
||||
};
|
||||
}, []);
|
||||
|
||||
|
||||
|
||||
return (
|
||||
<div className="min-h-screen flex flex-col bg-gray-100 p-2 sm:p-4">
|
||||
{/* Top Controls */}
|
||||
<div className="mb-4 flex flex-col sm:flex-row justify-center gap-2 sm:gap-4">
|
||||
<Button
|
||||
onClick={isRecording ? stopRecording : startRecording}
|
||||
variant={isRecording ? "destructive" : "default"}
|
||||
className={`px-4 sm:px-8 py-4 sm:py-6 text-base sm:text-lg font-semibold flex items-center justify-center gap-2 ${
|
||||
!isRecording ? 'bg-green-600 hover:bg-green-700' : ''
|
||||
}`}
|
||||
>
|
||||
{isRecording ? (
|
||||
<>
|
||||
<MicOff className="w-5 h-5 sm:w-6 sm:h-6" />
|
||||
Stop Recording
|
||||
</>
|
||||
) : (
|
||||
<>
|
||||
<Mic className="w-5 h-5 sm:w-6 sm:h-6" />
|
||||
Start Recording
|
||||
</>
|
||||
)}
|
||||
</Button>
|
||||
<Button
|
||||
onClick={clearLogs}
|
||||
variant="outline"
|
||||
className="px-4 sm:px-8 py-4 sm:py-6 text-base sm:text-lg font-semibold flex items-center justify-center gap-2"
|
||||
>
|
||||
<Trash2 className="w-5 h-5 sm:w-6 sm:h-6" />
|
||||
Clear Logs
|
||||
</Button>
|
||||
</div>
|
||||
|
||||
{/* Mobile Tabs - Only show on small screens */}
|
||||
<div className="lg:hidden mb-2">
|
||||
<div className="flex border-b border-gray-200">
|
||||
<TabButton
|
||||
active={activeTab === 'chat'}
|
||||
onClick={() => setActiveTab('chat')}
|
||||
>
|
||||
Chat History
|
||||
</TabButton>
|
||||
<TabButton
|
||||
active={activeTab === 'logs'}
|
||||
onClick={() => setActiveTab('logs')}
|
||||
>
|
||||
Logs
|
||||
</TabButton>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{/* Main Content */}
|
||||
<div className="flex flex-col lg:flex-row flex-1 gap-2 sm:gap-4">
|
||||
{/* Chat History Panel */}
|
||||
<div className={`w-full lg:w-1/2 h-[calc(100vh-12rem)] lg:h-[calc(90vh-5rem)] ${
|
||||
activeTab === 'chat' ? 'block' : 'hidden lg:block'
|
||||
}`}>
|
||||
<Card className="h-full">
|
||||
<CardHeader className="pb-2 hidden lg:block">
|
||||
<CardTitle className="text-center text-base sm:text-lg">Chat History</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent
|
||||
ref={chatHistoryRef}
|
||||
onScroll={handleChatScroll}
|
||||
className="h-full lg:h-[calc(100%-3rem)] overflow-y-auto pt-4 lg:pt-0"
|
||||
>
|
||||
<div className="flex flex-col space-y-4">
|
||||
{chatHistory.map((message, index) => (
|
||||
<div
|
||||
key={index}
|
||||
className={`flex ${
|
||||
message.role === 'assistant' ? 'bg-gray-50' :
|
||||
message.role === 'transcript' ? 'bg-blue-50' :
|
||||
'bg-white'
|
||||
} p-3 sm:p-4 rounded-lg animate-slide-in`}
|
||||
>
|
||||
<div className="w-6 h-6 sm:w-8 sm:h-8 rounded-full flex-shrink-0 mr-3 sm:mr-4">
|
||||
{message.role === 'assistant' ? (
|
||||
<div className="w-full h-full bg-green-600 rounded-full flex items-center justify-center text-white text-xs sm:text-sm">
|
||||
AI
|
||||
</div>
|
||||
) : message.role === 'transcript' ? (
|
||||
<div className="w-full h-full bg-blue-600 rounded-full flex items-center justify-center text-white text-xs sm:text-sm">
|
||||
T
|
||||
</div>
|
||||
) : (
|
||||
<div className="w-full h-full bg-gray-600 rounded-full flex items-center justify-center text-white text-xs sm:text-sm">
|
||||
U
|
||||
</div>
|
||||
)}
|
||||
</div>
|
||||
<div className="flex-1 prose prose-sm dark:prose-invert prose-p:my-3 prose-headings:mb-3 prose-headings:mt-6 prose-li:my-2 prose-pre:bg-gray-800 prose-pre:text-gray-100 max-w-none">
|
||||
<ReactMarkdown
|
||||
remarkPlugins={[remarkGfm]}
|
||||
components={{
|
||||
code: CodeBlock,
|
||||
}}
|
||||
>
|
||||
{message.content}
|
||||
</ReactMarkdown>
|
||||
{message.role === 'transcript' && !message.isFinal && (
|
||||
<span className="text-xs text-gray-500 ml-2 animate-fade-in">(typing...)</span>
|
||||
)}
|
||||
</div>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
|
||||
{/* Logs Panel */}
|
||||
<div className={`w-full lg:w-1/2 h-[calc(100vh-12rem)] lg:h-[calc(90vh-5rem)] ${
|
||||
activeTab === 'logs' ? 'block' : 'hidden lg:block'
|
||||
}`}>
|
||||
<Card className="h-full overflow-hidden">
|
||||
<CardHeader className="pb-2 hidden lg:block">
|
||||
<CardTitle className="text-center text-base sm:text-lg">Logs</CardTitle>
|
||||
</CardHeader>
|
||||
<CardContent
|
||||
ref={logsRef}
|
||||
onScroll={handleLogsScroll}
|
||||
className="h-full lg:h-[calc(100%-3rem)] bg-gray-900 p-2 lg:p-3 overflow-y-auto"
|
||||
>
|
||||
<div className="font-mono text-xs sm:text-sm">
|
||||
{logs.map((log, index) => {
|
||||
const time = new Date(log.timestamp).toISOString().split('T')[1].slice(0, -1);
|
||||
const baseClasses = "mb-1 font-mono";
|
||||
|
||||
const typeClasses = {
|
||||
error: "text-red-400",
|
||||
latency: "text-cyan-400",
|
||||
llm: "text-green-400",
|
||||
info: "text-gray-300"
|
||||
};
|
||||
|
||||
// Special styling for different event types
|
||||
const prefixColor = {
|
||||
vad: "text-purple-400",
|
||||
asr: "text-yellow-400",
|
||||
server: "text-blue-400"
|
||||
};
|
||||
|
||||
let content = log.message;
|
||||
let prefix = null;
|
||||
|
||||
// Extract prefix if message starts with [Something]
|
||||
const prefixMatch = log.message.match(/^\[(.*?)\]/);
|
||||
if (prefixMatch) {
|
||||
prefix = prefixMatch[0];
|
||||
content = log.message.slice(prefix.length);
|
||||
}
|
||||
|
||||
return (
|
||||
<div key={index} className={`${baseClasses} ${typeClasses[log.type]}`}>
|
||||
<span className="text-gray-500">[{time}]</span>{' '}
|
||||
{prefix && (
|
||||
<span className={
|
||||
Object.entries(prefixColor).find(([key]) =>
|
||||
prefix?.toLowerCase().includes(key))?.[1] || typeClasses[log.type]
|
||||
}>
|
||||
{prefix}
|
||||
</span>
|
||||
)}
|
||||
{log.type === 'latency' ? (
|
||||
formatLatencyLog(content)
|
||||
) : (
|
||||
<span>{content}</span>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
})}
|
||||
</div>
|
||||
</CardContent>
|
||||
</Card>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
module.exports = {
|
||||
plugins: {
|
||||
tailwindcss: {},
|
||||
autoprefixer: {},
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
class AudioProcessor extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.inputBuffer = [];
|
||||
|
||||
// Remove VAD parameters - now using server-side Silero VAD only
|
||||
// The client-side VAD was primarily for debugging and is no longer needed
|
||||
}
|
||||
|
||||
process(inputs, outputs, parameters) {
|
||||
const input = inputs[0];
|
||||
if (!input || !input[0]) return true;
|
||||
|
||||
// Convert to mono
|
||||
const monoInput = new Float32Array(input[0].length);
|
||||
for (let i = 0; i < input[0].length; i++) {
|
||||
let sum = 0;
|
||||
for (let channel = 0; channel < input.length; channel++) {
|
||||
sum += input[channel][i];
|
||||
}
|
||||
monoInput[i] = sum / input.length;
|
||||
}
|
||||
|
||||
// Process in chunks - removed VAD processing
|
||||
const CHUNK_SIZE = 1024;
|
||||
this.inputBuffer.push(...monoInput);
|
||||
|
||||
while (this.inputBuffer.length >= CHUNK_SIZE) {
|
||||
const chunk = this.inputBuffer.slice(0, CHUNK_SIZE);
|
||||
this.inputBuffer = this.inputBuffer.slice(CHUNK_SIZE);
|
||||
|
||||
// Convert to 16-bit PCM
|
||||
const pcmData = new Int16Array(chunk.length);
|
||||
for (let i = 0; i < chunk.length; i++) {
|
||||
const s = Math.max(-1, Math.min(1, chunk[i]));
|
||||
pcmData[i] = s < 0 ? s * 0x8000 : s * 0x7FFF;
|
||||
}
|
||||
|
||||
// Send the data - server-side Silero VAD will handle speech detection
|
||||
this.port.postMessage(pcmData.buffer, [pcmData.buffer]);
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
class EchoProcessor extends AudioWorkletProcessor {
|
||||
constructor() {
|
||||
super();
|
||||
this.audioBuffer = [];
|
||||
this.playbackPosition = 0;
|
||||
this.isPlaying = false;
|
||||
this.sampleRate = 16000;
|
||||
this.isMuted = false;
|
||||
this.outputBufferSize = 2048;
|
||||
this.outputBuffer = new Float32Array(this.outputBufferSize);
|
||||
this.outputBufferPosition = 0;
|
||||
this.hasNotifiedQueueEmpty = false; // Track if we've sent the queue empty notification
|
||||
|
||||
this.port.onmessage = (event) => {
|
||||
if (event.data instanceof Float32Array) {
|
||||
if (!this.isMuted) {
|
||||
const audioData = event.data;
|
||||
const newBuffer = new Float32Array(audioData.length);
|
||||
newBuffer.set(audioData);
|
||||
|
||||
if (!this.isPlaying) {
|
||||
this.audioBuffer = Array.from(newBuffer);
|
||||
this.playbackPosition = 0;
|
||||
this.outputBufferPosition = 0;
|
||||
this.hasNotifiedQueueEmpty = false; // Reset notification flag when starting new playback
|
||||
} else {
|
||||
this.audioBuffer.push(...Array.from(newBuffer));
|
||||
}
|
||||
|
||||
this.isPlaying = true;
|
||||
}
|
||||
} else if (event.data.type === 'clear') {
|
||||
// Clear the buffer and stop playback
|
||||
this.audioBuffer = [];
|
||||
this.playbackPosition = 0;
|
||||
this.outputBufferPosition = 0;
|
||||
this.isPlaying = false;
|
||||
this.isMuted = true;
|
||||
this.hasNotifiedQueueEmpty = false;
|
||||
// Notify that the queue is empty after clearing
|
||||
this.port.postMessage({ type: 'queue_empty' });
|
||||
} else if (event.data.type === 'unmute') {
|
||||
this.isMuted = false;
|
||||
this.hasNotifiedQueueEmpty = false;
|
||||
} else if (event.data.type === 'mute') {
|
||||
this.isMuted = true;
|
||||
this.audioBuffer = [];
|
||||
this.playbackPosition = 0;
|
||||
this.isPlaying = false;
|
||||
this.hasNotifiedQueueEmpty = false;
|
||||
// Notify that the queue is empty after muting
|
||||
this.port.postMessage({ type: 'queue_empty' });
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
process(inputs, outputs, parameters) {
|
||||
const output = outputs[0];
|
||||
|
||||
// If muted or not playing, output silence
|
||||
if (this.isMuted || !this.isPlaying || this.audioBuffer.length === 0) {
|
||||
for (let channel = 0; channel < output.length; channel++) {
|
||||
output[channel].fill(0);
|
||||
}
|
||||
|
||||
// Send queue_empty notification if we haven't already
|
||||
if (this.isPlaying && !this.hasNotifiedQueueEmpty) {
|
||||
this.port.postMessage({ type: 'queue_empty' });
|
||||
this.hasNotifiedQueueEmpty = true;
|
||||
this.isPlaying = false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
const outputChannel = output[0];
|
||||
const bufferSize = outputChannel.length;
|
||||
|
||||
if (this.isPlaying && this.audioBuffer.length > 0) {
|
||||
// Fill the output buffer
|
||||
for (let i = 0; i < bufferSize; i++) {
|
||||
if (this.playbackPosition < this.audioBuffer.length) {
|
||||
const sample = this.audioBuffer[this.playbackPosition];
|
||||
for (let channel = 0; channel < output.length; channel++) {
|
||||
output[channel][i] = sample;
|
||||
}
|
||||
this.playbackPosition++;
|
||||
} else {
|
||||
// End of buffer reached
|
||||
for (let channel = 0; channel < output.length; channel++) {
|
||||
output[channel][i] = 0;
|
||||
}
|
||||
|
||||
// If we've played everything, reset and notify
|
||||
if (this.playbackPosition >= this.audioBuffer.length && !this.hasNotifiedQueueEmpty) {
|
||||
this.isPlaying = false;
|
||||
this.playbackPosition = 0;
|
||||
this.audioBuffer = [];
|
||||
this.port.postMessage({ type: 'queue_empty' });
|
||||
this.hasNotifiedQueueEmpty = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Output silence if we're not playing
|
||||
for (let channel = 0; channel < output.length; channel++) {
|
||||
output[channel].fill(0);
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
registerProcessor('audio-processor', AudioProcessor);
|
||||
registerProcessor('echo-processor', EchoProcessor);
|
||||
@@ -0,0 +1,9 @@
|
||||
@tailwind base;
|
||||
@tailwind components;
|
||||
@tailwind utilities;
|
||||
|
||||
@layer base {
|
||||
.font-mono {
|
||||
font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "Liberation Mono", "Courier New", monospace;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
/** @type {import('tailwindcss').Config} */
|
||||
module.exports = {
|
||||
darkMode: ["class"],
|
||||
content: [
|
||||
'./pages/**/*.{js,ts,jsx,tsx,mdx}',
|
||||
'./components/**/*.{js,ts,jsx,tsx,mdx}',
|
||||
'./app/**/*.{js,ts,jsx,tsx,mdx}',
|
||||
'./src/**/*.{js,ts,jsx,tsx,mdx}',
|
||||
],
|
||||
theme: {
|
||||
container: {
|
||||
center: true,
|
||||
padding: "2rem",
|
||||
screens: {
|
||||
"2xl": "1400px",
|
||||
},
|
||||
},
|
||||
extend: {
|
||||
colors: {
|
||||
border: "hsl(var(--border))",
|
||||
background: "hsl(var(--background))",
|
||||
foreground: "hsl(var(--foreground))",
|
||||
},
|
||||
keyframes: {
|
||||
"accordion-down": {
|
||||
from: { height: 0 },
|
||||
to: { height: "var(--radix-accordion-content-height)" },
|
||||
},
|
||||
"accordion-up": {
|
||||
from: { height: "var(--radix-accordion-content-height)" },
|
||||
to: { height: 0 },
|
||||
},
|
||||
"fade-in": {
|
||||
'0%': { opacity: 0 },
|
||||
'100%': { opacity: 1 },
|
||||
},
|
||||
"slide-in": {
|
||||
'0%': { transform: 'translateY(5px)', opacity: 0 },
|
||||
'100%': { transform: 'translateY(0)', opacity: 1 },
|
||||
}
|
||||
},
|
||||
animation: {
|
||||
"accordion-down": "accordion-down 0.2s ease-out",
|
||||
"accordion-up": "accordion-up 0.2s ease-out",
|
||||
"fade-in": "fade-in 0.3s ease-out",
|
||||
"slide-in": "slide-in 0.3s ease-out",
|
||||
},
|
||||
typography: (theme) => ({
|
||||
DEFAULT: {
|
||||
css: {
|
||||
'--tw-prose-body': theme('colors.gray.900'),
|
||||
'--tw-prose-headings': theme('colors.gray.900'),
|
||||
'--tw-prose-links': theme('colors.blue.600'),
|
||||
'--tw-prose-code': theme('colors.gray.900'),
|
||||
maxWidth: 'none',
|
||||
color: 'var(--tw-prose-body)',
|
||||
fontSize: '1rem',
|
||||
lineHeight: '1.75',
|
||||
p: {
|
||||
marginTop: '1em',
|
||||
marginBottom: '1em',
|
||||
fontSize: '1rem',
|
||||
'&:first-child': {
|
||||
marginTop: 0,
|
||||
},
|
||||
'&:last-child': {
|
||||
marginBottom: 0,
|
||||
},
|
||||
},
|
||||
'ul, ol': {
|
||||
paddingLeft: '1.5em',
|
||||
marginTop: '0.5em',
|
||||
marginBottom: '0.5em',
|
||||
},
|
||||
li: {
|
||||
marginTop: '0.25em',
|
||||
marginBottom: '0.25em',
|
||||
fontSize: '1rem',
|
||||
lineHeight: '1.5',
|
||||
p: {
|
||||
marginTop: '0.375em',
|
||||
marginBottom: '0.375em',
|
||||
},
|
||||
},
|
||||
'h1, h2, h3, h4': {
|
||||
color: 'var(--tw-prose-headings)',
|
||||
marginTop: '1.5em',
|
||||
marginBottom: '0.5em',
|
||||
fontSize: '1.25rem',
|
||||
fontWeight: '600',
|
||||
lineHeight: '1.3',
|
||||
'&:first-child': {
|
||||
marginTop: 0,
|
||||
},
|
||||
},
|
||||
pre: {
|
||||
margin: '0.5em 0',
|
||||
padding: '0.5em',
|
||||
backgroundColor: 'transparent',
|
||||
borderRadius: '0.375rem',
|
||||
fontSize: '0.875rem',
|
||||
lineHeight: '1.5',
|
||||
overflowX: 'auto',
|
||||
},
|
||||
code: {
|
||||
color: 'var(--tw-prose-code)',
|
||||
backgroundColor: theme('colors.gray.100'),
|
||||
padding: '0.2em 0.4em',
|
||||
borderRadius: '0.25rem',
|
||||
fontSize: '0.875rem',
|
||||
fontWeight: '400',
|
||||
},
|
||||
'pre code': {
|
||||
backgroundColor: 'transparent',
|
||||
padding: 0,
|
||||
fontSize: '0.875rem',
|
||||
color: 'inherit',
|
||||
fontWeight: '400',
|
||||
},
|
||||
blockquote: {
|
||||
borderLeftWidth: '4px',
|
||||
borderLeftColor: theme('colors.gray.200'),
|
||||
paddingLeft: '1em',
|
||||
fontStyle: 'italic',
|
||||
marginTop: '1em',
|
||||
marginBottom: '1em',
|
||||
fontSize: '1rem',
|
||||
},
|
||||
hr: {
|
||||
marginTop: '2em',
|
||||
marginBottom: '2em',
|
||||
},
|
||||
a: {
|
||||
color: 'var(--tw-prose-links)',
|
||||
textDecoration: 'underline',
|
||||
'&:hover': {
|
||||
color: theme('colors.blue.700'),
|
||||
},
|
||||
},
|
||||
table: {
|
||||
width: '100%',
|
||||
marginTop: '1em',
|
||||
marginBottom: '1em',
|
||||
borderCollapse: 'collapse',
|
||||
fontSize: '0.875rem',
|
||||
lineHeight: '1.5',
|
||||
},
|
||||
'th, td': {
|
||||
padding: '0.5em',
|
||||
borderWidth: '1px',
|
||||
borderColor: theme('colors.gray.200'),
|
||||
},
|
||||
},
|
||||
},
|
||||
}),
|
||||
},
|
||||
},
|
||||
plugins: [
|
||||
require('@tailwindcss/typography'),
|
||||
require("tailwindcss-animate")
|
||||
],
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"baseUrl": ".",
|
||||
"paths": {
|
||||
"@/*": ["./*"]
|
||||
},
|
||||
"lib": [
|
||||
"dom",
|
||||
"dom.iterable",
|
||||
"esnext"
|
||||
],
|
||||
"allowJs": true,
|
||||
"skipLibCheck": true,
|
||||
"strict": false,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"noEmit": true,
|
||||
"incremental": true,
|
||||
"esModuleInterop": true,
|
||||
"module": "esnext",
|
||||
"moduleResolution": "node",
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"jsx": "preserve"
|
||||
},
|
||||
"include": [
|
||||
"next-env.d.ts",
|
||||
"**/*.ts",
|
||||
"**/*.tsx"
|
||||
],
|
||||
"exclude": [
|
||||
"node_modules"
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user