ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter8/sesame/README.md

@@ -0,0 +1,328 @@
+# Sesame CSM (1B) TTS - Text-to-Speech Fine-tuning
+
+## English
+
+This directory contains scripts for fine-tuning and running inference with the Sesame CSM text-to-speech model using Unsloth.
+
+## Files
+
+- `sesame_csm_sft_unsloth.py` - Training script for fine-tuning the model with LoRA
+- `inference.py` - Single inference script for generating speech from text
+- `batch_inference.py` - Batch inference script for processing multiple texts
+- `example_inputs.json` - Example input file for batch inference
+- `requirements.txt` - Python dependencies
+
+## Installation
+
+```bash
+# Use a separate project-local environment for Sesame.
+# This project pins transformers==4.52.3, while the root ch7 extra uses
+# transformers>=4.55 for MultilingualReasoning, so no single environment
+# can satisfy both contracts safely.
+# From the repository root:
+cd chapter8/sesame
+python -m venv .venv-sesame
+source .venv-sesame/bin/activate
+# Windows PowerShell: .\.venv-sesame\Scripts\Activate.ps1
+# Windows cmd: .venv-sesame\Scripts\activate.bat
+
+python -m pip install -r requirements.txt
+
+# For Conda users, install ffmpeg
+conda install -c conda-forge "ffmpeg>=6.0" -y
+conda install -c conda-forge libiconv -y
+```
+
+## Training
+
+To train a model with your own dataset:
+
+```bash
+python sesame_csm_sft_unsloth.py
+```
+
+This will:
+1. Load the base model `unsloth/csm-1b`
+2. Add LoRA adapters
+3. Fine-tune on your dataset
+4. Save the LoRA adapters to `lora_model/`
+
+## Inference
+
+### Single Text Inference
+
+Generate speech from a single text:
+
+```bash
+# Without context (simple generation)
+python inference.py \
+    --lora-path lora_model \
+    --text "We just finished fine tuning a text to speech model... and it's pretty good!" \
+    --output example_without_context_1.wav
+
+# With voice context (for voice consistency) using dataset index 3
+python inference.py \
+    --lora-path lora_model \
+    --text "Sesame is a super cool TTS model which can be fine tuned with Unsloth." \
+    --dataset-context-idx 3 \
+    --output example_with_context_1.wav
+
+# Using base model only (without LoRA)
+python inference.py \
+    --text "Hello world, this is a test of the text to speech system."
+
+# With custom speaker ID and longer generation
+python inference.py \
+    --lora-path lora_model \
+    --text "This is a longer sentence that needs more tokens." \
+    --speaker-id 0 \
+    --max-tokens 250 \
+    --output long_speech.wav
+
+# With 4-bit quantization (lower memory)
+python inference.py \
+    --lora-path lora_model \
+    --text "Memory efficient inference." \
+    --load-in-4bit
+```
+
+### Batch Inference
+
+Process multiple texts at once:
+
+```bash
+# From JSON file
+python batch_inference.py \
+    --lora-path lora_model \
+    --input-file example_inputs.json \
+    --output-dir batch_outputs
+
+# From plain text file (one text per line)
+python batch_inference.py \
+    --lora-path lora_model \
+    --input-file texts.txt \
+    --output-dir batch_outputs
+```
+
+#### Input File Formats
+
+**JSON format** (`example_inputs.json`):
+
+Without context:
+```json
+[
+    {
+        "text": "We just finished fine tuning a text to speech model... and it's pretty good!",
+        "speaker_id": 0,
+        "output": "example_without_context_1.wav"
+    },
+    {
+        "text": "Sesame is a super cool TTS model which can be fine tuned with Unsloth.",
+        "speaker_id": 0,
+        "output": "example_without_context_2.wav"
+    }
+]
+```
+
+With context (for voice consistency using dataset indices):
+```json
+[
+    {
+        "text": "Sesame is a super cool TTS model which can be fine tuned with Unsloth.",
+        "speaker_id": 0,
+        "dataset_context_idx": 3,
+        "output": "example_with_context_1.wav"
+    },
+    {
+        "text": "We just finished fine tuning a text to speech model... and it's pretty good!",
+        "speaker_id": 0,
+        "dataset_context_idx": 4,
+        "output": "example_with_context_2.wav"
+    }
+]
+```
+
+**Note**: `dataset_context_idx` refers to the index in the training dataset. The original `MrDragonFox/Elise` repository is now disabled, so the runnable defaults use its public `maxbsoft/mrdragonfox-elise` mirror. Indices 3 and 4 are used in the training script examples.
+
+**Plain text format** (one sentence per line, no context support):
+```
+Hello world, this is the first sentence.
+This is the second sentence.
+This is the third sentence.
+```
+
+## Parameters
+
+### Common Parameters
+
+- `--base-model`: Base model name or path (default: `unsloth/csm-1b`)
+- `--lora-path`: Path to saved LoRA adapters (optional)
+- `--load-in-4bit`: Load model in 4-bit quantization to reduce memory usage
+
+### Inference Parameters
+
+- `--text`: Text to convert to speech
+- `--speaker-id`: Speaker ID for multi-speaker models (default: 0)
+- `--output`: Output audio file path (default: `output.wav`)
+- `--max-tokens`: Maximum tokens to generate (125 ≈ 10 seconds of audio)
+- `--dataset-context-idx`: Dataset index to use for voice consistency (e.g., 3 or 4 from training examples)
+- `--dataset-name`: Dataset name to load context from (default: `maxbsoft/mrdragonfox-elise`)
+
+### Batch Inference Parameters
+
+- `--input-file`: Input file (JSON or plain text)
+- `--output-dir`: Output directory for audio files (default: `outputs`)
+
+## Model Information
+
+- **Base Model**: Sesame CSM (1B) - A compact text-to-speech model
+- **Output Format**: 24kHz WAV audio
+- **Token-to-Time Ratio**: Approximately 125 tokens = 10 seconds of audio
+- **Multi-speaker**: Supports multiple speakers via speaker IDs
+
+## Advanced Usage
+
+### Voice Consistency with Context
+
+For better voice consistency, you can provide audio context from the dataset (see `sesame_csm_sft_unsloth.py` lines 320-390 for examples):
+
+```python
+from inference import load_model, generate_speech
+
+model, processor = load_model("unsloth/csm-1b", "lora_model")
+
+# Use dataset index 3 for voice consistency (same as training script)
+generate_speech(
+    model=model,
+    processor=processor,
+    text="Sesame is a super cool TTS model which can be fine tuned with Unsloth.",
+    dataset_context_idx=3,
+    dataset_name="maxbsoft/mrdragonfox-elise",
+    output_path="output_with_context.wav"
+)
+```
+
+The inference scripts automatically load the audio and text from the specified dataset index, ensuring consistency with the training approach.
+
+## Memory Requirements
+
+- **Base model**: ~4-6GB VRAM
+- **Base model + LoRA training**: ~8-12GB VRAM
+- **With 4-bit quantization**: ~2-3GB VRAM
+- **Inference only**: ~2-4GB VRAM
+
+Use `--load-in-4bit` flag if you have limited GPU memory.
+
+## Troubleshooting
+
+### AssertionError during generation
+
+If you encounter `AssertionError` during `model.generate()`, ensure you're passing tensors correctly:
+
+```python
+# ❌ Wrong
+audio_values = model.generate(**inputs, max_new_tokens=125)
+
+# ✅ Correct
+audio_values = model.generate(
+    input_ids=inputs["input_ids"],
+    attention_mask=inputs.get("attention_mask"),
+    max_new_tokens=125,
+    output_audio=True
+)
+```
+
+### Out of memory
+
+- Use `--load-in-4bit` flag
+- Reduce `max_new_tokens`
+- Process texts one at a time instead of batching
+- Use a smaller batch size during training
+
+## Resources
+
+- [Unsloth Documentation](https://docs.unsloth.ai/)
+- [Unsloth TTS Guide](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning)
+- [Unsloth Discord](https://discord.gg/unsloth)
+- [Unsloth GitHub](https://github.com/unslothai/unsloth)
+
+## License
+
+This project uses the Unsloth library and Sesame CSM model. Please refer to their respective licenses.
+
+---
+
+## 中文
+
+# Sesame CSM（1B）TTS：文本转语音微调
+
+## 文件
+
+项目包含训练脚本、单条推理脚本、批量推理脚本、依赖文件以及数据处理相关配置。
+
+## 安装
+
+```bash
+# Sesame 请使用单独的项目本地环境。
+# 本项目固定 transformers==4.52.3，而根目录 ch7 extra 为
+# MultilingualReasoning 使用 transformers>=4.55；单一环境无法安全同时满足两者。
+# 从仓库根目录开始：
+cd chapter8/sesame
+python -m venv .venv-sesame
+source .venv-sesame/bin/activate
+# Windows PowerShell：.\.venv-sesame\Scripts\Activate.ps1
+# Windows cmd：.venv-sesame\Scripts\activate.bat
+
+python -m pip install -r requirements.txt
+
+# Conda 用户可安装 ffmpeg
+conda install -c conda-forge ffmpeg
+```
+
+请根据本机 CUDA 版本安装匹配的 PyTorch，并确认 FFmpeg 可以正常处理音频。
+
+## 训练
+
+训练脚本加载 Sesame CSM 1B 基础模型和语音数据集，通过 LoRA 进行参数高效微调。运行前请检查模型名称、数据集、输出目录、批量大小和训练轮数。
+
+## 推理
+
+### 单条文本推理
+
+推理脚本支持无上下文生成、使用数据集样本作为声音上下文、仅加载基础模型、自定义说话人 ID、更长生成长度以及 4-bit 量化。
+
+### 批量推理
+
+批量脚本可从 JSON 文件或逐行文本文件读取输入，并为每条文本生成独立音频。
+
+## 参数
+
+常用参数包括模型与适配器路径、输入文本、输出路径、说话人 ID、上下文样本索引、最大生成长度、温度和量化开关。批量模式还支持输入文件、输出目录和失败重试设置。
+
+## 模型信息
+
+CSM 是一类上下文语音模型，可利用文本、说话人标识和参考语音生成具有一致音色的语音。本实验围绕 1B 参数版本进行微调。
+
+## 高级用法
+
+### 使用上下文保持声音一致
+
+可选择训练数据中的音频作为上下文。训练与推理应使用一致的上下文格式和说话人 ID；英文示例使用数据集索引 3。
+
+## 显存需求
+
+实际显存取决于序列长度、上下文音频、批量大小和精度。4-bit 量化可降低推理显存，训练时可使用梯度累积与梯度检查点。
+
+## 故障排查
+
+### 生成期间出现 AssertionError
+
+确保说话人 ID、上下文列表和文本列表的长度及嵌套结构符合模型接口，不要把单个值误传为错误形状的批量输入。
+
+### 显存不足
+
+减小批量大小、上下文长度或最大生成长度，启用量化，并关闭其他占用 GPU 的进程。
+
+## 资源与许可
+
+模型和相关项目链接见英文部分。使用前请核对上游模型、数据集和代码许可。