liqiang/ai-agent-book
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
Details
dependency resolution / resolve (3.11) (push) Canceled after 0s
Details
dependency resolution / resolve (3.13) (push) Canceled after 0s
Details
deploy-pages / build (push) Canceled after 0s
Details
deploy-pages / deploy (push) Canceled after 0s
Details
i18n consistency check / check (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
Details
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
Details
web-search-agent tests / test (push) Canceled after 0s
Details
web-search-agent tests / agentbook (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
liqiang
2026-08-20 13:12:50 +00:00
commit b119135836
10275 changed files with 3284984 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/REPORT.md
+51
View File
@@ -0,0 +1,51 @@
# Experiment 8-6 strict acceptance report
Execution acceptance: **PASS**
This run trained two real LoRA adapters on an RTX PRO 6000. It used 128 Orpheus training utterances plus 16 held-out utterances, and 168 stratified Sesame training utterances plus 24 held-out utterances. Each track completed 60 optimizer updates at effective batch size four. Both adapters are identified by local SHA-256 inventories and public Hugging Face repositories.
## Execution gates
- PASS — `orpheus_128_train_examples`
- PASS — `orpheus_16_held_out_examples`
- PASS — `orpheus_60_optimizer_steps`
- PASS — `orpheus_remote_adapter_sha256_verified`
- PASS — `orpheus_16_valid_comparison_files`
- PASS — `sesame_128_train_examples`
- PASS — `sesame_tag_categories_present`
- PASS — `sesame_60_optimizer_steps`
- PASS — `sesame_remote_adapter_sha256_verified`
- PASS — `sesame_24_valid_comparison_files`
## Hypothesis results
- SUPPORTED — `orpheus_held_out_loss_decreased`
- NOT SUPPORTED — `orpheus_cross_sentence_timbre_proxy_improved`
- SUPPORTED — `sesame_held_out_loss_decreased`
- SUPPORTED — `sesame_adapted_mean_tag_score_is_positive`
- SUPPORTED — `sesame_tag_sensitivity_improved_over_base`
Execution completion and hypothesis support are intentionally separate. A completed campaign may produce a negative hypothesis result.
## Orpheus result
- Held-out loss: 5.237792 before → 4.865821 after.
- Mean cross-sentence MFCC-statistic cosine: 0.988627 base → 0.986702 adapted (Δ -0.001924).
- Eight unseen sentences were generated for each arm with matched seeds. This metric is a timbre-consistency proxy; it is not speaker-verification or a listening-test score.
## Sesame result
- Held-out loss: 128.230759 before → 124.342400 after.
- Mean matching AudioSet event-score difference (tagged − neutral): +0.000131 base → +0.001097 adapted (Δ +0.000966).
- Positive matched pairs: 3/6 base; 4/6 adapted.
- Six prompt pairs (laugh, giggle, sigh) were generated per arm with the same seed within each tagged/neutral pair. AudioSet scores are detector proxies, not proof of natural expression.
## Failure retention and limits
`failure_comparisons.json` retains silent/short outputs, each Orpheus arm's least-consistent sentence pair, and every Sesame pair where adding a tag did not raise the matching AudioSet score. `compatibility_failures.json` retains the disabled-source-dataset failure, current Unsloth CSM pad-token rejection, and Transformers bf16 codec merge failure, together with the exact standard-PEFT/float32 fallback. The Sesame held-out loss split contains laugh, sigh, and neutral examples but no giggle examples because all 32 available giggle-tagged rows were allocated to the substantive training split. The campaign does not include blinded human MOS, speaker-verification enrollment, confidence intervals over multiple training seeds, or deployment-scale data. Therefore it makes no claim of perceptual quality or generalization beyond this bounded run.
## Adapter identity
- Orpheus: https://huggingface.co/bojieli/exp8-6-orpheus-elise-lora/tree/536092e9479fa1717e2b8f9cc1be52728b273e95
- Sesame: https://huggingface.co/bojieli/exp8-6-sesame-elise-tags-lora/tree/f2e042be0f38d6078976ef7e16cf49b91097f756
- Exact revisions and every retained artifact hash are in `orpheus_manifest.json`, `sesame_manifest.json`, and `artifact_inventory.json`.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/orpheus/README.md
+208
View File
@@ -0,0 +1,208 @@
---
base_model: unsloth/orpheus-3b-0.1-ft
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:unsloth/orpheus-3b-0.1-ft
- lora
- transformers
- unsloth
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.0
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/orpheus/adapter_config.json
+52
View File
@@ -0,0 +1,52 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": {
"base_model_class": "LlamaForCausalLM",
"parent_library": "transformers.models.llama.modeling_llama",
"unsloth_fixed": true
},
"base_model_name_or_path": "unsloth/orpheus-3b-0.1-ft",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 16,
"lora_bias": false,
"lora_dropout": 0,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.19.0",
"qalora_group_size": 16,
"r": 16,
"rank_pattern": {},
"revision": null,
"target_modules": [
"gate_proj",
"up_proj",
"o_proj",
"down_proj",
"k_proj",
"q_proj",
"v_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/orpheus/chat_template.jinja
+93
View File
@@ -0,0 +1,93 @@
{{- bos_token }}
{%- if custom_tools is defined %}
{%- set tools = custom_tools %}
{%- endif %}
{%- if not tools_in_user_message is defined %}
{%- set tools_in_user_message = true %}
{%- endif %}
{%- if not date_string is defined %}
{%- if strftime_now is defined %}
{%- set date_string = strftime_now("%d %b %Y") %}
{%- else %}
{%- set date_string = "26 Jul 2024" %}
{%- endif %}
{%- endif %}
{%- if not tools is defined %}
{%- set tools = none %}
{%- endif %}
{#- This block extracts the system message, so we can slot it into the right place. #}
{%- if messages[0]['role'] == 'system' %}
{%- set system_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{%- set system_message = "" %}
{%- endif %}
{#- System message #}
{{- "<|start_header_id|>system<|end_header_id|>\n\n" }}
{%- if tools is not none %}
{{- "Environment: ipython\n" }}
{%- endif %}
{{- "Cutting Knowledge Date: December 2023\n" }}
{{- "Today Date: " + date_string + "\n\n" }}
{%- if tools is not none and not tools_in_user_message %}
{{- "You have access to the following functions. To call a function, please respond with JSON for a function call." }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{%- endif %}
{{- system_message }}
{{- "<|eot_id|>" }}
{#- Custom tools are passed in a user message with some extra guidance #}
{%- if tools_in_user_message and not tools is none %}
{#- Extract the first user message so we can plug it in here #}
{%- if messages | length != 0 %}
{%- set first_user_message = messages[0]['content']|trim %}
{%- set messages = messages[1:] %}
{%- else %}
{{- raise_exception("Cannot put tools in the first user message when there's no first user message!") }}
{%- endif %}
{{- '<|start_header_id|>user<|end_header_id|>\n\n' -}}
{{- "Given the following functions, please respond with a JSON for a function call " }}
{{- "with its proper arguments that best answers the given prompt.\n\n" }}
{{- 'Respond in the format {"name": function name, "parameters": dictionary of argument name and its value}.' }}
{{- "Do not use variables.\n\n" }}
{%- for t in tools %}
{{- t | tojson(indent=4) }}
{{- "\n\n" }}
{%- endfor %}
{{- first_user_message + "<|eot_id|>"}}
{%- endif %}
{%- for message in messages %}
{%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}
{{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' }}
{%- elif 'tool_calls' in message %}
{%- if not message.tool_calls|length == 1 %}
{{- raise_exception("This model only supports single tool-calls at once!") }}
{%- endif %}
{%- set tool_call = message.tool_calls[0].function %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' -}}
{{- '{"name": "' + tool_call.name + '", ' }}
{{- '"parameters": ' }}
{{- tool_call.arguments | tojson }}
{{- "}" }}
{{- "<|eot_id|>" }}
{%- elif message.role == "tool" or message.role == "ipython" %}
{{- "<|start_header_id|>ipython<|end_header_id|>\n\n" }}
{%- if message.content is mapping or message.content is iterable %}
{{- message.content | tojson }}
{%- else %}
{{- message.content }}
{%- endif %}
{{- "<|eot_id|>" }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|start_header_id|>assistant<|end_header_id|>\n\n' }}
{%- endif %}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/orpheus/special_tokens_map.json
+26
View File
@@ -0,0 +1,26 @@
{
"additional_special_tokens": [
"<|audio|>"
],
"bos_token": {
"content": "<|begin_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|eot_id|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|finetune_right_pad_id|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/sesame/README.md
+207
View File
@@ -0,0 +1,207 @@
---
base_model: unsloth/csm-1b
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:unsloth/csm-1b
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.0
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/sesame/adapter_config.json
+48
View File
@@ -0,0 +1,48 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "unsloth/csm-1b",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 16,
"lora_bias": false,
"lora_dropout": 0,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.19.0",
"qalora_group_size": 16,
"r": 16,
"rank_pattern": {},
"revision": null,
"target_modules": [
"down_proj",
"v_proj",
"q_proj",
"gate_proj",
"k_proj",
"o_proj",
"up_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/sesame/chat_template.jinja
+45
View File
@@ -0,0 +1,45 @@
{%- for message in messages %}
{#-- Validate role is a stringified integer --#}
{%- if not message['role'] is string or not message['role'].isdigit() %}
{{- raise_exception("The role must be an integer or a stringified integer (e.g. '0') designating the speaker id") }}
{%- endif %}
{#-- Validate content is a list --#}
{%- set content = message['content'] %}
{%- if content is not iterable or content is string %}
{{- raise_exception("The content must be a list") }}
{%- endif %}
{#-- Collect content types --#}
{%- set content_types = content | map(attribute='type') | list %}
{%- set is_last = loop.last %}
{#-- Last message validation --#}
{%- if is_last %}
{%- if 'text' not in content_types %}
{{- raise_exception("The last message must include one item of type 'text'") }}
{%- elif (content_types | select('equalto', 'text') | list | length > 1) or (content_types | select('equalto', 'audio') | list | length > 1) %}
{{- raise_exception("At most two items are allowed in the last message: one 'text' and one 'audio'") }}
{%- endif %}
{#-- All other messages validation --#}
{%- else %}
{%- if content_types | select('equalto', 'text') | list | length != 1
or content_types | select('equalto', 'audio') | list | length != 1 %}
{{- raise_exception("Each message (except the last) must contain exactly one 'text' and one 'audio' item") }}
{%- elif content_types | reject('in', ['text', 'audio']) | list | length > 0 %}
{{- raise_exception("Only 'text' and 'audio' types are allowed in content") }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- for message in messages %}
{{- bos_token }}
{{- '[' + message['role'] + ']' }}
{{- message['content'][0]['text'] }}
{{- eos_token }}
{%- if message['content']|length > 1 %}
{{- '<|AUDIO|><|audio_eos|>' }}
{%- endif %}
{%- endfor %}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/sesame/preprocessor_config.json
+11
View File
@@ -0,0 +1,11 @@
{
"chunk_length_s": null,
"feature_extractor_type": "EncodecFeatureExtractor",
"feature_size": 1,
"overlap": null,
"padding_side": "right",
"padding_value": 0.0,
"processor_class": "CsmProcessor",
"return_attention_mask": true,
"sampling_rate": 24000
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/adapters/sesame/special_tokens_map.json
+23
View File
@@ -0,0 +1,23 @@
{
"bos_token": {
"content": "<|begin_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"eos_token": {
"content": "<|end_of_text|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
},
"pad_token": {
"content": "<|finetune_right_pad_id|>",
"lstrip": false,
"normalized": false,
"rstrip": false,
"single_word": false
}
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/analysis.json
+723
View File
@@ -0,0 +1,723 @@
{
"experiment": "8-6",
"execution_acceptance": "PASS",
"execution_gates": {
"orpheus_128_train_examples": true,
"orpheus_16_held_out_examples": true,
"orpheus_60_optimizer_steps": true,
"orpheus_remote_adapter_sha256_verified": true,
"orpheus_16_valid_comparison_files": true,
"sesame_128_train_examples": true,
"sesame_tag_categories_present": true,
"sesame_60_optimizer_steps": true,
"sesame_remote_adapter_sha256_verified": true,
"sesame_24_valid_comparison_files": true
},
"hypothesis_results": {
"orpheus_held_out_loss_decreased": true,
"orpheus_cross_sentence_timbre_proxy_improved": false,
"sesame_held_out_loss_decreased": true,
"sesame_adapted_mean_tag_score_is_positive": true,
"sesame_tag_sensitivity_improved_over_base": true
},
"quality_claim": "No human naturalness or voice-identity quality claim; automatic metrics are reproducible proxies only.",
"remote_adapter_verification": {
"orpheus": {
"repository": "https://huggingface.co/bojieli/exp8-6-orpheus-elise-lora",
"revision": "536092e9479fa1717e2b8f9cc1be52728b273e95",
"expected_sha256": "fbd43cd5287a4b69398c29f88286e60146e9328cb48e3c22e15a2a305b25b29f",
"downloaded_sha256": "fbd43cd5287a4b69398c29f88286e60146e9328cb48e3c22e15a2a305b25b29f",
"verified": true
},
"sesame": {
"repository": "https://huggingface.co/bojieli/exp8-6-sesame-elise-tags-lora",
"revision": "f2e042be0f38d6078976ef7e16cf49b91097f756",
"expected_sha256": "23994b467b319f99bbc642f4dc2ba4d39718838cd7cbfe949311a42ebcb5480e",
"downloaded_sha256": "23994b467b319f99bbc642f4dc2ba4d39718838cd7cbfe949311a42ebcb5480e",
"verified": true
}
},
"orpheus": {
"base": {
"audio_count": 8,
"valid_audio_count": 8,
"mean_pairwise_mfcc_cosine": 0.9886267249073301,
"min_pairwise_mfcc_cosine": 0.9738007187843323,
"pairwise": [
{
"prompt_a": 1,
"prompt_b": 7,
"cosine": 0.9738007187843323
},
{
"prompt_a": 3,
"prompt_b": 7,
"cosine": 0.9751747846603394
},
{
"prompt_a": 5,
"prompt_b": 7,
"cosine": 0.9768423438072205
},
{
"prompt_a": 1,
"prompt_b": 6,
"cosine": 0.9821145534515381
},
{
"prompt_a": 5,
"prompt_b": 6,
"cosine": 0.9832608699798584
},
{
"prompt_a": 0,
"prompt_b": 7,
"cosine": 0.9844203591346741
},
{
"prompt_a": 3,
"prompt_b": 4,
"cosine": 0.9847696423530579
},
{
"prompt_a": 4,
"prompt_b": 5,
"cosine": 0.985227644443512
},
{
"prompt_a": 2,
"prompt_b": 7,
"cosine": 0.9852697849273682
},
{
"prompt_a": 1,
"prompt_b": 4,
"cosine": 0.9852963089942932
},
{
"prompt_a": 3,
"prompt_b": 6,
"cosine": 0.9860416650772095
},
{
"prompt_a": 0,
"prompt_b": 6,
"cosine": 0.9888354539871216
},
{
"prompt_a": 0,
"prompt_b": 1,
"cosine": 0.9906319379806519
},
{
"prompt_a": 0,
"prompt_b": 5,
"cosine": 0.9906508326530457
},
{
"prompt_a": 0,
"prompt_b": 4,
"cosine": 0.991395890712738
},
{
"prompt_a": 3,
"prompt_b": 5,
"cosine": 0.9918218851089478
},
{
"prompt_a": 1,
"prompt_b": 3,
"cosine": 0.9923202395439148
},
{
"prompt_a": 1,
"prompt_b": 2,
"cosine": 0.9923385977745056
},
{
"prompt_a": 2,
"prompt_b": 3,
"cosine": 0.9924396276473999
},
{
"prompt_a": 4,
"prompt_b": 7,
"cosine": 0.9930636882781982
},
{
"prompt_a": 2,
"prompt_b": 4,
"cosine": 0.9931535124778748
},
{
"prompt_a": 2,
"prompt_b": 5,
"cosine": 0.9931651949882507
},
{
"prompt_a": 6,
"prompt_b": 7,
"cosine": 0.9940599799156189
},
{
"prompt_a": 2,
"prompt_b": 6,
"cosine": 0.9941393136978149
},
{
"prompt_a": 0,
"prompt_b": 3,
"cosine": 0.9943145513534546
},
{
"prompt_a": 0,
"prompt_b": 2,
"cosine": 0.994386613368988
},
{
"prompt_a": 4,
"prompt_b": 6,
"cosine": 0.9950653314590454
},
{
"prompt_a": 1,
"prompt_b": 5,
"cosine": 0.9975469708442688
}
]
},
"adapted": {
"audio_count": 8,
"valid_audio_count": 8,
"mean_pairwise_mfcc_cosine": 0.9867024081093925,
"min_pairwise_mfcc_cosine": 0.9688550233840942,
"pairwise": [
{
"prompt_a": 1,
"prompt_b": 5,
"cosine": 0.9688550233840942
},
{
"prompt_a": 0,
"prompt_b": 5,
"cosine": 0.9705199003219604
},
{
"prompt_a": 3,
"prompt_b": 5,
"cosine": 0.9741944670677185
},
{
"prompt_a": 5,
"prompt_b": 6,
"cosine": 0.9782752990722656
},
{
"prompt_a": 4,
"prompt_b": 5,
"cosine": 0.9795999526977539
},
{
"prompt_a": 5,
"prompt_b": 7,
"cosine": 0.9798238277435303
},
{
"prompt_a": 0,
"prompt_b": 7,
"cosine": 0.9805128574371338
},
{
"prompt_a": 1,
"prompt_b": 2,
"cosine": 0.9849662184715271
},
{
"prompt_a": 0,
"prompt_b": 6,
"cosine": 0.9854229688644409
},
{
"prompt_a": 0,
"prompt_b": 2,
"cosine": 0.9863125681877136
},
{
"prompt_a": 0,
"prompt_b": 4,
"cosine": 0.9869258999824524
},
{
"prompt_a": 1,
"prompt_b": 4,
"cosine": 0.9872470498085022
},
{
"prompt_a": 2,
"prompt_b": 3,
"cosine": 0.9892308115959167
},
{
"prompt_a": 0,
"prompt_b": 1,
"cosine": 0.9895753860473633
},
{
"prompt_a": 2,
"prompt_b": 7,
"cosine": 0.9899733662605286
},
{
"prompt_a": 3,
"prompt_b": 6,
"cosine": 0.9900691509246826
},
{
"prompt_a": 0,
"prompt_b": 3,
"cosine": 0.9903662800788879
},
{
"prompt_a": 1,
"prompt_b": 3,
"cosine": 0.9910094141960144
},
{
"prompt_a": 3,
"prompt_b": 4,
"cosine": 0.9911966323852539
},
{
"prompt_a": 2,
"prompt_b": 6,
"cosine": 0.9914029836654663
},
{
"prompt_a": 2,
"prompt_b": 5,
"cosine": 0.9914451241493225
},
{
"prompt_a": 1,
"prompt_b": 7,
"cosine": 0.9918804168701172
},
{
"prompt_a": 4,
"prompt_b": 6,
"cosine": 0.9921923279762268
},
{
"prompt_a": 2,
"prompt_b": 4,
"cosine": 0.9927540421485901
},
{
"prompt_a": 1,
"prompt_b": 6,
"cosine": 0.992949903011322
},
{
"prompt_a": 3,
"prompt_b": 7,
"cosine": 0.993278980255127
},
{
"prompt_a": 6,
"prompt_b": 7,
"cosine": 0.9935612678527832
},
{
"prompt_a": 4,
"prompt_b": 7,
"cosine": 0.9941253066062927
}
]
},
"adapted_minus_base_mean_pairwise_mfcc_cosine": -0.0019243167979375864
},
"sesame": {
"classifier": "MIT/ast-finetuned-audioset-10-10-0.4593",
"labels": {
"laugh": {
"id": 16,
"name": "Laughter"
},
"giggle": {
"id": 18,
"name": "Giggle"
},
"sigh": {
"id": 26,
"name": "Sigh"
}
},
"base": {
"audio_count": 12,
"valid_audio_count": 12,
"mean_tagged_minus_neutral": 0.00013129252571767816,
"positive_pair_count": 3,
"pairs": [
{
"pair_id": "giggle_0",
"tag": "giggle",
"neutral_score": 0.0009356054943054914,
"tagged_score": 0.0006116584991104901,
"tagged_minus_neutral": -0.00032394699519500136
},
{
"pair_id": "giggle_1",
"tag": "giggle",
"neutral_score": 0.0008636609418317676,
"tagged_score": 0.0005855335621163249,
"tagged_minus_neutral": -0.00027812737971544266
},
{
"pair_id": "laugh_0",
"tag": "laugh",
"neutral_score": 0.0003021236334461719,
"tagged_score": 0.0003340380499139428,
"tagged_minus_neutral": 3.1914416467770934e-05
},
{
"pair_id": "laugh_1",
"tag": "laugh",
"neutral_score": 0.0002934956573881209,
"tagged_score": 0.0007533965981565416,
"tagged_minus_neutral": 0.0004599009407684207
},
{
"pair_id": "sigh_0",
"tag": "sigh",
"neutral_score": 0.0034571753349155188,
"tagged_score": 0.006142919883131981,
"tagged_minus_neutral": 0.002685744548216462
},
{
"pair_id": "sigh_1",
"tag": "sigh",
"neutral_score": 0.0024482603184878826,
"tagged_score": 0.0006605299422517419,
"tagged_minus_neutral": -0.0017877303762361407
}
]
},
"adapted": {
"audio_count": 12,
"valid_audio_count": 12,
"mean_tagged_minus_neutral": 0.001097210906057929,
"positive_pair_count": 4,
"pairs": [
{
"pair_id": "giggle_0",
"tag": "giggle",
"neutral_score": 0.001688887132331729,
"tagged_score": 0.0017779936315491796,
"tagged_minus_neutral": 8.910649921745062e-05
},
{
"pair_id": "giggle_1",
"tag": "giggle",
"neutral_score": 0.003269059816375375,
"tagged_score": 0.0007443267968483269,
"tagged_minus_neutral": -0.002524733019527048
},
{
"pair_id": "laugh_0",
"tag": "laugh",
"neutral_score": 0.00026441123918630183,
"tagged_score": 0.0012162356870248914,
"tagged_minus_neutral": 0.0009518244478385895
},
{
"pair_id": "laugh_1",
"tag": "laugh",
"neutral_score": 0.00033708245609886944,
"tagged_score": 0.0003236289485357702,
"tagged_minus_neutral": -1.3453507563099265e-05
},
{
"pair_id": "sigh_0",
"tag": "sigh",
"neutral_score": 0.0017972267232835293,
"tagged_score": 0.006373145151883364,
"tagged_minus_neutral": 0.0045759184285998344
},
{
"pair_id": "sigh_1",
"tag": "sigh",
"neutral_score": 0.0010693982476368546,
"tagged_score": 0.004574000835418701,
"tagged_minus_neutral": 0.0035046025877818465
}
]
},
"adapted_minus_base_mean_tag_sensitivity": 0.0009659183803402508,
"scores": [
{
"arm": "base",
"pair_id": "laugh_0",
"condition": "neutral",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.0003021236334461719,
"seconds": 2.72,
"rms": 0.06748455762863159,
"path": "audio/sesame/base/laugh_0_neutral.wav"
},
{
"arm": "base",
"pair_id": "laugh_0",
"condition": "tagged",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.0003340380499139428,
"seconds": 6.0,
"rms": 0.06531786173582077,
"path": "audio/sesame/base/laugh_0_tagged.wav"
},
{
"arm": "base",
"pair_id": "laugh_1",
"condition": "neutral",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.0002934956573881209,
"seconds": 2.08,
"rms": 0.23473648726940155,
"path": "audio/sesame/base/laugh_1_neutral.wav"
},
{
"arm": "base",
"pair_id": "laugh_1",
"condition": "tagged",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.0007533965981565416,
"seconds": 3.84,
"rms": 0.03746151924133301,
"path": "audio/sesame/base/laugh_1_tagged.wav"
},
{
"arm": "base",
"pair_id": "giggle_0",
"condition": "neutral",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.0009356054943054914,
"seconds": 1.68,
"rms": 0.14035280048847198,
"path": "audio/sesame/base/giggle_0_neutral.wav"
},
{
"arm": "base",
"pair_id": "giggle_0",
"condition": "tagged",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.0006116584991104901,
"seconds": 2.96,
"rms": 0.08793672919273376,
"path": "audio/sesame/base/giggle_0_tagged.wav"
},
{
"arm": "base",
"pair_id": "giggle_1",
"condition": "neutral",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.0008636609418317676,
"seconds": 2.96,
"rms": 0.049566783010959625,
"path": "audio/sesame/base/giggle_1_neutral.wav"
},
{
"arm": "base",
"pair_id": "giggle_1",
"condition": "tagged",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.0005855335621163249,
"seconds": 5.92,
"rms": 0.0142013905569911,
"path": "audio/sesame/base/giggle_1_tagged.wav"
},
{
"arm": "base",
"pair_id": "sigh_0",
"condition": "neutral",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.0034571753349155188,
"seconds": 2.32,
"rms": 0.06743967533111572,
"path": "audio/sesame/base/sigh_0_neutral.wav"
},
{
"arm": "base",
"pair_id": "sigh_0",
"condition": "tagged",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.006142919883131981,
"seconds": 6.0,
"rms": 0.05658983066678047,
"path": "audio/sesame/base/sigh_0_tagged.wav"
},
{
"arm": "base",
"pair_id": "sigh_1",
"condition": "neutral",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.0024482603184878826,
"seconds": 2.24,
"rms": 0.014161595143377781,
"path": "audio/sesame/base/sigh_1_neutral.wav"
},
{
"arm": "base",
"pair_id": "sigh_1",
"condition": "tagged",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.0006605299422517419,
"seconds": 3.36,
"rms": 0.010387708432972431,
"path": "audio/sesame/base/sigh_1_tagged.wav"
},
{
"arm": "adapted",
"pair_id": "laugh_0",
"condition": "neutral",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.00026441123918630183,
"seconds": 3.92,
"rms": 0.0402710996568203,
"path": "audio/sesame/adapted/laugh_0_neutral.wav"
},
{
"arm": "adapted",
"pair_id": "laugh_0",
"condition": "tagged",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.0012162356870248914,
"seconds": 3.6,
"rms": 0.06890808790922165,
"path": "audio/sesame/adapted/laugh_0_tagged.wav"
},
{
"arm": "adapted",
"pair_id": "laugh_1",
"condition": "neutral",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.00033708245609886944,
"seconds": 2.48,
"rms": 0.13957583904266357,
"path": "audio/sesame/adapted/laugh_1_neutral.wav"
},
{
"arm": "adapted",
"pair_id": "laugh_1",
"condition": "tagged",
"tag": "laugh",
"audioset_label": "Laughter",
"audioset_score": 0.0003236289485357702,
"seconds": 3.76,
"rms": 0.051783934235572815,
"path": "audio/sesame/adapted/laugh_1_tagged.wav"
},
{
"arm": "adapted",
"pair_id": "giggle_0",
"condition": "neutral",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.001688887132331729,
"seconds": 2.0,
"rms": 0.11314665526151657,
"path": "audio/sesame/adapted/giggle_0_neutral.wav"
},
{
"arm": "adapted",
"pair_id": "giggle_0",
"condition": "tagged",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.0017779936315491796,
"seconds": 2.72,
"rms": 0.07838206738233566,
"path": "audio/sesame/adapted/giggle_0_tagged.wav"
},
{
"arm": "adapted",
"pair_id": "giggle_1",
"condition": "neutral",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.003269059816375375,
"seconds": 3.28,
"rms": 0.10014152526855469,
"path": "audio/sesame/adapted/giggle_1_neutral.wav"
},
{
"arm": "adapted",
"pair_id": "giggle_1",
"condition": "tagged",
"tag": "giggle",
"audioset_label": "Giggle",
"audioset_score": 0.0007443267968483269,
"seconds": 4.88,
"rms": 0.07103203237056732,
"path": "audio/sesame/adapted/giggle_1_tagged.wav"
},
{
"arm": "adapted",
"pair_id": "sigh_0",
"condition": "neutral",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.0017972267232835293,
"seconds": 2.72,
"rms": 0.09723222255706787,
"path": "audio/sesame/adapted/sigh_0_neutral.wav"
},
{
"arm": "adapted",
"pair_id": "sigh_0",
"condition": "tagged",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.006373145151883364,
"seconds": 3.92,
"rms": 0.1117464229464531,
"path": "audio/sesame/adapted/sigh_0_tagged.wav"
},
{
"arm": "adapted",
"pair_id": "sigh_1",
"condition": "neutral",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.0010693982476368546,
"seconds": 2.32,
"rms": 0.15041208267211914,
"path": "audio/sesame/adapted/sigh_1_neutral.wav"
},
{
"arm": "adapted",
"pair_id": "sigh_1",
"condition": "tagged",
"tag": "sigh",
"audioset_label": "Sigh",
"audioset_score": 0.004574000835418701,
"seconds": 3.36,
"rms": 0.07697359472513199,
"path": "audio/sesame/adapted/sigh_1_tagged.wav"
}
]
}
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/artifact_inventory.json
+320
View File
@@ -0,0 +1,320 @@
[
{
"path": "adapters/orpheus/README.md",
"bytes": 5214,
"sha256": "d62b94e4ab068c9379c9d6a92fa73dc9468f85ce204615543fa20db6e453e4fb"
},
{
"path": "adapters/orpheus/adapter_config.json",
"bytes": 1238,
"sha256": "40bd5beba0cbeb72bbe0acf0371d0ab64f1217722d3590ee165f5f9d2f745ca9"
},
{
"path": "adapters/orpheus/chat_template.jinja",
"bytes": 3827,
"sha256": "5816fce10444e03c2e9ee1ef8a4a1ea61ae7e69e438613f3b17b69d0426223a4"
},
{
"path": "adapters/orpheus/special_tokens_map.json",
"bytes": 508,
"sha256": "f17390e64274b3117a7f99afa87eb1de93097b820842c68bd626aff53d73d7f5"
},
{
"path": "adapters/sesame/README.md",
"bytes": 5183,
"sha256": "65a8c53feecbfc1e7a8cd64bc6a79baedbffd1ef6b9cc17b1ea0a5529e4d74ec"
},
{
"path": "adapters/sesame/adapter_config.json",
"bytes": 1091,
"sha256": "44c3053b5037214ea8862003dfeea867554c9fbb83df5728ea80e9a0915e3ee2"
},
{
"path": "adapters/sesame/chat_template.jinja",
"bytes": 2002,
"sha256": "b43753fbbc23be93160cccf825d2cf97dec39538cb274350216940e7acbc0fe7"
},
{
"path": "adapters/sesame/preprocessor_config.json",
"bytes": 271,
"sha256": "8648040cda0469976874c733acbec88c0ef094cd95628ab63b32950f01872af9"
},
{
"path": "adapters/sesame/special_tokens_map.json",
"bytes": 459,
"sha256": "32f404d626cf7b1b6eea36c241ae7cbd6ec29c777a8701ea48c0fe9ebb94c9b1"
},
{
"path": "analysis.json",
"bytes": 19921,
"sha256": "958f7bb5188233867f50ee4920a4bc8d4ae46b1af1eb46ca8eef74cde88e3f1c"
},
{
"path": "audio/orpheus/adapted/prompt_00.wav",
"bytes": 221228,
"sha256": "24470415a507117b9f2b6a4b3f13350a453febca782f542e592ea6d41678f9da"
},
{
"path": "audio/orpheus/adapted/prompt_01.wav",
"bytes": 245804,
"sha256": "f31e8572ef2509e4537328d1b080e92f9a4d833a859068ce991220df790cf884"
},
{
"path": "audio/orpheus/adapted/prompt_02.wav",
"bytes": 196652,
"sha256": "4ec5e55e4aa7b085f61961340a701dd2e784787e98ef4d03111a2fddc1aab457"
},
{
"path": "audio/orpheus/adapted/prompt_03.wav",
"bytes": 188460,
"sha256": "07e7c5b0b9d37cc2e87cc70c31dee0adf55db8e76772e28661a33d8b14f1afe9"
},
{
"path": "audio/orpheus/adapted/prompt_04.wav",
"bytes": 155692,
"sha256": "9033e23dcb4a31f4deb2108b04674b811c7c3c8a4b3fc638e8076675fec428ce"
},
{
"path": "audio/orpheus/adapted/prompt_05.wav",
"bytes": 180268,
"sha256": "1de6449d583fb48f0db9d1ba19bbfefda60016f32b246d9562d34a6a9bd7e322"
},
{
"path": "audio/orpheus/adapted/prompt_06.wav",
"bytes": 196652,
"sha256": "0b54348eb7b0d46cbcbd1869afa0b96fde6997d79036b19e11b2b7854261e468"
},
{
"path": "audio/orpheus/adapted/prompt_07.wav",
"bytes": 204844,
"sha256": "e71ca35cf0c12b1fb40d30cb26a285d0207ce9f845239c5ecfdd246e605f7903"
},
{
"path": "audio/orpheus/base/prompt_00.wav",
"bytes": 147500,
"sha256": "04e89b14b5aed1dfd9ccd11ba50eb8ab6d9d4282995ca48cebc77256ecb90216"
},
{
"path": "audio/orpheus/base/prompt_01.wav",
"bytes": 139308,
"sha256": "7980d7b59dccc9b2dea59581280823586b69ca1d48862d6fac2e74a7a9bfabc2"
},
{
"path": "audio/orpheus/base/prompt_02.wav",
"bytes": 221228,
"sha256": "b98b50173b0a0964eedcfd1286b9e258bf5c0d304b6dcd881ac10c8bf9709918"
},
{
"path": "audio/orpheus/base/prompt_03.wav",
"bytes": 180268,
"sha256": "b91552513a2ed17fab55c484e16cffd70d917a360b4735b2aea250845563b4c3"
},
{
"path": "audio/orpheus/base/prompt_04.wav",
"bytes": 163884,
"sha256": "45421df4a7cec956ec603fbb64811cc87fe6947afc3a0335dd8b32d40ec9ccd8"
},
{
"path": "audio/orpheus/base/prompt_05.wav",
"bytes": 208940,
"sha256": "a2431d6ef66a4a798a528d185d0707f430d608764acda9586e91f82215544afb"
},
{
"path": "audio/orpheus/base/prompt_06.wav",
"bytes": 184364,
"sha256": "16af5a42ac769e4194002ea1465f0666e0990d4a231e370808cc2f9057665b7d"
},
{
"path": "audio/orpheus/base/prompt_07.wav",
"bytes": 196652,
"sha256": "143f28842c83448b8fcea73017526274be7ac061f0aa3b727ea623f9839bcaad"
},
{
"path": "audio/sesame/adapted/giggle_0_neutral.wav",
"bytes": 96044,
"sha256": "16cbff670bebf59188345a79fcc99952ee5f7ba925e214874c71399b8b684fb6"
},
{
"path": "audio/sesame/adapted/giggle_0_tagged.wav",
"bytes": 130604,
"sha256": "e0d7ef91ff0c52b71a8630aa598bfbc30971149f36915fd3252511cbeb3d9a2c"
},
{
"path": "audio/sesame/adapted/giggle_1_neutral.wav",
"bytes": 157484,
"sha256": "653c1ed9cf6e11412cc935dee7dff90f0d059a72137faec5a63c795459658e15"
},
{
"path": "audio/sesame/adapted/giggle_1_tagged.wav",
"bytes": 234284,
"sha256": "9b72e5252d8b6f7a6f4c7ec3bbbf0f45e39bb2a9e6a16db8e140a9c6e9f32c2c"
},
{
"path": "audio/sesame/adapted/laugh_0_neutral.wav",
"bytes": 188204,
"sha256": "464bbcf2b6292d8a259cbb594e3c14a0e5ba8ca531bcac391e6a6495f9a7420c"
},
{
"path": "audio/sesame/adapted/laugh_0_tagged.wav",
"bytes": 172844,
"sha256": "00cf9a83ce1694de4ddab137b668306acce45c5459cee81fa3702a3a5cc94359"
},
{
"path": "audio/sesame/adapted/laugh_1_neutral.wav",
"bytes": 119084,
"sha256": "4cba912639254c066d906b318531574ed1c2d664c2ece8e8b2f185fa85298e88"
},
{
"path": "audio/sesame/adapted/laugh_1_tagged.wav",
"bytes": 180524,
"sha256": "0b25fc0881603af5831526dd4fcfcfb7dfe1539b3185ddd4358b230d16b50ea8"
},
{
"path": "audio/sesame/adapted/sigh_0_neutral.wav",
"bytes": 130604,
"sha256": "1cbb54147d1375513821f170b229379c8285d264d7ba4b4c568fcdb75bf3c7f0"
},
{
"path": "audio/sesame/adapted/sigh_0_tagged.wav",
"bytes": 188204,
"sha256": "d432582bfebcd4961ca2d3dfcfd944df252ed4826172f5987325172bcb23fd4e"
},
{
"path": "audio/sesame/adapted/sigh_1_neutral.wav",
"bytes": 111404,
"sha256": "b00babd0052826c46bc90a4d1e5335069ad26f6670ad7e90669f71320f715a8a"
},
{
"path": "audio/sesame/adapted/sigh_1_tagged.wav",
"bytes": 161324,
"sha256": "6da2951a4629a2a809e8595bd3940d5da3fe494e4e89cec5c728dcc0fd9d22d9"
},
{
"path": "audio/sesame/base/giggle_0_neutral.wav",
"bytes": 80684,
"sha256": "b3a6df7fe20a4e713f19f961772e908477962b335419ac5289b05fea2d9adebd"
},
{
"path": "audio/sesame/base/giggle_0_tagged.wav",
"bytes": 142124,
"sha256": "fe8a33bf80d471d1812c6d37586e5989eadcc10ae61c48ed02e4e516eafd2dc3"
},
{
"path": "audio/sesame/base/giggle_1_neutral.wav",
"bytes": 142124,
"sha256": "73b02cd34995eb75549fd5941a67c1b621b7d4f613d140f83693955d9529cd44"
},
{
"path": "audio/sesame/base/giggle_1_tagged.wav",
"bytes": 284204,
"sha256": "c1f7efac3892a907c7e08ecef5c013fa83db23879a1a5d1d055f81ec7efd4348"
},
{
"path": "audio/sesame/base/laugh_0_neutral.wav",
"bytes": 130604,
"sha256": "beb4be0e62fa63ca5c81facb5f82f7a565e027f51e0df489d8f361664b49460f"
},
{
"path": "audio/sesame/base/laugh_0_tagged.wav",
"bytes": 288044,
"sha256": "1a475df9255e256277a1a4fc66fee780c604977381059a04f09b33f46ebc00ce"
},
{
"path": "audio/sesame/base/laugh_1_neutral.wav",
"bytes": 99884,
"sha256": "85f7ed5500add23a6d5a4e51df5c0c01e00ecb2c740863b0155cb22d547bf701"
},
{
"path": "audio/sesame/base/laugh_1_tagged.wav",
"bytes": 184364,
"sha256": "46f2c99ca4fd65fff0c724900e9baa1f7bfab25fbc6246c669c9678c463def96"
},
{
"path": "audio/sesame/base/sigh_0_neutral.wav",
"bytes": 111404,
"sha256": "661c54d5ffba92518ef690cade4d3338a7d5c6f8e67882bf024d557e2b5eeeff"
},
{
"path": "audio/sesame/base/sigh_0_tagged.wav",
"bytes": 288044,
"sha256": "90cbd9c00b4f27ce71723bc022d3a5a0503ea6039ed7456a2b50785d86986f30"
},
{
"path": "audio/sesame/base/sigh_1_neutral.wav",
"bytes": 107564,
"sha256": "eb79e2600f5c4c0d571ae6e6ba5dc26c84c1547a6fb5225a9467833789c61b8a"
},
{
"path": "audio/sesame/base/sigh_1_tagged.wav",
"bytes": 161324,
"sha256": "3e5b8de26910376f0906253ae1effff657b927efb6382a7ff4aaeea3080d0c58"
},
{
"path": "compatibility_failures.json",
"bytes": 2089,
"sha256": "b63d17feb8f4fa6d77b0930c27c2b5a11593c5f76e1e7681acbd4fc118f89725"
},
{
"path": "failure_comparisons.json",
"bytes": 1838,
"sha256": "4ca79650a568e6164c37fe2a76577b398245f9d372cbf7d640373d3eecfdb224"
},
{
"path": "orpheus_manifest.json",
"bytes": 9215,
"sha256": "ea5df497cb092ba6520cbe41d82209a0e688ae7f9ea74e7764132ff05b324d6a"
},
{
"path": "sesame_manifest.json",
"bytes": 12199,
"sha256": "86499cd9d2bcc89823432bb54b59072044bf232ccc47f34be5079e9289ed1542"
},
{
"path": "adapters/orpheus/adapter_model.safetensors",
"bytes": 97307544,
"sha256": "fbd43cd5287a4b69398c29f88286e60146e9328cb48e3c22e15a2a305b25b29f",
"storage": "huggingface",
"repository": "https://huggingface.co/bojieli/exp8-6-orpheus-elise-lora",
"revision": "536092e9479fa1717e2b8f9cc1be52728b273e95"
},
{
"path": "adapters/orpheus/tokenizer.json",
"bytes": 22849547,
"sha256": "fc3fecb199b4170636dbfab986d25f628157268d37b861f9cadaca60b1353bce",
"storage": "huggingface",
"repository": "https://huggingface.co/bojieli/exp8-6-orpheus-elise-lora",
"revision": "536092e9479fa1717e2b8f9cc1be52728b273e95"
},
{
"path": "adapters/orpheus/tokenizer_config.json",
"bytes": 5403483,
"sha256": "e2d66c40996da379756a3209c42283e7a8203b1238b54920b070ea384fdc3a9f",
"storage": "huggingface",
"repository": "https://huggingface.co/bojieli/exp8-6-orpheus-elise-lora",
"revision": "536092e9479fa1717e2b8f9cc1be52728b273e95"
},
{
"path": "adapters/sesame/adapter_model.safetensors",
"bytes": 58125112,
"sha256": "23994b467b319f99bbc642f4dc2ba4d39718838cd7cbfe949311a42ebcb5480e",
"storage": "huggingface",
"repository": "https://huggingface.co/bojieli/exp8-6-sesame-elise-tags-lora",
"revision": "f2e042be0f38d6078976ef7e16cf49b91097f756"
},
{
"path": "adapters/sesame/tokenizer.json",
"bytes": 17210159,
"sha256": "92487a7224af2e53dc421a3f73056078acea66f0e86734f8d41babce66bad5cc",
"storage": "huggingface",
"repository": "https://huggingface.co/bojieli/exp8-6-sesame-elise-tags-lora",
"revision": "f2e042be0f38d6078976ef7e16cf49b91097f756"
},
{
"path": "adapters/sesame/tokenizer_config.json",
"bytes": 50577,
"sha256": "eb03e82f89eebfd9b9e2c1a2741dfc34c8982c3c14d132b0ec42fed737b832d3",
"storage": "huggingface",
"repository": "https://huggingface.co/bojieli/exp8-6-sesame-elise-tags-lora",
"revision": "f2e042be0f38d6078976ef7e16cf49b91097f756"
}
]
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_00.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_01.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_02.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_03.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_04.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_05.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_06.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/adapted/prompt_07.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_00.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_01.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_02.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_03.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_04.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_05.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_06.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/orpheus/base/prompt_07.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/giggle_0_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/giggle_0_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/giggle_1_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/giggle_1_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/laugh_0_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/laugh_0_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/laugh_1_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/laugh_1_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/sigh_0_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/sigh_0_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/sigh_1_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/adapted/sigh_1_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/giggle_0_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/giggle_0_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/giggle_1_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/giggle_1_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/laugh_0_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/laugh_0_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/laugh_1_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/laugh_1_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/sigh_0_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/sigh_0_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/sigh_1_neutral.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/audio/sesame/base/sigh_1_tagged.wav
BIN
View File
Binary file not shown.
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/compatibility_failures.json
+30
View File
@@ -0,0 +1,30 @@
{
"experiment": "8-6",
"status": "resolved_without_reducing_campaign",
"failures": [
{
"stage": "dataset_load",
"component": "datasets 3.6.0 / Hugging Face Hub",
"attempted_source": "MrDragonFox/Elise",
"exception_type": "FileNotFoundError",
"message": "The dataset repository exists but Hugging Face marks it disabled, so load_dataset could not resolve its data files.",
"resolution": "Pinned the public non-disabled maxbsoft/mrdragonfox-elise mirror at revision 2cc657c3f94a83df18fcd968b7531ca1a19c7f88. Both training manifests record the substitution."
},
{
"stage": "sesame_model_load",
"component": "Unsloth 2026.8.2",
"exception_type": "RuntimeError",
"message": "Unsloth: Could not find a valid pad token for unsloth/csm-1b - please inspect the tokenizer. A temporary '<｜PAD▁TOKEN｜>' was added.",
"root_cause": "The upstream tokenizer aliases pad to EOS while CSM config declares existing token 128004 (<|finetune_right_pad_id|>) as padding; the 2026.8 safety guard also interprets CSM's audio-codebook vocab_size during validation.",
"resolution": "Used standard Transformers + PEFT LoRA for Sesame, with a temporary tokenizer metadata snapshot pointing pad_token to the existing model-configured token 128004. No vocabulary item or model weight was added."
},
{
"stage": "sesame_pre_training_evaluation",
"component": "Transformers 4.57.6 CsmForConditionalGeneration",
"exception_type": "RuntimeError",
"message": "Index put requires the source and destination dtypes match, got BFloat16 for the destination and Float for the source.",
"root_cause": "CSM's audio codec returned float32 audio embeddings while bf16 autocast produced bf16 text embedding slots in _merge_input_ids_with_input_values.",
"resolution": "Ran the 1B Sesame model and its LoRA campaign in float32 (bf16 disabled). The final campaign retained the same 168 training examples, 24 held-out examples, 60 optimizer steps, and 24 base/adapted comparison WAVs."
}
]
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/failure_comparisons.json
+68
View File
@@ -0,0 +1,68 @@
[
{
"track": "orpheus",
"arm": "base",
"reason": "lowest_cross_sentence_timbre_proxy",
"prompt_a": 1,
"prompt_b": 7,
"cosine": 0.9738007187843323
},
{
"track": "orpheus",
"arm": "adapted",
"reason": "lowest_cross_sentence_timbre_proxy",
"prompt_a": 1,
"prompt_b": 5,
"cosine": 0.9688550233840942
},
{
"track": "sesame",
"arm": "base",
"reason": "tag_did_not_raise_matching_audioset_score",
"pair_id": "giggle_0",
"tag": "giggle",
"neutral_score": 0.0009356054943054914,
"tagged_score": 0.0006116584991104901,
"tagged_minus_neutral": -0.00032394699519500136
},
{
"track": "sesame",
"arm": "base",
"reason": "tag_did_not_raise_matching_audioset_score",
"pair_id": "giggle_1",
"tag": "giggle",
"neutral_score": 0.0008636609418317676,
"tagged_score": 0.0005855335621163249,
"tagged_minus_neutral": -0.00027812737971544266
},
{
"track": "sesame",
"arm": "base",
"reason": "tag_did_not_raise_matching_audioset_score",
"pair_id": "sigh_1",
"tag": "sigh",
"neutral_score": 0.0024482603184878826,
"tagged_score": 0.0006605299422517419,
"tagged_minus_neutral": -0.0017877303762361407
},
{
"track": "sesame",
"arm": "adapted",
"reason": "tag_did_not_raise_matching_audioset_score",
"pair_id": "giggle_1",
"tag": "giggle",
"neutral_score": 0.003269059816375375,
"tagged_score": 0.0007443267968483269,
"tagged_minus_neutral": -0.002524733019527048
},
{
"track": "sesame",
"arm": "adapted",
"reason": "tag_did_not_raise_matching_audioset_score",
"pair_id": "laugh_1",
"tag": "laugh",
"neutral_score": 0.00033708245609886944,
"tagged_score": 0.0003236289485357702,
"tagged_minus_neutral": -1.3453507563099265e-05
}
]
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/orpheus_manifest.json
+264
View File
@@ -0,0 +1,264 @@
{
"experiment": "8-6",
"track": "orpheus_cross_sentence_voice_consistency",
"status": "trained_and_generated",
"seed": 7601,
"base_model": "unsloth/orpheus-3b-0.1-ft",
"base_model_revision": "eae2b6e5e429c81b95ac42a883ac64f126583d43",
"dataset": "maxbsoft/mrdragonfox-elise",
"dataset_revision": "2cc657c3f94a83df18fcd968b7531ca1a19c7f88",
"source_dataset_note": "Public non-disabled mirror of the disabled MrDragonFox/Elise dataset named by the upstream notebook.",
"train_examples_requested": 128,
"train_examples_encoded": 128,
"eval_examples_requested": 16,
"eval_examples_encoded": 16,
"train_failures": [],
"eval_failures": [],
"max_audio_seconds": 4.0,
"optimizer_steps": 60,
"effective_batch_size": 4,
"lora_rank": 16,
"pre_eval": {
"eval_loss": 5.237792491912842,
"eval_model_preparation_time": 0.0093,
"eval_runtime": 0.7249,
"eval_samples_per_second": 22.072,
"eval_steps_per_second": 22.072
},
"train_metrics": {
"train_runtime": 24.1374,
"train_samples_per_second": 9.943,
"train_steps_per_second": 2.486,
"total_flos": 1303634560167936.0,
"train_loss": 4.880102078119914,
"epoch": 1.875
},
"post_eval": {
"eval_loss": 4.86582088470459,
"eval_model_preparation_time": 0.0093,
"eval_runtime": 0.3247,
"eval_samples_per_second": 49.269,
"eval_steps_per_second": 49.269,
"epoch": 1.875
},
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
"peak_gpu_memory_bytes": 9277800448,
"wall_seconds": 202.15462517738342,
"adapter_local_files": [
{
"path": "adapters/orpheus/README.md",
"bytes": 5214,
"sha256": "d62b94e4ab068c9379c9d6a92fa73dc9468f85ce204615543fa20db6e453e4fb"
},
{
"path": "adapters/orpheus/adapter_config.json",
"bytes": 1238,
"sha256": "40bd5beba0cbeb72bbe0acf0371d0ab64f1217722d3590ee165f5f9d2f745ca9"
},
{
"path": "adapters/orpheus/adapter_model.safetensors",
"bytes": 97307544,
"sha256": "fbd43cd5287a4b69398c29f88286e60146e9328cb48e3c22e15a2a305b25b29f"
},
{
"path": "adapters/orpheus/chat_template.jinja",
"bytes": 3827,
"sha256": "5816fce10444e03c2e9ee1ef8a4a1ea61ae7e69e438613f3b17b69d0426223a4"
},
{
"path": "adapters/orpheus/special_tokens_map.json",
"bytes": 508,
"sha256": "f17390e64274b3117a7f99afa87eb1de93097b820842c68bd626aff53d73d7f5"
},
{
"path": "adapters/orpheus/tokenizer.json",
"bytes": 22849547,
"sha256": "fc3fecb199b4170636dbfab986d25f628157268d37b861f9cadaca60b1353bce"
},
{
"path": "adapters/orpheus/tokenizer_config.json",
"bytes": 5403483,
"sha256": "e2d66c40996da379756a3209c42283e7a8203b1238b54920b070ea384fdc3a9f"
}
],
"adapter_huggingface_repo": "https://huggingface.co/bojieli/exp8-6-orpheus-elise-lora",
"adapter_huggingface_revision": "536092e9479fa1717e2b8f9cc1be52728b273e95",
"audio": [
{
"prompt_id": 0,
"prompt": "The morning train crossed the bridge just before sunrise.",
"seed": 7601,
"path": "audio/orpheus/base/prompt_00.wav",
"sha256": "04e89b14b5aed1dfd9ccd11ba50eb8ab6d9d4282995ca48cebc77256ecb90216",
"samples": 73728,
"seconds": 3.072,
"decoded_frames": 36,
"first_invalid_frame": null
},
{
"prompt_id": 1,
"prompt": "Please leave the blue notebook beside the kitchen window.",
"seed": 7602,
"path": "audio/orpheus/base/prompt_01.wav",
"sha256": "7980d7b59dccc9b2dea59581280823586b69ca1d48862d6fac2e74a7a9bfabc2",
"samples": 69632,
"seconds": 2.9013333333333335,
"decoded_frames": 34,
"first_invalid_frame": null
},
{
"prompt_id": 2,
"prompt": "A patient astronomer mapped every bright star in the winter sky.",
"seed": 7603,
"path": "audio/orpheus/base/prompt_02.wav",
"sha256": "b98b50173b0a0964eedcfd1286b9e258bf5c0d304b6dcd881ac10c8bf9709918",
"samples": 110592,
"seconds": 4.608,
"decoded_frames": 54,
"first_invalid_frame": null
},
{
"prompt_id": 3,
"prompt": "We walked home slowly while the last shops turned off their lights.",
"seed": 7604,
"path": "audio/orpheus/base/prompt_03.wav",
"sha256": "b91552513a2ed17fab55c484e16cffd70d917a360b4735b2aea250845563b4c3",
"samples": 90112,
"seconds": 3.7546666666666666,
"decoded_frames": 44,
"first_invalid_frame": null
},
{
"prompt_id": 4,
"prompt": "Could you read the final paragraph one more time for the group?",
"seed": 7605,
"path": "audio/orpheus/base/prompt_04.wav",
"sha256": "45421df4a7cec956ec603fbb64811cc87fe6947afc3a0335dd8b32d40ec9ccd8",
"samples": 81920,
"seconds": 3.4133333333333336,
"decoded_frames": 40,
"first_invalid_frame": null
},
{
"prompt_id": 5,
"prompt": "The small garden stayed green even through the hottest week of July.",
"seed": 7606,
"path": "audio/orpheus/base/prompt_05.wav",
"sha256": "a2431d6ef66a4a798a528d185d0707f430d608764acda9586e91f82215544afb",
"samples": 104448,
"seconds": 4.352,
"decoded_frames": 51,
"first_invalid_frame": null
},
{
"prompt_id": 6,
"prompt": "I packed a warm coat, two apples, and a compass for the long hike.",
"seed": 7607,
"path": "audio/orpheus/base/prompt_06.wav",
"sha256": "16af5a42ac769e4194002ea1465f0666e0990d4a231e370808cc2f9057665b7d",
"samples": 92160,
"seconds": 3.84,
"decoded_frames": 45,
"first_invalid_frame": null
},
{
"prompt_id": 7,
"prompt": "Tomorrow's meeting begins at nine, so I will arrive a little early.",
"seed": 7608,
"path": "audio/orpheus/base/prompt_07.wav",
"sha256": "143f28842c83448b8fcea73017526274be7ac061f0aa3b727ea623f9839bcaad",
"samples": 98304,
"seconds": 4.096,
"decoded_frames": 48,
"first_invalid_frame": null
},
{
"prompt_id": 0,
"prompt": "The morning train crossed the bridge just before sunrise.",
"seed": 7601,
"path": "audio/orpheus/adapted/prompt_00.wav",
"sha256": "24470415a507117b9f2b6a4b3f13350a453febca782f542e592ea6d41678f9da",
"samples": 110592,
"seconds": 4.608,
"decoded_frames": 54,
"first_invalid_frame": null
},
{
"prompt_id": 1,
"prompt": "Please leave the blue notebook beside the kitchen window.",
"seed": 7602,
"path": "audio/orpheus/adapted/prompt_01.wav",
"sha256": "f31e8572ef2509e4537328d1b080e92f9a4d833a859068ce991220df790cf884",
"samples": 122880,
"seconds": 5.12,
"decoded_frames": 60,
"first_invalid_frame": null
},
{
"prompt_id": 2,
"prompt": "A patient astronomer mapped every bright star in the winter sky.",
"seed": 7603,
"path": "audio/orpheus/adapted/prompt_02.wav",
"sha256": "4ec5e55e4aa7b085f61961340a701dd2e784787e98ef4d03111a2fddc1aab457",
"samples": 98304,
"seconds": 4.096,
"decoded_frames": 48,
"first_invalid_frame": null
},
{
"prompt_id": 3,
"prompt": "We walked home slowly while the last shops turned off their lights.",
"seed": 7604,
"path": "audio/orpheus/adapted/prompt_03.wav",
"sha256": "07e7c5b0b9d37cc2e87cc70c31dee0adf55db8e76772e28661a33d8b14f1afe9",
"samples": 94208,
"seconds": 3.925333333333333,
"decoded_frames": 46,
"first_invalid_frame": null
},
{
"prompt_id": 4,
"prompt": "Could you read the final paragraph one more time for the group?",
"seed": 7605,
"path": "audio/orpheus/adapted/prompt_04.wav",
"sha256": "9033e23dcb4a31f4deb2108b04674b811c7c3c8a4b3fc638e8076675fec428ce",
"samples": 77824,
"seconds": 3.2426666666666666,
"decoded_frames": 38,
"first_invalid_frame": null
},
{
"prompt_id": 5,
"prompt": "The small garden stayed green even through the hottest week of July.",
"seed": 7606,
"path": "audio/orpheus/adapted/prompt_05.wav",
"sha256": "1de6449d583fb48f0db9d1ba19bbfefda60016f32b246d9562d34a6a9bd7e322",
"samples": 90112,
"seconds": 3.7546666666666666,
"decoded_frames": 44,
"first_invalid_frame": null
},
{
"prompt_id": 6,
"prompt": "I packed a warm coat, two apples, and a compass for the long hike.",
"seed": 7607,
"path": "audio/orpheus/adapted/prompt_06.wav",
"sha256": "0b54348eb7b0d46cbcbd1869afa0b96fde6997d79036b19e11b2b7854261e468",
"samples": 98304,
"seconds": 4.096,
"decoded_frames": 48,
"first_invalid_frame": null
},
{
"prompt_id": 7,
"prompt": "Tomorrow's meeting begins at nine, so I will arrive a little early.",
"seed": 7608,
"path": "audio/orpheus/adapted/prompt_07.wav",
"sha256": "e71ca35cf0c12b1fb40d30cb26a285d0207ce9f845239c5ecfdd246e605f7903",
"samples": 102400,
"seconds": 4.266666666666667,
"decoded_frames": 50,
"first_invalid_frame": null
}
]
}
chapter8/speech-sft-experiment/validation/exp8-6-20260804-v1/sesame_manifest.json
+369
View File
@@ -0,0 +1,369 @@
{
"experiment": "8-6",
"track": "sesame_paralinguistic_tags",
"status": "trained_and_generated",
"seed": 7602,
"base_model": "unsloth/csm-1b",
"base_model_revision": "39e8c756bec133ad7eb9ea1097c84a2bd891c949",
"dataset": "maxbsoft/mrdragonfox-elise",
"dataset_revision": "2cc657c3f94a83df18fcd968b7531ca1a19c7f88",
"source_dataset_note": "Public non-disabled mirror of the disabled MrDragonFox/Elise dataset named by the upstream notebook.",
"train_examples_selected": 168,
"train_examples_preprocessed": 168,
"train_category_counts": {
"laugh": 48,
"giggle": 32,
"sigh": 48,
"neutral": 40
},
"eval_examples_selected": 24,
"eval_examples_preprocessed": 24,
"eval_category_counts": {
"laugh": 8,
"giggle": 0,
"sigh": 8,
"neutral": 8
},
"train_failures": [],
"eval_failures": [],
"max_audio_seconds": 8.0,
"optimizer_steps": 60,
"effective_batch_size": 4,
"lora_rank": 16,
"pre_eval": {
"eval_loss": 128.2307586669922,
"eval_model_preparation_time": 0.0114,
"eval_runtime": 1.3794,
"eval_samples_per_second": 17.398,
"eval_steps_per_second": 17.398
},
"train_metrics": {
"train_runtime": 26.4676,
"train_samples_per_second": 9.068,
"train_steps_per_second": 2.267,
"total_flos": 460628125655040.0,
"train_loss": 124.45613021850586,
"epoch": 1.4285714285714286
},
"post_eval": {
"eval_loss": 124.34239959716797,
"eval_model_preparation_time": 0.0114,
"eval_runtime": 1.8573,
"eval_samples_per_second": 12.922,
"eval_steps_per_second": 12.922,
"epoch": 1.4285714285714286
},
"gpu": "NVIDIA RTX PRO 6000 Blackwell Workstation Edition",
"peak_gpu_memory_bytes": 10242490368,
"wall_seconds": 182.60698008537292,
"adapter_local_files": [
{
"path": "adapters/sesame/README.md",
"bytes": 5183,
"sha256": "65a8c53feecbfc1e7a8cd64bc6a79baedbffd1ef6b9cc17b1ea0a5529e4d74ec"
},
{
"path": "adapters/sesame/adapter_config.json",
"bytes": 1091,
"sha256": "44c3053b5037214ea8862003dfeea867554c9fbb83df5728ea80e9a0915e3ee2"
},
{
"path": "adapters/sesame/adapter_model.safetensors",
"bytes": 58125112,
"sha256": "23994b467b319f99bbc642f4dc2ba4d39718838cd7cbfe949311a42ebcb5480e"
},
{
"path": "adapters/sesame/chat_template.jinja",
"bytes": 2002,
"sha256": "b43753fbbc23be93160cccf825d2cf97dec39538cb274350216940e7acbc0fe7"
},
{
"path": "adapters/sesame/preprocessor_config.json",
"bytes": 271,
"sha256": "8648040cda0469976874c733acbec88c0ef094cd95628ab63b32950f01872af9"
},
{
"path": "adapters/sesame/special_tokens_map.json",
"bytes": 459,
"sha256": "32f404d626cf7b1b6eea36c241ae7cbd6ec29c777a8701ea48c0fe9ebb94c9b1"
},
{
"path": "adapters/sesame/tokenizer.json",
"bytes": 17210159,
"sha256": "92487a7224af2e53dc421a3f73056078acea66f0e86734f8d41babce66bad5cc"
},
{
"path": "adapters/sesame/tokenizer_config.json",
"bytes": 50577,
"sha256": "eb03e82f89eebfd9b9e2c1a2741dfc34c8982c3c14d132b0ec42fed737b832d3"
}
],
"adapter_huggingface_repo": "https://huggingface.co/bojieli/exp8-6-sesame-elise-tags-lora",
"adapter_huggingface_revision": "f2e042be0f38d6078976ef7e16cf49b91097f756",
"audio": [
{
"pair_id": "laugh_0",
"tag": "laugh",
"condition": "neutral",
"text": "I finally found the missing keys in my other pocket.",
"seed": 7602,
"path": "audio/sesame/base/laugh_0_neutral.wav",
"sha256": "beb4be0e62fa63ca5c81facb5f82f7a565e027f51e0df489d8f361664b49460f",
"samples": 65280,
"seconds": 2.72
},
{
"pair_id": "laugh_0",
"tag": "laugh",
"condition": "tagged",
"text": "I finally found the missing keys <laughs> in my other pocket.",
"seed": 7602,
"path": "audio/sesame/base/laugh_0_tagged.wav",
"sha256": "1a475df9255e256277a1a4fc66fee780c604977381059a04f09b33f46ebc00ce",
"samples": 144000,
"seconds": 6.0
},
{
"pair_id": "laugh_1",
"tag": "laugh",
"condition": "neutral",
"text": "That was the strangest joke I heard all week.",
"seed": 7603,
"path": "audio/sesame/base/laugh_1_neutral.wav",
"sha256": "85f7ed5500add23a6d5a4e51df5c0c01e00ecb2c740863b0155cb22d547bf701",
"samples": 49920,
"seconds": 2.08
},
{
"pair_id": "laugh_1",
"tag": "laugh",
"condition": "tagged",
"text": "That was the strangest joke <laughs> I heard all week.",
"seed": 7603,
"path": "audio/sesame/base/laugh_1_tagged.wav",
"sha256": "46f2c99ca4fd65fff0c724900e9baa1f7bfab25fbc6246c669c9678c463def96",
"samples": 92160,
"seconds": 3.84
},
{
"pair_id": "giggle_0",
"tag": "giggle",
"condition": "neutral",
"text": "You remembered the secret code after all.",
"seed": 7604,
"path": "audio/sesame/base/giggle_0_neutral.wav",
"sha256": "b3a6df7fe20a4e713f19f961772e908477962b335419ac5289b05fea2d9adebd",
"samples": 40320,
"seconds": 1.68
},
{
"pair_id": "giggle_0",
"tag": "giggle",
"condition": "tagged",
"text": "You remembered the secret code <giggles> after all.",
"seed": 7604,
"path": "audio/sesame/base/giggle_0_tagged.wav",
"sha256": "fe8a33bf80d471d1812c6d37586e5989eadcc10ae61c48ed02e4e516eafd2dc3",
"samples": 71040,
"seconds": 2.96
},
{
"pair_id": "giggle_1",
"tag": "giggle",
"condition": "neutral",
"text": "The tiny puppy tried to carry the enormous slipper.",
"seed": 7605,
"path": "audio/sesame/base/giggle_1_neutral.wav",
"sha256": "73b02cd34995eb75549fd5941a67c1b621b7d4f613d140f83693955d9529cd44",
"samples": 71040,
"seconds": 2.96
},
{
"pair_id": "giggle_1",
"tag": "giggle",
"condition": "tagged",
"text": "The tiny puppy <giggles> tried to carry the enormous slipper.",
"seed": 7605,
"path": "audio/sesame/base/giggle_1_tagged.wav",
"sha256": "c1f7efac3892a907c7e08ecef5c013fa83db23879a1a5d1d055f81ec7efd4348",
"samples": 142080,
"seconds": 5.92
},
{
"pair_id": "sigh_0",
"tag": "sigh",
"condition": "neutral",
"text": "The last bus left before we reached the corner.",
"seed": 7606,
"path": "audio/sesame/base/sigh_0_neutral.wav",
"sha256": "661c54d5ffba92518ef690cade4d3338a7d5c6f8e67882bf024d557e2b5eeeff",
"samples": 55680,
"seconds": 2.32
},
{
"pair_id": "sigh_0",
"tag": "sigh",
"condition": "tagged",
"text": "The last bus left <sighs> before we reached the corner.",
"seed": 7606,
"path": "audio/sesame/base/sigh_0_tagged.wav",
"sha256": "90cbd9c00b4f27ce71723bc022d3a5a0503ea6039ed7456a2b50785d86986f30",
"samples": 144000,
"seconds": 6.0
},
{
"pair_id": "sigh_1",
"tag": "sigh",
"condition": "neutral",
"text": "I suppose we need to finish the paperwork again.",
"seed": 7607,
"path": "audio/sesame/base/sigh_1_neutral.wav",
"sha256": "eb79e2600f5c4c0d571ae6e6ba5dc26c84c1547a6fb5225a9467833789c61b8a",
"samples": 53760,
"seconds": 2.24
},
{
"pair_id": "sigh_1",
"tag": "sigh",
"condition": "tagged",
"text": "I suppose <sighs> we need to finish the paperwork again.",
"seed": 7607,
"path": "audio/sesame/base/sigh_1_tagged.wav",
"sha256": "3e5b8de26910376f0906253ae1effff657b927efb6382a7ff4aaeea3080d0c58",
"samples": 80640,
"seconds": 3.36
},
{
"pair_id": "laugh_0",
"tag": "laugh",
"condition": "neutral",
"text": "I finally found the missing keys in my other pocket.",
"seed": 7602,
"path": "audio/sesame/adapted/laugh_0_neutral.wav",
"sha256": "464bbcf2b6292d8a259cbb594e3c14a0e5ba8ca531bcac391e6a6495f9a7420c",
"samples": 94080,
"seconds": 3.92
},
{
"pair_id": "laugh_0",
"tag": "laugh",
"condition": "tagged",
"text": "I finally found the missing keys <laughs> in my other pocket.",
"seed": 7602,
"path": "audio/sesame/adapted/laugh_0_tagged.wav",
"sha256": "00cf9a83ce1694de4ddab137b668306acce45c5459cee81fa3702a3a5cc94359",
"samples": 86400,
"seconds": 3.6
},
{
"pair_id": "laugh_1",
"tag": "laugh",
"condition": "neutral",
"text": "That was the strangest joke I heard all week.",
"seed": 7603,
"path": "audio/sesame/adapted/laugh_1_neutral.wav",
"sha256": "4cba912639254c066d906b318531574ed1c2d664c2ece8e8b2f185fa85298e88",
"samples": 59520,
"seconds": 2.48
},
{
"pair_id": "laugh_1",
"tag": "laugh",
"condition": "tagged",
"text": "That was the strangest joke <laughs> I heard all week.",
"seed": 7603,
"path": "audio/sesame/adapted/laugh_1_tagged.wav",
"sha256": "0b25fc0881603af5831526dd4fcfcfb7dfe1539b3185ddd4358b230d16b50ea8",
"samples": 90240,
"seconds": 3.76
},
{
"pair_id": "giggle_0",
"tag": "giggle",
"condition": "neutral",
"text": "You remembered the secret code after all.",
"seed": 7604,
"path": "audio/sesame/adapted/giggle_0_neutral.wav",
"sha256": "16cbff670bebf59188345a79fcc99952ee5f7ba925e214874c71399b8b684fb6",
"samples": 48000,
"seconds": 2.0
},
{
"pair_id": "giggle_0",
"tag": "giggle",
"condition": "tagged",
"text": "You remembered the secret code <giggles> after all.",
"seed": 7604,
"path": "audio/sesame/adapted/giggle_0_tagged.wav",
"sha256": "e0d7ef91ff0c52b71a8630aa598bfbc30971149f36915fd3252511cbeb3d9a2c",
"samples": 65280,
"seconds": 2.72
},
{
"pair_id": "giggle_1",
"tag": "giggle",
"condition": "neutral",
"text": "The tiny puppy tried to carry the enormous slipper.",
"seed": 7605,
"path": "audio/sesame/adapted/giggle_1_neutral.wav",
"sha256": "653c1ed9cf6e11412cc935dee7dff90f0d059a72137faec5a63c795459658e15",
"samples": 78720,
"seconds": 3.28
},
{
"pair_id": "giggle_1",
"tag": "giggle",
"condition": "tagged",
"text": "The tiny puppy <giggles> tried to carry the enormous slipper.",
"seed": 7605,
"path": "audio/sesame/adapted/giggle_1_tagged.wav",
"sha256": "9b72e5252d8b6f7a6f4c7ec3bbbf0f45e39bb2a9e6a16db8e140a9c6e9f32c2c",
"samples": 117120,
"seconds": 4.88
},
{
"pair_id": "sigh_0",
"tag": "sigh",
"condition": "neutral",
"text": "The last bus left before we reached the corner.",
"seed": 7606,
"path": "audio/sesame/adapted/sigh_0_neutral.wav",
"sha256": "1cbb54147d1375513821f170b229379c8285d264d7ba4b4c568fcdb75bf3c7f0",
"samples": 65280,
"seconds": 2.72
},
{
"pair_id": "sigh_0",
"tag": "sigh",
"condition": "tagged",
"text": "The last bus left <sighs> before we reached the corner.",
"seed": 7606,
"path": "audio/sesame/adapted/sigh_0_tagged.wav",
"sha256": "d432582bfebcd4961ca2d3dfcfd944df252ed4826172f5987325172bcb23fd4e",
"samples": 94080,
"seconds": 3.92
},
{
"pair_id": "sigh_1",
"tag": "sigh",
"condition": "neutral",
"text": "I suppose we need to finish the paperwork again.",
"seed": 7607,
"path": "audio/sesame/adapted/sigh_1_neutral.wav",
"sha256": "b00babd0052826c46bc90a4d1e5335069ad26f6670ad7e90669f71320f715a8a",
"samples": 55680,
"seconds": 2.32
},
{
"pair_id": "sigh_1",
"tag": "sigh",
"condition": "tagged",
"text": "I suppose <sighs> we need to finish the paperwork again.",
"seed": 7607,
"path": "audio/sesame/adapted/sigh_1_tagged.wav",
"sha256": "6da2951a4629a2a809e8595bd3940d5da3fe494e4e89cec5c728dcc0fd9d22d9",
"samples": 80640,
"seconds": 3.36
}
]
}