ai-agent-book 精选快照(<2MB 代码与文档,来自 github.com/bojieli/ai-agent-book)
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,55 @@
|
||||
# AI 写作风格 Skill:开放式提炼 + LLM-as-a-judge
|
||||
|
||||
本项目把用户对「AI 味」的 before/after 纠正持续提炼为写作规则。提炼代码不再包含
|
||||
`PATTERN_LIBRARY` 或 detector 白名单:模型可以从反馈中发现词表之外的语义、句法、
|
||||
语气和篇章问题。所有候选规则统一交给 LLM judge,并且只有通过独立人工金标校准后
|
||||
才能进入最终 Skill。
|
||||
|
||||
默认模型是 OpenAI `gpt-5.6-sol`,OpenAI 直连使用 Responses API 的 JSON 输出。
|
||||
|
||||
```bash
|
||||
# 从仓库根目录开始
|
||||
uv sync --locked --python 3.12 --extra ch8
|
||||
source .venv/bin/activate
|
||||
source ~/.zshrc # 载入本机 OPENAI_API_KEY
|
||||
|
||||
cd chapter9/ai-style-skill
|
||||
python -m pytest -q test_pipeline.py
|
||||
python run_ai_style_skill.py # 默认 --provider openai --model gpt-5.6-sol
|
||||
```
|
||||
|
||||
单元测试用假的批量 judge,因而无需 API key;完整验收必须调用真实模型。原始请求、响应、
|
||||
Token 用量、延迟和哈希写入 `validation/<run>/evidence.json`,凭据值不会落盘。
|
||||
|
||||
## 完整循环
|
||||
|
||||
1. **收集**(`data/feedback_pairs.json`):26 条用户纠正,除了原有案例,还加入长句信息堆叠、重复「值得注意的是」和连续被动语态等库外问题。
|
||||
2. **开放式提炼**(`extract_rules.py`):模型一次比较完整反馈语料,在同一次语义判断中归并重复现象、分开不同问题。这样不会因批次顺序把一个概念拆成多个 id,或把两个相似表面形式误并。代码验证 source id 和正反例确实来自输入,并把 detector 固定为 `{"type": "llm"}`。
|
||||
3. **合并**(`skill_manager.py`):模型输出必须使用唯一规则 id;管理器只做机械的 id 去重与来源合并,不再按预置 detector 指纹过滤。模型仍只能提出候选,不能自行激活规则。
|
||||
4. **校准**(`judge.py`、`data/golden_set.json`):每条规则用与其反馈来源关联的独立人工正反例校准;一致率低于 0.8 或没有金标覆盖时拒绝上线。
|
||||
5. **评估**(`evaluate.py`、`data/eval_texts.json`):人工标注使用反馈来源而不是写死模型生成的规则 id。这样规则名称可动态变化,同时库外保留样本仍能揭示漏检。
|
||||
6. **改写**(`rewrite_demo.py`):只把通过校准的 active 规则交给模型改写,不再提供预置换写模式。
|
||||
|
||||
judge 会在一次请求里评判一段文本与全部 active 规则,避免为每条规则分别调用 API;
|
||||
校准时也会批量评判一条规则的全部金标样本。解析失败或缺失 verdict 一律按校准不一致处理。
|
||||
|
||||
## 防止新的自洽闭环
|
||||
|
||||
- 提炼输入只有用户纠正和当前已提炼规则,没有八类模式清单。
|
||||
- 候选不会因为无法映射到已有 detector 而回退或丢弃。
|
||||
- 金标集和 boundary/retention 集是独立人工文本,不复用提炼用的 before/after。
|
||||
- 评估集明确保留三类原模式库之外的问题,并为相似但合理的写法提供 retention 反例。
|
||||
- 激活、阈值门槛、证据校验和 API 回执仍由模型外部代码控制。
|
||||
|
||||
第三方 OpenAI 兼容端点仍可通过 `--provider ark` 或 `--provider openrouter` 使用;本项目的
|
||||
OpenAI 默认路径和验收基线使用 `gpt-5.6-sol`。
|
||||
|
||||
## 真实验收结果(2026-08-18)
|
||||
|
||||
使用 `--provider openrouter --model openai/gpt-5.6-sol` 完成真实运行,证据见
|
||||
`validation/real_20260818T130450Z/evidence.json`,`validation/latest.json` 为同一份结果:
|
||||
|
||||
- 完整语料开放式提炼出 11 条唯一规则,11 条全部通过独立金标校准,一致率均为 1.0。
|
||||
- boundary 检出 10/11,超过 0.85 门槛;retention 误伤 0/11。
|
||||
- 长句信息堆叠、重复强调套话和连续被动语态三类库外保留样本全部命中对应新规则。
|
||||
- 35 次真实 API 调用均保留回执,共使用 86,002 tokens;证据文件不记录凭据值。
|
||||
@@ -0,0 +1,244 @@
|
||||
[
|
||||
{
|
||||
"id": "rule-excessive-em-dash-separators",
|
||||
"name": "避免用连续破折号代替常规断句",
|
||||
"definition": "当一句或一段中连续使用多个破折号来串联普通并列信息、步骤或分句,导致层次不清、阅读节奏拖长时命中。应改用句号、逗号、冒号或分句。若破折号只偶尔用于插入说明、语意转折或强调,且前后关系明确,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "关于季度复盘会议——时间定于本周五下午三点——地点在三号会议室——请提前准备数据报表——如有冲突请及时告知——谢谢配合。",
|
||||
"good_example": "季度复盘定在这周五下午三点,三号会议室。请提前准备好数据报表,时间冲突的话提前跟我说。",
|
||||
"rewrite_hint": "先判断各部分是并列信息还是独立句意;并列项用逗号,完整意思用句号,只在确有插入或强调作用时保留破折号。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-001",
|
||||
"fp-008",
|
||||
"fp-017"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-repeated-not-but-contrast",
|
||||
"name": "避免反复使用“不是……而是……”对仗",
|
||||
"definition": "当“不是……而是……”被连续用于定义产品、解释概念或制造升华,尤其连续出现两次以上并形成整齐对仗时命中。这类写法常以否定铺垫代替直接说明。若上下文确实需要纠正一种明确误解,并且只使用一次来表达真实对立,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "这不是一次简单的版本更新,而是一次体验的全面革新;不是功能的堆砌,而是对用户需求的深度回应。",
|
||||
"good_example": "这次版本更新改动很大:界面重做,核心流程缩短了两步,都来自上一版用户反馈里最高频的问题。",
|
||||
"rewrite_hint": "删去否定式铺垫,直接说明对象是什么、具体改了什么,并用事实或例子支撑判断。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿",
|
||||
"公众号文章"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-002",
|
||||
"fp-009",
|
||||
"fp-015"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-repeated-let-us-slogans",
|
||||
"name": "避免连用“让我们”式号召口号",
|
||||
"definition": "当邮件、README 或产品稿连续以“让我们”开头,使用“共同见证”“携手并进”等口号来代替具体通知、邀请或行动说明时命中。若只出现一次,且确实是在自然地发出明确、可执行的共同邀请,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "让我们共同见证这个激动人心的时刻!让我们一起开启数字化转型的全新篇章。让我们携手并进,共创辉煌明天。",
|
||||
"good_example": "新版本今天上线了,期待大家试用之后多提意见。",
|
||||
"rewrite_hint": "直接交代发生了什么,以及希望读者采取什么具体行动,如试用、反馈、提 issue 或提交 PR。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-003",
|
||||
"fp-010",
|
||||
"fp-018"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-mechanical-sequence-markers",
|
||||
"name": "避免机械堆叠顺序连接词",
|
||||
"definition": "当短段落逐句套用“首先、其次、再次、然后、最后、总而言之”等连接词,实际内容本可自然串联,因而呈现模板化或公文腔时命中。若步骤顺序严格、跳步会造成错误,或连接词确实用于澄清复杂层级,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "首先,克隆本仓库到本地。其次,安装所需依赖。再次,配置环境变量。然后,运行初始化脚本。最后,启动开发服务器即可。",
|
||||
"good_example": "克隆仓库后安装依赖,把 .env.example 复制为 .env 并填好密钥,再运行 init.sh 初始化,最后 make dev 启动服务。",
|
||||
"rewrite_hint": "删除不提供额外信息的序号套话,改用动作之间的真实关系连接;必要时只保留少量“再”“最后”,或改成清晰的步骤列表。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-004",
|
||||
"fp-012",
|
||||
"fp-019"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-remove-generic-era-openings",
|
||||
"name": "删除空泛的“在……时代”开场",
|
||||
"definition": "当文章以“在……的今天”“在这个……的时代”等宏大背景开头,但该背景没有提供具体时间、事件或因果信息,只用于烘托气氛时命中。若时代或日期本身是论证所需的关键背景,并有具体事实支撑,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "在这个快节奏的时代,高效的沟通显得尤为珍贵。在这个充满变化的时代,稳定的协作关系更值得珍惜。",
|
||||
"good_example": "最近项目节奏快,咱们沟通尽量简短高效;也希望协作关系保持稳定,有问题随时同步。",
|
||||
"rewrite_hint": "直接写当前发生的具体变化、影响和要求,用“最近”“这两年”等可核实时间范围替代泛化时代判断。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-005",
|
||||
"fp-010",
|
||||
"fp-016"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-control-emoji-density",
|
||||
"name": "控制 emoji 的使用密度与场合",
|
||||
"definition": "当正式产品稿中使用 emoji,或公众号段落中几乎每个信息点都附带表情,造成视觉干扰、削弱专业感时命中。正式稿件应去除 emoji;轻松社交场景中少量、与语义直接相关且不妨碍阅读的 emoji 不应一概判定为问题。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "🎉 重磅来袭!🔥 全新智能手表正式发布!⌚ 超长续航 14 天 🔋,心率血氧全天候监测 ❤️,50 米防水 🏊,现在下单立减 200 元 💰,错过再等一年 ⏰!",
|
||||
"good_example": "新款智能手表发布:续航 14 天,支持心率血氧监测和 50 米防水,首发价立减 200 元。",
|
||||
"rewrite_hint": "正式文本删除全部 emoji;其他场景先去掉装饰性表情,只在确有语气或分类作用时保留极少量。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"公众号文章"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-006",
|
||||
"fp-011"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 2
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-forced-parallelism",
|
||||
"name": "避免刻意堆叠同构排比",
|
||||
"definition": "当三句或更多分句反复使用完全相同的开头和句法框架,如“读书可以……”“它让……”“每一次……都是……”,且内容多为抽象赞美、缺少信息增量时命中。若排比用于必要的结构对照、篇幅适度且每项都有实质信息,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "读书可以拓宽视野,读书可以沉淀心灵,读书可以启迪智慧,读书可以点亮人生。",
|
||||
"good_example": "读书的好处很多:视野会变宽,心能静下来,看问题也会多几个角度。",
|
||||
"rewrite_hint": "保留一个总述,将各项改成不同句式或合并为并列成分;删去重复、空泛或意义相近的项。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"公众号文章"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-007",
|
||||
"fp-014",
|
||||
"fp-017"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-replace-empty-metaphors-with-facts",
|
||||
"name": "用具体事实替代空洞抒情比喻",
|
||||
"definition": "当“智者、屏障、明灯、灯塔”等比喻只负责抬高语气,却没有说明产品效果、处理进度或实际价值,甚至替代了读者需要的关键信息时命中。若比喻准确、简洁,并能帮助理解陌生概念或符合文学性场景,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "您的建议宛如一盏明灯,照亮了我们产品改进的前路;您的信任仿佛一座灯塔,指引着我们不断前行。",
|
||||
"good_example": "您提的建议很具体,我们已经记进需求池,下个版本会先改其中两条。谢谢您的信任。",
|
||||
"rewrite_hint": "删除装饰性喻体,补上可验证的信息,如实际体验、使用场景、已采取的动作和后续安排。",
|
||||
"scope": [
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-013",
|
||||
"fp-020"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-split-overloaded-sentences",
|
||||
"name": "拆分承载过多层次的长句",
|
||||
"definition": "当一句话同时塞入背景、原因、条件、多个动作、结果和要求,主要依靠“且、并、同时、为了、让”等连续连接,读者难以一次识别主干时命中。句子较长但主干清楚、修饰关系单一且不存在理解负担时不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "这次更新在保留原有工作台布局的基础上通过重新组织导航层级并合并重复入口同时优化首次加载和列表刷新策略让用户从打开应用到完成一次记录的整个过程都比以前更连贯也更省时间。",
|
||||
"good_example": "这次更新保留了原有工作台布局,同时重新组织导航层级、合并重复入口。首次加载和列表刷新也做了优化,从打开应用到完成记录会更连贯。",
|
||||
"rewrite_hint": "先提取背景、改动、要求和结果,再按层次拆成两到三句;每句只保留一个主要动作或判断。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-021",
|
||||
"fp-024"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 2
|
||||
},
|
||||
{
|
||||
"id": "rule-remove-repeated-attention-prefaces",
|
||||
"name": "删除反复出现的提醒性套话",
|
||||
"definition": "当连续句子都用“值得注意的是”“需要特别注意的是”“更值得注意的是”等元话语开头,只是重复强调而没有建立新的信息层级时命中。若只在关键风险首次出现时使用一次,且确实需要提示读者改变注意焦点,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "值得注意的是,本周五是最终截止时间。需要特别注意的是,逾期提交将影响下游联调。还值得注意的是,附件命名必须统一。",
|
||||
"good_example": "最终截止时间是本周五,逾期会影响下游联调。附件请按统一格式命名。",
|
||||
"rewrite_hint": "删掉“值得注意的是”等前缀,直接写结论、截止时间、后果或操作要求;需要突出时依靠信息顺序而非重复提醒。",
|
||||
"scope": [
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-022",
|
||||
"fp-025"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-passive-voice-chains",
|
||||
"name": "避免连续使用“被”字被动句",
|
||||
"definition": "当多个相邻分句都采用“对象被执行者处理”的结构,且执行者明确、适合直接充当主语时命中。若执行者未知、不重要,或文本确实需要突出受影响对象,则单个被动句不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "配置文件会被启动器读取,依赖会被安装脚本自动下载,数据库会被迁移工具初始化,服务会被进程管理器拉起。",
|
||||
"good_example": "启动器读取配置文件,安装脚本自动下载依赖,迁移工具初始化数据库,进程管理器随后拉起服务。",
|
||||
"rewrite_hint": "把工具、模块或责任方移到主语位置,改写为“谁做什么”;只有确需突出承受结果的对象时才保留被动表达。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-023",
|
||||
"fp-026"
|
||||
],
|
||||
"status": "candidate",
|
||||
"last_confirmed_batch": 3
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,138 @@
|
||||
{
|
||||
"boundary": [
|
||||
{
|
||||
"id": "b1",
|
||||
"scene": "产品发布稿",
|
||||
"expected_sources": ["fp-001", "fp-008"],
|
||||
"text": "全新降噪耳机——沉浸式体验——一触即达的智能降噪——40 小时超长续航——让音乐回归纯粹——即刻预约——享首发礼遇!"
|
||||
},
|
||||
{
|
||||
"id": "b2",
|
||||
"scene": "公众号文章",
|
||||
"expected_sources": ["fp-002", "fp-009", "fp-015"],
|
||||
"text": "坚持不是咬牙硬撑,而是找到自己的节奏;努力不是做给别人看,而是对得起自己的选择。"
|
||||
},
|
||||
{
|
||||
"id": "b3",
|
||||
"scene": "公众号文章",
|
||||
"expected_sources": ["fp-007", "fp-014"],
|
||||
"text": "我们优化了搜索,我们重构了推荐,我们升级了播放,我们改进了评论。"
|
||||
},
|
||||
{
|
||||
"id": "b4",
|
||||
"scene": "邮件",
|
||||
"expected_sources": ["fp-003", "fp-018"],
|
||||
"text": "让我们携手开启新财年的征程。让我们以饱满的热情迎接每一个挑战。"
|
||||
},
|
||||
{
|
||||
"id": "b5",
|
||||
"scene": "README 段落",
|
||||
"expected_sources": ["fp-004", "fp-012", "fp-019"],
|
||||
"text": "首先安装依赖,其次配置密钥,再次运行迁移脚本,最后启动服务。"
|
||||
},
|
||||
{
|
||||
"id": "b6",
|
||||
"scene": "产品发布稿",
|
||||
"expected_sources": ["fp-006", "fp-011", "fp-007", "fp-014"],
|
||||
"text": "新品上市 🎉🎉!超美配色 🌈,超强性能 🚀,超值价格 💰,快来抢购吧 🛒✨!"
|
||||
},
|
||||
{
|
||||
"id": "b7",
|
||||
"scene": "公众号文章",
|
||||
"expected_sources": ["fp-005", "fp-016"],
|
||||
"text": "在自媒体蓬勃发展的今天,内容创作者站在了流量风口。在这个注意力稀缺的时代,好内容愈发珍贵。"
|
||||
},
|
||||
{
|
||||
"id": "b8",
|
||||
"scene": "邮件",
|
||||
"expected_sources": ["fp-001", "fp-008", "fp-002", "fp-009", "fp-015"],
|
||||
"text": "这次改版——不是小修小补——而是全面升级——不是功能的叠加——而是体验的飞跃。"
|
||||
},
|
||||
{
|
||||
"id": "b9",
|
||||
"scene": "README 段落",
|
||||
"expected_sources": ["fp-021", "fp-024"],
|
||||
"text": "为了确保首次部署时各项依赖能够按照正确顺序完成安装并让初始化脚本拿到全部必要参数同时避免网络波动导致中途失败建议在运行命令前逐项检查配置文件和代理设置然后再启动自动安装流程。"
|
||||
},
|
||||
{
|
||||
"id": "b10",
|
||||
"scene": "产品发布稿",
|
||||
"expected_sources": ["fp-022", "fp-025"],
|
||||
"text": "值得注意的是,新版本支持离线模式。更值得注意的是,缓存空间可以自行设置。还值得注意的是,旧数据不会被删除。"
|
||||
},
|
||||
{
|
||||
"id": "b11",
|
||||
"scene": "邮件",
|
||||
"expected_sources": ["fp-023", "fp-026"],
|
||||
"text": "需求已被产品组确认,接口将被后端团队调整,测试用例会被质量组补齐,上线时间则会被项目经理重新安排。"
|
||||
}
|
||||
],
|
||||
"retention": [
|
||||
{
|
||||
"id": "r1",
|
||||
"scene": "技术文档",
|
||||
"note": "一处必要的补充说明,成对使用两个破折号是合法用法",
|
||||
"text": "内存分配器采用伙伴系统——把空闲块按 2 的幂次分级管理——这样既能快速合并,又能减少外部碎片。"
|
||||
},
|
||||
{
|
||||
"id": "r2",
|
||||
"scene": "邮件",
|
||||
"note": "真正构成对比的单次「不是……而是……」",
|
||||
"text": "这个方案不是技术上不可行,而是排期成本太高,建议放到下个季度再评估。"
|
||||
},
|
||||
{
|
||||
"id": "r3",
|
||||
"scene": "公众号文章",
|
||||
"note": "人类作者克制的两句对仗,不构成三连排比",
|
||||
"text": "写得慢的稿子,读者未必看得出来;写得急的稿子,读者一眼就能看出来。"
|
||||
},
|
||||
{
|
||||
"id": "r4",
|
||||
"scene": "技术文档",
|
||||
"note": "两步教程中「首先/其次」各出现一次",
|
||||
"text": "配置分两步:首先把 API 密钥写进配置文件,其次确认网络能访问服务端点。其他选项保持默认即可。"
|
||||
},
|
||||
{
|
||||
"id": "r5",
|
||||
"scene": "公众号文章",
|
||||
"note": "生活化随笔里零星两个 emoji",
|
||||
"text": "今天试了家附近的社区食堂,十五块钱两荤两素,味道超出预期 😋。老板娘说开了八年,附近的老人几乎天天来。这样的小店能多开几家就好了 👍。"
|
||||
},
|
||||
{
|
||||
"id": "r6",
|
||||
"scene": "公众号文章",
|
||||
"note": "结尾单次「让我们」是真诚的读者互动",
|
||||
"text": "如果你也有类似的踩坑经历,让我们在评论区交流一下,互相省点时间。"
|
||||
},
|
||||
{
|
||||
"id": "r7",
|
||||
"scene": "公众号文章",
|
||||
"note": "历史叙述中的举例补充,成对破折号合法;开头不是「在……的时代」套话",
|
||||
"text": "上世纪八十年代,个体户刚刚出现在街头。他们中的不少人——比如那位卖服装起家的陈老板——后来成了第一批民营企业家。"
|
||||
},
|
||||
{
|
||||
"id": "r8",
|
||||
"scene": "邮件",
|
||||
"note": "单次对比加单处破折号补充原因",
|
||||
"text": "他不是不想来,而是台风把航班全取消了——人在机场等了六个小时。"
|
||||
},
|
||||
{
|
||||
"id": "r9",
|
||||
"scene": "技术文档",
|
||||
"note": "句子较长,但条件、动作和结果关系清楚,没有堆叠互不相干的信息",
|
||||
"text": "当客户端连续三次请求失败时,SDK 会进入退避状态,并在下一次重试前等待最多三十秒。"
|
||||
},
|
||||
{
|
||||
"id": "r10",
|
||||
"scene": "安全公告",
|
||||
"note": "只用一次强调语提示真正重要的兼容性风险",
|
||||
"text": "值得注意的是,本次升级会改变令牌格式,依赖旧格式的客户端必须同步更新。"
|
||||
},
|
||||
{
|
||||
"id": "r11",
|
||||
"scene": "事故报告",
|
||||
"note": "被动语态用于未知执行者的单一客观事实,没有连续堆砌",
|
||||
"text": "凌晨两点,主数据库被意外重启,具体操作来源仍在调查。"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,210 @@
|
||||
[
|
||||
{
|
||||
"id": "fp-001",
|
||||
"batch": 1,
|
||||
"scene": "产品发布稿",
|
||||
"before": "我们隆重推出全新智能笔记应用——一款真正懂你的效率神器——它不仅能记录灵感——还能自动整理、智能归类、跨端同步——让你的每一条想法都不再流失——现在下载,即刻开启高效人生——限时免费,错过再等一年!",
|
||||
"after": "新版智能笔记应用上线了。它能记录灵感,自动整理和归类,支持多设备同步。现在下载可以免费试用一个月。",
|
||||
"correction": "破折号太多了,一句话里五六个,读起来喘不过气,全改掉"
|
||||
},
|
||||
{
|
||||
"id": "fp-002",
|
||||
"batch": 1,
|
||||
"scene": "公众号文章",
|
||||
"before": "真正的成长,不是年龄的增加,而是认知的升级;不是知识的堆砌,而是思维的重构;不是经验的重复,而是边界的突破。",
|
||||
"after": "真正的成长,往往体现在认知的变化上:看问题的角度变了,做事的方法跟着变,最后连自己能做什么的边界都被重新画了一遍。",
|
||||
"correction": "不要「不是……而是……」的句式,连用三次太 AI 了"
|
||||
},
|
||||
{
|
||||
"id": "fp-003",
|
||||
"batch": 1,
|
||||
"scene": "邮件",
|
||||
"before": "让我们共同见证这个激动人心的时刻!让我们一起开启数字化转型的全新篇章。让我们携手并进,共创辉煌明天。",
|
||||
"after": "新版本今天上线了,期待大家试用之后多提意见。",
|
||||
"correction": "邮件里别连用「让我们」开头,像机器人演讲"
|
||||
},
|
||||
{
|
||||
"id": "fp-004",
|
||||
"batch": 1,
|
||||
"scene": "README 段落",
|
||||
"before": "首先,克隆本仓库到本地。其次,安装所需依赖。再次,配置环境变量。然后,运行初始化脚本。最后,启动开发服务器即可。",
|
||||
"after": "克隆仓库后安装依赖,把 .env.example 复制为 .env 并填好密钥,再运行 init.sh 初始化,最后 make dev 启动服务。",
|
||||
"correction": "首先其次再次然后最后,模板腔太重,连成句子写"
|
||||
},
|
||||
{
|
||||
"id": "fp-005",
|
||||
"batch": 1,
|
||||
"scene": "公众号文章",
|
||||
"before": "在人工智能飞速发展的今天,每个人都面临前所未有的机遇与挑战。在这个信息爆炸的时代,如何保持独立思考显得尤为重要。",
|
||||
"after": "这两年 AI 工具更新得飞快,很多人一边兴奋一边焦虑。信息越多,越需要自己能做判断。",
|
||||
"correction": "「在……的今天」「在……的时代」这种开头直接删掉"
|
||||
},
|
||||
{
|
||||
"id": "fp-006",
|
||||
"batch": 1,
|
||||
"scene": "产品发布稿",
|
||||
"before": "🎉 重磅来袭!🔥 全新智能手表正式发布!⌚ 超长续航 14 天 🔋,心率血氧全天候监测 ❤️,50 米防水 🏊,现在下单立减 200 元 💰,错过再等一年 ⏰!",
|
||||
"after": "新款智能手表发布:续航 14 天,支持心率血氧监测和 50 米防水,首发价立减 200 元。",
|
||||
"correction": "emoji 泛滥,正式稿件里一个都别放"
|
||||
},
|
||||
{
|
||||
"id": "fp-007",
|
||||
"batch": 2,
|
||||
"scene": "公众号文章",
|
||||
"before": "读书可以拓宽视野,读书可以沉淀心灵,读书可以启迪智慧,读书可以点亮人生。",
|
||||
"after": "读书的好处很多:视野会变宽,心能静下来,看问题也会多几个角度。",
|
||||
"correction": "四连排比太刻意了,写散一点"
|
||||
},
|
||||
{
|
||||
"id": "fp-008",
|
||||
"batch": 2,
|
||||
"scene": "邮件",
|
||||
"before": "关于季度复盘会议——时间定于本周五下午三点——地点在三号会议室——请提前准备数据报表——如有冲突请及时告知——谢谢配合。",
|
||||
"after": "季度复盘定在这周五下午三点,三号会议室。请提前准备好数据报表,时间冲突的话提前跟我说。",
|
||||
"correction": "把破折号当逗号用是吧?全部改成正常标点"
|
||||
},
|
||||
{
|
||||
"id": "fp-009",
|
||||
"batch": 2,
|
||||
"scene": "README 段落",
|
||||
"before": "本项目不是简单的爬虫框架,而是一套完整的数据采集解决方案;不是一次性的脚本集合,而是可长期维护的工程化平台。",
|
||||
"after": "本项目提供一套可扩展的数据采集方案,支持插件化开发,适合作为长期维护的工程使用。",
|
||||
"correction": "不要「不是……而是……」的对仗,直接说它是什么就行"
|
||||
},
|
||||
{
|
||||
"id": "fp-010",
|
||||
"batch": 2,
|
||||
"scene": "产品发布稿",
|
||||
"before": "在这个万物互联的时代,让我们共同拥抱智能家居的美好未来!让我们一起,让科技温暖每一个家庭。",
|
||||
"after": "智能家居新品系列今天发布,覆盖灯光、安防和环境控制,希望能让家更省心。",
|
||||
"correction": "开头「在这个……的时代」和连喊「让我们」都去掉"
|
||||
},
|
||||
{
|
||||
"id": "fp-011",
|
||||
"batch": 2,
|
||||
"scene": "公众号文章",
|
||||
"before": "早餐这样吃才健康 🍞🥛!营养师推荐的三餐搭配来了 🍚🥗,坚持一个月,气色肉眼可见变好 ✨✨,姐妹们快收藏 💖!",
|
||||
"after": "早餐怎么吃更健康?整理了一份营养师的三餐搭配建议,坚持一段时间会有改善,需要的可以收藏。",
|
||||
"correction": "emoji 收一收,一整段全是表情没法看"
|
||||
},
|
||||
{
|
||||
"id": "fp-012",
|
||||
"batch": 2,
|
||||
"scene": "邮件",
|
||||
"before": "首先感谢各位本季度的付出。其次,需要指出交付节奏仍有优化空间。再次,请各组提交改进方案。最后,期待下季度再创佳绩。",
|
||||
"after": "感谢大家这个季度的付出。交付节奏上还有改进空间,请各组本周内提交改进方案,我们下个季度接着努力。",
|
||||
"correction": "「首先/其次/再次/最后」公文八股味,内部邮件别这么写"
|
||||
},
|
||||
{
|
||||
"id": "fp-013",
|
||||
"batch": 3,
|
||||
"scene": "公众号文章",
|
||||
"before": "这款耳机仿佛一位贴心的智者,在你喧嚣的世界里低语;它的降噪功能宛如一道温柔的屏障,将浮躁隔绝在外。",
|
||||
"after": "这款耳机的降噪效果确实好,地铁里戴上能安静不少,报站的人声也能压得住。",
|
||||
"correction": "「仿佛一位智者」这种空洞比喻删掉,说人话"
|
||||
},
|
||||
{
|
||||
"id": "fp-014",
|
||||
"batch": 3,
|
||||
"scene": "README 段落",
|
||||
"before": "它让配置更简单,它让部署更快速,它让监控更全面,它让运维更安心。",
|
||||
"after": "配置、部署、监控和告警都内置了默认方案,运维成本明显更低。",
|
||||
"correction": "「它让……它让……」排比改掉,平铺直叙"
|
||||
},
|
||||
{
|
||||
"id": "fp-015",
|
||||
"batch": 3,
|
||||
"scene": "产品发布稿",
|
||||
"before": "这不是一次简单的版本更新,而是一次体验的全面革新;不是功能的堆砌,而是对用户需求的深度回应。",
|
||||
"after": "这次版本更新改动很大:界面重做,核心流程缩短了两步,都来自上一版用户反馈里最高频的问题。",
|
||||
"correction": "「不是……而是……」连用两句,AI 味很重"
|
||||
},
|
||||
{
|
||||
"id": "fp-016",
|
||||
"batch": 3,
|
||||
"scene": "邮件",
|
||||
"before": "在这个快节奏的时代,高效的沟通显得尤为珍贵。在这个充满变化的时代,稳定的协作关系更值得珍惜。",
|
||||
"after": "最近项目节奏快,咱们沟通尽量简短高效;也希望协作关系保持稳定,有问题随时同步。",
|
||||
"correction": "别用「在这个……的时代」起头,直接说事"
|
||||
},
|
||||
{
|
||||
"id": "fp-017",
|
||||
"batch": 3,
|
||||
"scene": "公众号文章",
|
||||
"before": "成长是一场修行——它教会我们坚持——教会我们取舍——教会我们感恩。每一次跌倒都是财富,每一次爬起都是勋章,每一次坚持都是胜利。",
|
||||
"after": "成长是场慢功夫。能坚持下来的人,多半都慢慢学会了取舍,也开始记得感谢路上帮过自己的人。摔过跤的地方,印象总是最深。",
|
||||
"correction": "破折号加「每一次……都是……」排比,都太浓了"
|
||||
},
|
||||
{
|
||||
"id": "fp-018",
|
||||
"batch": 3,
|
||||
"scene": "README 段落",
|
||||
"before": "让我们一起构建更好的开源生态!让我们从 star 本项目开始,共同参与这场技术盛宴。",
|
||||
"after": "如果这个项目对你有帮助,欢迎 star;也欢迎提 issue 和 PR。",
|
||||
"correction": "README 里别喊「让我们」口号"
|
||||
},
|
||||
{
|
||||
"id": "fp-019",
|
||||
"batch": 3,
|
||||
"scene": "产品发布稿",
|
||||
"before": "首先,全新设计语言带来视觉革新。其次,性能提升高达 40%。再次,续航焦虑彻底解决。总而言之,这是迄今为止最值得升级的一代。",
|
||||
"after": "这一代换了全新的外观设计,性能比上代提升 40%,电池也够用一整天了。如果是老用户,这次值得升级。",
|
||||
"correction": "「首先其次再次总而言之」八股味,删掉套话"
|
||||
},
|
||||
{
|
||||
"id": "fp-020",
|
||||
"batch": 3,
|
||||
"scene": "邮件",
|
||||
"before": "您的建议宛如一盏明灯,照亮了我们产品改进的前路;您的信任仿佛一座灯塔,指引着我们不断前行。",
|
||||
"after": "您提的建议很具体,我们已经记进需求池,下个版本会先改其中两条。谢谢您的信任。",
|
||||
"correction": "「明灯」「灯塔」这种假抒情删掉,回邮件说正事"
|
||||
},
|
||||
{
|
||||
"id": "fp-021",
|
||||
"batch": 1,
|
||||
"scene": "产品发布稿",
|
||||
"before": "这次更新在保留原有工作台布局的基础上通过重新组织导航层级并合并重复入口同时优化首次加载和列表刷新策略让用户从打开应用到完成一次记录的整个过程都比以前更连贯也更省时间。",
|
||||
"after": "这次更新保留了原有工作台布局,同时重新组织导航层级、合并重复入口。首次加载和列表刷新也做了优化,从打开应用到完成记录会更连贯。",
|
||||
"correction": "一句话塞了太多层意思,长得读不到头,拆成两三句"
|
||||
},
|
||||
{
|
||||
"id": "fp-022",
|
||||
"batch": 1,
|
||||
"scene": "公众号文章",
|
||||
"before": "值得注意的是,团队规模并不是越大越好。值得注意的是,沟通成本会随人数增加。更值得注意的是,真正影响交付的是协作方式。",
|
||||
"after": "团队并非越大越好,人数增加也会抬高沟通成本。真正影响交付的,还是协作方式。",
|
||||
"correction": "别每句都拿「值得注意的是」开头,直接说结论"
|
||||
},
|
||||
{
|
||||
"id": "fp-023",
|
||||
"batch": 2,
|
||||
"scene": "README 段落",
|
||||
"before": "配置文件会被启动器读取,依赖会被安装脚本自动下载,数据库会被迁移工具初始化,服务会被进程管理器拉起。",
|
||||
"after": "启动器读取配置文件,安装脚本自动下载依赖,迁移工具初始化数据库,进程管理器随后拉起服务。",
|
||||
"correction": "四句全是「被」字被动语态,改成谁做什么"
|
||||
},
|
||||
{
|
||||
"id": "fp-024",
|
||||
"batch": 2,
|
||||
"scene": "邮件",
|
||||
"before": "考虑到本季度多个项目同时进入交付阶段且测试环境近期频繁被占用为了避免各组在最后一周集中提交导致验证排队请大家在完成核心功能后尽早预约测试窗口并把预计时间同步到项目群。",
|
||||
"after": "本季度有多个项目同时交付,测试环境最近也经常被占用。请各组完成核心功能后尽早预约测试窗口,并把预计时间同步到项目群,避免最后一周排队。",
|
||||
"correction": "整段只有一个句号,条件原因要求全挤一起了,拆开写"
|
||||
},
|
||||
{
|
||||
"id": "fp-025",
|
||||
"batch": 3,
|
||||
"scene": "邮件",
|
||||
"before": "值得注意的是,本周五是最终截止时间。需要特别注意的是,逾期提交将影响下游联调。还值得注意的是,附件命名必须统一。",
|
||||
"after": "最终截止时间是本周五,逾期会影响下游联调。附件请按统一格式命名。",
|
||||
"correction": "「值得注意的是」反复强调,像自动生成的提醒,删掉套话"
|
||||
},
|
||||
{
|
||||
"id": "fp-026",
|
||||
"batch": 3,
|
||||
"scene": "产品发布稿",
|
||||
"before": "页面被全新的设计系统重构,搜索结果被智能算法重新排序,常用操作被快捷入口集中展示,用户的等待时间被大幅缩短。",
|
||||
"after": "全新的设计系统重构了页面,智能算法重新排序搜索结果,快捷入口集中展示常用操作,整体等待时间也明显缩短。",
|
||||
"correction": "连续被动句很生硬,改成明确主语的主动表达"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,222 @@
|
||||
[
|
||||
{
|
||||
"id": "g-dash-1",
|
||||
"labels": [{"source_ids": ["fp-001", "fp-008"], "expected": true}],
|
||||
"text": "年度新品——更轻——更快——更智能——现在预约——立即享受专属优惠。"
|
||||
},
|
||||
{
|
||||
"id": "g-dash-2",
|
||||
"labels": [{"source_ids": ["fp-001", "fp-008"], "expected": true}],
|
||||
"text": "会议改到周四——还是下午三点——地点不变——材料提前发——请大家准时参加。"
|
||||
},
|
||||
{
|
||||
"id": "g-dash-3",
|
||||
"labels": [{"source_ids": ["fp-001", "fp-008"], "expected": false}],
|
||||
"text": "这项机制——也就是写时复制——只在数据真正变化时创建副本。"
|
||||
},
|
||||
{
|
||||
"id": "g-dash-4",
|
||||
"labels": [{"source_ids": ["fp-001", "fp-008"], "expected": false}],
|
||||
"text": "她只补充了一点——预算必须在月底前确认。"
|
||||
},
|
||||
{
|
||||
"id": "g-not-but-1",
|
||||
"labels": [{"source_ids": ["fp-002", "fp-009", "fp-015"], "expected": true}],
|
||||
"text": "写作不是表达,而是连接;阅读不是消费,而是成长;分享不是炫耀,而是传递。"
|
||||
},
|
||||
{
|
||||
"id": "g-not-but-2",
|
||||
"labels": [{"source_ids": ["fp-002", "fp-009", "fp-015"], "expected": true}],
|
||||
"text": "这不是修补,而是重塑。不是妥协,而是选择。不是终点,而是起点。"
|
||||
},
|
||||
{
|
||||
"id": "g-not-but-3",
|
||||
"labels": [{"source_ids": ["fp-002", "fp-009", "fp-015"], "expected": false}],
|
||||
"text": "问题不是磁盘空间不足,而是日志轮转没有启动。"
|
||||
},
|
||||
{
|
||||
"id": "g-not-but-4",
|
||||
"labels": [{"source_ids": ["fp-002", "fp-009", "fp-015"], "expected": false}],
|
||||
"text": "我们比较了两个方案,最终选择延迟更低的那个。"
|
||||
},
|
||||
{
|
||||
"id": "g-lets-1",
|
||||
"labels": [{"source_ids": ["fp-003", "fp-018"], "expected": true}],
|
||||
"text": "让我们勇敢出发!让我们携手向前!让我们共同书写新的篇章!"
|
||||
},
|
||||
{
|
||||
"id": "g-lets-2",
|
||||
"labels": [{"source_ids": ["fp-003", "fp-018"], "expected": true}],
|
||||
"text": "让我们见证改变,让我们拥抱未来,让我们一起创造无限可能。"
|
||||
},
|
||||
{
|
||||
"id": "g-lets-3",
|
||||
"labels": [{"source_ids": ["fp-003", "fp-018"], "expected": false}],
|
||||
"text": "如果你复现了这个问题,让我们在 issue 里核对一下环境信息。"
|
||||
},
|
||||
{
|
||||
"id": "g-lets-4",
|
||||
"labels": [{"source_ids": ["fp-003", "fp-018"], "expected": false}],
|
||||
"text": "欢迎提交问题和补丁,维护者通常会在两个工作日内回复。"
|
||||
},
|
||||
{
|
||||
"id": "g-sequence-1",
|
||||
"labels": [{"source_ids": ["fp-004", "fp-012", "fp-019"], "expected": true}],
|
||||
"text": "首先分析需求,其次制定方案,再次协调资源,最后推动落地,总而言之我们会全力以赴。"
|
||||
},
|
||||
{
|
||||
"id": "g-sequence-2",
|
||||
"labels": [{"source_ids": ["fp-004", "fp-012", "fp-019"], "expected": true}],
|
||||
"text": "首先,外观焕然一新。其次,性能全面升级。再次,体验大幅优化。总而言之,这是一款划时代产品。"
|
||||
},
|
||||
{
|
||||
"id": "g-sequence-3",
|
||||
"labels": [{"source_ids": ["fp-004", "fp-012", "fp-019"], "expected": false}],
|
||||
"text": "安装分两步:首先下载压缩包,其次执行安装命令。"
|
||||
},
|
||||
{
|
||||
"id": "g-sequence-4",
|
||||
"labels": [{"source_ids": ["fp-004", "fp-012", "fp-019"], "expected": false}],
|
||||
"text": "先备份数据库,再执行迁移;确认结果后启动新版本。"
|
||||
},
|
||||
{
|
||||
"id": "g-era-1",
|
||||
"labels": [{"source_ids": ["fp-005", "fp-016"], "expected": true}],
|
||||
"text": "在这个信息爆炸的时代,专注力显得尤为珍贵。"
|
||||
},
|
||||
{
|
||||
"id": "g-era-2",
|
||||
"labels": [{"source_ids": ["fp-005", "fp-016"], "expected": true}],
|
||||
"text": "在科技飞速发展的今天,创新已经成为企业前进的不竭动力。"
|
||||
},
|
||||
{
|
||||
"id": "g-era-3",
|
||||
"labels": [{"source_ids": ["fp-005", "fp-016"], "expected": false}],
|
||||
"text": "人工智能工具越来越多,这份调查比较了其中五款的实际响应时间。"
|
||||
},
|
||||
{
|
||||
"id": "g-era-4",
|
||||
"labels": [{"source_ids": ["fp-005", "fp-016"], "expected": false}],
|
||||
"text": "在上世纪九十年代的上海,寻呼机曾是常见的通信工具。"
|
||||
},
|
||||
{
|
||||
"id": "g-emoji-1",
|
||||
"labels": [{"source_ids": ["fp-006", "fp-011"], "expected": true}],
|
||||
"text": "新品来了 🎉🎉,颜值爆表 ✨,性能起飞 🚀,价格惊喜 💰,快来抢购 🛒!"
|
||||
},
|
||||
{
|
||||
"id": "g-emoji-2",
|
||||
"labels": [{"source_ids": ["fp-006", "fp-011"], "expected": false}],
|
||||
"text": "早起打卡 ☀️,早餐打卡 🥐,运动打卡 🏃,阅读打卡 📚,今天也要加油 💪!"
|
||||
},
|
||||
{
|
||||
"id": "g-emoji-3",
|
||||
"labels": [{"source_ids": ["fp-006", "fp-011"], "expected": false}],
|
||||
"text": "这家小店的面很好吃 😋,下次还想带朋友来。"
|
||||
},
|
||||
{
|
||||
"id": "g-emoji-4",
|
||||
"labels": [{"source_ids": ["fp-006", "fp-011"], "expected": false}],
|
||||
"text": "构建已经通过 ✅,可以合并了。"
|
||||
},
|
||||
{
|
||||
"id": "g-parallel-1",
|
||||
"labels": [{"source_ids": ["fp-007", "fp-014"], "expected": true}],
|
||||
"text": "它让工作更轻松,它让协作更顺畅,它让决策更高效,它让未来更清晰。"
|
||||
},
|
||||
{
|
||||
"id": "g-parallel-2",
|
||||
"labels": [{"source_ids": ["fp-007", "fp-014"], "expected": true}],
|
||||
"text": "每一次尝试都是成长,每一次失败都是积累,每一次坚持都是突破。"
|
||||
},
|
||||
{
|
||||
"id": "g-parallel-3",
|
||||
"labels": [{"source_ids": ["fp-007", "fp-014"], "expected": false}],
|
||||
"text": "我喜欢清晨的安静,也喜欢傍晚的热闹。"
|
||||
},
|
||||
{
|
||||
"id": "g-parallel-4",
|
||||
"labels": [{"source_ids": ["fp-007", "fp-014"], "expected": false}],
|
||||
"text": "搜索速度提升了,索引占用则略有增加,两项变化都在预期范围内。"
|
||||
},
|
||||
{
|
||||
"id": "g-metaphor-1",
|
||||
"labels": [{"source_ids": ["fp-013", "fp-020"], "expected": true}],
|
||||
"text": "这款音箱仿佛一位懂你的智者,总能在恰当的时刻送上恰当的音乐。"
|
||||
},
|
||||
{
|
||||
"id": "g-metaphor-2",
|
||||
"labels": [{"source_ids": ["fp-013", "fp-020"], "expected": true}],
|
||||
"text": "她的建议宛如一座灯塔,照亮了团队前行的道路。"
|
||||
},
|
||||
{
|
||||
"id": "g-metaphor-3",
|
||||
"labels": [{"source_ids": ["fp-013", "fp-020"], "expected": false}],
|
||||
"text": "这座灯塔建于 1903 年,至今仍为进港船只导航。"
|
||||
},
|
||||
{
|
||||
"id": "g-metaphor-4",
|
||||
"labels": [{"source_ids": ["fp-013", "fp-020"], "expected": false}],
|
||||
"text": "新算法把平均搜索时间从 800 毫秒降到了 120 毫秒。"
|
||||
},
|
||||
{
|
||||
"id": "g-long-sentence-1",
|
||||
"labels": [{"source_ids": ["fp-021", "fp-024"], "expected": true}],
|
||||
"text": "由于需求在评审后又发生变化并且接口文档直到昨晚才最终确认导致前端无法按原计划完成联调所以本周的提测时间需要顺延到下周二请相关同事同步调整后续安排。"
|
||||
},
|
||||
{
|
||||
"id": "g-long-sentence-2",
|
||||
"labels": [{"source_ids": ["fp-021", "fp-024"], "expected": true}],
|
||||
"text": "新版本通过整合搜索推荐收藏和历史记录入口并重新设计信息层级以及优化缓存更新逻辑让用户能够在更少操作里找到需要的内容同时降低首次打开页面时的等待时间。"
|
||||
},
|
||||
{
|
||||
"id": "g-long-sentence-3",
|
||||
"labels": [{"source_ids": ["fp-021", "fp-024"], "expected": false}],
|
||||
"text": "当请求连续失败三次时,客户端会暂停重试,并在等待三十秒后重新建立连接。"
|
||||
},
|
||||
{
|
||||
"id": "g-long-sentence-4",
|
||||
"labels": [{"source_ids": ["fp-021", "fp-024"], "expected": false}],
|
||||
"text": "测试环境正在升级。联调改到明天下午,地址不变。"
|
||||
},
|
||||
{
|
||||
"id": "g-worth-noting-1",
|
||||
"labels": [{"source_ids": ["fp-022", "fp-025"], "expected": true}],
|
||||
"text": "值得注意的是,价格已经调整。还值得注意的是,套餐内容也有变化。尤其值得注意的是,优惠只到月底。"
|
||||
},
|
||||
{
|
||||
"id": "g-worth-noting-2",
|
||||
"labels": [{"source_ids": ["fp-022", "fp-025"], "expected": true}],
|
||||
"text": "需要强调的是,数据不会迁移。需要特别强调的是,旧账号无法登录。再次强调,用户必须重新注册。"
|
||||
},
|
||||
{
|
||||
"id": "g-worth-noting-3",
|
||||
"labels": [{"source_ids": ["fp-022", "fp-025"], "expected": false}],
|
||||
"text": "值得注意的是,这个安全补丁会使旧版客户端无法连接,升级前必须通知所有用户。"
|
||||
},
|
||||
{
|
||||
"id": "g-worth-noting-4",
|
||||
"labels": [{"source_ids": ["fp-022", "fp-025"], "expected": false}],
|
||||
"text": "价格已经调整,套餐内容不变,优惠持续到月底。"
|
||||
},
|
||||
{
|
||||
"id": "g-passive-1",
|
||||
"labels": [{"source_ids": ["fp-023", "fp-026"], "expected": true}],
|
||||
"text": "方案被委员会审议,预算被财务部门核定,合同被法务团队复核,最终日期被项目组确认。"
|
||||
},
|
||||
{
|
||||
"id": "g-passive-2",
|
||||
"labels": [{"source_ids": ["fp-023", "fp-026"], "expected": true}],
|
||||
"text": "文件会被程序读取,字段会被转换器处理,结果会被缓存模块保存,通知会被任务系统发送。"
|
||||
},
|
||||
{
|
||||
"id": "g-passive-3",
|
||||
"labels": [{"source_ids": ["fp-023", "fp-026"], "expected": false}],
|
||||
"text": "凌晨两点,备用节点被意外重启,操作来源仍在调查。"
|
||||
},
|
||||
{
|
||||
"id": "g-passive-4",
|
||||
"labels": [{"source_ids": ["fp-023", "fp-026"], "expected": false}],
|
||||
"text": "调度器读取任务列表,工作进程执行任务,监控服务记录结果。"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,8 @@
|
||||
#!/usr/bin/env python3
|
||||
"""兼容入口:运行真实的开放式提炼与 LLM-as-a-judge 完整流程。"""
|
||||
|
||||
from run_ai_style_skill import main
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,7 @@
|
||||
# Default: OpenAI Responses API with GPT-5.6 Sol
|
||||
OPENAI_API_KEY=your_api_key_here
|
||||
|
||||
# Optional compatible providers
|
||||
ARK_API_KEY=your_api_key_here
|
||||
ARK_MODEL=doubao-seed-1-6-250615
|
||||
OPENROUTER_API_KEY=your_api_key_here
|
||||
@@ -0,0 +1,126 @@
|
||||
"""用外部人工标注的 boundary/retention 集评估 active 规则。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from judge import JudgeFn, llm_judge, score_text
|
||||
from llm_client import default_model
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
DATA_DIR = ROOT / "data"
|
||||
OUTPUT_DIR = ROOT / "output"
|
||||
|
||||
|
||||
def load_eval_texts(path: Path | None = None) -> Dict[str, List[Dict[str, Any]]]:
|
||||
return json.loads((path or DATA_DIR / "eval_texts.json").read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def _expected_rule_ids(item: Dict[str, Any], rules: List[Dict[str, Any]]) -> set[str]:
|
||||
"""把人工标注的反馈来源映射到本次动态提炼出的规则 id。"""
|
||||
expected_sources = set(item.get("expected_sources", []))
|
||||
return {
|
||||
rule["id"]
|
||||
for rule in rules
|
||||
if expected_sources & set(rule.get("source_ids", []))
|
||||
}
|
||||
|
||||
|
||||
def evaluate_rules(
|
||||
rules: List[Dict[str, Any]],
|
||||
eval_texts: Dict[str, List[Dict[str, Any]]],
|
||||
judge_fn: JudgeFn,
|
||||
) -> Dict[str, Any]:
|
||||
"""在保留集上逐段调用一次 LLM judge,不使用词表或正则探针。"""
|
||||
rule_ids = [rule["id"] for rule in rules]
|
||||
tp = {rule_id: 0 for rule_id in rule_ids}
|
||||
fp = {rule_id: 0 for rule_id in rule_ids}
|
||||
fn = {rule_id: 0 for rule_id in rule_ids}
|
||||
|
||||
boundary_details = []
|
||||
detected = 0
|
||||
for item in eval_texts.get("boundary", []):
|
||||
expected = _expected_rule_ids(item, rules)
|
||||
fired = set(score_text(item["text"], rules, judge_fn, text_id=item["id"]))
|
||||
matched = expected & fired
|
||||
boundary_details.append({
|
||||
"id": item["id"],
|
||||
"expected_sources": item.get("expected_sources", []),
|
||||
"expected": sorted(expected),
|
||||
"fired": sorted(fired),
|
||||
})
|
||||
if matched:
|
||||
detected += 1
|
||||
for rule_id in rule_ids:
|
||||
if rule_id in fired and rule_id in expected:
|
||||
tp[rule_id] += 1
|
||||
elif rule_id in fired:
|
||||
fp[rule_id] += 1
|
||||
elif rule_id in expected:
|
||||
fn[rule_id] += 1
|
||||
|
||||
retention_details = []
|
||||
harmed = 0
|
||||
for item in eval_texts.get("retention", []):
|
||||
fired = score_text(item["text"], rules, judge_fn, text_id=item["id"])
|
||||
retention_details.append({"id": item["id"], "fired": sorted(fired)})
|
||||
if fired:
|
||||
harmed += 1
|
||||
for rule_id in fired:
|
||||
fp[rule_id] += 1
|
||||
|
||||
per_rule = {}
|
||||
for rule_id in rule_ids:
|
||||
precision = tp[rule_id] / (tp[rule_id] + fp[rule_id]) if tp[rule_id] + fp[rule_id] else 1.0
|
||||
recall = tp[rule_id] / (tp[rule_id] + fn[rule_id]) if tp[rule_id] + fn[rule_id] else 1.0
|
||||
per_rule[rule_id] = {
|
||||
"tp": tp[rule_id],
|
||||
"fp": fp[rule_id],
|
||||
"fn": fn[rule_id],
|
||||
"precision": round(precision, 3),
|
||||
"recall": round(recall, 3),
|
||||
}
|
||||
|
||||
boundary_total = len(eval_texts.get("boundary", []))
|
||||
retention_total = len(eval_texts.get("retention", []))
|
||||
return {
|
||||
"per_rule": per_rule,
|
||||
"boundary_detection_rate": detected / boundary_total if boundary_total else 0.0,
|
||||
"boundary_detected": detected,
|
||||
"boundary_total": boundary_total,
|
||||
"retention_harm_rate": harmed / retention_total if retention_total else 0.0,
|
||||
"retention_harmed": harmed,
|
||||
"retention_total": retention_total,
|
||||
"boundary_details": boundary_details,
|
||||
"retention_details": retention_details,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--provider", choices=("ark", "openrouter", "openai"), default="openai")
|
||||
parser.add_argument("--model", default=None)
|
||||
args = parser.parse_args()
|
||||
|
||||
rules_path = ROOT / "skill" / "rules.json"
|
||||
rules = json.loads(rules_path.read_text(encoding="utf-8"))
|
||||
judge_fn = llm_judge(provider=args.provider, model=args.model)
|
||||
metrics = evaluate_rules(rules, load_eval_texts(), judge_fn)
|
||||
report = {
|
||||
"provider": args.provider,
|
||||
"model": args.model or default_model(args.provider),
|
||||
**metrics,
|
||||
}
|
||||
OUTPUT_DIR.mkdir(exist_ok=True)
|
||||
(OUTPUT_DIR / "eval_report.json").write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,143 @@
|
||||
"""从用户纠正的 before/after 对中开放式提炼候选写作规则。
|
||||
|
||||
本模块不包含预置模式库。模型直接比较用户给出的 before/after 与纠正原话,
|
||||
发现已有清单之外的新规律。所有候选都统一使用 LLM judge;模型只能提出候选,
|
||||
是否合并、校准和激活仍由模型外部代码决定。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
from llm_client import chat
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
DATA_DIR = ROOT / "data"
|
||||
|
||||
|
||||
def load_pairs(path: Path | None = None) -> List[Dict[str, Any]]:
|
||||
return json.loads((path or DATA_DIR / "feedback_pairs.json").read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
_LLM_EXTRACT_PROMPT = """你是写作规范的提炼助手。请比较本批用户纠正的 before/after 对,
|
||||
从反馈本身归纳具体、可复核的写作规则。不要依赖任何预置的模式清单,也不要把规则限制在
|
||||
正则表达式能检测的现象;语义、句法、语气和篇章层面的新规律都可以提出。
|
||||
|
||||
返回 JSON:{{"rules": [{{"id": "rule-<英文短横线命名>", "name": "...",
|
||||
"definition": "清楚说明什么情况下命中,以及什么相似情况不应命中",
|
||||
"bad_example": "取自 before 的原文片段", "good_example": "对应的 after 原文片段",
|
||||
"rewrite_hint": "具体改写建议", "scope": ["适用场景"],
|
||||
"source_ids": ["支撑该规则的反馈对 id"]}}]}}
|
||||
|
||||
要求:
|
||||
1. 每条规则必须至少有一条 source_ids,且 id、坏例、好例都只能来自输入;不要编造证据。
|
||||
2. 同一现象只返回一条规则。规则要区分“滥用”和合理使用,不能写成一刀切禁令。
|
||||
3. 如果本批现象与“当前规则”语义相同,必须复用当前规则的 id;只有发现新规律才创建新 id。
|
||||
4. 返回的候选将全部交给外部 LLM judge,用独立人工金标集校准后才可能激活。
|
||||
|
||||
当前规则(可能为空):
|
||||
{existing_rules}
|
||||
|
||||
本批反馈:
|
||||
{pairs}
|
||||
"""
|
||||
|
||||
|
||||
def _strip_json_fence(content: str) -> str:
|
||||
return re.sub(r"^```(?:json)?\s*|\s*```$", "", content.strip(), flags=re.I)
|
||||
|
||||
|
||||
def _validate_candidate(
|
||||
rule: Dict[str, Any], pair_by_id: Dict[str, Dict[str, Any]]
|
||||
) -> Dict[str, Any] | None:
|
||||
required = (
|
||||
"id", "name", "definition", "bad_example", "good_example",
|
||||
"rewrite_hint", "source_ids",
|
||||
)
|
||||
if not all(rule.get(key) for key in required):
|
||||
return None
|
||||
if not re.fullmatch(r"rule-[a-z0-9]+(?:-[a-z0-9]+)*", str(rule["id"])):
|
||||
return None
|
||||
|
||||
source_ids = list(dict.fromkeys(rule["source_ids"]))
|
||||
if not source_ids or any(source_id not in pair_by_id for source_id in source_ids):
|
||||
return None
|
||||
|
||||
bad_example = str(rule["bad_example"])
|
||||
good_example = str(rule["good_example"])
|
||||
supported = any(
|
||||
bad_example in pair_by_id[source_id]["before"]
|
||||
and good_example in pair_by_id[source_id]["after"]
|
||||
for source_id in source_ids
|
||||
)
|
||||
if not supported:
|
||||
return None
|
||||
|
||||
return {
|
||||
"id": rule["id"],
|
||||
"name": str(rule["name"]),
|
||||
"definition": str(rule["definition"]),
|
||||
"detector": {"type": "llm"},
|
||||
"bad_example": bad_example,
|
||||
"good_example": good_example,
|
||||
"rewrite_hint": str(rule["rewrite_hint"]),
|
||||
"scope": sorted({str(item) for item in rule.get("scope", [])}),
|
||||
"source_ids": source_ids,
|
||||
"status": "candidate",
|
||||
}
|
||||
|
||||
|
||||
def extract_with_llm(
|
||||
pairs: List[Dict[str, Any]],
|
||||
*,
|
||||
provider: str,
|
||||
model: str | None = None,
|
||||
seed: int = 8901,
|
||||
existing_rules: List[Dict[str, Any]] | None = None,
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
|
||||
"""开放式提炼候选规则,返回 ``(候选列表, API 证据回执)``。"""
|
||||
brief = [
|
||||
{
|
||||
"id": pair["id"],
|
||||
"scene": pair["scene"],
|
||||
"before": pair["before"],
|
||||
"after": pair["after"],
|
||||
"correction": pair["correction"],
|
||||
}
|
||||
for pair in pairs
|
||||
]
|
||||
current = [
|
||||
{"id": rule["id"], "name": rule["name"], "definition": rule["definition"]}
|
||||
for rule in (existing_rules or [])
|
||||
]
|
||||
content, receipt = chat(
|
||||
[{
|
||||
"role": "user",
|
||||
"content": _LLM_EXTRACT_PROMPT.format(
|
||||
pairs=json.dumps(brief, ensure_ascii=False, indent=2),
|
||||
existing_rules=json.dumps(current, ensure_ascii=False, indent=2),
|
||||
),
|
||||
}],
|
||||
provider=provider,
|
||||
model=model,
|
||||
seed=seed,
|
||||
max_tokens=16000,
|
||||
)
|
||||
payload = json.loads(_strip_json_fence(content))
|
||||
pair_by_id = {pair["id"]: pair for pair in pairs}
|
||||
candidates = []
|
||||
for rule in payload.get("rules", []):
|
||||
candidate = _validate_candidate(rule, pair_by_id)
|
||||
if candidate is not None:
|
||||
candidates.append(candidate)
|
||||
return candidates, receipt
|
||||
|
||||
|
||||
def write_candidates(candidates: List[Dict[str, Any]], path: Path | None = None) -> Path:
|
||||
out = path or DATA_DIR / "candidate_rules.json"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(json.dumps(candidates, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return out
|
||||
@@ -0,0 +1,194 @@
|
||||
"""用一个外部 LLM judge 校准并评估所有候选写作规则。
|
||||
|
||||
规则本身来自开放式提炼,不携带正则或预置模式。judge 根据规则定义、正反例与
|
||||
作用域作语义判定。每条规则必须先在独立人工金标上达到一致率门槛,之后才可激活。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
from typing import Any, Callable, Dict, List, Tuple
|
||||
|
||||
from llm_client import chat
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
DATA_DIR = ROOT / "data"
|
||||
|
||||
Verdicts = Dict[Tuple[str, str], Dict[str, Any]]
|
||||
JudgeFn = Callable[[List[Dict[str, Any]], List[Dict[str, str]]], Verdicts]
|
||||
|
||||
_JUDGE_PROMPT = """你是独立的中文写作质量评判者。请逐一判断每段待评文本是否命中每条规则。
|
||||
严格依据规则的定义、适用范围和正反例:相似词语本身不等于命中,只有规则所描述的滥用
|
||||
确实出现才判 true。不要因为文本来自评估集而猜标签。
|
||||
|
||||
必须为每个 text_id 与 rule_id 的组合返回一项。只返回 JSON:
|
||||
{{"verdicts": [{{"text_id": "...", "rule_id": "...", "hit": true,
|
||||
"evidence": "命中时摘录最短证据;未命中时为空字符串"}}]}}
|
||||
|
||||
规则:
|
||||
{rules}
|
||||
|
||||
待评文本:
|
||||
{texts}
|
||||
"""
|
||||
|
||||
|
||||
def _strip_json_fence(content: str) -> str:
|
||||
return re.sub(r"^```(?:json)?\s*|\s*```$", "", content.strip(), flags=re.I)
|
||||
|
||||
|
||||
def llm_judge(
|
||||
*,
|
||||
provider: str,
|
||||
model: str | None = None,
|
||||
seed: int = 8901,
|
||||
receipts: List[Dict[str, Any]] | None = None,
|
||||
) -> JudgeFn:
|
||||
"""创建批量 LLM judge;一次调用可判定多条规则或多段文本。"""
|
||||
|
||||
def judge(rules: List[Dict[str, Any]], texts: List[Dict[str, str]]) -> Verdicts:
|
||||
brief_rules = [
|
||||
{
|
||||
"id": rule["id"],
|
||||
"name": rule["name"],
|
||||
"definition": rule["definition"],
|
||||
"bad_example": rule.get("bad_example", ""),
|
||||
"good_example": rule.get("good_example", ""),
|
||||
"scope": rule.get("scope", []),
|
||||
}
|
||||
for rule in rules
|
||||
]
|
||||
content, receipt = chat(
|
||||
[{
|
||||
"role": "user",
|
||||
"content": _JUDGE_PROMPT.format(
|
||||
rules=json.dumps(brief_rules, ensure_ascii=False, indent=2),
|
||||
texts=json.dumps(texts, ensure_ascii=False, indent=2),
|
||||
),
|
||||
}],
|
||||
provider=provider,
|
||||
model=model,
|
||||
seed=seed,
|
||||
# Responses API 的 max_output_tokens 也覆盖 reasoning tokens;小上限会让
|
||||
# JSON 在中途被截断。给 reasoning 与每个 verdict 都留出明确余量。
|
||||
max_tokens=max(2000, 1000 + len(rules) * len(texts) * 120),
|
||||
)
|
||||
if receipts is not None:
|
||||
receipts.append(receipt)
|
||||
|
||||
expected = {(rule["id"], text["id"]) for rule in rules for text in texts}
|
||||
try:
|
||||
payload = json.loads(_strip_json_fence(content))
|
||||
except (json.JSONDecodeError, AttributeError):
|
||||
return {
|
||||
key: {"hit": False, "evidence": "", "parse_error": True}
|
||||
for key in expected
|
||||
}
|
||||
|
||||
verdicts: Verdicts = {}
|
||||
for item in payload.get("verdicts", []):
|
||||
key = (str(item.get("rule_id", "")), str(item.get("text_id", "")))
|
||||
if key not in expected or not isinstance(item.get("hit"), bool):
|
||||
continue
|
||||
verdicts[key] = {
|
||||
"hit": item["hit"],
|
||||
"evidence": str(item.get("evidence", "")),
|
||||
}
|
||||
for key in expected - set(verdicts):
|
||||
verdicts[key] = {"hit": False, "evidence": "", "missing": True}
|
||||
return verdicts
|
||||
|
||||
return judge
|
||||
|
||||
|
||||
def _calibration_cases(
|
||||
rule: Dict[str, Any], golden_set: List[Dict[str, Any]]
|
||||
) -> List[Dict[str, Any]]:
|
||||
sources = set(rule.get("source_ids", []))
|
||||
cases = []
|
||||
for item in golden_set:
|
||||
matching = [
|
||||
label["expected"]
|
||||
for label in item.get("labels", [])
|
||||
if sources & set(label.get("source_ids", []))
|
||||
]
|
||||
if not matching:
|
||||
continue
|
||||
if len(set(matching)) != 1:
|
||||
raise ValueError(f"金标 {item['id']} 对规则 {rule['id']} 给出了冲突标签")
|
||||
cases.append({"id": item["id"], "text": item["text"], "expected": matching[0]})
|
||||
return cases
|
||||
|
||||
|
||||
def calibrate(
|
||||
rule: Dict[str, Any],
|
||||
golden_set: List[Dict[str, Any]],
|
||||
judge_fn: JudgeFn,
|
||||
*,
|
||||
threshold: float = 0.8,
|
||||
) -> Dict[str, Any]:
|
||||
"""用与候选来源关联的独立人工金标校准一条规则。"""
|
||||
labeled = _calibration_cases(rule, golden_set)
|
||||
texts = [{"id": item["id"], "text": item["text"]} for item in labeled]
|
||||
verdicts = judge_fn([rule], texts) if texts else {}
|
||||
cases = []
|
||||
agree = 0
|
||||
for item in labeled:
|
||||
verdict = verdicts.get((rule["id"], item["id"]), {"hit": False, "missing": True})
|
||||
got = bool(verdict["hit"])
|
||||
ok = got == item["expected"] and not verdict.get("missing") and not verdict.get("parse_error")
|
||||
agree += int(ok)
|
||||
cases.append({
|
||||
"id": item["id"],
|
||||
"expected": item["expected"],
|
||||
"judged": got,
|
||||
"evidence": verdict.get("evidence", ""),
|
||||
"missing": bool(verdict.get("missing")),
|
||||
"parse_error": bool(verdict.get("parse_error")),
|
||||
"agree": ok,
|
||||
})
|
||||
total = len(cases)
|
||||
agreement = agree / total if total else 0.0
|
||||
decision = "activate" if total and agreement >= threshold else "reject"
|
||||
return {
|
||||
"rule_id": rule["id"],
|
||||
"cases": cases,
|
||||
"total": total,
|
||||
"agree": agree,
|
||||
"agreement": agreement,
|
||||
"threshold": threshold,
|
||||
"decision": decision,
|
||||
"note": (
|
||||
"judge 与独立人工金标的一致率达到阈值,允许上线"
|
||||
if decision == "activate"
|
||||
else "金标覆盖不足或一致率低于阈值,拒绝上线该规则"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def score_text(
|
||||
text: str,
|
||||
rules: List[Dict[str, Any]],
|
||||
judge_fn: JudgeFn,
|
||||
*,
|
||||
text_id: str = "target",
|
||||
) -> Dict[str, Any]:
|
||||
"""用 LLM judge 对全部 active 规则一次性打分。"""
|
||||
if not rules:
|
||||
return {}
|
||||
verdicts = judge_fn(rules, [{"id": text_id, "text": text}])
|
||||
result: Dict[str, Any] = {}
|
||||
for rule in rules:
|
||||
verdict = verdicts.get((rule["id"], text_id), {})
|
||||
if verdict.get("hit"):
|
||||
result[rule["id"]] = {
|
||||
"verdict": True,
|
||||
"evidence": verdict.get("evidence", ""),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def load_golden_set(path: Path | None = None) -> List[Dict[str, Any]]:
|
||||
return json.loads((path or DATA_DIR / "golden_set.json").read_text(encoding="utf-8"))
|
||||
@@ -0,0 +1,118 @@
|
||||
"""OpenAI Responses / 兼容 Chat Completions 客户端:统一证据回执。
|
||||
|
||||
约定与 chapter8/self-modifying-agent/llm_generator.py 一致:每次真实调用返回
|
||||
(content, receipt),receipt 含原始请求、原始响应、Token 用量、延迟与
|
||||
请求/响应哈希,不记录凭据值。凭证从环境变量读取,支持 ark / openrouter / openai。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
_PROVIDERS = {
|
||||
"openrouter": ("OPENROUTER_API_KEY", "https://openrouter.ai/api/v1"),
|
||||
"ark": ("ARK_API_KEY", "https://ark.cn-beijing.volces.com/api/v3"),
|
||||
"openai": ("OPENAI_API_KEY", None),
|
||||
}
|
||||
|
||||
_DEFAULT_MODELS = {
|
||||
"openrouter": "openai/gpt-5.6-sol",
|
||||
"ark": "doubao-seed-1-6-250615",
|
||||
"openai": "gpt-5.6-sol",
|
||||
}
|
||||
|
||||
|
||||
def make_client(provider: str) -> Tuple[Any, Dict[str, Any]]:
|
||||
# openai 包只在真实路径才需要,惰性导入保证离线路径零依赖。
|
||||
from openai import OpenAI
|
||||
|
||||
if provider not in _PROVIDERS:
|
||||
raise ValueError(f"不支持的 provider:{provider}(可选:{sorted(_PROVIDERS)})")
|
||||
env_name, base_url = _PROVIDERS[provider]
|
||||
key = os.getenv(env_name)
|
||||
if not key:
|
||||
raise RuntimeError(f"真实 LLM 路径需要设置环境变量 {env_name}")
|
||||
client = OpenAI(api_key=key, base_url=base_url) if base_url else OpenAI(api_key=key)
|
||||
api = "responses" if provider in {"openai", "openrouter"} else "chat/completions"
|
||||
backend = {
|
||||
"provider": provider,
|
||||
"endpoint": (base_url or "https://api.openai.com/v1") + f"/{api}",
|
||||
"credential_env": env_name,
|
||||
}
|
||||
return client, backend
|
||||
|
||||
|
||||
def default_model(provider: str) -> str:
|
||||
if provider == "ark":
|
||||
return os.getenv("ARK_MODEL", _DEFAULT_MODELS["ark"])
|
||||
return _DEFAULT_MODELS[provider]
|
||||
|
||||
|
||||
def chat(
|
||||
messages: List[Dict[str, str]],
|
||||
*,
|
||||
provider: str,
|
||||
model: str | None = None,
|
||||
seed: int = 8901,
|
||||
max_tokens: int = 5000,
|
||||
) -> Tuple[str, Dict[str, Any]]:
|
||||
"""发起一次结构化 JSON 调用,返回 ``(文本内容, 证据回执)``。"""
|
||||
client, backend = make_client(provider)
|
||||
selected = model or default_model(provider)
|
||||
started = time.perf_counter()
|
||||
if provider in {"openai", "openrouter"}:
|
||||
request = {
|
||||
"model": selected,
|
||||
"input": messages,
|
||||
"reasoning": {"effort": "medium"},
|
||||
"max_output_tokens": max_tokens,
|
||||
"text": {"format": {"type": "json_object"}},
|
||||
"store": False,
|
||||
}
|
||||
response = client.responses.create(**request)
|
||||
content = response.output_text
|
||||
else:
|
||||
request = {
|
||||
"model": selected,
|
||||
"messages": messages,
|
||||
"temperature": 0,
|
||||
"seed": seed,
|
||||
"max_tokens": max_tokens,
|
||||
"response_format": {"type": "json_object"},
|
||||
}
|
||||
response = client.chat.completions.create(**request)
|
||||
content = response.choices[0].message.content or ""
|
||||
elapsed = time.perf_counter() - started
|
||||
raw = response.model_dump(mode="json", exclude_none=True)
|
||||
usage = raw.get("usage") or {}
|
||||
cost = usage.get("cost")
|
||||
prompt_tokens = usage.get("input_tokens", usage.get("prompt_tokens", 0))
|
||||
completion_tokens = usage.get("output_tokens", usage.get("completion_tokens", 0))
|
||||
receipt = {
|
||||
"backend": {**backend, "model": selected, "credential_value_recorded": False},
|
||||
"request": request,
|
||||
"response": raw,
|
||||
"request_sha256": hashlib.sha256(
|
||||
json.dumps(request, sort_keys=True).encode()
|
||||
).hexdigest(),
|
||||
"response_sha256": hashlib.sha256(
|
||||
json.dumps(raw, sort_keys=True).encode()
|
||||
).hexdigest(),
|
||||
"elapsed_seconds": round(elapsed, 6),
|
||||
"usage": {
|
||||
"prompt_tokens": int(prompt_tokens or 0),
|
||||
"completion_tokens": int(completion_tokens or 0),
|
||||
"total_tokens": int(usage.get("total_tokens") or 0),
|
||||
"provider_reported_cost_usd": float(cost) if cost is not None else None,
|
||||
"cost_qualification": (
|
||||
"provider-native usage.cost"
|
||||
if cost is not None
|
||||
else "provider did not expose monetary cost; no price was guessed"
|
||||
),
|
||||
},
|
||||
}
|
||||
return content, receipt
|
||||
@@ -0,0 +1,3 @@
|
||||
# OpenAI 直连使用 Responses API;单元测试通过 mock 保持离线。
|
||||
openai>=1.97.1
|
||||
pytest>=8.3.0
|
||||
@@ -0,0 +1,85 @@
|
||||
"""把 active 规则连同原文交给 LLM 改写,返回 before/after 与证据回执。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
from llm_client import chat
|
||||
|
||||
# 演示用原文:刻意集齐八类 AI 味。
|
||||
SAMPLE_TEXT = (
|
||||
"在这个效率至上的时代,让我们一起重新认识这款笔记工具——它不是简单的记录软件——"
|
||||
"而是你的第二大脑。首先,它能自动整理灵感;其次,它让检索快如闪电;"
|
||||
"最后,它让分享毫无门槛——仿佛一位永不疲倦的管家。总而言之,让我们从现在开始,"
|
||||
"把每一条灵感都安顿好 🚀✨💡。"
|
||||
)
|
||||
|
||||
_REWRITE_PROMPT = """你是中文文案改写助手。请按下面的写作规则改写给出的文案,消除所有违规之处,
|
||||
保持原意,不要增加新事实。只返回 JSON:{{"rewritten": "改写后的全文",
|
||||
"applied_rules": ["实际应用的规则 id"]}}
|
||||
|
||||
写作规则:
|
||||
{rules}
|
||||
|
||||
待改写文案:
|
||||
{text}
|
||||
"""
|
||||
|
||||
|
||||
def rewrite_with_llm(
|
||||
text: str,
|
||||
rules: List[Dict[str, Any]],
|
||||
*,
|
||||
provider: str,
|
||||
model: str | None = None,
|
||||
seed: int = 8901,
|
||||
) -> Tuple[Dict[str, Any], Dict[str, Any]]:
|
||||
"""真实 LLM 改写路径,返回 (改写结果, 证据回执)。"""
|
||||
brief_rules = [
|
||||
{"id": r["id"], "name": r["name"], "definition": r["definition"],
|
||||
"bad_example": r.get("bad_example", ""), "good_example": r.get("good_example", "")}
|
||||
for r in rules
|
||||
]
|
||||
content, receipt = chat(
|
||||
[{"role": "user", "content": _REWRITE_PROMPT.format(
|
||||
rules=json.dumps(brief_rules, ensure_ascii=False, indent=2), text=text)}],
|
||||
provider=provider, model=model, seed=seed,
|
||||
)
|
||||
payload = json.loads(re.sub(r"^```(?:json)?\s*|\s*```$", "", content.strip(), flags=re.I))
|
||||
return {
|
||||
"mode": "real_llm",
|
||||
"original": text,
|
||||
"rewritten": payload.get("rewritten", ""),
|
||||
"applied_rules": payload.get("applied_rules", []),
|
||||
}, receipt
|
||||
|
||||
|
||||
def _load_active_rules() -> List[Dict[str, Any]]:
|
||||
import json as _json
|
||||
from pathlib import Path
|
||||
|
||||
rules_path = Path(__file__).resolve().parent / "skill" / "rules.json"
|
||||
if not rules_path.exists():
|
||||
raise SystemExit("请先生成 Skill:python demo.py 或 python run_ai_style_skill.py")
|
||||
return _json.loads(rules_path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
def main() -> int:
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--provider", choices=("ark", "openrouter", "openai"), default="openai")
|
||||
parser.add_argument("--model", default=None)
|
||||
args = parser.parse_args()
|
||||
rules = _load_active_rules()
|
||||
result, _ = rewrite_with_llm(
|
||||
SAMPLE_TEXT, rules, provider=args.provider, model=args.model
|
||||
)
|
||||
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,195 @@
|
||||
#!/usr/bin/env python3
|
||||
"""把「AI 味」反馈开放式提炼为规则,并由 LLM judge 校准和评估。
|
||||
|
||||
默认使用 OpenAI GPT-5.6 Sol:
|
||||
|
||||
python run_ai_style_skill.py
|
||||
|
||||
流程:全量反馈 → LLM 开放式提炼并做语义归并 → 每条规则用独立人工
|
||||
金标校准 LLM judge → 生成 Skill → boundary/retention 评估 → 改写演示。
|
||||
代码中不包含预置模式库,也不会用 detector 指纹过滤模型发现的新规律。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from evaluate import evaluate_rules, load_eval_texts
|
||||
from extract_rules import extract_with_llm, load_pairs, write_candidates
|
||||
from judge import calibrate, llm_judge, load_golden_set
|
||||
from llm_client import default_model
|
||||
from rewrite_demo import SAMPLE_TEXT, rewrite_with_llm
|
||||
from skill_manager import merge_rules, prune_rules, write_archive, write_skill
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
# 验收门槛(模型外部代码,LLM 不可修改)。
|
||||
GATE_BOUNDARY_RATE = 0.85
|
||||
GATE_RETENTION_HARM = 0.15
|
||||
|
||||
|
||||
def run_pipeline(
|
||||
*,
|
||||
provider: str = "openai",
|
||||
model: str | None = None,
|
||||
seed: int = 8901,
|
||||
batches: int = 3,
|
||||
) -> Dict[str, Any]:
|
||||
pairs = load_pairs()
|
||||
eval_texts = load_eval_texts()
|
||||
golden_set = load_golden_set()
|
||||
receipts: List[Dict[str, Any]] = []
|
||||
|
||||
# 1) 一次查看全部反馈,避免批次顺序让同一概念被拆分、不同概念被误并。
|
||||
candidates, receipt = extract_with_llm(
|
||||
pairs,
|
||||
provider=provider,
|
||||
model=model,
|
||||
seed=seed,
|
||||
)
|
||||
receipts.append(receipt)
|
||||
rules, merge_report = merge_rules([], candidates)
|
||||
pair_batches = {pair["id"]: pair.get("batch", 1) for pair in pairs}
|
||||
for rule in rules:
|
||||
rule["last_confirmed_batch"] = max(
|
||||
(pair_batches[source] for source in rule.get("source_ids", []) if source in pair_batches),
|
||||
default=batches,
|
||||
)
|
||||
curve = [
|
||||
{
|
||||
"batch": batch_no,
|
||||
"rules_with_evidence": sum(
|
||||
any(pair_batches.get(source) == batch_no for source in rule.get("source_ids", []))
|
||||
for rule in rules
|
||||
),
|
||||
"cumulative_rule_count": sum(
|
||||
any(pair_batches.get(source, batches + 1) <= batch_no for source in rule.get("source_ids", []))
|
||||
for rule in rules
|
||||
),
|
||||
}
|
||||
for batch_no in range(1, batches + 1)
|
||||
]
|
||||
total_candidates = len(candidates)
|
||||
write_candidates(candidates)
|
||||
|
||||
# 2) 所有规则都由同一个批量 LLM judge 校准;无金标或不达标都拒绝上线。
|
||||
calibration = []
|
||||
active: List[Dict[str, Any]] = []
|
||||
judge_fn = llm_judge(
|
||||
provider=provider, model=model, seed=seed, receipts=receipts
|
||||
)
|
||||
for rule in rules:
|
||||
result = calibrate(rule, golden_set, judge_fn)
|
||||
calibration.append(result)
|
||||
if result["decision"] == "activate":
|
||||
rule["status"] = "active"
|
||||
active.append(rule)
|
||||
else:
|
||||
rule["status"] = "rejected"
|
||||
|
||||
# 3) prune 演示:被证据推翻或长期未触发的规则归档(离线数据下无归档)。
|
||||
active, archived = prune_rules(active, current_batch=batches, contradicted_ids=set())
|
||||
|
||||
# 4) 生成 Skill 并评估。
|
||||
skill_path = write_skill(active)
|
||||
archive_path = write_archive(archived)
|
||||
metrics = evaluate_rules(active, eval_texts, judge_fn)
|
||||
|
||||
# 5) 改写演示。
|
||||
rewrite, receipt = rewrite_with_llm(
|
||||
SAMPLE_TEXT, active, provider=provider, model=model, seed=seed
|
||||
)
|
||||
receipts.append(receipt)
|
||||
|
||||
gates = {
|
||||
"boundary_detection_rate >= 0.85": metrics["boundary_detection_rate"] >= GATE_BOUNDARY_RATE,
|
||||
"retention_harm_rate <= 0.15": metrics["retention_harm_rate"] <= GATE_RETENTION_HARM,
|
||||
"corpus_rules_have_unique_ids": (
|
||||
len(rules) == total_candidates
|
||||
and len({rule["id"] for rule in rules}) == len(rules)
|
||||
and not merge_report["merged"]
|
||||
),
|
||||
"all_active_rules_use_llm_judge": all(
|
||||
rule.get("detector") == {"type": "llm"} for rule in active
|
||||
),
|
||||
"all_active_rules_calibrated": all(
|
||||
any(c["rule_id"] == rule["id"] and c["decision"] == "activate" for c in calibration)
|
||||
for rule in active
|
||||
),
|
||||
"out_of_library_cases_detected": all(
|
||||
any(detail["expected"] and set(detail["expected"]) & set(detail["fired"])
|
||||
for detail in metrics["boundary_details"] if detail["id"] == case_id)
|
||||
for case_id in ("b9", "b10", "b11")
|
||||
),
|
||||
"real_llm_called_with_receipts": bool(receipts) and all(
|
||||
receipt["response"].get("id") for receipt in receipts
|
||||
),
|
||||
}
|
||||
|
||||
report = {
|
||||
"experiment": "ai-style-skill",
|
||||
"executed_at": datetime.now(timezone.utc).isoformat(),
|
||||
"execution_mode": "real_llm_judge",
|
||||
"provider": provider,
|
||||
"model": model or default_model(provider),
|
||||
"growth_curve": curve,
|
||||
"total_candidates": total_candidates,
|
||||
"final_rule_count": len(rules),
|
||||
"active_rules": [r["id"] for r in active],
|
||||
"archived_rules": [r["id"] for r in archived],
|
||||
"calibration": calibration,
|
||||
"skill_path": str(skill_path.relative_to(ROOT)),
|
||||
"archive_path": str(archive_path.relative_to(ROOT)) if archive_path else None,
|
||||
"metrics": metrics,
|
||||
"rewrite_demo": rewrite,
|
||||
"raw_api_receipts": receipts,
|
||||
"gates": gates,
|
||||
"accepted": all(gates.values()),
|
||||
}
|
||||
return report
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--provider", choices=("ark", "openrouter", "openai"), default="openai")
|
||||
parser.add_argument("--model", default=None)
|
||||
parser.add_argument("--seed", type=int, default=8901)
|
||||
args = parser.parse_args()
|
||||
|
||||
report = run_pipeline(provider=args.provider, model=args.model, seed=args.seed)
|
||||
|
||||
stamp = datetime.now(timezone.utc).strftime("real_%Y%m%dT%H%M%SZ")
|
||||
out_dir = ROOT / "validation" / stamp
|
||||
out_dir.mkdir(parents=True, exist_ok=False)
|
||||
evidence_path = out_dir / "evidence.json"
|
||||
evidence_path.write_text(
|
||||
json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
canonical = ROOT / "validation" / "latest.json"
|
||||
shutil.copyfile(evidence_path, canonical)
|
||||
print(f"证据回执:{evidence_path.relative_to(ROOT)}(validation/latest.json 已指向)")
|
||||
|
||||
metrics = report["metrics"]
|
||||
print(json.dumps({
|
||||
"mode": report["execution_mode"],
|
||||
"accepted": report["accepted"],
|
||||
"boundary_detection_rate": f"{metrics['boundary_detected']}/{metrics['boundary_total']}",
|
||||
"retention_harm_rate": f"{metrics['retention_harmed']}/{metrics['retention_total']}",
|
||||
"candidates_to_rules": f"{report['total_candidates']} -> {report['final_rule_count']}",
|
||||
"active_rules": report["active_rules"],
|
||||
"calibration": [
|
||||
{"rule_id": c["rule_id"], "agreement": c["agreement"], "decision": c["decision"]}
|
||||
for c in report["calibration"]
|
||||
],
|
||||
"gates": report["gates"],
|
||||
}, ensure_ascii=False, indent=2))
|
||||
return 0 if report["accepted"] else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
name: ai-style
|
||||
description: 中文文案去「AI 味」检查清单,由用户纠正反馈持续提炼而来
|
||||
---
|
||||
|
||||
# 去 AI 味写作 Skill
|
||||
|
||||
## 何时加载
|
||||
|
||||
当任务是用中文撰写或改写面向读者的文案(产品发布稿、公众号文章、邮件、README 等),
|
||||
或用户反馈文字「AI 味太重」「不像人写的」时,加载本 Skill。
|
||||
|
||||
## 使用方式
|
||||
|
||||
起草或改写时逐条对照下面的规则自查。每条规则都给出定义、可检查的检测方法、
|
||||
坏例与好例;规则只在声明的作用域内生效,作用域之外的文体不要套用。
|
||||
|
||||
## 规则清单(共 11 条)
|
||||
|
||||
### 规则 1:避免用连续破折号代替常规断句(`rule-excessive-em-dash-separators`)
|
||||
|
||||
- **定义**:当一句或一段中连续使用多个破折号来串联普通并列信息、步骤或分句,导致层次不清、阅读节奏拖长时命中。应改用句号、逗号、冒号或分句。若破折号只偶尔用于插入说明、语意转折或强调,且前后关系明确,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:关于季度复盘会议——时间定于本周五下午三点——地点在三号会议室——请提前准备数据报表——如有冲突请及时告知——谢谢配合。
|
||||
- **好例**:季度复盘定在这周五下午三点,三号会议室。请提前准备好数据报表,时间冲突的话提前跟我说。
|
||||
- **改写建议**:先判断各部分是并列信息还是独立句意;并列项用逗号,完整意思用句号,只在确有插入或强调作用时保留破折号。
|
||||
- **作用域**:产品发布稿、公众号文章、邮件
|
||||
- **来源反馈**:fp-001, fp-008, fp-017
|
||||
|
||||
### 规则 2:避免反复使用“不是……而是……”对仗(`rule-avoid-repeated-not-but-contrast`)
|
||||
|
||||
- **定义**:当“不是……而是……”被连续用于定义产品、解释概念或制造升华,尤其连续出现两次以上并形成整齐对仗时命中。这类写法常以否定铺垫代替直接说明。若上下文确实需要纠正一种明确误解,并且只使用一次来表达真实对立,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:这不是一次简单的版本更新,而是一次体验的全面革新;不是功能的堆砌,而是对用户需求的深度回应。
|
||||
- **好例**:这次版本更新改动很大:界面重做,核心流程缩短了两步,都来自上一版用户反馈里最高频的问题。
|
||||
- **改写建议**:删去否定式铺垫,直接说明对象是什么、具体改了什么,并用事实或例子支撑判断。
|
||||
- **作用域**:README 段落、产品发布稿、公众号文章
|
||||
- **来源反馈**:fp-002, fp-009, fp-015
|
||||
|
||||
### 规则 3:避免连用“让我们”式号召口号(`rule-avoid-repeated-let-us-slogans`)
|
||||
|
||||
- **定义**:当邮件、README 或产品稿连续以“让我们”开头,使用“共同见证”“携手并进”等口号来代替具体通知、邀请或行动说明时命中。若只出现一次,且确实是在自然地发出明确、可执行的共同邀请,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:让我们共同见证这个激动人心的时刻!让我们一起开启数字化转型的全新篇章。让我们携手并进,共创辉煌明天。
|
||||
- **好例**:新版本今天上线了,期待大家试用之后多提意见。
|
||||
- **改写建议**:直接交代发生了什么,以及希望读者采取什么具体行动,如试用、反馈、提 issue 或提交 PR。
|
||||
- **作用域**:README 段落、产品发布稿、邮件
|
||||
- **来源反馈**:fp-003, fp-010, fp-018
|
||||
|
||||
### 规则 4:避免机械堆叠顺序连接词(`rule-avoid-mechanical-sequence-markers`)
|
||||
|
||||
- **定义**:当短段落逐句套用“首先、其次、再次、然后、最后、总而言之”等连接词,实际内容本可自然串联,因而呈现模板化或公文腔时命中。若步骤顺序严格、跳步会造成错误,或连接词确实用于澄清复杂层级,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:首先,克隆本仓库到本地。其次,安装所需依赖。再次,配置环境变量。然后,运行初始化脚本。最后,启动开发服务器即可。
|
||||
- **好例**:克隆仓库后安装依赖,把 .env.example 复制为 .env 并填好密钥,再运行 init.sh 初始化,最后 make dev 启动服务。
|
||||
- **改写建议**:删除不提供额外信息的序号套话,改用动作之间的真实关系连接;必要时只保留少量“再”“最后”,或改成清晰的步骤列表。
|
||||
- **作用域**:README 段落、产品发布稿、邮件
|
||||
- **来源反馈**:fp-004, fp-012, fp-019
|
||||
|
||||
### 规则 5:删除空泛的“在……时代”开场(`rule-remove-generic-era-openings`)
|
||||
|
||||
- **定义**:当文章以“在……的今天”“在这个……的时代”等宏大背景开头,但该背景没有提供具体时间、事件或因果信息,只用于烘托气氛时命中。若时代或日期本身是论证所需的关键背景,并有具体事实支撑,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:在这个快节奏的时代,高效的沟通显得尤为珍贵。在这个充满变化的时代,稳定的协作关系更值得珍惜。
|
||||
- **好例**:最近项目节奏快,咱们沟通尽量简短高效;也希望协作关系保持稳定,有问题随时同步。
|
||||
- **改写建议**:直接写当前发生的具体变化、影响和要求,用“最近”“这两年”等可核实时间范围替代泛化时代判断。
|
||||
- **作用域**:产品发布稿、公众号文章、邮件
|
||||
- **来源反馈**:fp-005, fp-010, fp-016
|
||||
|
||||
### 规则 6:控制 emoji 的使用密度与场合(`rule-control-emoji-density`)
|
||||
|
||||
- **定义**:当正式产品稿中使用 emoji,或公众号段落中几乎每个信息点都附带表情,造成视觉干扰、削弱专业感时命中。正式稿件应去除 emoji;轻松社交场景中少量、与语义直接相关且不妨碍阅读的 emoji 不应一概判定为问题。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:🎉 重磅来袭!🔥 全新智能手表正式发布!⌚ 超长续航 14 天 🔋,心率血氧全天候监测 ❤️,50 米防水 🏊,现在下单立减 200 元 💰,错过再等一年 ⏰!
|
||||
- **好例**:新款智能手表发布:续航 14 天,支持心率血氧监测和 50 米防水,首发价立减 200 元。
|
||||
- **改写建议**:正式文本删除全部 emoji;其他场景先去掉装饰性表情,只在确有语气或分类作用时保留极少量。
|
||||
- **作用域**:产品发布稿、公众号文章
|
||||
- **来源反馈**:fp-006, fp-011
|
||||
|
||||
### 规则 7:避免刻意堆叠同构排比(`rule-avoid-forced-parallelism`)
|
||||
|
||||
- **定义**:当三句或更多分句反复使用完全相同的开头和句法框架,如“读书可以……”“它让……”“每一次……都是……”,且内容多为抽象赞美、缺少信息增量时命中。若排比用于必要的结构对照、篇幅适度且每项都有实质信息,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:读书可以拓宽视野,读书可以沉淀心灵,读书可以启迪智慧,读书可以点亮人生。
|
||||
- **好例**:读书的好处很多:视野会变宽,心能静下来,看问题也会多几个角度。
|
||||
- **改写建议**:保留一个总述,将各项改成不同句式或合并为并列成分;删去重复、空泛或意义相近的项。
|
||||
- **作用域**:README 段落、公众号文章
|
||||
- **来源反馈**:fp-007, fp-014, fp-017
|
||||
|
||||
### 规则 8:用具体事实替代空洞抒情比喻(`rule-replace-empty-metaphors-with-facts`)
|
||||
|
||||
- **定义**:当“智者、屏障、明灯、灯塔”等比喻只负责抬高语气,却没有说明产品效果、处理进度或实际价值,甚至替代了读者需要的关键信息时命中。若比喻准确、简洁,并能帮助理解陌生概念或符合文学性场景,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:您的建议宛如一盏明灯,照亮了我们产品改进的前路;您的信任仿佛一座灯塔,指引着我们不断前行。
|
||||
- **好例**:您提的建议很具体,我们已经记进需求池,下个版本会先改其中两条。谢谢您的信任。
|
||||
- **改写建议**:删除装饰性喻体,补上可验证的信息,如实际体验、使用场景、已采取的动作和后续安排。
|
||||
- **作用域**:公众号文章、邮件
|
||||
- **来源反馈**:fp-013, fp-020
|
||||
|
||||
### 规则 9:拆分承载过多层次的长句(`rule-split-overloaded-sentences`)
|
||||
|
||||
- **定义**:当一句话同时塞入背景、原因、条件、多个动作、结果和要求,主要依靠“且、并、同时、为了、让”等连续连接,读者难以一次识别主干时命中。句子较长但主干清楚、修饰关系单一且不存在理解负担时不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:这次更新在保留原有工作台布局的基础上通过重新组织导航层级并合并重复入口同时优化首次加载和列表刷新策略让用户从打开应用到完成一次记录的整个过程都比以前更连贯也更省时间。
|
||||
- **好例**:这次更新保留了原有工作台布局,同时重新组织导航层级、合并重复入口。首次加载和列表刷新也做了优化,从打开应用到完成记录会更连贯。
|
||||
- **改写建议**:先提取背景、改动、要求和结果,再按层次拆成两到三句;每句只保留一个主要动作或判断。
|
||||
- **作用域**:产品发布稿、邮件
|
||||
- **来源反馈**:fp-021, fp-024
|
||||
|
||||
### 规则 10:删除反复出现的提醒性套话(`rule-remove-repeated-attention-prefaces`)
|
||||
|
||||
- **定义**:当连续句子都用“值得注意的是”“需要特别注意的是”“更值得注意的是”等元话语开头,只是重复强调而没有建立新的信息层级时命中。若只在关键风险首次出现时使用一次,且确实需要提示读者改变注意焦点,则不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:值得注意的是,本周五是最终截止时间。需要特别注意的是,逾期提交将影响下游联调。还值得注意的是,附件命名必须统一。
|
||||
- **好例**:最终截止时间是本周五,逾期会影响下游联调。附件请按统一格式命名。
|
||||
- **改写建议**:删掉“值得注意的是”等前缀,直接写结论、截止时间、后果或操作要求;需要突出时依靠信息顺序而非重复提醒。
|
||||
- **作用域**:公众号文章、邮件
|
||||
- **来源反馈**:fp-022, fp-025
|
||||
|
||||
### 规则 11:避免连续使用“被”字被动句(`rule-avoid-passive-voice-chains`)
|
||||
|
||||
- **定义**:当多个相邻分句都采用“对象被执行者处理”的结构,且执行者明确、适合直接充当主语时命中。若执行者未知、不重要,或文本确实需要突出受影响对象,则单个被动句不命中。
|
||||
- **检测方法**:LLM judge 语义判定(上线前须通过独立人工金标集校准)
|
||||
- **坏例**:配置文件会被启动器读取,依赖会被安装脚本自动下载,数据库会被迁移工具初始化,服务会被进程管理器拉起。
|
||||
- **好例**:启动器读取配置文件,安装脚本自动下载依赖,迁移工具初始化数据库,进程管理器随后拉起服务。
|
||||
- **改写建议**:把工具、模块或责任方移到主语位置,改写为“谁做什么”;只有确需突出承受结果的对象时才保留被动表达。
|
||||
- **作用域**:README 段落、产品发布稿
|
||||
- **来源反馈**:fp-023, fp-026
|
||||
@@ -0,0 +1,244 @@
|
||||
[
|
||||
{
|
||||
"id": "rule-excessive-em-dash-separators",
|
||||
"name": "避免用连续破折号代替常规断句",
|
||||
"definition": "当一句或一段中连续使用多个破折号来串联普通并列信息、步骤或分句,导致层次不清、阅读节奏拖长时命中。应改用句号、逗号、冒号或分句。若破折号只偶尔用于插入说明、语意转折或强调,且前后关系明确,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "关于季度复盘会议——时间定于本周五下午三点——地点在三号会议室——请提前准备数据报表——如有冲突请及时告知——谢谢配合。",
|
||||
"good_example": "季度复盘定在这周五下午三点,三号会议室。请提前准备好数据报表,时间冲突的话提前跟我说。",
|
||||
"rewrite_hint": "先判断各部分是并列信息还是独立句意;并列项用逗号,完整意思用句号,只在确有插入或强调作用时保留破折号。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-001",
|
||||
"fp-008",
|
||||
"fp-017"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-repeated-not-but-contrast",
|
||||
"name": "避免反复使用“不是……而是……”对仗",
|
||||
"definition": "当“不是……而是……”被连续用于定义产品、解释概念或制造升华,尤其连续出现两次以上并形成整齐对仗时命中。这类写法常以否定铺垫代替直接说明。若上下文确实需要纠正一种明确误解,并且只使用一次来表达真实对立,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "这不是一次简单的版本更新,而是一次体验的全面革新;不是功能的堆砌,而是对用户需求的深度回应。",
|
||||
"good_example": "这次版本更新改动很大:界面重做,核心流程缩短了两步,都来自上一版用户反馈里最高频的问题。",
|
||||
"rewrite_hint": "删去否定式铺垫,直接说明对象是什么、具体改了什么,并用事实或例子支撑判断。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿",
|
||||
"公众号文章"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-002",
|
||||
"fp-009",
|
||||
"fp-015"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-repeated-let-us-slogans",
|
||||
"name": "避免连用“让我们”式号召口号",
|
||||
"definition": "当邮件、README 或产品稿连续以“让我们”开头,使用“共同见证”“携手并进”等口号来代替具体通知、邀请或行动说明时命中。若只出现一次,且确实是在自然地发出明确、可执行的共同邀请,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "让我们共同见证这个激动人心的时刻!让我们一起开启数字化转型的全新篇章。让我们携手并进,共创辉煌明天。",
|
||||
"good_example": "新版本今天上线了,期待大家试用之后多提意见。",
|
||||
"rewrite_hint": "直接交代发生了什么,以及希望读者采取什么具体行动,如试用、反馈、提 issue 或提交 PR。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-003",
|
||||
"fp-010",
|
||||
"fp-018"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-mechanical-sequence-markers",
|
||||
"name": "避免机械堆叠顺序连接词",
|
||||
"definition": "当短段落逐句套用“首先、其次、再次、然后、最后、总而言之”等连接词,实际内容本可自然串联,因而呈现模板化或公文腔时命中。若步骤顺序严格、跳步会造成错误,或连接词确实用于澄清复杂层级,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "首先,克隆本仓库到本地。其次,安装所需依赖。再次,配置环境变量。然后,运行初始化脚本。最后,启动开发服务器即可。",
|
||||
"good_example": "克隆仓库后安装依赖,把 .env.example 复制为 .env 并填好密钥,再运行 init.sh 初始化,最后 make dev 启动服务。",
|
||||
"rewrite_hint": "删除不提供额外信息的序号套话,改用动作之间的真实关系连接;必要时只保留少量“再”“最后”,或改成清晰的步骤列表。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-004",
|
||||
"fp-012",
|
||||
"fp-019"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-remove-generic-era-openings",
|
||||
"name": "删除空泛的“在……时代”开场",
|
||||
"definition": "当文章以“在……的今天”“在这个……的时代”等宏大背景开头,但该背景没有提供具体时间、事件或因果信息,只用于烘托气氛时命中。若时代或日期本身是论证所需的关键背景,并有具体事实支撑,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "在这个快节奏的时代,高效的沟通显得尤为珍贵。在这个充满变化的时代,稳定的协作关系更值得珍惜。",
|
||||
"good_example": "最近项目节奏快,咱们沟通尽量简短高效;也希望协作关系保持稳定,有问题随时同步。",
|
||||
"rewrite_hint": "直接写当前发生的具体变化、影响和要求,用“最近”“这两年”等可核实时间范围替代泛化时代判断。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-005",
|
||||
"fp-010",
|
||||
"fp-016"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-control-emoji-density",
|
||||
"name": "控制 emoji 的使用密度与场合",
|
||||
"definition": "当正式产品稿中使用 emoji,或公众号段落中几乎每个信息点都附带表情,造成视觉干扰、削弱专业感时命中。正式稿件应去除 emoji;轻松社交场景中少量、与语义直接相关且不妨碍阅读的 emoji 不应一概判定为问题。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "🎉 重磅来袭!🔥 全新智能手表正式发布!⌚ 超长续航 14 天 🔋,心率血氧全天候监测 ❤️,50 米防水 🏊,现在下单立减 200 元 💰,错过再等一年 ⏰!",
|
||||
"good_example": "新款智能手表发布:续航 14 天,支持心率血氧监测和 50 米防水,首发价立减 200 元。",
|
||||
"rewrite_hint": "正式文本删除全部 emoji;其他场景先去掉装饰性表情,只在确有语气或分类作用时保留极少量。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"公众号文章"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-006",
|
||||
"fp-011"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 2
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-forced-parallelism",
|
||||
"name": "避免刻意堆叠同构排比",
|
||||
"definition": "当三句或更多分句反复使用完全相同的开头和句法框架,如“读书可以……”“它让……”“每一次……都是……”,且内容多为抽象赞美、缺少信息增量时命中。若排比用于必要的结构对照、篇幅适度且每项都有实质信息,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "读书可以拓宽视野,读书可以沉淀心灵,读书可以启迪智慧,读书可以点亮人生。",
|
||||
"good_example": "读书的好处很多:视野会变宽,心能静下来,看问题也会多几个角度。",
|
||||
"rewrite_hint": "保留一个总述,将各项改成不同句式或合并为并列成分;删去重复、空泛或意义相近的项。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"公众号文章"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-007",
|
||||
"fp-014",
|
||||
"fp-017"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-replace-empty-metaphors-with-facts",
|
||||
"name": "用具体事实替代空洞抒情比喻",
|
||||
"definition": "当“智者、屏障、明灯、灯塔”等比喻只负责抬高语气,却没有说明产品效果、处理进度或实际价值,甚至替代了读者需要的关键信息时命中。若比喻准确、简洁,并能帮助理解陌生概念或符合文学性场景,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "您的建议宛如一盏明灯,照亮了我们产品改进的前路;您的信任仿佛一座灯塔,指引着我们不断前行。",
|
||||
"good_example": "您提的建议很具体,我们已经记进需求池,下个版本会先改其中两条。谢谢您的信任。",
|
||||
"rewrite_hint": "删除装饰性喻体,补上可验证的信息,如实际体验、使用场景、已采取的动作和后续安排。",
|
||||
"scope": [
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-013",
|
||||
"fp-020"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-split-overloaded-sentences",
|
||||
"name": "拆分承载过多层次的长句",
|
||||
"definition": "当一句话同时塞入背景、原因、条件、多个动作、结果和要求,主要依靠“且、并、同时、为了、让”等连续连接,读者难以一次识别主干时命中。句子较长但主干清楚、修饰关系单一且不存在理解负担时不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "这次更新在保留原有工作台布局的基础上通过重新组织导航层级并合并重复入口同时优化首次加载和列表刷新策略让用户从打开应用到完成一次记录的整个过程都比以前更连贯也更省时间。",
|
||||
"good_example": "这次更新保留了原有工作台布局,同时重新组织导航层级、合并重复入口。首次加载和列表刷新也做了优化,从打开应用到完成记录会更连贯。",
|
||||
"rewrite_hint": "先提取背景、改动、要求和结果,再按层次拆成两到三句;每句只保留一个主要动作或判断。",
|
||||
"scope": [
|
||||
"产品发布稿",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-021",
|
||||
"fp-024"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 2
|
||||
},
|
||||
{
|
||||
"id": "rule-remove-repeated-attention-prefaces",
|
||||
"name": "删除反复出现的提醒性套话",
|
||||
"definition": "当连续句子都用“值得注意的是”“需要特别注意的是”“更值得注意的是”等元话语开头,只是重复强调而没有建立新的信息层级时命中。若只在关键风险首次出现时使用一次,且确实需要提示读者改变注意焦点,则不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "值得注意的是,本周五是最终截止时间。需要特别注意的是,逾期提交将影响下游联调。还值得注意的是,附件命名必须统一。",
|
||||
"good_example": "最终截止时间是本周五,逾期会影响下游联调。附件请按统一格式命名。",
|
||||
"rewrite_hint": "删掉“值得注意的是”等前缀,直接写结论、截止时间、后果或操作要求;需要突出时依靠信息顺序而非重复提醒。",
|
||||
"scope": [
|
||||
"公众号文章",
|
||||
"邮件"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-022",
|
||||
"fp-025"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
},
|
||||
{
|
||||
"id": "rule-avoid-passive-voice-chains",
|
||||
"name": "避免连续使用“被”字被动句",
|
||||
"definition": "当多个相邻分句都采用“对象被执行者处理”的结构,且执行者明确、适合直接充当主语时命中。若执行者未知、不重要,或文本确实需要突出受影响对象,则单个被动句不命中。",
|
||||
"detector": {
|
||||
"type": "llm"
|
||||
},
|
||||
"bad_example": "配置文件会被启动器读取,依赖会被安装脚本自动下载,数据库会被迁移工具初始化,服务会被进程管理器拉起。",
|
||||
"good_example": "启动器读取配置文件,安装脚本自动下载依赖,迁移工具初始化数据库,进程管理器随后拉起服务。",
|
||||
"rewrite_hint": "把工具、模块或责任方移到主语位置,改写为“谁做什么”;只有确需突出承受结果的对象时才保留被动表达。",
|
||||
"scope": [
|
||||
"README 段落",
|
||||
"产品发布稿"
|
||||
],
|
||||
"source_ids": [
|
||||
"fp-023",
|
||||
"fp-026"
|
||||
],
|
||||
"status": "active",
|
||||
"last_confirmed_batch": 3
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Skill 的增量维护:按开放式规则 id 合并、prune 与 SKILL.md 生成。
|
||||
|
||||
防膨胀原则:提炼模型看到当前规则,语义相同时复用稳定 id;本模块按 id 合并来源,
|
||||
而不是再用预置 detector 指纹把新发现筛掉。
|
||||
长期未被新证据确认、或被评估证据推翻的规则归档到 skill/archive/,不再进入
|
||||
SKILL.md。所有合并/激活/归档决定都发生在这里,不交给生成候选的模型。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Set, Tuple
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
SKILL_DIR = ROOT / "skill"
|
||||
|
||||
def rule_signature(rule: Dict[str, Any]) -> str:
|
||||
"""模型在已有规则语义相同时复用 id;该稳定 id 就是合并键。"""
|
||||
return str(rule.get("id", "")).strip().lower()
|
||||
|
||||
|
||||
def merge_rules(
|
||||
existing: List[Dict[str, Any]], candidates: List[Dict[str, Any]]
|
||||
) -> Tuple[List[Dict[str, Any]], Dict[str, Any]]:
|
||||
"""把候选规则合并进现有规则集,返回 (规则集, 合并报告)。"""
|
||||
rules = [dict(rule) for rule in existing]
|
||||
report: Dict[str, Any] = {"added": [], "merged": [], "conflicts": []}
|
||||
for cand in candidates:
|
||||
sig = rule_signature(cand)
|
||||
match = next(
|
||||
(rule for rule in rules if rule_signature(rule) == sig), None
|
||||
)
|
||||
if match is None:
|
||||
rules.append(cand)
|
||||
report["added"].append(cand["id"])
|
||||
continue
|
||||
# 去重合并:并集来源与作用域,保留首次通过 schema 检查的定义与范例。
|
||||
match["source_ids"] = sorted(set(match.get("source_ids", [])) | set(cand.get("source_ids", [])))
|
||||
match["scope"] = sorted(set(match.get("scope", [])) | set(cand.get("scope", [])))
|
||||
report["merged"].append(cand["id"])
|
||||
return rules, report
|
||||
|
||||
|
||||
def prune_rules(
|
||||
rules: List[Dict[str, Any]],
|
||||
*,
|
||||
current_batch: int,
|
||||
idle_batches: int = 2,
|
||||
contradicted_ids: Set[str] | None = None,
|
||||
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
|
||||
"""把长期未被新证据确认、或被证据推翻的规则归档,返回 (存活, 归档)。"""
|
||||
contradicted = contradicted_ids or set()
|
||||
active, archived = [], []
|
||||
for rule in rules:
|
||||
last = rule.get("last_confirmed_batch", current_batch)
|
||||
reason = None
|
||||
if rule["id"] in contradicted:
|
||||
reason = "被评估证据推翻(误伤率过高或与金标集冲突)"
|
||||
elif current_batch - last > idle_batches:
|
||||
reason = f"连续超过 {idle_batches} 批反馈未再被触发"
|
||||
if reason:
|
||||
archived.append({**rule, "status": "archived", "archive_reason": reason})
|
||||
else:
|
||||
active.append(rule)
|
||||
return active, archived
|
||||
|
||||
|
||||
def _describe_detector(detector: Dict[str, Any]) -> str:
|
||||
if detector.get("type") != "llm":
|
||||
return "无效检测器(规则不会激活)"
|
||||
return "LLM judge 语义判定(上线前须通过独立人工金标集校准)"
|
||||
|
||||
|
||||
def render_skill_md(rules: List[Dict[str, Any]]) -> str:
|
||||
"""按 house 风格渲染 SKILL.md:何时加载 + 每条规则的定义/坏例/好例/作用域。"""
|
||||
lines = [
|
||||
"---",
|
||||
"name: ai-style",
|
||||
"description: 中文文案去「AI 味」检查清单,由用户纠正反馈持续提炼而来",
|
||||
"---",
|
||||
"",
|
||||
"# 去 AI 味写作 Skill",
|
||||
"",
|
||||
"## 何时加载",
|
||||
"",
|
||||
"当任务是用中文撰写或改写面向读者的文案(产品发布稿、公众号文章、邮件、README 等),",
|
||||
"或用户反馈文字「AI 味太重」「不像人写的」时,加载本 Skill。",
|
||||
"",
|
||||
"## 使用方式",
|
||||
"",
|
||||
"起草或改写时逐条对照下面的规则自查。每条规则都给出定义、可检查的检测方法、",
|
||||
"坏例与好例;规则只在声明的作用域内生效,作用域之外的文体不要套用。",
|
||||
"",
|
||||
f"## 规则清单(共 {len(rules)} 条)",
|
||||
"",
|
||||
]
|
||||
for i, rule in enumerate(rules, 1):
|
||||
lines += [
|
||||
f"### 规则 {i}:{rule['name']}(`{rule['id']}`)",
|
||||
"",
|
||||
f"- **定义**:{rule['definition']}",
|
||||
f"- **检测方法**:{_describe_detector(rule['detector'])}",
|
||||
f"- **坏例**:{rule.get('bad_example', '')}",
|
||||
f"- **好例**:{rule.get('good_example', '')}",
|
||||
f"- **改写建议**:{rule.get('rewrite_hint', '按定义改写。')}",
|
||||
f"- **作用域**:{'、'.join(rule.get('scope', [])) or '通用'}",
|
||||
f"- **来源反馈**:{', '.join(rule.get('source_ids', [])) or '无'}",
|
||||
"",
|
||||
]
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def write_skill(rules: List[Dict[str, Any]], skill_dir: Path | None = None) -> Path:
|
||||
out_dir = skill_dir or SKILL_DIR
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
path = out_dir / "SKILL.md"
|
||||
path.write_text(render_skill_md(rules), encoding="utf-8")
|
||||
# 同步保存机器可读的规则集,供 evaluate/judge 直接加载。
|
||||
(out_dir / "rules.json").write_text(
|
||||
json.dumps(rules, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def write_archive(archived: List[Dict[str, Any]], skill_dir: Path | None = None) -> Path | None:
|
||||
if not archived:
|
||||
return None
|
||||
out_dir = (skill_dir or SKILL_DIR) / "archive"
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
lines = ["# 已归档规则", ""]
|
||||
for rule in archived:
|
||||
lines += [
|
||||
f"## {rule['name']}(`{rule['id']}`)",
|
||||
f"- 归档原因:{rule.get('archive_reason', '未说明')}",
|
||||
f"- 原定义:{rule['definition']}",
|
||||
f"- 来源反馈:{', '.join(rule.get('source_ids', [])) or '无'}",
|
||||
"",
|
||||
]
|
||||
path = out_dir / "ARCHIVED.md"
|
||||
path.write_text("\n".join(lines), encoding="utf-8")
|
||||
return path
|
||||
@@ -0,0 +1,204 @@
|
||||
"""开放式规则提炼与 LLM-as-a-judge 流程的离线单元测试。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
|
||||
import extract_rules
|
||||
from evaluate import evaluate_rules
|
||||
from judge import calibrate, score_text
|
||||
from llm_client import default_model
|
||||
from skill_manager import merge_rules, prune_rules, render_skill_md, write_skill
|
||||
|
||||
|
||||
def _rule(rule_id="rule-worth-noting", sources=None):
|
||||
return {
|
||||
"id": rule_id,
|
||||
"name": "避免重复强调套话",
|
||||
"definition": "连续用空泛的强调短语引出普通信息时命中;一次必要的风险提示不命中。",
|
||||
"detector": {"type": "llm"},
|
||||
"bad_example": "值得注意的是,甲。值得注意的是,乙。",
|
||||
"good_example": "甲,乙。",
|
||||
"rewrite_hint": "删除重复引导语,直接陈述信息。",
|
||||
"scope": ["邮件"],
|
||||
"source_ids": sources or ["fp-022"],
|
||||
"status": "candidate",
|
||||
}
|
||||
|
||||
|
||||
def test_extract_accepts_open_ended_rule_and_forces_llm_judge(monkeypatch):
|
||||
pairs = [{
|
||||
"id": "fp-022",
|
||||
"scene": "邮件",
|
||||
"before": "值得注意的是,甲。值得注意的是,乙。",
|
||||
"after": "甲,乙。",
|
||||
"correction": "别重复强调",
|
||||
}]
|
||||
payload = {
|
||||
"rules": [{
|
||||
"id": "rule-worth-noting",
|
||||
"name": "避免重复强调套话",
|
||||
"definition": "重复强调普通信息时命中。",
|
||||
"detector": {"type": "regex", "pattern": "值得注意的是"},
|
||||
"bad_example": pairs[0]["before"],
|
||||
"good_example": pairs[0]["after"],
|
||||
"rewrite_hint": "直接说结论。",
|
||||
"scope": ["邮件"],
|
||||
"source_ids": ["fp-022"],
|
||||
}]
|
||||
}
|
||||
|
||||
def fake_chat(messages, **kwargs):
|
||||
assert "不要依赖任何预置的模式清单" in messages[0]["content"]
|
||||
return json.dumps(payload, ensure_ascii=False), {"response": {"id": "resp_test"}}
|
||||
|
||||
monkeypatch.setattr(extract_rules, "chat", fake_chat)
|
||||
candidates, receipt = extract_rules.extract_with_llm(
|
||||
pairs, provider="openai", model="gpt-5.6-sol"
|
||||
)
|
||||
assert receipt["response"]["id"] == "resp_test"
|
||||
assert [candidate["id"] for candidate in candidates] == ["rule-worth-noting"]
|
||||
assert candidates[0]["detector"] == {"type": "llm"}
|
||||
|
||||
|
||||
def test_extract_rejects_hallucinated_sources_and_examples(monkeypatch):
|
||||
pairs = [{
|
||||
"id": "fp-001", "scene": "邮件", "before": "原文", "after": "改文", "correction": "修改"
|
||||
}]
|
||||
payload = {"rules": [
|
||||
{
|
||||
"id": "rule-invented", "name": "n", "definition": "d",
|
||||
"bad_example": "编造坏例", "good_example": "编造好例",
|
||||
"rewrite_hint": "h", "scope": [], "source_ids": ["fp-999"],
|
||||
}
|
||||
]}
|
||||
monkeypatch.setattr(
|
||||
extract_rules, "chat", lambda *args, **kwargs: (json.dumps(payload), {})
|
||||
)
|
||||
candidates, _ = extract_rules.extract_with_llm(pairs, provider="openai")
|
||||
assert candidates == []
|
||||
|
||||
|
||||
def test_merge_uses_stable_rule_id_not_detector_type():
|
||||
first = _rule("rule-worth-noting", ["fp-022"])
|
||||
repeated = _rule("rule-worth-noting", ["fp-025"])
|
||||
novel = _rule("rule-passive-stacking", ["fp-023"])
|
||||
novel["name"] = "避免连续被动句"
|
||||
|
||||
rules, report = merge_rules([], [first])
|
||||
rules, report = merge_rules(rules, [repeated, novel])
|
||||
assert len(rules) == 2
|
||||
assert report["merged"] == ["rule-worth-noting"]
|
||||
assert report["added"] == ["rule-passive-stacking"]
|
||||
merged = next(rule for rule in rules if rule["id"] == "rule-worth-noting")
|
||||
assert set(merged["source_ids"]) == {"fp-022", "fp-025"}
|
||||
|
||||
|
||||
def test_prune_archives_idle_and_contradicted():
|
||||
rules = [
|
||||
{"id": "rule-a", "definition": "a", "source_ids": ["fp-001"], "last_confirmed_batch": 1},
|
||||
{"id": "rule-b", "definition": "b", "source_ids": ["fp-002"], "last_confirmed_batch": 3},
|
||||
{"id": "rule-c", "definition": "c", "source_ids": ["fp-003"], "last_confirmed_batch": 3},
|
||||
]
|
||||
active, archived = prune_rules(
|
||||
rules, current_batch=4, idle_batches=2, contradicted_ids={"rule-c"}
|
||||
)
|
||||
assert [rule["id"] for rule in active] == ["rule-b"]
|
||||
assert {rule["id"] for rule in archived} == {"rule-a", "rule-c"}
|
||||
|
||||
|
||||
def test_calibration_uses_external_labels_linked_by_feedback_source():
|
||||
rule = _rule(sources=["fp-022", "fp-025"])
|
||||
golden = [
|
||||
{
|
||||
"id": "g1", "text": "坏:重复强调",
|
||||
"labels": [{"source_ids": ["fp-022", "fp-025"], "expected": True}],
|
||||
},
|
||||
{
|
||||
"id": "g2", "text": "好:单次必要提示",
|
||||
"labels": [{"source_ids": ["fp-022", "fp-025"], "expected": False}],
|
||||
},
|
||||
]
|
||||
|
||||
def good_judge(rules, texts):
|
||||
return {
|
||||
(item_rule["id"], text["id"]): {
|
||||
"hit": text["text"].startswith("坏"), "evidence": "重复强调"
|
||||
}
|
||||
for item_rule in rules for text in texts
|
||||
}
|
||||
|
||||
result = calibrate(rule, golden, good_judge)
|
||||
assert result["total"] == 2
|
||||
assert result["agreement"] == 1.0
|
||||
assert result["decision"] == "activate"
|
||||
|
||||
|
||||
def test_calibration_rejects_missing_gold_or_bad_judge():
|
||||
rule = _rule()
|
||||
no_coverage = calibrate(rule, [], lambda rules, texts: {})
|
||||
assert no_coverage["decision"] == "reject"
|
||||
|
||||
golden = [{
|
||||
"id": "g1", "text": "应命中",
|
||||
"labels": [{"source_ids": ["fp-022"], "expected": True}],
|
||||
}]
|
||||
bad = calibrate(
|
||||
rule,
|
||||
golden,
|
||||
lambda rules, texts: {(rule["id"], "g1"): {"hit": False, "evidence": ""}},
|
||||
)
|
||||
assert bad["decision"] == "reject"
|
||||
|
||||
|
||||
def test_score_text_batches_all_rules_in_one_judge_call():
|
||||
rules = [_rule("rule-a", ["fp-a"]), _rule("rule-b", ["fp-b"])]
|
||||
calls = []
|
||||
|
||||
def judge(batch_rules, texts):
|
||||
calls.append((batch_rules, texts))
|
||||
return {
|
||||
("rule-a", "sample"): {"hit": True, "evidence": "证据"},
|
||||
("rule-b", "sample"): {"hit": False, "evidence": ""},
|
||||
}
|
||||
|
||||
fired = score_text("待评文本", rules, judge, text_id="sample")
|
||||
assert list(fired) == ["rule-a"]
|
||||
assert fired["rule-a"]["evidence"] == "证据"
|
||||
assert len(calls) == 1 and len(calls[0][0]) == 2
|
||||
|
||||
|
||||
def test_evaluation_maps_human_source_labels_to_dynamic_rule_ids():
|
||||
rules = [_rule("rule-dynamic-name", ["fp-new"])]
|
||||
eval_texts = {
|
||||
"boundary": [{"id": "b1", "text": "坏文本", "expected_sources": ["fp-new"]}],
|
||||
"retention": [{"id": "r1", "text": "好文本"}],
|
||||
}
|
||||
|
||||
def judge(batch_rules, texts):
|
||||
text = texts[0]
|
||||
return {
|
||||
(batch_rules[0]["id"], text["id"]): {
|
||||
"hit": text["id"] == "b1", "evidence": "坏文本" if text["id"] == "b1" else ""
|
||||
}
|
||||
}
|
||||
|
||||
metrics = evaluate_rules(rules, eval_texts, judge)
|
||||
assert metrics["boundary_detection_rate"] == 1.0
|
||||
assert metrics["retention_harm_rate"] == 0.0
|
||||
|
||||
|
||||
def test_skill_md_describes_llm_judge_and_structure(tmp_path):
|
||||
rules = [_rule()]
|
||||
markdown = render_skill_md(rules)
|
||||
assert "LLM judge 语义判定" in markdown
|
||||
assert "预置" not in markdown
|
||||
for section in ("**定义**", "**坏例**", "**好例**", "**作用域**", "**检测方法**"):
|
||||
assert section in markdown
|
||||
path = write_skill(rules, tmp_path)
|
||||
assert path.exists()
|
||||
assert json.loads((tmp_path / "rules.json").read_text(encoding="utf-8"))[0]["id"]
|
||||
|
||||
|
||||
def test_openai_default_is_gpt_5_6_sol():
|
||||
assert default_model("openai") == "gpt-5.6-sol"
|
||||
@@ -0,0 +1,10 @@
|
||||
# 无编号补充案例 结果(2026-08-07)
|
||||
|
||||
## 执行结果
|
||||
|
||||
- 16 项机制测试全部通过。
|
||||
- 离线路径:21 条候选规则合并为 8 条;未完成任务集检出 8/8;正常文本保留集误伤 0/8;金标校准 10/10。
|
||||
- 第一次真实 LLM 运行直接采用模型规则,结果为检出 0/8、误伤 7/8,说明模型提出的规则不能未经检查直接进入 Skill。
|
||||
- 修改后,模型外部代码只接纳能映射到受支持检测器的候选;无法映射时回退到确定性规则。最终真实路径为 21 条候选合并成 8 条,检出 8/8、误伤 0/8、金标校准 10/10,整体验收通过。
|
||||
|
||||
最终证据见 `real_20260807T160547Z/evidence.json`。该结果说明,LLM 适合提出写作规则候选,但规则合并、检测器选择、校准和发布仍应由模型外部代码控制。
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user