ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,426 @@
|
||||
# AndroidWorld T3A Evaluation Notes / AndroidWorld T3A 评估分析笔记
|
||||
|
||||
> Companion material for *AI Agents in Depth*, Chapter 7 — **Experiment 7-12: Evaluate and improve on AndroidWorld**.
|
||||
> 配套《深入理解 AI Agent》第 7 章 **实验 7-12 ★★★：AndroidWorld 的评估和改进**。
|
||||
|
||||
← [Chapter 7 index / 返回第 7 章目录](../README.md) · 📖 [Read the chapter / 读本章正文](../../book/chapter7.md)（[EN](../../book-en/chapter7.md)）
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
### What this directory is
|
||||
|
||||
This folder is **not** a copy of the [AndroidWorld](https://github.com/google-research/android_world) benchmark codebase. It contains **evaluation artifacts and analysis notes** for a **T3A** (Text-only / accessibility-tree style mobile agent) run plus a companion runner that executes the book's full **diagnose → hypothesize → experiment → decide → iterate** loop against a separate, unmodified upstream checkout.
|
||||
|
||||
| Path | Role |
|
||||
| --- | --- |
|
||||
| [`t3a_summary.md`](t3a_summary.md) | High-level report: per-task outcomes + capability-tag × difficulty matrix, strengths/weaknesses |
|
||||
| [`t3a_failed_analysis.md`](t3a_failed_analysis.md) | Failure taxonomy with root-cause write-ups (transcription, complex UI, math/counting, etc.) |
|
||||
| [`t3a.md`](t3a.md) | Full step traces for runs (including successes): per-step `Action` / `Reason` / `Summary` records |
|
||||
| [`t3a_failed.md`](t3a_failed.md) | Step traces focused on failed tasks (useful for root-cause replay) |
|
||||
| [`experiment_core.py`](experiment_core.py) | Evidence aggregation, success/cost decisions, strict completion gates, and five-stage report rendering |
|
||||
| [`run_controlled_experiment.py`](run_controlled_experiment.py) | Real AndroidWorld control/treatment and candidate-rerun runner; no mock fallback |
|
||||
| [`merge_candidate_shards.py`](merge_candidate_shards.py) | Strict merger for independent trial shards; rejects overlap, provenance drift, missing reference setup, and duplicate episodes |
|
||||
| [`test_experiment.py`](test_experiment.py) | Offline checks for redaction, cost decisions, and non-overclaiming gates |
|
||||
| [`requirements.txt`](requirements.txt) | Installs the adjacent upstream checkout plus the OpenAI-compatible API client |
|
||||
| `validation/` | Machine-readable real-run evidence and the reports generated from it |
|
||||
|
||||
To execute the controlled loop, first clone and configure upstream AndroidWorld (see [Reproduce the benchmark](#reproduce-the-benchmark-optional) below). The large `t3a*.md` files remain reading/analysis inputs; the runner and `validation/` artifacts are the executable evidence layer.
|
||||
|
||||
### Background: AndroidWorld + T3A
|
||||
|
||||
- **AndroidWorld** evaluates agents that complete real tasks on Android apps (navigation, UI interaction, multi-app flows). Tasks are often **parameterized templates** (anti-contamination, diverse instances) and are scored by **final UI / environment state**, not by matching a fixed action sequence.
|
||||
- The notes here analyze a **T3A** agent run (logged as `t3a_claude4_sonnet` in the summary tables): the agent plans from UI state (accessibility tree / similar structured observations) and issues discrete actions (`open_app`, `click`, `status`, …).
|
||||
|
||||
### Snapshot results (from the included report)
|
||||
|
||||
Numbers below come from [`t3a_summary.md`](t3a_summary.md) (116 tasks, one trial each; agent `t3a_claude4_sonnet`, run on 2025-07-02):
|
||||
|
||||
| Metric | Value (approx.) |
|
||||
| --- | --- |
|
||||
| Overall success rate | **~88%** |
|
||||
| Fail rate | **~12%** |
|
||||
| Mean episode length (successful) | **~13.5** steps |
|
||||
|
||||
**Where it succeeds:** structured, linear flows—camera/clock/contacts, file ops, Markor notes, many system toggles, multi-app and short-term memorization on easier tags.
|
||||
|
||||
**Where it fails (clustered):** SMS reply edge cases, Wi-Fi / combined connectivity, Tasks app queries, VLC playlists, and tasks needing **transcription**, **math/counting**, **complex UI understanding**, **information retrieval**, or **requires_setup**.
|
||||
|
||||
### Capability portrait
|
||||
|
||||
From the tag × difficulty matrix in the summary:
|
||||
|
||||
| Strengths | Critical weaknesses |
|
||||
| --- | --- |
|
||||
| `multi_app`, `memorization` (easy ~1.0) | `transcription` (~0.0) |
|
||||
| Decent `search` on medium | `math_counting` (easy ~0.0) |
|
||||
| Reliable on standard UI flows | `complex_ui_understanding`, `information_retrieval` (very low) |
|
||||
| | `requires_setup` (easy ~0.0) |
|
||||
|
||||
**One-line portrait:** a strong “operator” on standard linear tasks; weak as a “thinker” when deep vision, counting, non-standard UI, or fragile multi-step state is required.
|
||||
|
||||
### Failure categories (see detailed analysis)
|
||||
|
||||
Condensed from [`t3a_failed_analysis.md`](t3a_failed_analysis.md):
|
||||
|
||||
1. **Transcription** — Navigates gallery/VLC correctly but cannot OCR image/video text; may invent plausible data and “fake success.”
|
||||
2. **Complex UI** — Sees widgets but lacks a mental model of control logic (e.g. timer digit entry loops after detecting invalid `63s`).
|
||||
3. **App first-run overhead** — Tutorials / permission wizards burn step budget before the real goal.
|
||||
4. **Math / counting** — Can scroll and “see” list items but fails to filter + count or sum durations under step limits.
|
||||
5. **Retrieval + planning** — Dense UIs (calendar grid), multi-delete with state tracking; inefficient recovery (day-by-day instead of reselecting).
|
||||
|
||||
Many failures surface as **max steps** (`Agent did not indicate task is done. Reached max number of steps.`)—symptom of loops, inefficient recovery, or missing perception, not merely “too few steps.”
|
||||
|
||||
### How to use this material (Experiment 7-12)
|
||||
|
||||
Follow the book’s five-step loop:
|
||||
|
||||
1. **Diagnose** — Cross the per-task table with the capability matrix; map surface failures to capability gaps.
|
||||
2. **Hypothesize** — Layered ideas (surface → mid → deep), e.g. settings navigation hints, fix multimodal input pipe, add UI tree + screenshot, stronger vision model, conditional thinking for count tasks.
|
||||
3. **Experiment** — Cheap ablations first; measure success **and** latency/cost side effects.
|
||||
4. **Decide** — Deploy high ROI fixes; reject global “always think” if only a small tag set benefits.
|
||||
5. **Iterate** — Re-run the suite; new residual failures become the next report.
|
||||
|
||||
### Executed controlled loop (2026-07-29 to 2026-08-04)
|
||||
|
||||
The companion runner now makes the book's loop executable while leaving the adjacent upstream checkout unmodified. It records the real AndroidWorld evaluator reward, explicit agent termination, actions, steps, wall time, LLM calls, token use, estimated token cost, exact model/runtime provenance, and the installed version of every required app after every episode. A bounded final analysis is requested from the same real configured LLM; the JSON evidence, not that prose, remains authoritative.
|
||||
|
||||
The first low-cost phase tested **H1**, a Wi-Fi navigation/state-verification guideline, against the untouched upstream T3A prompt. Its four matched task pairs completed with no runtime errors:
|
||||
|
||||
| Phase 1 result | Control | H1 treatment |
|
||||
| --- | ---: | ---: |
|
||||
| Successful episodes | 1 / 4 | 1 / 4 |
|
||||
| Mean evaluator reward | 0.50 | 0.50 |
|
||||
| Mean latency | 233.47 s | 156.98 s |
|
||||
| Input + output tokens | 442,619 | 210,039 |
|
||||
|
||||
H1 reduced observed latency and token use but produced **no paired success gain**, so it was not promoted. See [phase-1 evidence](validation/paired_wifi_api35_20260729/evidence.json) and its [report](validation/paired_wifi_api35_20260729/report.md).
|
||||
|
||||
The residual traces exposed an API-35 observation issue: AndroidWorld's gRPC accessibility feed often returned only status-bar elements after opening the Internet panel, while an independent UIAutomator dump showed the full real Settings hierarchy. **H5** therefore tests a middle-layer input-pipeline change: upstream's `A11yMethod.UIAUTOMATOR` versus the gRPC forwarder, with the same base T3A prompt in both arms. This is an AndroidWorld-supported observation path selected from the companion runner, not an edit to upstream source.
|
||||
|
||||
H5 recovered the four-task slice from `1/4` control successes to `4/4` UIAutomator successes with no paired regression and a `0.788×` latency ratio. It was still restricted because its `2.498×` mean-token ratio exceeded the `1.5×` guardrail. The resulting cost-refinement hypothesis **H5C** keeps real UIAutomator observations/actions/evaluators but filters non-semantic container elements before T3A formats the prompt.
|
||||
|
||||
The completed H5C paired run preserved `4/4` successes in both arms. Compact UIAutomator used `70,557.5` mean tokens versus `139,439.5` for raw UIAutomator (`0.506×`) and `99.18s` versus `101.20s` mean latency (`0.980×`). It therefore passed the stricter H5C subset gate and became eligible only for a full-suite candidate rerun. At that stage it was **not** deployment approval and did not complete Experiment 7-12's 116-task × five-seed requirement. See the [H5C evidence](validation/paired_h5c_compact_api35_20260729/evidence.json) and [report](validation/paired_h5c_compact_api35_20260729/report.md).
|
||||
|
||||
The final reference-environment campaign subsequently completed all five gates: 580/580 unique episodes, 116 tasks × trials 1–5, zero runtime errors, official setup completed, and the same 24/24 required package versions on every Pixel 6/API-33 shard. The canonical [merged evidence](validation/candidate_h5c_api33_local_qwen_20260804/evidence.json) and [generated report](validation/candidate_h5c_api33_local_qwen_20260804/report.md) record:
|
||||
|
||||
| Full candidate result | Value |
|
||||
| --- | ---: |
|
||||
| Strict T3A successes | 26 / 580 (`4.4828%`) |
|
||||
| Evaluator rewards | 77 full (`1.0`) + 1 partial (`0.5`) |
|
||||
| Mean evaluator reward | `0.133621` |
|
||||
| Mean steps / LLM calls | `9.672414` / `18.998276` |
|
||||
| Mean latency | `109.860845s` |
|
||||
| Mean tokens | `169,069.563793` |
|
||||
| Total input / output tokens | `97,384,410` / `675,937` |
|
||||
| Estimated API cost | `$0.00` (local inference) |
|
||||
|
||||
Strict success follows the upstream minimal-runner rule: the final evaluator state must equal `1.0` **and** the agent must explicitly declare completion. This is why the 26 strict successes are fewer than the 77 full-reward final states; one additional episode received partial reward `0.5`. Evaluator failures were retained as experimental outcomes; they were not rerun. The merged evidence sets `scope.direct_episode_gate_completed`, `scope.full_suite_completed`, `scope.manuscript_five_seed_gate_completed`, and `experiment_complete` to `true`, but `decision.deployment_approved` remains `false` because the observed result is poor and there is no valid full-suite control comparison.
|
||||
|
||||
The candidate used local `qwen2.5-7b-instruct-local` revision `a09a35458c702b33eeacc393d103063234e8bc28`, served by vLLM 0.19.0 on an NVIDIA RTX PRO 6000 Blackwell 96 GB. The H5C paired source used `doubao-seed-1-6-250615`. Consequently, this campaign completes the direct execution/evidence requirement for the promoted observation treatment, but it is **not** a same-model extension and establishes neither comparative uplift nor noninferiority.
|
||||
|
||||
The following compatibility treatments are explicitly part of the result boundary:
|
||||
|
||||
1. `ContactsNewContactDraft`: UIAutomator does not populate `state.forest`, so `state.ui_elements` is passed to the unchanged official contact predicate.
|
||||
2. Clipper foreground race: the unchanged clipboard get/set operation is retried once after one second only for the exact documented foreground-access error.
|
||||
3. `SimpleSmsReplyMostRecent`: the inbox is polled for five additional seconds; if emulator-console injection still leaves it empty, the exact last injected address/body is inserted into the same SMS SQLite database that upstream already clears, then the unchanged evaluator query runs.
|
||||
4. `RetroPlayingQueue`: only the exact missing `playing_queue` table error from the pinned APK maps to an empty observed queue; the unchanged exact-queue predicate then records evaluator failure.
|
||||
5. Native 32,768-token context overflow: only after a real provider context error, a deterministic retry retains at most 12,000 characters from the ends of the action-selection UI description or 6,000 from each before/after summary UI description. The goal, history, action, reason, guidance, output format, original retained UI indices, and per-episode truncation/removal counters remain intact. There were 63 such truncations, removing 7,390,498 UI-description characters in total.
|
||||
6. Runtime-error retries reuse the exact parameters saved in the failed checkpoint, preventing upstream generator drift from changing the task. Completed checkpoints remain canonical when later parameter regeneration drifts, and resume on the same live emulator preserves the completed setup state rather than rerunning setup.
|
||||
|
||||
Phase 1 command (shown for reproducibility):
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 GRPC_VERBOSITY=ERROR \
|
||||
python run_controlled_experiment.py \
|
||||
--mode paired --hypothesis H1 \
|
||||
--tasks SystemWifiTurnOff,SystemWifiTurnOffVerify,SystemWifiTurnOn,SystemWifiTurnOnVerify \
|
||||
--trials 1 --seed 42 --model-seed 42 --max-steps 10 \
|
||||
--transition-pause 0.5 --skip-device-time \
|
||||
--output-dir validation/paired_wifi_api35_20260729
|
||||
```
|
||||
|
||||
Phase 2 command:
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 GRPC_VERBOSITY=ERROR \
|
||||
python run_controlled_experiment.py \
|
||||
--mode paired --hypothesis H5 \
|
||||
--source-phase1-evidence validation/paired_wifi_api35_20260729/evidence.json \
|
||||
--tasks SystemWifiTurnOff,SystemWifiTurnOffVerify,SystemWifiTurnOn,SystemWifiTurnOnVerify \
|
||||
--trials 1 --seed 42 --model-seed 42 --max-steps 10 \
|
||||
--transition-pause 0.5 --skip-device-time \
|
||||
--output-dir validation/paired_h5_a11y_api35_20260729
|
||||
```
|
||||
|
||||
Cost-refinement command:
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 GRPC_VERBOSITY=ERROR \
|
||||
python run_controlled_experiment.py \
|
||||
--mode paired --hypothesis H5C \
|
||||
--source-phase1-evidence validation/paired_wifi_api35_20260729/evidence.json \
|
||||
--source-phase2-evidence validation/paired_h5_a11y_api35_20260729/evidence.json \
|
||||
--tasks SystemWifiTurnOff,SystemWifiTurnOffVerify,SystemWifiTurnOn,SystemWifiTurnOnVerify \
|
||||
--trials 1 --seed 42 --model-seed 42 --max-steps 10 \
|
||||
--transition-pause 0.5 --skip-device-time \
|
||||
--output-dir validation/paired_h5c_compact_api35_REPRODUCE
|
||||
```
|
||||
|
||||
The H1/H5 decision gate requires at least four complete pairs, a positive net success delta, no paired regression, and no more than `1.5×` mean latency or token use. H5C instead requires all four compact-treatment pairs to succeed, no regression, at most `1.5×` latency, and at most `0.75×` raw-UIAutomator tokens. Passing either gate permits only a **candidate rerun**, never deployment. Candidate reruns must supply the actual promoted paired-evidence file; a run ID string alone is insufficient. Full experiment completion additionally requires 580 direct candidate records: all 116 tasks × five distinct trial seeds, with no episode error.
|
||||
|
||||
For parallel reference-environment execution, keep `--trials 5` and assign one or more 1-based trials with `--trial-indices`; use a distinct `--execution-shard` label for every emulator. Each shard remains incomplete by itself. `merge_candidate_shards.py` accepts completion only when the shard union contains trials 1–5 exactly once for every task, with the same model, source decision, API-33 environment, completed upstream app setup, and identical required-app versions. Evaluator failures are direct results and are retained; only runtime-error records are eligible for `--resume --retry-errors`.
|
||||
|
||||
The historical paired slices used a Pixel 9 Pro API-35 AVD without the complete third-party app bundle; `--skip-device-time` was restricted to those time-independent Wi-Fi evaluators. The full candidate campaign instead uses five isolated Pixel 6/API-33 emulators, the completed upstream setup procedure, and the same 24 required app packages and versions on every shard. The evidence keeps those two environments separate rather than treating the reference-environment candidate run as a same-environment extension of the API-35 slice.
|
||||
|
||||
Concrete example trajectories for root-cause practice:
|
||||
|
||||
| Example task | File | Lesson |
|
||||
| --- | --- | --- |
|
||||
| `ExpenseAddMultipleFromGallery` | failed analysis + `t3a_failed.md` | OCR / multimodal gap; fabricated expenses |
|
||||
| `ClockTimerEntry` | same | No durable UI model; repeats bad digit sequence |
|
||||
| `MarkorTranscribeVideo` | same | Video navigation OK, content blind |
|
||||
| `SportsTracker*Count*` / duration | same | Perception without arithmetic |
|
||||
| Successful short flows (`CameraTakeVideo`, stopwatch) | `t3a.md` | What “good” step traces look like |
|
||||
|
||||
### Directory layout
|
||||
|
||||
```text
|
||||
chapter7/android-world/
|
||||
├── README.md # This file
|
||||
├── experiment_core.py # Evidence, decisions, completion gates, report renderer
|
||||
├── run_controlled_experiment.py # Real AndroidWorld paired/candidate runner
|
||||
├── merge_candidate_shards.py # Validates and merges parallel trial shards
|
||||
├── test_experiment.py # Focused offline integrity tests
|
||||
├── requirements.txt # Adjacent upstream + API client dependency
|
||||
├── t3a_summary.md # Aggregated metrics + capability matrix
|
||||
├── t3a_failed_analysis.md # Failure taxonomy & root causes
|
||||
├── t3a.md # Full (large) run logs
|
||||
├── t3a_failed.md # Failed-task run logs
|
||||
└── validation/ # Real evidence.json + generated report.md artifacts
|
||||
```
|
||||
|
||||
### Reproduce the benchmark (optional)
|
||||
|
||||
The controlled runner expects a separate adjacent AndroidWorld checkout (the current workspace uses `chapter7/android_world`) plus its configured emulator and model credential. For a clean reproduction:
|
||||
|
||||
1. Clone [google-research/android_world](https://github.com/google-research/android_world) (or the fork your course materials specify).
|
||||
2. Provide an Android emulator / device environment as required by that project.
|
||||
3. Install the companion requirements in that environment, set the selected provider credential (the default is `ARK_API_KEY`), and run one of the commands above. For the retained local run, the OpenAI-compatible endpoint was `http://127.0.0.1:18111/v1` on the host (`http://host.docker.internal:18111/v1` from the emulator containers). Set `LOCAL_API_KEY` only in the launching process environment; the runner records only its variable name and never persists its value.
|
||||
4. Provision the exact upstream Pixel 6 / API-33 apps before attempting `--full-suite`; do not use the API-35 Wi-Fi-only deviations for the full benchmark.
|
||||
|
||||
Run one trial per isolated emulator, changing `<N>`, ports, and output directory for shards 1–5:
|
||||
|
||||
```bash
|
||||
python run_controlled_experiment.py \
|
||||
--android-world-checkout /workspace/android_world \
|
||||
--mode candidate-rerun --hypothesis H5C \
|
||||
--source-paired-evidence validation/paired_h5c_compact_api35_20260729/evidence.json \
|
||||
--full-suite --trials 5 --trial-indices <N> --execution-shard shard-<N>-of-5 \
|
||||
--seed 42 --model-seed 42 --max-steps 10 --transition-pause 0.5 \
|
||||
--provider local-vllm --model qwen2.5-7b-instruct-local \
|
||||
--base-url http://host.docker.internal:18111/v1 --api-key-env LOCAL_API_KEY \
|
||||
--max-model-tokens 1024 --model-timeout-s 90 --model-retries 2 \
|
||||
--model-source local_gpu \
|
||||
--model-revision a09a35458c702b33eeacc393d103063234e8bc28 \
|
||||
--model-runtime vllm-0.19.0 \
|
||||
--accelerator NVIDIA_RTX_PRO_6000_Blackwell_96GB \
|
||||
--perform-emulator-setup \
|
||||
--output-dir validation/candidate_h5c_api33_local_shard<N>
|
||||
```
|
||||
|
||||
Merge only after all five shards have completed. The merger rejects overlap, missing trials, provenance/setup/app-version drift, runtime errors, and duplicate task/trial keys:
|
||||
|
||||
```bash
|
||||
python merge_candidate_shards.py \
|
||||
validation/candidate_h5c_api33_local_shard{1,2,3,4,5}/evidence.json \
|
||||
--source-paired-evidence validation/paired_h5c_compact_api35_20260729/evidence.json \
|
||||
--output-dir validation/candidate_h5c_api33_local_qwen_20260804
|
||||
```
|
||||
|
||||
Reading order if you only study the notes: **`t3a_summary.md` → `t3a_failed_analysis.md` → sample episodes in `t3a_failed.md` / `t3a.md`**.
|
||||
|
||||
### Related chapter projects
|
||||
|
||||
| Project | Relation |
|
||||
| --- | --- |
|
||||
| Upstream `android_world` (external) | Runnable benchmark environment |
|
||||
| [model-benchmark](../model-benchmark/) | API latency / reliability dimensions of “evaluation” |
|
||||
| [elo-leaderboard](../elo-leaderboard/) | Pairwise ranking instead of absolute task success |
|
||||
| [public-health-reporting-eval](../public-health-reporting-eval/) | Another structured eval harness in-repo |
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 本目录是什么
|
||||
|
||||
本目录**不是** [AndroidWorld](https://github.com/google-research/android_world) 基准的源码拷贝。它既包含 **T3A** 类移动 Agent 的**评估产物与分析笔记**，也包含一个连接独立、未修改上游 checkout 的配套 runner，用于真实执行书中的完整闭环：**诊断 → 假设 → 实验 → 决策 → 迭代**（对应**实验 7-12**）。
|
||||
|
||||
| 路径 | 作用 |
|
||||
| --- | --- |
|
||||
| [`t3a_summary.md`](t3a_summary.md) | 总览：逐任务结果 + 能力标签 × 难度矩阵、优势与短板 |
|
||||
| [`t3a_failed_analysis.md`](t3a_failed_analysis.md) | 失败分类与根因（转录、复杂 UI、数学/计数等） |
|
||||
| [`t3a.md`](t3a.md) | 完整逐步轨迹（含成功案例）：每步记录 `Action` / `Reason` / `Summary` |
|
||||
| [`t3a_failed.md`](t3a_failed.md) | 失败任务轨迹（适合回放根因） |
|
||||
| [`experiment_core.py`](experiment_core.py) | 证据聚合、成功/成本决策、严格完成门槛与五阶段报告渲染 |
|
||||
| [`run_controlled_experiment.py`](run_controlled_experiment.py) | 真实 AndroidWorld 对照/处理与候选重跑 runner；没有 mock fallback |
|
||||
| [`merge_candidate_shards.py`](merge_candidate_shards.py) | 严格合并独立 trial 分片，并拒绝重叠、来源漂移、缺少参考环境 setup 或重复 episode |
|
||||
| [`test_experiment.py`](test_experiment.py) | 脱机检查脱敏、成本决策与防止夸大结论的门槛 |
|
||||
| [`requirements.txt`](requirements.txt) | 安装相邻上游 checkout 与 OpenAI 兼容 API 客户端 |
|
||||
| `validation/` | 真实运行的机器可读证据及据此生成的报告 |
|
||||
|
||||
若要**自己跑**基准，请按上游仓库克隆与配置（见下文[复现基准](#复现基准可选)）。本目录以**阅读与分析**为主。
|
||||
|
||||
### 背景：AndroidWorld 与 T3A
|
||||
|
||||
- **AndroidWorld**：在真实 Android 应用上评测 Agent 的导航、UI 交互与多应用任务。任务多为**参数化模板**（降低泄漏、增加多样性），按**最终 UI / 环境状态**判分，而不是比对固定操作序列。
|
||||
- 笔记分析的是一次 **T3A** 运行（摘要表中记为 `t3a_claude4_sonnet`）：主要依据 UI 状态（无障碍树等结构化观察）规划，并输出离散动作（`open_app`、`click`、`status` 等）。
|
||||
|
||||
### 结果快照（来自随附报告）
|
||||
|
||||
数据摘自 [`t3a_summary.md`](t3a_summary.md)（116 个任务，每任务 1 次 trial；Agent 为 `t3a_claude4_sonnet`，运行于 2025-07-02）：
|
||||
|
||||
| 指标 | 约值 |
|
||||
| --- | --- |
|
||||
| 总体成功率 | **~88%** |
|
||||
| 失败率 | **~12%** |
|
||||
| 成功任务平均步数 | **~13.5** |
|
||||
|
||||
**擅长：** 结构化、线性流程——相机/时钟/联系人、文件操作、Markor 笔记、多数系统开关；在较简单标签上，跨应用与短时记忆表现好。
|
||||
|
||||
**短板（失败扎堆）：** 短信回复边缘、Wi-Fi/组合连接、Tasks 查询、VLC 播放列表，以及需要**转录**、**数学/计数**、**复杂 UI 理解**、**信息检索**、**requires_setup** 的任务。
|
||||
|
||||
### 能力画像
|
||||
|
||||
| 优势 | 关键短板 |
|
||||
| --- | --- |
|
||||
| `multi_app`、`memorization`（easy ~1.0） | `transcription`（~0.0） |
|
||||
| `search` 在 medium 上较好 | `math_counting`（easy ~0.0） |
|
||||
| 标准 UI 流程稳定 | `complex_ui_understanding`、`information_retrieval` 很低 |
|
||||
| | `requires_setup`（easy ~0.0） |
|
||||
|
||||
**一句话：** 在标准线性任务上是高效的「操作手」；在深度视觉、计数、非标 UI、脆弱多步状态维护上，「思考者」能力明显不足。
|
||||
|
||||
### 失败类别（详见分析文）
|
||||
|
||||
浓缩自 [`t3a_failed_analysis.md`](t3a_failed_analysis.md)：
|
||||
|
||||
1. **转录失败** — 图库/VLC 导航正确，但无法 OCR 图/视频文字；可能捏造合理数据「假装成功」。
|
||||
2. **复杂 UI** — 看得见控件，却没有控件逻辑的心智模型（如计时器输入，发现 `63s` 非法后仍重复错误序列）。
|
||||
3. **应用首次启动开销** — 教程/权限向导吃掉步数预算。
|
||||
4. **数学/计数** — 能滚动「看见」列表，却完不成筛选+计数或时长求和。
|
||||
5. **检索与规划** — 密集日历格、去重删除的状态维护；恢复策略低效（逐天点而不是回月视图重选）。
|
||||
|
||||
大量失败以**步数耗尽**呈现（`Reached max number of steps`）——根因往往是循环、低效恢复或感知缺失，而不仅是「上限太小」。
|
||||
|
||||
### 如何使用（实验 7-12）
|
||||
|
||||
按书中五步闭环：
|
||||
|
||||
1. **诊断** — 交叉逐任务表与能力矩阵，把表面失败映射到能力缺陷。
|
||||
2. **假设** — 表层 → 中层 → 深层（如设置导航提示、修复多模态输入管道、截图+UI 树、更强视觉模型、仅对计数任务开思考）。
|
||||
3. **实验** — 先做低成本对照；同时量成功率与时延/成本副作用。
|
||||
4. **决策** — 优先部署高 ROI；拒绝为少数标签让全局任务承担数倍延迟/成本。
|
||||
5. **迭代** — 重跑全集，新失败模式成为下一轮起点。
|
||||
|
||||
### 已执行的对照闭环（2026-07-29 至 2026-08-04）
|
||||
|
||||
配套 runner 会逐 episode 记录真实 AndroidWorld evaluator reward、Agent 是否显式结束、动作、步数、耗时、LLM 调用、token、估算 token 成本、模型/运行时来源，以及每个必需 App 的安装版本。运行结束后，同一个真实配置模型会对聚合证据做受约束分析；JSON 证据始终是权威来源，LLM 文本不能覆盖它。
|
||||
|
||||
第一阶段测试低成本表层假设 **H1**：对照组使用原始 T3A prompt，处理组只增加 Wi-Fi 导航和最终状态确认指南。四个配对任务全部正常结束；两组均只成功 `1/4`，平均 evaluator reward 都是 `0.50`。处理组平均延迟由 `233.47s` 降至 `156.98s`，输入+输出 token 由 `442,619` 降至 `210,039`，但**没有配对成功增益**，因此不晋级。证据见 [phase-1 evidence](validation/paired_wifi_api35_20260729/evidence.json) 与 [report](validation/paired_wifi_api35_20260729/report.md)。
|
||||
|
||||
残余轨迹暴露了 API 35 观察兼容问题：打开 Internet 面板后，gRPC 无障碍树经常只剩状态栏元素，而独立 UIAutomator dump 能看到完整的真实 Settings 层级。因此第二阶段中层假设 **H5** 对比 gRPC forwarder 与 AndroidWorld 上游已有的 `A11yMethod.UIAUTOMATOR`，两组保持相同原始 T3A prompt、参数、seed、模型和 evaluator。H5 将该四任务切片从对照组 `1/4` 成功提升到 UIAutomator 的 `4/4`，但平均 token 比达到 `2.498×`，超过 `1.5×` 门槛，因此没有晋级。
|
||||
|
||||
随后执行的成本优化假设 **H5C** 对比原始 UIAutomator 与过滤非语义容器节点的紧凑 UIAutomator。两组都保持 `4/4` 成功；紧凑组平均 token 从 `139,439.5` 降至 `70,557.5`（`0.506×`），平均延迟从 `101.20s` 降至 `99.18s`（`0.980×`）。该结果通过了 H5C 的四任务候选门槛；在当时它仅表示可以在完整参考环境中进行候选重跑，尚不是部署批准，也尚未完成 116 任务 × 5 轮要求。证据见 [H5C JSON](validation/paired_h5c_compact_api35_20260729/evidence.json) 与 [报告](validation/paired_h5c_compact_api35_20260729/report.md)；英文部分列出了精确复现命令。
|
||||
|
||||
最终参考环境 campaign 已完成全部五项执行门槛：580/580 条唯一 episode（116 任务 × 1–5 轮）、零运行时错误、官方 setup 完成，且五个 Pixel 6/API-33 分片均安装相同版本的 24/24 个必需应用。权威结果见[合并 evidence](validation/candidate_h5c_api33_local_qwen_20260804/evidence.json)与[生成报告](validation/candidate_h5c_api33_local_qwen_20260804/report.md)：
|
||||
|
||||
| 完整候选结果 | 数值 |
|
||||
| --- | ---: |
|
||||
| 严格 T3A 成功 | 26 / 580（`4.4828%`） |
|
||||
| Evaluator reward | 77 条满分（`1.0`）+ 1 条部分分（`0.5`） |
|
||||
| 平均 evaluator reward | `0.133621` |
|
||||
| 平均步数 / LLM 调用 | `9.672414` / `18.998276` |
|
||||
| 平均延迟 | `109.860845s` |
|
||||
| 平均 token | `169,069.563793` |
|
||||
| 总输入 / 输出 token | `97,384,410` / `675,937` |
|
||||
| 估算 API 成本 | `$0.00`（本地推理） |
|
||||
|
||||
严格成功遵循上游 minimal runner 规则：最终 evaluator 必须为 `1.0`，并且 Agent 必须显式宣告完成。因此 26 条严格成功少于 77 条 evaluator 满分的最终状态；另有 1 条部分 reward `0.5`。Evaluator 失败作为实验结果被完整保留，没有重跑。合并证据中 `scope.direct_episode_gate_completed`、`scope.full_suite_completed`、`scope.manuscript_five_seed_gate_completed` 与 `experiment_complete` 均为 `true`；但由于观测成绩很低，且没有有效的全集对照，`decision.deployment_approved` 仍为 `false`。
|
||||
|
||||
候选运行使用本地 `qwen2.5-7b-instruct-local`（revision `a09a35458c702b33eeacc393d103063234e8bc28`），由 NVIDIA RTX PRO 6000 Blackwell 96 GB 上的 vLLM 0.19.0 提供服务；H5C 配对源则使用 `doubao-seed-1-6-250615`。因此本 campaign 完成了已晋级观察方案的直接执行/证据要求，但**不是**同模型延续，不能证明比较提升或非劣性。
|
||||
|
||||
本结果包含以下明示的兼容处理边界：
|
||||
|
||||
1. `ContactsNewContactDraft`：UIAutomator 不填充 `state.forest`，因此把 `state.ui_elements` 传给未改动的官方联系人 predicate。
|
||||
2. Clipper 前台竞态：仅对文档中精确的前台访问错误，在一秒后重试一次未改动的剪贴板读/写操作。
|
||||
3. `SimpleSmsReplyMostRecent`：多轮询收件箱五秒；若 emulator console 注入后仍为空，将最后一次注入的精确地址/正文写入上游本就直接清理的同一 SMS SQLite 数据库，然后运行未改动的 evaluator query。
|
||||
4. `RetroPlayingQueue`：仅将固定 APK 缺失 `playing_queue` 表的精确错误映射为空观察队列，再由未改动的精确队列 predicate 记录 evaluator 失败。
|
||||
5. 原生 32,768-token 上下文溢出：仅在真实 provider context error 之后，确定性重试最多保留 action-selection UI 描述两端共 12,000 个字符，或 before/after summary UI 描述各 6,000 个字符；goal、history、action、reason、guidance、输出格式、保留 UI 的原索引与逐 episode 计数都保留。共发生 63 次截断，移除 7,390,498 个 UI 描述字符。
|
||||
6. 运行时错误重试复用失败 checkpoint 中保存的精确参数，避免上游生成器漂移改变任务；若后续重生成参数发生漂移，已完成 checkpoint 仍为权威记录。同一存活 emulator 上 resume 时保留已完成的 setup 状态，不重复 setup。
|
||||
|
||||
H1/H5 决策门槛要求至少四个完整 pair、净成功增益为正、零配对退化，且平均延迟与 token 都不超过对照的 `1.5×`。H5C 则要求四个紧凑处理组全部成功、零退化、延迟不超过 `1.5×`、token 不超过原始 UIAutomator 的 `0.75×`。通过门槛只允许进入**候选重跑**，绝不等于部署。候选重跑必须提供真实晋级 pair 的 evidence 文件；仅提供 run ID 不够。实验完成还必须有 580 条直接候选记录，即 116 个任务 × 五个不同 trial seed，且没有 episode error。
|
||||
|
||||
若要在多个参考环境上并行执行，请保留 `--trials 5`，用 `--trial-indices` 分配一个或多个从 1 开始的 trial，并为每个 emulator 设置不同的 `--execution-shard`。单个分片本身永远不算完成。`merge_candidate_shards.py` 只有在 1–5 号 trial 对每个任务恰好出现一次，并且模型、晋级来源、API-33 环境、上游 App setup 完成状态与必需 App 版本一致时才接受合并。Evaluator 失败属于直接实验结果，必须保留；只有运行时 error 才可通过 `--resume --retry-errors` 重试。
|
||||
|
||||
历史配对切片运行在 Pixel 9 Pro API-35 AVD 上，缺少完整第三方 App bundle；`--skip-device-time` 仅用于这些与时间无关的 Wi-Fi evaluator。完整候选 campaign 则改用五个相互隔离的 Pixel 6/API-33 emulator，执行完整上游 setup，并在所有分片上保持相同的 24 个必需 App 包及版本。证据将两种环境明确分开，不把参考环境候选运行描述成 API-35 切片的同环境延伸。
|
||||
|
||||
适合精读的轨迹示例：
|
||||
|
||||
| 任务 | 材料 | 启示 |
|
||||
| --- | --- | --- |
|
||||
| `ExpenseAddMultipleFromGallery` | 失败分析 + `t3a_failed.md` | OCR/多模态缺口；伪造开销条目 |
|
||||
| `ClockTimerEntry` | 同上 | 无稳定 UI 模型；重复错误输入 |
|
||||
| `MarkorTranscribeVideo` | 同上 | 会播视频但「看不见」内容 |
|
||||
| `SportsTracker*` 计数/时长 | 同上 | 有感知无算术 |
|
||||
| 成功短流程（摄像、秒表等） | `t3a.md` | 对照「正常」轨迹长什么样 |
|
||||
|
||||
### 目录结构
|
||||
|
||||
```text
|
||||
chapter7/android-world/
|
||||
├── README.md # 本文件
|
||||
├── experiment_core.py # 证据、决策、完成门槛、报告渲染
|
||||
├── run_controlled_experiment.py # 真实 AndroidWorld 配对/候选 runner
|
||||
├── merge_candidate_shards.py # 校验并合并并行 trial 分片
|
||||
├── test_experiment.py # 聚焦的脱机完整性测试
|
||||
├── requirements.txt # 相邻上游 + API 客户端依赖
|
||||
├── t3a_summary.md # 汇总指标与能力矩阵
|
||||
├── t3a_failed_analysis.md # 失败分类与根因
|
||||
├── t3a.md # 完整运行日志（体积大）
|
||||
├── t3a_failed.md # 失败任务日志
|
||||
└── validation/ # 真实 evidence.json + 生成的 report.md
|
||||
```
|
||||
|
||||
### 复现基准（可选）
|
||||
|
||||
配套 runner 需要一个独立的相邻 AndroidWorld checkout（当前工作区使用 `chapter7/android_world`）、已配置模拟器以及真实模型凭证。自行重跑请：
|
||||
|
||||
1. 克隆 [google-research/android_world](https://github.com/google-research/android_world)（或课程指定 fork）。
|
||||
2. 按上游文档准备模拟器/真机环境。
|
||||
3. 在对应环境中安装配套依赖，设置所选 provider 凭证（默认 `ARK_API_KEY`），运行英文部分给出的命令。本地保留运行在 host 使用 `http://127.0.0.1:18111/v1`，container 内使用 `http://host.docker.internal:18111/v1`；`LOCAL_API_KEY` 只设在启动进程环境中，runner 只记录变量名，不保存其值。
|
||||
4. 只有在完整配置上游 Pixel 6 / API-33 App 后才能尝试 `--full-suite`；不要把 API-35 的 Wi-Fi 专用偏差用于完整 benchmark。
|
||||
|
||||
五分片的本地 GPU 命令与严格合并流程见英文复现节；每个隔离 emulator 运行一个 `--trial-indices`，全部完成后再调用 `merge_candidate_shards.py`。
|
||||
|
||||
仅做笔记研读的推荐顺序：**`t3a_summary.md` → `t3a_failed_analysis.md` → 抽读 `t3a_failed.md` / `t3a.md` 中的若干 episode**。
|
||||
|
||||
### 相关项目
|
||||
|
||||
| 项目 | 关系 |
|
||||
| --- | --- |
|
||||
| 上游 `android_world`（外部） | 可运行的评测环境 |
|
||||
| [model-benchmark](../model-benchmark/) | API 时延/可用性维度的评测 |
|
||||
| [elo-leaderboard](../elo-leaderboard/) | 成对比较式排行，而非绝对任务成功率 |
|
||||
| [public-health-reporting-eval](../public-health-reporting-eval/) | 仓库内另一套结构化评测脚手架 |
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Log files can be **very large** (`t3a.md` ~1MB+). Prefer summary + failed analysis first.
|
||||
- 日志文件体积很大，建议先读摘要与失败分析。
|
||||
- Project type: historical **reading / analysis notes** plus a runnable companion that requires a separately provisioned upstream AndroidWorld environment.
|
||||
- 项目类型：历史**阅读/分析材料** + 可运行配套工具；后者依赖另行配置的上游 AndroidWorld 环境。
|
||||
@@ -0,0 +1,588 @@
|
||||
"""Pure reporting helpers for the Experiment 7-12 AndroidWorld loop.
|
||||
|
||||
The runtime runner deliberately keeps AndroidWorld imports out of this module so
|
||||
the evidence checks and report generation can be tested without an emulator.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from collections import defaultdict
|
||||
import json
|
||||
import re
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
|
||||
BASELINE_TASK_COUNT = 116
|
||||
WIFI_TASKS = (
|
||||
"SystemWifiTurnOff",
|
||||
"SystemWifiTurnOffVerify",
|
||||
"SystemWifiTurnOn",
|
||||
"SystemWifiTurnOnVerify",
|
||||
)
|
||||
|
||||
_SECRET_PATTERNS = (
|
||||
re.compile(r"(?i)(authorization\s*[:=]\s*bearer\s+)[^\s,;]+"),
|
||||
re.compile(r"(?i)((?:api[_-]?key|access[_-]?token|secret)\s*[:=]\s*)[^\s,;]+"),
|
||||
re.compile(r"\b(?:sk|ak)-[A-Za-z0-9_-]{12,}\b"),
|
||||
)
|
||||
|
||||
|
||||
def redact_text(value: object, secrets: Iterable[str] = ()) -> str:
|
||||
"""Returns a printable error/message with likely credentials removed."""
|
||||
text = str(value)
|
||||
for secret in secrets:
|
||||
if secret:
|
||||
text = text.replace(secret, "[REDACTED]")
|
||||
text = _SECRET_PATTERNS[0].sub(r"\1[REDACTED]", text)
|
||||
text = _SECRET_PATTERNS[1].sub(r"\1[REDACTED]", text)
|
||||
text = _SECRET_PATTERNS[2].sub("[REDACTED]", text)
|
||||
return text
|
||||
|
||||
|
||||
def _mean(values: Iterable[float]) -> float | None:
|
||||
items = list(values)
|
||||
if not items:
|
||||
return None
|
||||
return round(sum(items) / len(items), 6)
|
||||
|
||||
|
||||
def aggregate_episodes(episodes: Iterable[Mapping[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
"""Aggregates real episode records by arm."""
|
||||
groups: dict[str, list[Mapping[str, Any]]] = defaultdict(list)
|
||||
for episode in episodes:
|
||||
groups[str(episode["arm"])].append(episode)
|
||||
|
||||
output: dict[str, dict[str, Any]] = {}
|
||||
for arm, rows in sorted(groups.items()):
|
||||
completed = [row for row in rows if row.get("status") == "completed"]
|
||||
output[arm] = {
|
||||
"episodes": len(rows),
|
||||
"completed_episodes": len(completed),
|
||||
"error_episodes": len(rows) - len(completed),
|
||||
"successes": sum(bool(row.get("success")) for row in completed),
|
||||
"success_rate": _mean(float(bool(row.get("success"))) for row in completed),
|
||||
"mean_evaluator_reward": _mean(
|
||||
float(row.get("evaluator_reward", 0.0)) for row in completed
|
||||
),
|
||||
"mean_steps": _mean(float(row.get("steps", 0)) for row in completed),
|
||||
"mean_latency_s": _mean(
|
||||
float(row.get("elapsed_s", 0.0)) for row in completed
|
||||
),
|
||||
"mean_llm_calls": _mean(
|
||||
float(row.get("llm", {}).get("calls", 0)) for row in completed
|
||||
),
|
||||
"mean_llm_latency_s": _mean(
|
||||
float(row.get("llm", {}).get("latency_s", 0.0)) for row in completed
|
||||
),
|
||||
"mean_total_tokens": _mean(
|
||||
float(row.get("llm", {}).get("input_tokens", 0))
|
||||
+ float(row.get("llm", {}).get("output_tokens", 0))
|
||||
for row in completed
|
||||
),
|
||||
"total_input_tokens": sum(
|
||||
int(row.get("llm", {}).get("input_tokens", 0)) for row in completed
|
||||
),
|
||||
"total_output_tokens": sum(
|
||||
int(row.get("llm", {}).get("output_tokens", 0)) for row in completed
|
||||
),
|
||||
"total_tokens": sum(
|
||||
int(row.get("llm", {}).get("input_tokens", 0))
|
||||
+ int(row.get("llm", {}).get("output_tokens", 0))
|
||||
for row in completed
|
||||
),
|
||||
"estimated_cost_usd": round(
|
||||
sum(
|
||||
float(row.get("llm", {}).get("estimated_cost_usd", 0.0))
|
||||
for row in completed
|
||||
),
|
||||
9,
|
||||
),
|
||||
}
|
||||
return output
|
||||
|
||||
|
||||
def paired_rows(episodes: Iterable[Mapping[str, Any]]) -> list[dict[str, Any]]:
|
||||
"""Builds paired control/treatment comparisons without inventing missing arms."""
|
||||
groups: dict[str, dict[str, Mapping[str, Any]]] = defaultdict(dict)
|
||||
for episode in episodes:
|
||||
if episode.get("arm") in ("control", "treatment"):
|
||||
groups[str(episode["pair_id"])][str(episode["arm"])] = episode
|
||||
|
||||
rows = []
|
||||
for pair_id, arms in sorted(groups.items()):
|
||||
if set(arms) != {"control", "treatment"}:
|
||||
continue
|
||||
control = arms["control"]
|
||||
treatment = arms["treatment"]
|
||||
if control.get("status") != "completed" or treatment.get("status") != "completed":
|
||||
continue
|
||||
rows.append({
|
||||
"pair_id": pair_id,
|
||||
"task": control["task"],
|
||||
"trial": control["trial"],
|
||||
"control_success": bool(control.get("success")),
|
||||
"treatment_success": bool(treatment.get("success")),
|
||||
"success_delta": int(bool(treatment.get("success"))) - int(bool(control.get("success"))),
|
||||
"control_reward": float(control.get("evaluator_reward", 0.0)),
|
||||
"treatment_reward": float(treatment.get("evaluator_reward", 0.0)),
|
||||
"reward_delta": round(
|
||||
float(treatment.get("evaluator_reward", 0.0))
|
||||
- float(control.get("evaluator_reward", 0.0)),
|
||||
6,
|
||||
),
|
||||
"control_steps": int(control.get("steps", 0)),
|
||||
"treatment_steps": int(treatment.get("steps", 0)),
|
||||
"control_latency_s": float(control.get("elapsed_s", 0.0)),
|
||||
"treatment_latency_s": float(treatment.get("elapsed_s", 0.0)),
|
||||
})
|
||||
return rows
|
||||
|
||||
|
||||
def choose_decision(
|
||||
arm_summary: Mapping[str, Mapping[str, Any]],
|
||||
pairs: Iterable[Mapping[str, Any]],
|
||||
*,
|
||||
minimum_pairs: int = 4,
|
||||
maximum_latency_ratio: float = 1.5,
|
||||
maximum_token_ratio: float = 1.5,
|
||||
) -> dict[str, Any]:
|
||||
"""Makes a conservative success/cost candidate decision from paired evidence."""
|
||||
pair_list = list(pairs)
|
||||
control = arm_summary.get("control")
|
||||
treatment = arm_summary.get("treatment")
|
||||
if not control or not treatment or len(pair_list) < minimum_pairs:
|
||||
return {
|
||||
"outcome": "insufficient_evidence",
|
||||
"promote_to_full_suite_candidate": False,
|
||||
"deployment_approved": False,
|
||||
"reason": f"Need at least {minimum_pairs} completed pairs; observed {len(pair_list)}.",
|
||||
}
|
||||
|
||||
improvement_count = sum(int(row["success_delta"]) for row in pair_list)
|
||||
regressions = sum(row["success_delta"] < 0 for row in pair_list)
|
||||
control_latency = control.get("mean_latency_s")
|
||||
treatment_latency = treatment.get("mean_latency_s")
|
||||
latency_ratio = None
|
||||
if control_latency and treatment_latency is not None:
|
||||
latency_ratio = round(float(treatment_latency) / float(control_latency), 6)
|
||||
control_tokens = control.get("mean_total_tokens")
|
||||
treatment_tokens = treatment.get("mean_total_tokens")
|
||||
token_ratio = None
|
||||
if control_tokens and treatment_tokens is not None:
|
||||
token_ratio = round(float(treatment_tokens) / float(control_tokens), 6)
|
||||
control_calls = control.get("mean_llm_calls")
|
||||
treatment_calls = treatment.get("mean_llm_calls")
|
||||
call_ratio = None
|
||||
if control_calls and treatment_calls is not None:
|
||||
call_ratio = round(float(treatment_calls) / float(control_calls), 6)
|
||||
|
||||
acceptable_cost = (
|
||||
latency_ratio is not None
|
||||
and token_ratio is not None
|
||||
and latency_ratio <= maximum_latency_ratio
|
||||
and token_ratio <= maximum_token_ratio
|
||||
)
|
||||
|
||||
if improvement_count > 0 and regressions == 0 and acceptable_cost:
|
||||
outcome = "promote_candidate_to_full_suite_rerun"
|
||||
promote = True
|
||||
reason = (
|
||||
f"Treatment improved {improvement_count} net paired task(s) with no paired regression. "
|
||||
"This is a candidate decision, not deployment approval or a full-suite result."
|
||||
)
|
||||
elif improvement_count > 0 and regressions == 0:
|
||||
outcome = "restrict_candidate_due_to_cost"
|
||||
promote = False
|
||||
reason = (
|
||||
"Treatment improved paired success without regressions, but exceeded the "
|
||||
f"latency/token guardrails ({maximum_latency_ratio:.2f}x / "
|
||||
f"{maximum_token_ratio:.2f}x). Restrict it to targeted follow-up; do not "
|
||||
"promote it to the full suite yet."
|
||||
)
|
||||
elif improvement_count < 0 or regressions:
|
||||
outcome = "reject_candidate"
|
||||
promote = False
|
||||
reason = (
|
||||
f"Treatment has {regressions} paired regression(s) and net success delta "
|
||||
f"{improvement_count}; do not promote."
|
||||
)
|
||||
else:
|
||||
outcome = "inconclusive_no_success_gain"
|
||||
promote = False
|
||||
reason = "Treatment produced no paired success gain; keep the upstream control prompt."
|
||||
|
||||
return {
|
||||
"outcome": outcome,
|
||||
"promote_to_full_suite_candidate": promote,
|
||||
"deployment_approved": False,
|
||||
"reason": reason,
|
||||
"completed_pairs": len(pair_list),
|
||||
"net_success_delta": improvement_count,
|
||||
"paired_regressions": regressions,
|
||||
"mean_latency_ratio_treatment_over_control": latency_ratio,
|
||||
"mean_token_ratio_treatment_over_control": token_ratio,
|
||||
"mean_llm_call_ratio_treatment_over_control": call_ratio,
|
||||
"guardrails": {
|
||||
"maximum_latency_ratio": maximum_latency_ratio,
|
||||
"maximum_token_ratio": maximum_token_ratio,
|
||||
"passed": acceptable_cost,
|
||||
},
|
||||
"scope_recommendation": (
|
||||
"full_suite_candidate_only" if promote else "do_not_deploy"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def choose_efficiency_decision(
|
||||
arm_summary: Mapping[str, Mapping[str, Any]],
|
||||
pairs: Iterable[Mapping[str, Any]],
|
||||
*,
|
||||
minimum_pairs: int = 4,
|
||||
maximum_latency_ratio: float = 1.5,
|
||||
maximum_token_ratio: float = 0.75,
|
||||
) -> dict[str, Any]:
|
||||
"""Promotes a cost refinement only when H5 success is preserved and cost falls."""
|
||||
pair_list = list(pairs)
|
||||
control = arm_summary.get("control")
|
||||
treatment = arm_summary.get("treatment")
|
||||
if not control or not treatment or len(pair_list) < minimum_pairs:
|
||||
return {
|
||||
"outcome": "insufficient_evidence",
|
||||
"promote_to_full_suite_candidate": False,
|
||||
"deployment_approved": False,
|
||||
"reason": f"Need at least {minimum_pairs} completed pairs; observed {len(pair_list)}.",
|
||||
}
|
||||
|
||||
net_success_delta = sum(int(row["success_delta"]) for row in pair_list)
|
||||
regressions = sum(row["success_delta"] < 0 for row in pair_list)
|
||||
treatment_successes = sum(bool(row["treatment_success"]) for row in pair_list)
|
||||
required_treatment_successes = len(pair_list)
|
||||
success_preserved = treatment_successes == required_treatment_successes
|
||||
control_latency = control.get("mean_latency_s")
|
||||
treatment_latency = treatment.get("mean_latency_s")
|
||||
latency_ratio = (
|
||||
round(float(treatment_latency) / float(control_latency), 6)
|
||||
if control_latency and treatment_latency is not None else None
|
||||
)
|
||||
control_tokens = control.get("mean_total_tokens")
|
||||
treatment_tokens = treatment.get("mean_total_tokens")
|
||||
token_ratio = (
|
||||
round(float(treatment_tokens) / float(control_tokens), 6)
|
||||
if control_tokens and treatment_tokens is not None else None
|
||||
)
|
||||
control_calls = control.get("mean_llm_calls")
|
||||
treatment_calls = treatment.get("mean_llm_calls")
|
||||
call_ratio = (
|
||||
round(float(treatment_calls) / float(control_calls), 6)
|
||||
if control_calls and treatment_calls is not None else None
|
||||
)
|
||||
passed = (
|
||||
regressions == 0
|
||||
and net_success_delta >= 0
|
||||
and success_preserved
|
||||
and latency_ratio is not None
|
||||
and token_ratio is not None
|
||||
and latency_ratio <= maximum_latency_ratio
|
||||
and token_ratio <= maximum_token_ratio
|
||||
)
|
||||
if passed:
|
||||
outcome = "promote_efficient_candidate_to_full_suite_rerun"
|
||||
reason = (
|
||||
"Treatment preserved paired success with no regression and passed the "
|
||||
"latency/token efficiency guardrails. This is a candidate decision only."
|
||||
)
|
||||
elif not success_preserved or regressions or net_success_delta < 0:
|
||||
outcome = "reject_efficiency_candidate_due_to_regression"
|
||||
reason = (
|
||||
f"Treatment succeeded on {treatment_successes}/{required_treatment_successes} "
|
||||
f"completed pairs, with {regressions} paired regression(s) and net success "
|
||||
f"delta {net_success_delta}; it did not preserve the H5 success baseline."
|
||||
)
|
||||
else:
|
||||
outcome = "reject_efficiency_candidate_due_to_cost"
|
||||
reason = (
|
||||
"Treatment preserved success but did not reduce tokens to the required "
|
||||
f"{maximum_token_ratio:.2f}x ratio within the latency guardrail."
|
||||
)
|
||||
return {
|
||||
"outcome": outcome,
|
||||
"promote_to_full_suite_candidate": passed,
|
||||
"deployment_approved": False,
|
||||
"reason": reason,
|
||||
"completed_pairs": len(pair_list),
|
||||
"net_success_delta": net_success_delta,
|
||||
"paired_regressions": regressions,
|
||||
"treatment_successes": treatment_successes,
|
||||
"required_treatment_successes": required_treatment_successes,
|
||||
"success_preservation_passed": success_preserved,
|
||||
"mean_latency_ratio_treatment_over_control": latency_ratio,
|
||||
"mean_token_ratio_treatment_over_control": token_ratio,
|
||||
"mean_llm_call_ratio_treatment_over_control": call_ratio,
|
||||
"guardrails": {
|
||||
"objective": "success_noninferiority_and_token_reduction",
|
||||
"require_all_treatment_pairs_successful": True,
|
||||
"maximum_latency_ratio": maximum_latency_ratio,
|
||||
"maximum_token_ratio": maximum_token_ratio,
|
||||
"passed": passed,
|
||||
},
|
||||
"scope_recommendation": (
|
||||
"full_suite_candidate_only" if passed else "do_not_deploy"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def enforce_scope_claims(evidence: dict[str, Any]) -> None:
|
||||
"""Sets completion gates from direct episode evidence, never from counters."""
|
||||
scope = evidence.setdefault("scope", {})
|
||||
distinct_tasks = len(set(scope.get("tasks", [])))
|
||||
configured_trials = int(scope.get("trials_per_task", 0))
|
||||
mode = scope.get("mode")
|
||||
tasks = list(dict.fromkeys(str(task) for task in scope.get("tasks", [])))
|
||||
expected_episode_keys = {
|
||||
(task, trial)
|
||||
for task in tasks
|
||||
for trial in range(1, configured_trials + 1)
|
||||
}
|
||||
episodes = evidence.get("episodes", [])
|
||||
actual_episode_keys = {
|
||||
(str(row.get("task")), int(row.get("trial", 0))) for row in episodes
|
||||
}
|
||||
direct_episode_gate = (
|
||||
len(episodes) == len(expected_episode_keys)
|
||||
and actual_episode_keys == expected_episode_keys
|
||||
and all(
|
||||
row.get("arm") == "candidate"
|
||||
and row.get("status") == "completed"
|
||||
and row.get("evaluator_reward") is not None
|
||||
and isinstance(row.get("pair_seed"), int)
|
||||
for row in episodes
|
||||
)
|
||||
and all(
|
||||
len({
|
||||
row["pair_seed"] for row in episodes if row.get("task") == task
|
||||
}) == configured_trials
|
||||
for task in tasks
|
||||
)
|
||||
)
|
||||
full_suite = (
|
||||
mode == "candidate_rerun"
|
||||
and distinct_tasks == BASELINE_TASK_COUNT
|
||||
and configured_trials >= 5
|
||||
and direct_episode_gate
|
||||
and evidence.get("environment", {}).get("api_level") == 33
|
||||
and evidence.get("environment", {}).get("emulator_setup_completed") is True
|
||||
and evidence.get("environment", {})
|
||||
.get("app_provisioning", {})
|
||||
.get("complete")
|
||||
)
|
||||
scope["direct_episode_gate_completed"] = direct_episode_gate
|
||||
scope["full_suite_completed"] = full_suite
|
||||
scope["manuscript_five_seed_gate_completed"] = full_suite
|
||||
evidence["experiment_complete"] = bool(
|
||||
full_suite and evidence.get("decision", {}).get("source_paired_run_id")
|
||||
)
|
||||
|
||||
|
||||
def _fmt(value: Any) -> str:
|
||||
if value is None:
|
||||
return "n/a"
|
||||
if isinstance(value, float):
|
||||
return f"{value:.3f}"
|
||||
return str(value)
|
||||
|
||||
|
||||
def render_report(evidence: Mapping[str, Any]) -> str:
|
||||
"""Renders the five-stage report from machine-readable evidence."""
|
||||
scope = evidence["scope"]
|
||||
environment = evidence["environment"]
|
||||
arm_summary = evidence.get("arm_summary", {})
|
||||
decision = evidence.get("decision", {})
|
||||
pairs = evidence.get("paired_comparison", [])
|
||||
blockers = evidence.get("environment_boundaries", [])
|
||||
hypotheses = evidence.get("diagnosis", {}).get("layered_hypotheses", [])
|
||||
phase = evidence.get("phase", {})
|
||||
llm_analysis = evidence.get("llm_analysis", {})
|
||||
controls = (
|
||||
"same checkout, model, task parameters, generated seed policy, step budget, "
|
||||
"Pixel 6/API-33 device class, upstream setup, and app versions across isolated shards."
|
||||
if environment.get("shard_devices")
|
||||
else "same checkout, model, task parameters, generated seed, step budget, and "
|
||||
"emulator; arm order alternates by pair."
|
||||
)
|
||||
|
||||
lines = [
|
||||
"# Experiment 7-12 AndroidWorld iteration report",
|
||||
"",
|
||||
f"- Run ID: `{evidence['run_id']}`",
|
||||
f"- Generated (UTC): `{evidence['generated_at_utc']}`",
|
||||
f"- Upstream commit: `{environment.get('android_world_commit', 'not reached')}`",
|
||||
f"- Device: `{environment.get('device_model', 'not reached')}`, API "
|
||||
f"`{environment.get('api_level', 'not reached')}` (upstream tested reference: API "
|
||||
f"`{environment.get('upstream_tested_api_level', 33)}`)",
|
||||
f"- Observation method: `{environment.get('a11y_method', 'a11y_forwarder_app')}`",
|
||||
f"- Provider/model: `{evidence['model']['provider']}` / `{evidence['model']['model']}`",
|
||||
f"- Model source/runtime: `{evidence['model'].get('source', 'not recorded')}` / "
|
||||
f"`{evidence['model'].get('runtime', 'not recorded')}`",
|
||||
f"- Accelerator: `{evidence['model'].get('accelerator', 'not recorded')}`",
|
||||
f"- Required apps: "
|
||||
f"`{environment.get('app_provisioning', {}).get('installed_required_package_count', 'not reached')}/"
|
||||
f"{environment.get('app_provisioning', {}).get('required_package_count', 'not reached')}`",
|
||||
f"- Scope: {len(scope['tasks'])} task(s), {scope['trials_per_task']} trial(s), "
|
||||
f"mode `{scope['mode']}`",
|
||||
f"- Full 116-task × 5-seed suite completed: **{str(scope['full_suite_completed']).lower()}**",
|
||||
"",
|
||||
"The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% "
|
||||
"numbers are explicitly hypothetical and are not used as rerun results here.",
|
||||
"",
|
||||
"## 1. Diagnose",
|
||||
"",
|
||||
]
|
||||
for item in evidence["diagnosis"]["findings"]:
|
||||
lines.append(f"- {item}")
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
"## 2. Hypothesis",
|
||||
"",
|
||||
"The diagnosis produced explicit surface, middle, and deep hypotheses. Only one "
|
||||
"variable is changed in this run; the other hypotheses remain untested.",
|
||||
"",
|
||||
"| Layer / ID | Proposed change | Target | Verification | Status |",
|
||||
"| --- | --- | --- | --- | --- |",
|
||||
])
|
||||
for row in hypotheses:
|
||||
lines.append(
|
||||
f"| {row.get('layer', 'n/a')} / `{row.get('id', 'n/a')}` | "
|
||||
f"{row.get('idea', 'n/a')} | {row.get('target', 'n/a')} | "
|
||||
f"{row.get('verification', 'n/a')} | {row.get('status', 'not tested')} |"
|
||||
)
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
f"Selected hypothesis: `{evidence['hypothesis']['id']}`",
|
||||
f"- Change: {evidence['hypothesis']['change']}",
|
||||
f"- Expected measurable result: {evidence['hypothesis']['expected_result']}",
|
||||
f"- Guardrails: {evidence['hypothesis']['guardrails']}",
|
||||
"",
|
||||
"## 3. Controlled experiment",
|
||||
"",
|
||||
f"- Phase: `{phase.get('id', 'phase_1_surface')}` — "
|
||||
f"{phase.get('description', 'low-cost surface prompt ablation')}",
|
||||
f"- Independent variable: {phase.get('independent_variable', 'task-specific T3A guidelines')}",
|
||||
f"- Controls: {controls}",
|
||||
"",
|
||||
"| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |",
|
||||
"| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |",
|
||||
])
|
||||
for arm, row in sorted(arm_summary.items()):
|
||||
lines.append(
|
||||
f"| {arm} | {row['completed_episodes']}/{row['episodes']} | "
|
||||
f"{_fmt(row['success_rate'])} | {_fmt(row['mean_evaluator_reward'])} | "
|
||||
f"{_fmt(row['mean_steps'])} | {_fmt(row['mean_latency_s'])} | "
|
||||
f"{_fmt(row['mean_llm_calls'])} | {_fmt(row.get('mean_total_tokens'))} | "
|
||||
f"{row['total_input_tokens']} / {row['total_output_tokens']} | "
|
||||
f"{row.get('estimated_cost_usd', 0.0):.6f} |"
|
||||
)
|
||||
|
||||
if pairs:
|
||||
lines.extend([
|
||||
"",
|
||||
"| Task / trial | Control | Treatment | Δ success | Control→treatment steps |",
|
||||
"| --- | ---: | ---: | ---: | ---: |",
|
||||
])
|
||||
for row in pairs:
|
||||
lines.append(
|
||||
f"| {row['task']} / {row['trial']} | {int(row['control_success'])} | "
|
||||
f"{int(row['treatment_success'])} | {row['success_delta']:+d} | "
|
||||
f"{row['control_steps']}→{row['treatment_steps']} |"
|
||||
)
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
"## 4. Data-driven decision",
|
||||
"",
|
||||
f"- Outcome: **`{decision.get('outcome', 'not_applicable')}`**",
|
||||
f"- Reason: {decision.get('reason', 'This artifact is a candidate rerun, not a paired decision run.')}",
|
||||
f"- Treatment/control mean latency ratio: "
|
||||
f"{_fmt(decision.get('mean_latency_ratio_treatment_over_control'))}",
|
||||
f"- Treatment/control mean token ratio: "
|
||||
f"{_fmt(decision.get('mean_token_ratio_treatment_over_control'))}",
|
||||
f"- Treatment/control mean LLM-call ratio: "
|
||||
f"{_fmt(decision.get('mean_llm_call_ratio_treatment_over_control'))}",
|
||||
f"- Cost guardrails passed: **{str(decision.get('guardrails', {}).get('passed', False)).lower()}**",
|
||||
f"- Deployment approved: **{str(decision.get('deployment_approved', False)).lower()}**",
|
||||
"",
|
||||
"## 5. Rerun and next report",
|
||||
"",
|
||||
])
|
||||
if scope["full_suite_completed"]:
|
||||
lines.append(
|
||||
"The complete 116-task, five-trial candidate rerun gate is satisfied by direct episode evidence."
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
"This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. "
|
||||
"The next gate is a conditionally enabled candidate rerun over all 116 tasks with five "
|
||||
"seeds after provisioning the upstream API-33 app environment."
|
||||
)
|
||||
failed = [
|
||||
episode for episode in evidence.get("episodes", [])
|
||||
if episode.get("status") != "completed" or not episode.get("success")
|
||||
]
|
||||
if failed:
|
||||
lines.append("")
|
||||
lines.append("Observed residual failures:")
|
||||
for episode in failed:
|
||||
if episode.get("error"):
|
||||
detail = episode["error"]
|
||||
elif episode.get("evaluator_reward") == 1.0 and not episode.get("agent_declared_done"):
|
||||
detail = "final evaluator state passed, but the agent never declared completion"
|
||||
elif episode.get("agent_declared_done") and episode.get("evaluator_reward") != 1.0:
|
||||
detail = "agent declared completion, but the real evaluator state failed"
|
||||
else:
|
||||
detail = "evaluator reward / completion gate was not satisfied"
|
||||
lines.append(f"- `{episode['arm']} / {episode['task']} / trial {episode['trial']}`: {detail}")
|
||||
|
||||
lines.extend(["", "### LLM analysis of this run", ""])
|
||||
if llm_analysis.get("status") == "completed":
|
||||
lines.append(
|
||||
"The following bounded interpretation was produced by the configured real LLM from "
|
||||
"the aggregate evidence (the JSON remains authoritative):"
|
||||
)
|
||||
lines.append("")
|
||||
lines.append(f"- Summary: {llm_analysis.get('summary', 'n/a')}")
|
||||
lines.append(
|
||||
f"- Cost/benefit interpretation: {llm_analysis.get('cost_benefit_interpretation', 'n/a')}"
|
||||
)
|
||||
for item in llm_analysis.get("observed_failure_pattern", []):
|
||||
lines.append(f"- Residual pattern: {item}")
|
||||
next_hypothesis = llm_analysis.get("next_hypothesis", {})
|
||||
if next_hypothesis:
|
||||
lines.append(
|
||||
f"- Next hypothesis `{next_hypothesis.get('id', 'n/a')}` "
|
||||
f"({next_hypothesis.get('layer', 'n/a')}): {next_hypothesis.get('idea', 'n/a')} "
|
||||
f"Target: {next_hypothesis.get('target', 'n/a')} Verification: "
|
||||
f"{next_hypothesis.get('verification', 'n/a')}"
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
f"No LLM analysis was accepted: {llm_analysis.get('error', 'analysis was not requested for this artifact')}"
|
||||
)
|
||||
|
||||
lines.extend(["", "## Environment boundaries", ""])
|
||||
if blockers:
|
||||
lines.extend(f"- {item}" for item in blockers)
|
||||
else:
|
||||
lines.append("- None recorded.")
|
||||
lines.extend([
|
||||
"",
|
||||
"The JSON beside this report is the authoritative evidence. It contains episode-level "
|
||||
"evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; "
|
||||
"credentials and raw prompts are not stored.",
|
||||
"",
|
||||
])
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def dumps_json(evidence: Mapping[str, Any]) -> str:
|
||||
return json.dumps(evidence, ensure_ascii=False, indent=2, sort_keys=True) + "\n"
|
||||
@@ -0,0 +1,271 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Merge independently executed Experiment 7-12 trial shards without hiding failures."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from experiment_core import (
|
||||
BASELINE_TASK_COUNT,
|
||||
aggregate_episodes,
|
||||
dumps_json,
|
||||
enforce_scope_claims,
|
||||
paired_rows,
|
||||
render_report,
|
||||
)
|
||||
from run_controlled_experiment import (
|
||||
EXPERIMENT_ID,
|
||||
OpenAICompatibleLlm,
|
||||
_generate_llm_analysis,
|
||||
_utc_now,
|
||||
)
|
||||
|
||||
|
||||
def _parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("shards", nargs="+", type=Path)
|
||||
parser.add_argument("--output-dir", type=Path, required=True)
|
||||
parser.add_argument("--source-paired-evidence", type=Path, required=True)
|
||||
parser.add_argument("--analysis-base-url")
|
||||
parser.add_argument("--analysis-api-key-env", default="LOCAL_API_KEY")
|
||||
parser.add_argument("--analysis-model")
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def _load(path: Path) -> dict[str, Any]:
|
||||
data = path.read_bytes()
|
||||
evidence = json.loads(data)
|
||||
if evidence.get("experiment") != EXPERIMENT_ID:
|
||||
raise RuntimeError(f"Not direct Experiment {EXPERIMENT_ID} evidence: {path}")
|
||||
return evidence
|
||||
|
||||
|
||||
def _app_versions(evidence: dict[str, Any]) -> list[tuple[str, Any, Any]]:
|
||||
apps = evidence.get("environment", {}).get("app_provisioning", {}).get("apps", [])
|
||||
return sorted(
|
||||
(row.get("package"), row.get("version_code"), row.get("version_name"))
|
||||
for row in apps
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
args = _parse_args()
|
||||
if args.output_dir.exists():
|
||||
raise RuntimeError(f"Output directory already exists: {args.output_dir}")
|
||||
paths = [path.resolve() for path in args.shards]
|
||||
shards = [_load(path) for path in paths]
|
||||
if len(shards) < 2:
|
||||
raise RuntimeError("At least two independent shards are required")
|
||||
|
||||
reference = shards[0]
|
||||
reference_tasks = reference.get("scope", {}).get("tasks", [])
|
||||
reference_trials = int(reference.get("scope", {}).get("trials_per_task", 0))
|
||||
if len(reference_tasks) != BASELINE_TASK_COUNT or reference_trials < 5:
|
||||
raise RuntimeError("Shards must declare the complete 116-task, five-trial scope")
|
||||
if reference.get("scope", {}).get("mode") != "candidate_rerun":
|
||||
raise RuntimeError("Only candidate-rerun shards can be merged")
|
||||
|
||||
selected_trials: set[int] = set()
|
||||
episodes: list[dict[str, Any]] = []
|
||||
seen_keys: set[tuple[str, int, str]] = set()
|
||||
shard_rows = []
|
||||
reference_model = reference.get("model")
|
||||
reference_source = reference.get("decision", {}).get("source_paired_run_id")
|
||||
reference_versions = _app_versions(reference)
|
||||
merged_boundaries: list[str] = []
|
||||
merged_retry_history: list[dict[str, Any]] = []
|
||||
merged_parameter_drift: list[dict[str, Any]] = []
|
||||
merged_retry_parameter_drift: list[dict[str, Any]] = []
|
||||
for path, shard in zip(paths, shards):
|
||||
scope = shard.get("scope", {})
|
||||
if scope.get("tasks") != reference_tasks:
|
||||
raise RuntimeError(f"Task ordering differs in shard: {path}")
|
||||
if int(scope.get("trials_per_task", 0)) != reference_trials:
|
||||
raise RuntimeError(f"Trial scope differs in shard: {path}")
|
||||
if shard.get("model") != reference_model:
|
||||
raise RuntimeError(f"Model configuration differs in shard: {path}")
|
||||
if shard.get("decision", {}).get("source_paired_run_id") != reference_source:
|
||||
raise RuntimeError(f"Promoted paired source differs in shard: {path}")
|
||||
environment = shard.get("environment", {})
|
||||
if environment.get("api_level") != 33:
|
||||
raise RuntimeError(f"Shard is not on reference API 33: {path}")
|
||||
if environment.get("emulator_setup_completed") is not True:
|
||||
raise RuntimeError(f"Shard did not complete official emulator/app setup: {path}")
|
||||
if not environment.get("app_provisioning", {}).get("complete"):
|
||||
raise RuntimeError(f"Shard has an incomplete official app bundle: {path}")
|
||||
if _app_versions(shard) != reference_versions:
|
||||
raise RuntimeError(f"Official app versions differ in shard: {path}")
|
||||
if shard.get("credentials_persisted") is not False:
|
||||
raise RuntimeError(f"Shard does not attest credential-free evidence: {path}")
|
||||
shard_trials = {int(value) for value in scope.get("selected_trials", [])}
|
||||
if not shard_trials or selected_trials.intersection(shard_trials):
|
||||
raise RuntimeError(f"Missing or overlapping selected trials in shard: {path}")
|
||||
selected_trials.update(shard_trials)
|
||||
for episode in shard.get("episodes", []):
|
||||
trial = int(episode.get("trial", 0))
|
||||
key = (str(episode.get("task")), trial, str(episode.get("arm")))
|
||||
if trial not in shard_trials:
|
||||
raise RuntimeError(f"Episode lies outside its declared trial shard: {key}")
|
||||
if key in seen_keys:
|
||||
raise RuntimeError(f"Duplicate direct episode across shards: {key}")
|
||||
seen_keys.add(key)
|
||||
episodes.append(copy.deepcopy(episode))
|
||||
shard_rows.append({
|
||||
"path": str(path),
|
||||
"sha256": hashlib.sha256(path.read_bytes()).hexdigest(),
|
||||
"run_id": shard.get("run_id"),
|
||||
"execution_shard": environment.get("execution_shard"),
|
||||
"selected_trials": sorted(shard_trials),
|
||||
"episodes": len(shard.get("episodes", [])),
|
||||
"completed_episodes": sum(
|
||||
row.get("status") == "completed" for row in shard.get("episodes", [])
|
||||
),
|
||||
"error_episodes": sum(
|
||||
row.get("status") != "completed" for row in shard.get("episodes", [])
|
||||
),
|
||||
})
|
||||
for boundary in shard.get("environment_boundaries", []):
|
||||
if boundary not in merged_boundaries:
|
||||
merged_boundaries.append(boundary)
|
||||
for retry in shard.get("retry_history", []):
|
||||
merged_retry_history.append({
|
||||
"execution_shard": environment.get("execution_shard"),
|
||||
**copy.deepcopy(retry),
|
||||
})
|
||||
for drift in shard.get("resume_parameter_drift", []):
|
||||
row = {
|
||||
"execution_shard": environment.get("execution_shard"),
|
||||
**copy.deepcopy(drift),
|
||||
}
|
||||
if row not in merged_parameter_drift:
|
||||
merged_parameter_drift.append(row)
|
||||
for drift in shard.get("retry_parameter_drift", []):
|
||||
row = {
|
||||
"execution_shard": environment.get("execution_shard"),
|
||||
**copy.deepcopy(drift),
|
||||
}
|
||||
if row not in merged_retry_parameter_drift:
|
||||
merged_retry_parameter_drift.append(row)
|
||||
|
||||
expected_trials = set(range(1, reference_trials + 1))
|
||||
if selected_trials != expected_trials:
|
||||
raise RuntimeError(
|
||||
f"Shard trial union is {sorted(selected_trials)}; expected {sorted(expected_trials)}"
|
||||
)
|
||||
task_order = {task: index for index, task in enumerate(reference_tasks)}
|
||||
episodes.sort(key=lambda row: (task_order[str(row["task"])], int(row["trial"])))
|
||||
|
||||
merged = copy.deepcopy(reference)
|
||||
merged["run_id"] = "exp7-12-merged-" + _utc_now().replace(":", "").replace("-", "")
|
||||
merged["generated_at_utc"] = _utc_now()
|
||||
merged["command"] = ["merge_candidate_shards.py", *map(str, paths)]
|
||||
merged["scope"]["selected_trials"] = sorted(selected_trials)
|
||||
merged["episodes"] = episodes
|
||||
merged["shards"] = shard_rows
|
||||
merged["environment_boundaries"] = merged_boundaries
|
||||
merged["retry_history"] = merged_retry_history
|
||||
if merged_parameter_drift:
|
||||
merged["resume_parameter_drift"] = merged_parameter_drift
|
||||
else:
|
||||
merged.pop("resume_parameter_drift", None)
|
||||
if merged_retry_parameter_drift:
|
||||
merged["retry_parameter_drift"] = merged_retry_parameter_drift
|
||||
else:
|
||||
merged.pop("retry_parameter_drift", None)
|
||||
merged["environment"]["execution_shard"] = "merged"
|
||||
merged["environment"]["shard_devices"] = [
|
||||
{
|
||||
"run_id": shard.get("run_id"),
|
||||
"execution_shard": shard.get("environment", {}).get("execution_shard"),
|
||||
"device_serial": shard.get("environment", {}).get("device_serial"),
|
||||
"avd_name": shard.get("environment", {}).get("avd_name"),
|
||||
"api_level": shard.get("environment", {}).get("api_level"),
|
||||
}
|
||||
for shard in shards
|
||||
]
|
||||
merged["scope"]["completed_episodes"] = sum(
|
||||
row.get("status") == "completed" for row in episodes
|
||||
)
|
||||
merged["scope"]["error_episodes"] = sum(
|
||||
row.get("status") != "completed" for row in episodes
|
||||
)
|
||||
merged["arm_summary"] = aggregate_episodes(episodes)
|
||||
merged["paired_comparison"] = paired_rows(episodes)
|
||||
enforce_scope_claims(merged)
|
||||
paired_source = json.loads(args.source_paired_evidence.read_text(encoding="utf-8"))
|
||||
if paired_source.get("run_id") != reference_source:
|
||||
raise RuntimeError("Supplied paired evidence does not match the shard source run ID")
|
||||
merged["decision"]["source_paired_evidence"] = str(
|
||||
args.source_paired_evidence.resolve()
|
||||
)
|
||||
merged["decision"]["source_paired_model"] = paired_source.get("model")
|
||||
paired_model = paired_source.get("model", {}).get("model")
|
||||
candidate_model = merged.get("model", {}).get("model")
|
||||
if paired_model and paired_model != candidate_model:
|
||||
message = (
|
||||
f"The full-suite candidate uses model {candidate_model}, while the promoted paired "
|
||||
f"H5C source used {paired_model}. This user-requested local-GPU campaign evaluates "
|
||||
"the promoted observation treatment but is not a same-model extension of the paired result."
|
||||
)
|
||||
if message not in merged.setdefault("environment_boundaries", []):
|
||||
merged["environment_boundaries"].append(message)
|
||||
if merged["scope"]["full_suite_completed"]:
|
||||
merged["decision"].update({
|
||||
"outcome": "full_candidate_rerun_completed",
|
||||
"deployment_approved": False,
|
||||
"reason": (
|
||||
"The direct 116-task x five-trial candidate rerun completed on five "
|
||||
"independent reference-environment shards. Negative evaluator results are retained."
|
||||
),
|
||||
})
|
||||
else:
|
||||
merged["decision"].update({
|
||||
"outcome": "candidate_rerun_has_errors",
|
||||
"deployment_approved": False,
|
||||
"reason": (
|
||||
"The merged candidate evidence contains missing or error episodes and does not "
|
||||
"satisfy the strict completion gate."
|
||||
),
|
||||
})
|
||||
|
||||
if args.analysis_base_url and args.analysis_model:
|
||||
import os
|
||||
|
||||
api_key = os.environ.get(args.analysis_api_key_env)
|
||||
if not api_key:
|
||||
raise RuntimeError(
|
||||
f"Analysis credential variable is unset: {args.analysis_api_key_env}"
|
||||
)
|
||||
llm = OpenAICompatibleLlm(
|
||||
api_key=api_key,
|
||||
base_url=args.analysis_base_url,
|
||||
model=args.analysis_model,
|
||||
seed=int(reference_model.get("seed", 42)),
|
||||
max_tokens=int(reference_model.get("max_tokens", 1024)),
|
||||
timeout_s=120,
|
||||
retries=1,
|
||||
input_cost_per_million_usd=0.0,
|
||||
output_cost_per_million_usd=0.0,
|
||||
)
|
||||
merged["llm_analysis"] = _generate_llm_analysis(merged, llm, [api_key])
|
||||
else:
|
||||
merged["llm_analysis"] = {
|
||||
"status": "not_run",
|
||||
"error": "Merged analysis endpoint was not configured.",
|
||||
}
|
||||
|
||||
args.output_dir.mkdir(parents=True)
|
||||
(args.output_dir / "evidence.json").write_text(dumps_json(merged), encoding="utf-8")
|
||||
(args.output_dir / "report.md").write_text(render_report(merged), encoding="utf-8")
|
||||
print(f"Evidence: {args.output_dir / 'evidence.json'}")
|
||||
print(f"Report: {args.output_dir / 'report.md'}")
|
||||
return 0 if merged["scope"]["full_suite_completed"] else 2
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,2 @@
|
||||
-e ../android_world
|
||||
openai>=1.30.0,<3
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,147 @@
|
||||
### 类别一：转录失败 (Transcription Failure)
|
||||
|
||||
这类失败的核心是 Agent 无法从非文本格式（如图片、视频）中提取和理解文字信息。
|
||||
|
||||
**1. 任务: `ExpenseAddMultipleFromGallery` (task 13)**
|
||||
* **目标**: 从图库中的 `expenses.jpg` 图片里读取开销项目，并将其添加到“Pro Expense”应用中。
|
||||
* **失败过程分析**:
|
||||
1. **操作正确**: Agent 成功地打开了“Simple Gallery Pro”，找到了 `expenses.jpg` 并将其全屏显示。这一系列的UI导航操作是完全正确的。
|
||||
2. **认知断层**: 在第8步，Agent 的思考（`Reason`）是：“我需要先打开开销追踪应用，为添加开销数据做准备”。这表明它**知道**下一步该做什么，但完全**没有提及**它从图片中看到了什么。它无法“读取”图片上的文字。
|
||||
3. **行为“伪装”**: 从第13步开始，Agent 在“Pro Expense”应用中输入了“Coffee”、“4.50”、“Lunch”、“12.75”等条目。这些数据并非来自图片，而是 Agent 根据任务描述“添加开销”而**凭空生成或从其训练数据中联想出的通用示例**。它在模拟一个成功的流程，尽管它没有获取到完成任务所必需的真实信息。
|
||||
4. **最终失败**: 由于 Agent 添加的数据与 `expenses.jpg` 中的真实数据完全不符，最终的任务验证（通过检查应用数据库）失败。
|
||||
* **根本原因**: Agent 的视觉模型缺乏 OCR 能力。它能识别UI元素（按钮、列表），但无法将图片中的像素信息转译为结构化的文本数据。
|
||||
|
||||
**2. 任务: `MarkorTranscribeVideo` (task 36)**
|
||||
* **目标**: 观看 `ZwUN_moment_70_.mp4` 视频，并将视频帧上显示的文字序列，以逗号分隔的形式记录到 `Markor` 笔记中。
|
||||
* **失败过程分析**:
|
||||
1. **导航能力出色**: Agent 展现了复杂的导航能力。它成功打开VLC，处理了权限请求，浏览到 `Download` 文件夹，并正确地播放了目标视频。它甚至还处理了VLC首次启动时的教程浮层。
|
||||
2. **内容理解为零**: 在第12步到第20步之间，Agent 反复尝试与播放器交互（点击播放、暂停），但其思考日志中从未提及视频中出现的任何文字内容。它就像一个盲人，可以熟练地操作播放器，却看不见屏幕上播放的内容。
|
||||
3. **陷入无效循环**: 它似乎知道任务尚未完成，所以不断重复“点击播放器”这个动作，希望能触发下一步，但由于无法获取新信息（视频文字），它无法推进任务。
|
||||
4. **最终失败**: 任务因步骤超限而失败。
|
||||
* **根本原因**: 与图片转录类似，Agent 完全不具备从视频帧中提取视觉信息并将其转换为文本的能力。这是其感知模块的根本性缺陷。
|
||||
|
||||
---
|
||||
|
||||
### 类别二：复杂UI理解失败 (Complex UI Understanding Failure)
|
||||
|
||||
这类失败的核心是 Agent 无法理解非标准、复杂或有特殊状态逻辑的UI。它能看到按钮，但不知道按下按钮后会发生什么，或者如何通过一系列操作达到特定UI状态。
|
||||
|
||||
**1. 任务: `ClockTimerEntry` (task 9)**
|
||||
* **目标**: 设置一个“0小时16分35秒”的计时器。
|
||||
* **失败过程分析**:
|
||||
1. **初步操作正确**: Agent 打开时钟，切换到计时器标签页，准备输入。
|
||||
2. **发现错误**: 它尝试输入`1635`。在输入`1`、`6`、`3`后，屏幕显示为 `00h 01m 63s`。在第5步，它的思考日志写道：“我看到当前显示的是‘00h 01m 63s’，这是一个无效的时间格式”。这表明它**具备基础的UI阅读能力**，能识别出错误,但是并不理解输入显示的逻辑应当是什么样的，缺失基于预期的长久规划，所以错误的终止了继续输入的行为，而是转向了错误处理。
|
||||
3. **能够纠正，但不能学习**: 它接着正确地使用了三次退格键，将输入清零。
|
||||
4. **陷入逻辑循环**: 从第9步开始，它完全重复了第一次的错误输入序列，再次输入`1`、`6`...。它没有从上一次的失败中学习到这个UI的输入机制
|
||||
* **根本原因**: Agent 对这个UI的理解是表层的。它知道“输入数字”和“删除”，但它没有形成一个关于“计时器数字如何进位”的心智模型。因此，当它的简单策略失败时，它无法调整策略，只能重复无效的尝试。
|
||||
|
||||
### 类别三：应用初始化导致的步数限制
|
||||
|
||||
**1. 任务: `MarkorAddNoteHeader` (task 23)**
|
||||
* **目标**: 在一个笔记文件的现有内容**之前**，添加一行新文字和一个空行。
|
||||
* **失败过程分析**:
|
||||
1. **导航与初级编辑正确**: Agent 成功打开 `Markor`，找到并打开了目标笔记。
|
||||
2. Step 1-7: 处理首次启动。Agent 打开 Markor 应用，然后像一个真实用户一样，耐心地点击了5次“下一步”来完成教程，最后关闭了弹出的更新日志。这7步是正确的，但对于完成核心任务来说是“开销”。
|
||||
3. **高级操作的困惑**: 在第11和12步，它通过长按和点击，成功地“全选”了所有文本。这是一个相对复杂的操作，它执行得很好。
|
||||
* **根本原因**: 应用初次启动时需要进行初始化设置，这部分步数消耗了模型的有效步数。
|
||||
|
||||
---
|
||||
|
||||
### 类别四：数学/计数失败 (Math/Counting Failure)
|
||||
|
||||
这类失败的核心是 Agent 缺乏在UI操作环境中执行基本数学或逻辑运算的能力。
|
||||
|
||||
**1. 任务: `SportsTrackerActivitiesCountForWeek` (task 87)**
|
||||
* **目标**: 计算“本周”（从周一开始）总共有多少次“跑步”活动。
|
||||
* **失败过程分析**:
|
||||
1. **数据收集成功**: Agent 成功打开 `OpenTracks` 应用，并在日志中通过上下滚动，完整地浏览了本周的所有活动列表。它把完成任务所需的所有原始信息都“看”了一遍。
|
||||
2. **认知处理失败**: 在第10步，它给出了答案 `2`。令人疑惑的地方在于：Agent 声称自己完成了任务，而错误的原因却是：Agent did not indicate task is done. Reached max number of steps.
|
||||
|
||||
* **可能原因**: Agent 能够完成感知（看到列表）和操作（滚动列表）的部分，但没能完成认知处理（筛选“跑步”活动，并对结果进行**计数**）。
|
||||
|
||||
**2. 任务: `SportsTrackerTotalDurationForCategoryThisWeek` (task 92)**
|
||||
* **目标**: 计算本周所有“跑步”活动的总时长。
|
||||
* **失败过程分析**:
|
||||
1. **数据收集成功**: 和上一个任务一样，Agent 成功打开应用并浏览了所有本周的活动。它看到了每个活动的名称和时长（例如 `30:00`, `1:15:00`）。
|
||||
2. **认知处理失败**: 它没能在限定步数内完成这个任务，最终超时。因为它需要执行比计数更复杂的认知操作：
|
||||
* **筛选**: 找到所有“跑步”活动。
|
||||
* **提取**: 记下每个跑步活动的时长。
|
||||
* **求和**: 将所有时长加在一起。
|
||||
* **可能原因**: Agent 不具备执行数学**求和**的能力。它可以看到数字，但无法在操作的上下文中对它们进行计算。
|
||||
|
||||
---
|
||||
|
||||
### 类别五：信息检索与复杂逻辑失败
|
||||
|
||||
这类失败的核心是 Agent 无法在复杂场景下定位信息，或无法执行需要多个逻辑步骤和状态维护的复杂任务。
|
||||
|
||||
**1. 任务: `SimpleCalendarAnyEventsOnDate` (task 68)**
|
||||
* **目标**: 检查“10月28日”是否有任何事件。
|
||||
* **失败过程分析**:
|
||||
1. **界面理解失败**: Agent 打开了日历的月视图。这是一个信息密集的界面。在第2步，它试图点击10月28日，但错误地点到了10月13日。
|
||||
2. **低效的恢复策略**: 发现错误后，它没有返回月视图重新选择，而是陷入了一种非常低效的策略：在日视图中，一次又一次地点击“向后一天”的箭头，试图从13日逐天“走到”28日。
|
||||
3. **最终失败**: 这种策略耗时过长，导致任务在到达目标日期前就因步骤超限而失败。
|
||||
* **根本原因**: 这是**信息检索**和**复杂UI理解**的复合失败。首先，它无法在密集网格中准确定位目标（检索失败）。其次，它没有更好的错误恢复逻辑，只会采用最原始、最低效的线性导航策略。
|
||||
|
||||
**2. 任务: `RecipeDeleteDuplicateRecipes3` (task 52)**
|
||||
* **目标**: 删除所有重复的食谱，但要确保每种独特的食谱至少保留一个实例。
|
||||
* **失败过程分析**:
|
||||
1. **子任务成功**: 日志显示，Agent 在初期是成功的。它能识别出“Avocado Toast with Egg”是重复的，并成功删除了一个副本。它也能识别并删除一个“BBQ Chicken Quesadillas”的副本。
|
||||
2. **状态维护失败**: **失败点在于**，在成功执行了几次删除后，它似乎“忘记”了自己已经处理过哪些食谱，或者被列表更新后的新布局搞糊涂了。它开始陷入循环，反复尝试删除“Butternut Squash Soup”，或者在已经清理过的项目上再次尝试，最终在长达34步的无效操作后超时。
|
||||
* **根本原因**: 这是一个典型的**复杂任务规划和状态维护**的失败。Agent 可以执行“删除一个重复项”这个简单的子程序，但它无法管理一个更高级的、需要记忆和迭代的循环逻辑（“*当还有未处理的重复项时，找到下一组并处理*”）。它的任务计划非常脆弱，一旦UI状态因自身操作而改变（列表项变少），它就无法适应新的状态，导致逻辑混乱。
|
||||
*
|
||||
|
||||
### 超时类型
|
||||
|
||||
好的，我们来对 `t3a_failed.md` 文件中所有任务进行一次全面的统计，专门找出那些因为“达到最大步数限制”而失败的案例。
|
||||
|
||||
失败日志中对应的标志性语句是：
|
||||
`Agent did not indicate task is done. Reached max number of steps.`
|
||||
|
||||
经过对整个 `t3a_failed.md` 文件内容的统计，在所有失败的任务中，**总共有 28 个任务**是因为步数耗尽而失败的。
|
||||
|
||||
单纯说“步数不足”是表象，更深层的原因是 Agent **为何会**耗尽步数。我们可以将这 28 个失败案例归为以下几种典型的“低效模式”：
|
||||
|
||||
#### 1. 陷入无效的逻辑循环或低效策略 (9个任务)
|
||||
|
||||
Agent 知道目标，但它采用的策略是错误的或极其低效的，导致它在重复的无用功中耗尽了所有步数。
|
||||
|
||||
* **`ClockTimerEntry`**: 发现输入错误后，能够清零，但随即又**重复了一遍完全相同的错误输入**，陷入循环。
|
||||
* **`RecipeDeleteDuplicateRecipes` (3个相关任务)**: 能够成功删除一两个重复项，但随后就**逻辑混乱**，无法系统性地清理所有重复项，开始无效点击，直到超时。
|
||||
* **`SimpleCalendar...` (4个相关任务)**: 在日历中定位日期时，一旦首次点击错误，就陷入了“从A日逐天点击到B日”的**最低效导航策略**，而不是返回月视图重新选择，快速消耗了步数。
|
||||
* **`MarkorTranscribeVideo`**: 由于无法从视频中获取信息，它只能反复“点击播放器”，期待状态改变，陷入了**无信息输入的无效等待循环**。
|
||||
|
||||
#### 2. 被复杂或非标UI交互所困 (7个任务)
|
||||
|
||||
Agent 在面对不熟悉的、需要精细操作的UI控件时，会反复尝试错误的交互方式，直至失败。
|
||||
|
||||
* **`MarkorAddNoteHeader`, `MarkorChangeNoteContent`, `MarkorEditNote`**: 这三个任务都失败在**文本编辑器**上。Agent 不理解如何精确地定位光标、插入文本、替换部分文本，而是错误地选择了“全选”等不适用的策略。
|
||||
* **`MarkorMoveNote`**: 在文件移动的对话框中，虽然最终找到了正确的文件夹，但之前的导航步骤过于繁琐，耗尽了步数。
|
||||
* **`OsmAndMarker`**: 在地图应用中，它完全无法理解如何添加“标记点”，在长按、点击等操作中浪费了所有步数。
|
||||
* **`SimpleDrawProCreateDrawing`**: 在保存文件的对话框中，它无法正确地**清空并重命名**文件，反复进行无效的删除和输入尝试。
|
||||
|
||||
#### 3. 信息处理能力缺失导致的操作停滞 (5个任务)
|
||||
|
||||
Agent 能够看到数据，但无法在认知层面进行处理（计数、求和、比较），导致它只能反复地、漫无目的地滚动浏览，无法给出答案。
|
||||
|
||||
* **`SportsTrackerActivitiesCountForWeek`**
|
||||
* **`SportsTrackerActivityDuration`**
|
||||
* **`SportsTrackerLongestDistanceActivity`**
|
||||
* **`SportsTrackerTotalDistanceForCategoryOverInterval`**
|
||||
* **`SportsTrackerTotalDurationForCategoryThisWeek`**
|
||||
|
||||
在这5个任务中，Agent 的行为模式高度一致：成功打开应用，然后花费大量步数（通常是7-8步）反复上下滚动列表，最后因为无法进行**计数、求和、比较大小**等数学运算而超时失败。
|
||||
|
||||
#### 4. 简单的导航或执行失败 (7个任务)
|
||||
|
||||
在这些看似直接的任务中，Agent 依然会因为某些原因“迷路”或卡住，最终超时。
|
||||
|
||||
* **`MarkorCreateFolder`, `MarkorCreateNoteAndSms`, `MarkorCreateNoteFromClipboard`**: 在创建文件/文件夹的流程中，Agent 在处理文件名、扩展名或后续分享步骤时被卡住。
|
||||
* **`MarkorDeleteNewestNote`, `MarkorDeleteNote`**: 在删除文件的流程中，它在排序或选择文件后，未能及时执行删除操作。
|
||||
* **`RetroPlaylistDuration`**: 在添加歌曲到播放列表时，它能成功添加几首，但似乎没有一个明确的停止条件和检查机制（检查总时长），导致无限添加下去直至超时。
|
||||
* **`SimpleCalendarDeleteEvents`**: 在删除多个事件时，它成功删除了一个，但之后没能有效地继续删除下一个，浪费了步骤。
|
||||
|
||||
### 结论
|
||||
|
||||
**“步数耗尽”更像是一个“症状”而非“病因”。**
|
||||
这 28 个案例清晰地表明，Agent 的失败并非因为步数限制本身过于苛刻，而是因为它缺乏高效解决问题的核心能力。当面对其能力短板（如复杂UI理解、数学运算、高级任务规划）时，它无法制定出简洁有效的执行路径，只能通过大量低效甚至无效的尝试来“凑步数”，最终不可避免地走向失败。
|
||||
@@ -0,0 +1,254 @@
|
||||
### 核心内容概括
|
||||
|
||||
1. **逐任务性能列表 (Per-Task Performance List):**
|
||||
- 详细列出了 Agent 在116个不同任务上的表现。
|
||||
- 关键指标包括：成功率、平均任务长度（步数）、耗时等。
|
||||
2. **按能力标签与难度划分的性能分析 (Performance Analysis by Capability Tag and Difficulty):**
|
||||
- 将所有任务按照所需的核心能力和难度进行分类。
|
||||
- 统计并展示 Agent 在每种能力/难度组合下的平均成功率。
|
||||
|
||||
### 第一部分：逐任务性能分析
|
||||
|
||||
---
|
||||
|
||||
### **1. 表格列释义**
|
||||
|
||||
我们首先明确每个数据列的含义：
|
||||
|
||||
- `task`: 任务的唯一名称，清晰地描述了任务目标。例如 `AudioRecorderRecordAudio` (录音机录制音频) 或 `ContactsAddContact` (联系人应用中添加联系人)。
|
||||
- `task_num`: 任务的数字编号，从0到115。
|
||||
- `num_complete_trials`: 已完成的尝试次数。在这里，每个任务都只尝试了1次。
|
||||
- `mean_success_rate`: 平均成功率。由于每个任务只尝试1次，`1.00` 代表成功，`0.00` 代表失败。
|
||||
- `mean_episode_length`: 平均回合长度。一个“回合”指 Agent 为完成一次任务所执行的**总步数**（如点击、输入、滑动等）。这个数值越高，通常意味着任务越复杂或 Agent 走了弯路。对于失败的任务，此值为 `NaN` (Not a Number)，表示没有成功的记录。
|
||||
- `total_runtime_s`: 完成该任务总共消耗的时间，单位是秒 (s)。
|
||||
- `num_fail_trials`: 失败的尝试次数。`0.00` 表示该次尝试成功，`1.00` 表示失败。
|
||||
|
||||
### **2. 洞察与发现**
|
||||
|
||||
1. 压倒性的成功率：
|
||||
|
||||
从 task_num 0 到 81，以及从 93 到 101，Agent 的 mean_success_rate 均为 1.00。这表明 Agent 在处理绝大多数结构化、明确的任务时表现得非常出色且稳定。这些任务涵盖了：
|
||||
|
||||
- **基础应用操作**：相机、时钟、录音机、联系人。
|
||||
- **文件管理**：删除文件、移动文件。
|
||||
- **内容创作与编辑**：在 `Markor`（一个笔记应用）中创建/编辑笔记。
|
||||
- **系统设置**：开关蓝牙、调节亮度。
|
||||
2. 明确的失败群：
|
||||
|
||||
失败的案例非常集中，主要出现在 task_num 82 以及 102 至 115 的任务中。
|
||||
|
||||
- **`SimpleSmsReplyMostRecent` (task 82):** 回复最近一条短信，失败。
|
||||
- **`SystemWifiTurn...` (tasks 102, 103, 104, 105):** 与开关Wi-Fi及验证状态相关的任务，有多次失败。
|
||||
- **`Tasks...` (tasks 106 - 111):** 与 `Tasks`（待办事项）应用相关的一系列查询任务，全部失败。
|
||||
- **`Turn...` (tasks 112, 113):** 涉及Wi-Fi和蓝牙组合操作的任务，失败。
|
||||
- **`Vlc...` (tasks 114, 115):** 在VLC播放器中创建播放列表的任务，失败。
|
||||
3. 任务复杂度的体现：
|
||||
|
||||
观察 mean_episode_length（平均步数），我们可以评估任务的复杂度。
|
||||
|
||||
- **简单任务**：`ClockStopWatchPausedVerify` (task 7) 仅需 3 步。
|
||||
- **复杂任务**：`RecipeAddMultipleRecipesFromMarkor2` (task 48) 需要 44 步，`OsmAndTrack` (task 44) 需要 41 步。这说明 Agent 能够维持一个较长的操作序列来完成复杂的目标。
|
||||
|
||||
**d. 总体性能平均值 (Average)：**
|
||||
|
||||
- `mean_success_rate`: **`0.88`**，即 88% 的总体成功率。这是一个非常高的指标，说明 Agent 具有很强的泛化能力和执行能力。
|
||||
- `num_fail_trials`: **`0.12`**，即 12% 的失败率，与成功率相对应。
|
||||
- `mean_episode_length`: **`13.45`** 步，所有成功任务的平均操作步数。
|
||||
|
||||
### 第二部分：按能力标签与难度划分的性能分析
|
||||
|
||||
### **1. 表格解读**
|
||||
|
||||
这张表格是一个诊断矩阵。它不再关注单个任务的成败，而是将任务解构成一系列所需的核心能力（`tags`），并在不同的难度等级（`easy`, `medium`, `hard`）下评估 Agent 的表现。表格中的数值是对应类别下所有任务的**平均成功率**。
|
||||
|
||||
- **`tags`**: 描述任务所需要的一种或多种核心能力。例如：
|
||||
- `complex_ui_understanding`: 理解复杂或非标准的界面布局。
|
||||
- `math_counting`: 需要进行数学计算或计数。
|
||||
- `transcription`: 需要从一种形式（如图像、视频）转录信息到文本。
|
||||
- `information_retrieval`: 从屏幕上寻找并提取特定信息。
|
||||
- **`difficulty`**: 任务的难度级别。
|
||||
- **数值**: 该 `tag` 和 `difficulty` 组合下所有任务的平均成功率。`1.0` 表示100%成功, `0.0` 表示0%成功, 或 `NaN` 表示数据集中没有该组合的任务。
|
||||
|
||||
### **2. Agent 的能力画像**
|
||||
|
||||
### **核心优势 (Core Strengths)**
|
||||
|
||||
Agent 在以下几个方面表现出了近乎完美或非常强的能力：
|
||||
|
||||
- **跨应用操作 (`multi_app`)**: 在 `easy` 级别下成功率为 `1.00`。这表明 Agent 能够可靠地在不同应用程序之间进行切换以完成任务。
|
||||
- **记忆 (`memorization`)**: 在 `easy` 级别下成功率为 `1.00`。Agent 具备有效的短期记忆能力，能够记住先前步骤的信息（例如，一个文件名）并在后续步骤中使用它。
|
||||
- **搜索 (`search`)**: 在 `medium` 级别下成功率为 `1.00`，在 `easy` 级别下也有 `0.60` 的表现。这说明 Agent 擅长使用应用内或系统级的搜索功能来定位信息。
|
||||
|
||||
### **关键短板 (Critical Weaknesses)**
|
||||
|
||||
- **转录 (`transcription`)**: **成功率为 `0.00`**。这是最严重的失败，表明 Agent **完全不具备**从图像或视频等非结构化源头准确提取并转录信息到文本字段的能力。这可能源于其视觉模型（Vision Model）在光学字符识别（OCR）上的缺陷。
|
||||
- **数学/计数 (`math_counting`)**: 在 `easy` 级别下成功率为 **`0.00`**，在中等和困难级别下也仅有 `0.33`。这是一个重大的认知缺陷。Agent 似乎无法在手机操作的情境中执行简单的数学运算或对界面元素进行计数。
|
||||
- **需要预设 (`requires_setup`)**: 在 `easy` 级别下成功率为 **`0.00`**。Agent 无法处理那些需要先进行特定环境设置（例如，确保某个文件存在或某个设置开启）才能开始的任务。它可能缺乏检查前置条件并根据情况采取修正动作的能力。
|
||||
- **复杂UI理解 (`complex_ui_understanding`)**: 成功率普遍很低 (`easy` 0.17, `hard` 0.14)。这是另一个核心弱点。Agent 的操作严重依赖于**标准、规范的UI设计**。一旦遇到布局复杂、控件非主流或信息密度高的界面，它就很容易“迷路”，无法准确定位到正确的交互元素。
|
||||
- **信息检索 (`information_retrieval`)**: 在 `easy` 级别下成功率仅为 `0.17`。这与 `complex_ui_understanding` 弱点高度相关。即使在简单的场景下，如果信息没有以一种简单明了的方式呈现，Agent 也很难从中找到并提取出需要的内容。
|
||||
|
||||
---
|
||||
|
||||
### 第三部分：总体结论与推断
|
||||
|
||||
现在，我们可以将两部分的分析联系起来，形成一个完整的结论。
|
||||
|
||||
**`t3a_claude4_sonnet` Agent 的总体画像是：一个在执行标准、线性流程任务方面非常高效的“操作手”，但在需要深度视觉理解、逻辑推理和适应非标准环境等高级认知能力的“思考者”角色上存在明显不足。**
|
||||
|
||||
**为什么那些任务会失败？**
|
||||
|
||||
- **`SystemWifiTurn...` (tasks 102-105) 和 `Tasks...` (tasks 106-111) 等的失败**：可以高度归因于 **`complex_ui_understanding`** 和 **`information_retrieval`** 的双重失败。系统设置界面、某些设计不佳的应用（可能是 `Tasks` 应用）的UI可能不符合Agent的“预期”，导致它无法找到正确的开关或读取到正确的状态信息。
|
||||
- **`SimpleSmsReplyMostRecent` (task 82) 的失败**：可能涉及 `information_retrieval`（需要准确识别出“哪一条是最近的”）和 `complex_ui_understanding`。
|
||||
- 所有涉及**数学、转录、需要预设**的任务失败，其根本原因已在第二部分中清晰揭示。
|
||||
|
||||
### **下一步的分析与优化建议**
|
||||
|
||||
基于以上分析，后续的优化路径非常清晰：
|
||||
|
||||
1. **根因分析 (Root Cause Analysis)**: 我们目前是基于统计摘要进行的宏观推断。下一步最关键的操作，就是深入到**具体失败任务的详细操作日志**中。通过逐帧查看 Agent 的“观察 (`Observation`) -> 思考 (`Thought`) -> 行动 (`Action`)”链条，我们可以精确地看到它是在哪一步、因为什么样的错误感知或推理而导致了任务失败。
|
||||
2. **模型与算法优化 (Model & Algorithm Optimization)**:
|
||||
- 针对 **`complex_ui`** 问题，需要用更多样化、更复杂的UI布局数据来训练 Agent，或者开发更鲁棒的UI解析模块（例如，将UI元素解析为图结构而非仅仅是位置和文本）。
|
||||
- 针对 **`transcription`** 和 **`math_counting`** 问题，可能需要在 Agent 的工具集中集成更强大的专用工具，例如一个高精度的OCR服务或一个计算器工具，并教会 Agent 何时以及如何调用这些工具。
|
||||
|
||||
```shell
|
||||
task_num num_complete_trials mean_success_rate mean_episode_length total_runtime_s num_fail_trials
|
||||
task
|
||||
AudioRecorderRecordAudio 0 1.00 0.0 10.00 101.80 0.00
|
||||
AudioRecorderRecordAudioWithFileName 1 1.00 0.0 20.00 326.70 0.00
|
||||
BrowserDraw 2 1.00 0.0 20.00 391.40 0.00
|
||||
BrowserMaze 3 1.00 1.0 16.00 195.50 0.00
|
||||
BrowserMultiply 4 1.00 1.0 15.00 177.30 0.00
|
||||
CameraTakePhoto 5 1.00 1.0 4.00 41.20 0.00
|
||||
CameraTakeVideo 6 1.00 1.0 7.00 83.80 0.00
|
||||
ClockStopWatchPausedVerify 7 1.00 1.0 3.00 30.10 0.00
|
||||
ClockStopWatchRunning 8 1.00 1.0 4.00 40.80 0.00
|
||||
ClockTimerEntry 9 1.00 0.0 10.00 127.10 0.00
|
||||
ContactsAddContact 10 1.00 1.0 8.00 106.60 0.00
|
||||
ContactsNewContactDraft 11 1.00 1.0 8.00 108.40 0.00
|
||||
ExpenseAddMultiple 12 1.00 1.0 25.00 356.10 0.00
|
||||
ExpenseAddMultipleFromGallery 13 1.00 0.0 32.00 455.00 0.00
|
||||
ExpenseAddMultipleFromMarkor 14 1.00 0.0 24.00 325.70 0.00
|
||||
ExpenseAddSingle 15 1.00 1.0 10.00 140.40 0.00
|
||||
ExpenseDeleteDuplicates 16 1.00 1.0 10.00 151.50 0.00
|
||||
ExpenseDeleteDuplicates2 17 1.00 1.0 15.00 381.10 0.00
|
||||
ExpenseDeleteMultiple 18 1.00 1.0 11.00 152.90 0.00
|
||||
ExpenseDeleteMultiple2 19 1.00 1.0 14.00 198.30 0.00
|
||||
ExpenseDeleteSingle 20 1.00 1.0 5.00 65.30 0.00
|
||||
FilesDeleteFile 21 1.00 1.0 10.00 129.60 0.00
|
||||
FilesMoveFile 22 1.00 1.0 13.00 164.00 0.00
|
||||
MarkorAddNoteHeader 23 1.00 0.0 12.00 175.70 0.00
|
||||
MarkorChangeNoteContent 24 1.00 0.0 12.00 174.10 0.00
|
||||
MarkorCreateFolder 25 1.00 0.0 10.00 137.10 0.00
|
||||
MarkorCreateNote 26 1.00 1.0 13.00 190.50 0.00
|
||||
MarkorCreateNoteAndSms 27 1.00 0.0 18.00 264.50 0.00
|
||||
MarkorCreateNoteFromClipboard 28 1.00 0.0 14.00 194.60 0.00
|
||||
MarkorDeleteAllNotes 29 1.00 1.0 13.00 154.80 0.00
|
||||
MarkorDeleteNewestNote 30 1.00 0.0 10.00 124.70 0.00
|
||||
MarkorDeleteNote 31 1.00 0.0 10.00 120.80 0.00
|
||||
MarkorEditNote 32 1.00 0.0 12.00 169.60 0.00
|
||||
MarkorMergeNotes 33 1.00 0.0 31.00 452.30 0.00
|
||||
MarkorMoveNote 34 1.00 0.0 14.00 185.70 0.00
|
||||
MarkorTranscribeReceipt 35 1.00 0.0 18.00 232.30 0.00
|
||||
MarkorTranscribeVideo 36 1.00 0.0 20.00 266.70 0.00
|
||||
NotesIsTodo 37 1.00 1.0 3.00 43.30 0.00
|
||||
NotesMeetingAttendeeCount 38 1.00 1.0 7.00 87.80 0.00
|
||||
NotesRecipeIngredientCount 39 1.00 1.0 6.00 82.70 0.00
|
||||
NotesTodoItemCount 40 1.00 1.0 7.00 102.50 0.00
|
||||
OpenAppTaskEval 41 1.00 1.0 3.00 35.20 0.00
|
||||
OsmAndFavorite 42 1.00 1.0 10.00 148.60 0.00
|
||||
OsmAndMarker 43 1.00 0.0 20.00 275.00 0.00
|
||||
OsmAndTrack 44 1.00 0.0 41.00 607.30 0.00
|
||||
RecipeAddMultipleRecipes 45 1.00 1.0 36.00 540.30 0.00
|
||||
RecipeAddMultipleRecipesFromImage 46 1.00 0.0 24.00 316.70 0.00
|
||||
RecipeAddMultipleRecipesFromMarkor 47 1.00 0.0 25.00 363.80 0.00
|
||||
RecipeAddMultipleRecipesFromMarkor2 48 1.00 0.0 44.00 719.30 0.00
|
||||
RecipeAddSingleRecipe 49 1.00 1.0 13.00 190.90 0.00
|
||||
RecipeDeleteDuplicateRecipes 50 1.00 0.0 10.00 153.60 0.00
|
||||
RecipeDeleteDuplicateRecipes2 51 1.00 0.0 24.00 348.60 0.00
|
||||
RecipeDeleteDuplicateRecipes3 52 1.00 0.0 34.00 433.90 0.00
|
||||
RecipeDeleteMultipleRecipes 53 1.00 0.0 24.00 328.80 0.00
|
||||
RecipeDeleteMultipleRecipesWithConstraint 54 1.00 1.0 11.00 164.00 0.00
|
||||
RecipeDeleteMultipleRecipesWithNoise 55 1.00 1.0 20.00 252.80 0.00
|
||||
RecipeDeleteSingleRecipe 56 1.00 1.0 6.00 68.50 0.00
|
||||
RecipeDeleteSingleWithRecipeWithNoise 57 1.00 1.0 7.00 84.60 0.00
|
||||
RetroCreatePlaylist 58 1.00 1.0 21.00 270.50 0.00
|
||||
RetroPlayingQueue 59 1.00 1.0 17.00 235.80 0.00
|
||||
RetroPlaylistDuration 60 1.00 0.0 30.00 386.10 0.00
|
||||
RetroSavePlaylist 61 1.00 1.0 27.00 321.30 0.00
|
||||
SaveCopyOfReceiptTaskEval 62 1.00 1.0 11.00 132.40 0.00
|
||||
SimpleCalendarAddOneEvent 63 1.00 0.0 14.00 202.30 0.00
|
||||
SimpleCalendarAddOneEventInTwoWeeks 64 1.00 0.0 18.00 233.70 0.00
|
||||
SimpleCalendarAddOneEventRelativeDay 65 1.00 0.0 17.00 231.50 0.00
|
||||
SimpleCalendarAddOneEventTomorrow 66 1.00 0.0 18.00 235.70 0.00
|
||||
SimpleCalendarAddRepeatingEvent 67 1.00 0.0 16.00 224.10 0.00
|
||||
SimpleCalendarAnyEventsOnDate 68 1.00 0.0 10.00 164.80 0.00
|
||||
SimpleCalendarDeleteEvents 69 1.00 0.0 14.00 210.00 0.00
|
||||
SimpleCalendarDeleteEventsOnRelativeDay 70 1.00 0.0 3.00 45.60 0.00
|
||||
SimpleCalendarDeleteOneEvent 71 1.00 0.0 12.00 186.20 0.00
|
||||
SimpleCalendarEventOnDateAtTime 72 1.00 0.0 6.00 104.20 0.00
|
||||
SimpleCalendarEventsInNextWeek 73 1.00 0.0 6.00 85.80 0.00
|
||||
SimpleCalendarEventsInTimeRange 74 1.00 0.0 10.00 174.50 0.00
|
||||
SimpleCalendarEventsOnDate 75 1.00 1.0 6.00 198.00 0.00
|
||||
SimpleCalendarFirstEventAfterStartTime 76 1.00 0.0 10.00 167.50 0.00
|
||||
SimpleCalendarLocationOfEvent 77 1.00 1.0 6.00 110.80 0.00
|
||||
SimpleCalendarNextEvent 78 1.00 0.0 7.00 105.50 0.00
|
||||
SimpleCalendarNextMeetingWithPerson 79 1.00 0.0 4.00 60.60 0.00
|
||||
SimpleDrawProCreateDrawing 80 1.00 0.0 18.00 307.90 0.00
|
||||
SimpleSmsReply 81 1.00 0.0 9.00 211.30 0.00
|
||||
SimpleSmsReplyMostRecent 82 0.00 NaN NaN 17.30 1.00
|
||||
SimpleSmsResend 83 1.00 0.0 8.00 142.40 0.00
|
||||
SimpleSmsSend 84 1.00 0.0 8.00 132.70 0.00
|
||||
SimpleSmsSendClipboardContent 85 1.00 0.0 8.00 132.50 0.00
|
||||
SimpleSmsSendReceivedAddress 86 1.00 0.0 18.00 276.50 0.00
|
||||
SportsTrackerActivitiesCountForWeek 87 1.00 0.0 10.00 180.20 0.00
|
||||
SportsTrackerActivitiesOnDate 88 1.00 0.0 5.00 83.40 0.00
|
||||
SportsTrackerActivityDuration 89 1.00 0.0 10.00 156.40 0.00
|
||||
SportsTrackerLongestDistanceActivity 90 1.00 0.0 10.00 191.60 0.00
|
||||
SportsTrackerTotalDistanceForCategoryOverInterval 91 1.00 0.0 20.00 351.00 0.00
|
||||
SportsTrackerTotalDurationForCategoryThisWeek 92 1.00 0.0 10.00 187.50 0.00
|
||||
SystemBluetoothTurnOff 93 1.00 1.0 8.00 90.20 0.00
|
||||
SystemBluetoothTurnOffVerify 94 1.00 1.0 7.00 82.20 0.00
|
||||
SystemBluetoothTurnOn 95 1.00 1.0 5.00 68.80 0.00
|
||||
SystemBluetoothTurnOnVerify 96 1.00 1.0 4.00 57.10 0.00
|
||||
SystemBrightnessMax 97 1.00 0.0 10.00 120.50 0.00
|
||||
SystemBrightnessMaxVerify 98 1.00 0.0 10.00 131.20 0.00
|
||||
SystemBrightnessMin 99 1.00 0.0 10.00 125.60 0.00
|
||||
SystemBrightnessMinVerify 100 1.00 0.0 10.00 125.80 0.00
|
||||
SystemCopyToClipboard 101 1.00 0.0 5.00 79.30 0.00
|
||||
SystemWifiTurnOff 102 1.00 0.0 10.00 146.10 0.00
|
||||
SystemWifiTurnOffVerify 103 0.00 NaN NaN 29.10 1.00
|
||||
SystemWifiTurnOn 104 0.00 NaN NaN 29.90 1.00
|
||||
SystemWifiTurnOnVerify 105 0.00 NaN NaN 28.80 1.00
|
||||
TasksCompletedTasksForDate 106 0.00 NaN NaN 40.30 1.00
|
||||
TasksDueNextWeek 107 0.00 NaN NaN 30.60 1.00
|
||||
TasksDueOnDate 108 0.00 NaN NaN 30.30 1.00
|
||||
TasksHighPriorityTasks 109 0.00 NaN NaN 33.80 1.00
|
||||
TasksHighPriorityTasksDueOnDate 110 0.00 NaN NaN 29.20 1.00
|
||||
TasksIncompleteTasksOnDate 111 0.00 NaN NaN 30.40 1.00
|
||||
TurnOffWifiAndTurnOnBluetooth 112 0.00 NaN NaN 29.30 1.00
|
||||
TurnOnWifiAndOpenApp 113 0.00 NaN NaN 39.20 1.00
|
||||
VlcCreatePlaylist 114 0.00 NaN NaN 9.10 1.00
|
||||
VlcCreateTwoPlaylists 115 0.00 NaN NaN 9.30 1.00
|
||||
========= Average ========= 0 0.88 0.4 13.45 174.96 0.12
|
||||
|
||||
|
||||
mean_success_rate
|
||||
difficulty easy medium hard
|
||||
tags
|
||||
complex_ui_understanding 0.17 0.5 0.14
|
||||
data_edit 0.36 0.5 0.0
|
||||
data_entry 0.27 0.44 0.0
|
||||
game_playing 0.50 - -
|
||||
information_retrieval 0.17 0.4 0.0
|
||||
math_counting 0.00 0.33 0.33
|
||||
memorization 1.00 0.0 0.25
|
||||
multi_app 1.00 0.0 0.0
|
||||
parameterized 0.37 0.44 0.22
|
||||
repetition 0.50 0.5 0.4
|
||||
requires_setup 0.00 0.5 0.0
|
||||
screen_reading 0.40 0.5 0.33
|
||||
search 0.60 1.0 0.17
|
||||
transcription 0.00 0.0 0.0
|
||||
untagged 0.60 1.0 -
|
||||
verification 0.60 - -
|
||||
```
|
||||
@@ -0,0 +1,474 @@
|
||||
"""Focused, offline checks for Experiment 7-12 evidence/reporting."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from argparse import Namespace
|
||||
import sqlite3
|
||||
|
||||
import pytest
|
||||
|
||||
from experiment_core import (
|
||||
BASELINE_TASK_COUNT,
|
||||
aggregate_episodes,
|
||||
choose_decision,
|
||||
choose_efficiency_decision,
|
||||
enforce_scope_claims,
|
||||
paired_rows,
|
||||
redact_text,
|
||||
render_report,
|
||||
)
|
||||
from run_controlled_experiment import (
|
||||
_context_safe_output_cap,
|
||||
_missing_retro_queue_as_empty,
|
||||
_read_nonempty_with_retry,
|
||||
_retry_clipper_foreground,
|
||||
_truncate_current_ui_section,
|
||||
_validate_resume_evidence,
|
||||
)
|
||||
|
||||
|
||||
def _episode(arm: str, task: str, success: bool, latency: float) -> dict:
|
||||
return {
|
||||
"pair_id": task + ":trial-1",
|
||||
"task": task,
|
||||
"trial": 1,
|
||||
"arm": arm,
|
||||
"status": "completed",
|
||||
"success": success,
|
||||
"evaluator_reward": float(success),
|
||||
"steps": 3 if success else 10,
|
||||
"elapsed_s": latency,
|
||||
"llm": {"calls": 4, "input_tokens": 100, "output_tokens": 20},
|
||||
}
|
||||
|
||||
|
||||
def test_redaction_covers_explicit_and_pattern_credentials() -> None:
|
||||
secret = "definitely-not-for-output"
|
||||
text = redact_text(
|
||||
f"api_key={secret} Authorization: Bearer abcdefghijk sk-example123456789",
|
||||
[secret],
|
||||
)
|
||||
assert secret not in text
|
||||
assert "abcdefghijk" not in text
|
||||
assert "sk-example123456789" not in text
|
||||
assert text.count("[REDACTED]") >= 3
|
||||
|
||||
|
||||
def test_retro_missing_queue_schema_becomes_empty_observation() -> None:
|
||||
def missing_queue(_env: object) -> list[str]:
|
||||
raise sqlite3.OperationalError("no such table: playing_queue")
|
||||
|
||||
assert _missing_retro_queue_as_empty(missing_queue)(object()) == []
|
||||
|
||||
|
||||
def test_retro_compatibility_does_not_hide_other_sqlite_errors() -> None:
|
||||
def corrupt_database(_env: object) -> list[str]:
|
||||
raise sqlite3.OperationalError("database disk image is malformed")
|
||||
|
||||
with pytest.raises(sqlite3.OperationalError, match="malformed"):
|
||||
_missing_retro_queue_as_empty(corrupt_database)(object())
|
||||
|
||||
|
||||
def test_context_cap_keeps_headroom_for_provider_lower_bound() -> None:
|
||||
error = (
|
||||
"This model's maximum context length is 32768 tokens. However, you "
|
||||
"requested 1024 output tokens and your prompt contains at least 31745 "
|
||||
"input tokens."
|
||||
)
|
||||
assert _context_safe_output_cap(error, 1024) == 991
|
||||
assert _context_safe_output_cap("unrelated provider error", 1024) is None
|
||||
|
||||
|
||||
def test_context_truncation_is_limited_to_middle_of_current_ui() -> None:
|
||||
prefix = "prefix and goal"
|
||||
ui = "A" * 9000 + "M" * 16384 + "Z" * 9000
|
||||
suffix = "guidance and output format"
|
||||
prompt = (
|
||||
prefix
|
||||
+ "\n\nHere is a list of descriptions for some UI elements on the current screen:\n"
|
||||
+ ui
|
||||
+ "\nHere are some useful guidelines you need to follow:\n"
|
||||
+ suffix
|
||||
)
|
||||
result = _truncate_current_ui_section(prompt)
|
||||
assert result is not None
|
||||
truncated, removed = result
|
||||
assert prefix in truncated and suffix in truncated
|
||||
assert "A" * 1000 in truncated and "Z" * 1000 in truncated
|
||||
assert removed > 0
|
||||
assert len(truncated) < len(prompt)
|
||||
|
||||
|
||||
def test_context_truncation_handles_before_and_after_summary_ui() -> None:
|
||||
before = "B" * 12000
|
||||
after = "A" * 12000
|
||||
prompt = (
|
||||
"goal and summary rules\n"
|
||||
"Here is the description for the before screenshot:\n"
|
||||
+ before
|
||||
+ "\nHere is the description for the after screenshot:\n"
|
||||
+ after
|
||||
+ "\nThis is the action you picked: click\nBased on the reason: test"
|
||||
)
|
||||
result = _truncate_current_ui_section(prompt)
|
||||
assert result is not None
|
||||
truncated, removed = result
|
||||
assert "goal and summary rules" in truncated
|
||||
assert "This is the action you picked: click" in truncated
|
||||
assert "B" * 500 in truncated and "A" * 500 in truncated
|
||||
assert removed > 0
|
||||
|
||||
|
||||
def test_sms_inbox_poll_preserves_empty_then_observed_result() -> None:
|
||||
reads = iter([[], [], ["Row: 0, address=123, body=hello"]])
|
||||
assert _read_nonempty_with_retry(
|
||||
lambda: next(reads), attempts=3, delay_s=0
|
||||
) == ["Row: 0, address=123, body=hello"]
|
||||
|
||||
|
||||
def test_clipper_retry_is_limited_to_exact_foreground_error() -> None:
|
||||
attempts = iter([
|
||||
RuntimeError(
|
||||
"Clipper app must be in the foreground to access clipboard. "
|
||||
"Additionally, app privileges must be granted manually."
|
||||
),
|
||||
"clipboard value",
|
||||
])
|
||||
|
||||
def flaky_call() -> str:
|
||||
result = next(attempts)
|
||||
if isinstance(result, Exception):
|
||||
raise result
|
||||
return result
|
||||
|
||||
assert _retry_clipper_foreground(flaky_call, delay_s=0) == "clipboard value"
|
||||
with pytest.raises(RuntimeError, match="unrelated"):
|
||||
_retry_clipper_foreground(
|
||||
lambda: (_ for _ in ()).throw(RuntimeError("unrelated")), delay_s=0
|
||||
)
|
||||
|
||||
|
||||
def test_paired_comparison_and_conservative_candidate_decision() -> None:
|
||||
episodes = []
|
||||
for index in range(4):
|
||||
task = f"wifi-{index}"
|
||||
episodes.extend([
|
||||
_episode("control", task, index > 0, 10.0),
|
||||
_episode("treatment", task, True, 11.0),
|
||||
])
|
||||
summary = aggregate_episodes(episodes)
|
||||
pairs = paired_rows(episodes)
|
||||
decision = choose_decision(summary, pairs)
|
||||
assert len(pairs) == 4
|
||||
assert decision["net_success_delta"] == 1
|
||||
assert decision["paired_regressions"] == 0
|
||||
assert decision["promote_to_full_suite_candidate"] is True
|
||||
assert decision["outcome"] == "promote_candidate_to_full_suite_rerun"
|
||||
|
||||
|
||||
def test_subset_can_never_claim_full_suite_completion() -> None:
|
||||
evidence = {
|
||||
"scope": {
|
||||
"mode": "candidate_rerun",
|
||||
"tasks": ["a", "b", "c", "d"],
|
||||
"trials_per_task": 5,
|
||||
"completed_episodes": 20,
|
||||
"error_episodes": 0,
|
||||
},
|
||||
"decision": {"source_paired_run_id": "paired-real"},
|
||||
}
|
||||
enforce_scope_claims(evidence)
|
||||
assert evidence["scope"]["full_suite_completed"] is False
|
||||
assert evidence["experiment_complete"] is False
|
||||
|
||||
|
||||
def test_full_suite_gate_requires_direct_116_by_5_evidence() -> None:
|
||||
tasks = [f"task-{index}" for index in range(BASELINE_TASK_COUNT)]
|
||||
episodes = [
|
||||
{
|
||||
"task": task,
|
||||
"trial": trial,
|
||||
"pair_seed": task_index * 1009 + trial,
|
||||
"arm": "candidate",
|
||||
"status": "completed",
|
||||
"evaluator_reward": 1.0,
|
||||
}
|
||||
for task_index, task in enumerate(tasks)
|
||||
for trial in range(1, 6)
|
||||
]
|
||||
evidence = {
|
||||
"scope": {
|
||||
"mode": "candidate_rerun",
|
||||
"tasks": tasks,
|
||||
"trials_per_task": 5,
|
||||
"completed_episodes": BASELINE_TASK_COUNT * 5,
|
||||
"error_episodes": 0,
|
||||
},
|
||||
"episodes": episodes,
|
||||
"decision": {"source_paired_run_id": "paired-real"},
|
||||
"environment": {
|
||||
"api_level": 33,
|
||||
"emulator_setup_completed": True,
|
||||
"app_provisioning": {"complete": True},
|
||||
},
|
||||
}
|
||||
enforce_scope_claims(evidence)
|
||||
assert evidence["scope"]["full_suite_completed"] is True
|
||||
assert evidence["experiment_complete"] is True
|
||||
|
||||
|
||||
def test_full_suite_gate_requires_reference_api_and_apps() -> None:
|
||||
tasks = [f"task-{index}" for index in range(BASELINE_TASK_COUNT)]
|
||||
episodes = [
|
||||
{
|
||||
"task": task,
|
||||
"trial": trial,
|
||||
"pair_seed": task_index * 1009 + trial,
|
||||
"arm": "candidate",
|
||||
"status": "completed",
|
||||
"evaluator_reward": 0.0,
|
||||
}
|
||||
for task_index, task in enumerate(tasks)
|
||||
for trial in range(1, 6)
|
||||
]
|
||||
evidence = {
|
||||
"scope": {
|
||||
"mode": "candidate_rerun",
|
||||
"tasks": tasks,
|
||||
"trials_per_task": 5,
|
||||
},
|
||||
"episodes": episodes,
|
||||
"decision": {"source_paired_run_id": "paired-real"},
|
||||
"environment": {
|
||||
"api_level": 35,
|
||||
"emulator_setup_completed": False,
|
||||
"app_provisioning": {"complete": False},
|
||||
},
|
||||
}
|
||||
enforce_scope_claims(evidence)
|
||||
assert evidence["scope"]["direct_episode_gate_completed"] is True
|
||||
assert evidence["scope"]["full_suite_completed"] is False
|
||||
assert evidence["experiment_complete"] is False
|
||||
|
||||
|
||||
def test_full_suite_gate_rejects_counters_without_direct_episodes() -> None:
|
||||
evidence = {
|
||||
"scope": {
|
||||
"mode": "candidate_rerun",
|
||||
"tasks": [f"task-{index}" for index in range(BASELINE_TASK_COUNT)],
|
||||
"trials_per_task": 5,
|
||||
"completed_episodes": BASELINE_TASK_COUNT * 5,
|
||||
"error_episodes": 0,
|
||||
},
|
||||
"episodes": [],
|
||||
"decision": {"source_paired_run_id": "paired-real"},
|
||||
}
|
||||
enforce_scope_claims(evidence)
|
||||
assert evidence["scope"]["direct_episode_gate_completed"] is False
|
||||
assert evidence["scope"]["full_suite_completed"] is False
|
||||
assert evidence["experiment_complete"] is False
|
||||
|
||||
|
||||
def test_success_gain_over_cost_guardrail_is_not_promoted() -> None:
|
||||
episodes = []
|
||||
for index in range(4):
|
||||
task = f"wifi-{index}"
|
||||
control = _episode("control", task, index > 0, 10.0)
|
||||
treatment = _episode("treatment", task, True, 20.0)
|
||||
treatment["llm"]["input_tokens"] = 1000
|
||||
treatment["llm"]["output_tokens"] = 200
|
||||
episodes.extend([control, treatment])
|
||||
summary = aggregate_episodes(episodes)
|
||||
decision = choose_decision(summary, paired_rows(episodes))
|
||||
assert decision["outcome"] == "restrict_candidate_due_to_cost"
|
||||
assert decision["guardrails"]["passed"] is False
|
||||
assert decision["promote_to_full_suite_candidate"] is False
|
||||
assert decision["deployment_approved"] is False
|
||||
|
||||
|
||||
def test_efficiency_refinement_can_promote_without_inventing_success_gain() -> None:
|
||||
episodes = []
|
||||
for index in range(4):
|
||||
task = f"wifi-{index}"
|
||||
control = _episode("control", task, True, 10.0)
|
||||
treatment = _episode("treatment", task, True, 9.0)
|
||||
treatment["llm"]["input_tokens"] = 40
|
||||
treatment["llm"]["output_tokens"] = 10
|
||||
episodes.extend([control, treatment])
|
||||
decision = choose_efficiency_decision(
|
||||
aggregate_episodes(episodes), paired_rows(episodes)
|
||||
)
|
||||
assert decision["net_success_delta"] == 0
|
||||
assert decision["paired_regressions"] == 0
|
||||
assert decision["guardrails"]["passed"] is True
|
||||
assert decision["promote_to_full_suite_candidate"] is True
|
||||
assert decision["deployment_approved"] is False
|
||||
|
||||
|
||||
def test_efficiency_refinement_rejects_cheap_but_unsuccessful_treatment() -> None:
|
||||
episodes = []
|
||||
for index in range(4):
|
||||
task = f"wifi-{index}"
|
||||
control = _episode("control", task, False, 10.0)
|
||||
treatment = _episode("treatment", task, False, 9.0)
|
||||
treatment["llm"]["input_tokens"] = 40
|
||||
treatment["llm"]["output_tokens"] = 10
|
||||
episodes.extend([control, treatment])
|
||||
decision = choose_efficiency_decision(
|
||||
aggregate_episodes(episodes), paired_rows(episodes)
|
||||
)
|
||||
assert decision["mean_token_ratio_treatment_over_control"] < 0.75
|
||||
assert decision["success_preservation_passed"] is False
|
||||
assert decision["guardrails"]["passed"] is False
|
||||
assert decision["outcome"] == "reject_efficiency_candidate_due_to_regression"
|
||||
assert decision["promote_to_full_suite_candidate"] is False
|
||||
|
||||
|
||||
def test_report_labels_historical_and_hypothetical_numbers() -> None:
|
||||
evidence = {
|
||||
"run_id": "test-run",
|
||||
"generated_at_utc": "2026-07-29T00:00:00Z",
|
||||
"environment": {
|
||||
"android_world_commit": "abc123",
|
||||
"device_model": "emulator",
|
||||
"api_level": 35,
|
||||
"upstream_tested_api_level": 33,
|
||||
},
|
||||
"model": {"provider": "real-provider", "model": "real-model"},
|
||||
"scope": {
|
||||
"tasks": ["SystemWifiTurnOn"],
|
||||
"trials_per_task": 1,
|
||||
"mode": "paired",
|
||||
"full_suite_completed": False,
|
||||
},
|
||||
"diagnosis": {"findings": ["Historical finding."]},
|
||||
"hypothesis": {
|
||||
"id": "H1",
|
||||
"change": "Add a task guideline.",
|
||||
"expected_result": "Improve paired reward.",
|
||||
"guardrails": "Same task and model.",
|
||||
},
|
||||
"arm_summary": {},
|
||||
"paired_comparison": [],
|
||||
"decision": {
|
||||
"outcome": "insufficient_evidence",
|
||||
"reason": "Need four pairs.",
|
||||
},
|
||||
"episodes": [],
|
||||
"environment_boundaries": ["API mismatch."],
|
||||
"llm_analysis": {
|
||||
"status": "completed",
|
||||
"summary": "Observed subset summary.",
|
||||
"observed_failure_pattern": ["One bounded residual pattern."],
|
||||
"cost_benefit_interpretation": "No deployment approval.",
|
||||
"next_hypothesis": {
|
||||
"id": "H5",
|
||||
"layer": "middle",
|
||||
"idea": "Test the input path.",
|
||||
"target": "One paired gain.",
|
||||
"verification": "Matched paired run.",
|
||||
},
|
||||
},
|
||||
}
|
||||
report = render_report(evidence)
|
||||
assert "historical input evidence" in report
|
||||
assert "explicitly hypothetical" in report
|
||||
assert "not the complete AndroidWorld benchmark" in report
|
||||
assert "Full 116-task × 5-seed suite completed: **false**" in report
|
||||
assert "Observed subset summary." in report
|
||||
assert "No deployment approval." in report
|
||||
|
||||
|
||||
def test_resume_rejects_changed_configuration() -> None:
|
||||
evidence = {
|
||||
"experiment": "7-12",
|
||||
"hypothesis": {"id": "H5"},
|
||||
"scope": {
|
||||
"mode": "paired",
|
||||
"tasks": ["SystemWifiTurnOff"],
|
||||
"trials_per_task": 1,
|
||||
"max_steps": 10,
|
||||
},
|
||||
"model": {
|
||||
"model": "real-model",
|
||||
"seed": 42,
|
||||
"provider": "real-provider",
|
||||
"base_url": "https://provider.invalid/v1",
|
||||
"max_tokens": 1024,
|
||||
},
|
||||
"environment": {
|
||||
"skip_device_time": True,
|
||||
"device_serial": "emulator-5554",
|
||||
"grpc_port": 8554,
|
||||
},
|
||||
"episodes": [],
|
||||
}
|
||||
args = Namespace(
|
||||
tasks="SystemWifiTurnOff",
|
||||
hypothesis="H5",
|
||||
mode="paired",
|
||||
trials=1,
|
||||
max_steps=11,
|
||||
model="real-model",
|
||||
model_seed=42,
|
||||
provider="real-provider",
|
||||
base_url="https://provider.invalid/v1",
|
||||
max_model_tokens=1024,
|
||||
transition_pause=None,
|
||||
skip_device_time=True,
|
||||
console_port=5554,
|
||||
grpc_port=8554,
|
||||
seed=42,
|
||||
)
|
||||
with pytest.raises(RuntimeError, match="max_steps"):
|
||||
_validate_resume_evidence(evidence, args)
|
||||
|
||||
|
||||
def test_resume_rejects_changed_pair_seed() -> None:
|
||||
evidence = {
|
||||
"experiment": "7-12",
|
||||
"hypothesis": {"id": "H5C"},
|
||||
"scope": {
|
||||
"mode": "paired",
|
||||
"tasks": ["SystemWifiTurnOff"],
|
||||
"trials_per_task": 1,
|
||||
"max_steps": 10,
|
||||
},
|
||||
"model": {
|
||||
"model": "real-model",
|
||||
"seed": 42,
|
||||
"provider": "real-provider",
|
||||
"base_url": "https://provider.invalid/v1",
|
||||
"max_tokens": 1024,
|
||||
},
|
||||
"environment": {
|
||||
"skip_device_time": True,
|
||||
"device_serial": "emulator-5554",
|
||||
"grpc_port": 8554,
|
||||
},
|
||||
"episodes": [{
|
||||
"task": "SystemWifiTurnOff",
|
||||
"trial": 1,
|
||||
"arm": "control",
|
||||
"pair_seed": 42,
|
||||
}],
|
||||
}
|
||||
args = Namespace(
|
||||
tasks="SystemWifiTurnOff",
|
||||
hypothesis="H5C",
|
||||
mode="paired",
|
||||
trials=1,
|
||||
max_steps=10,
|
||||
model="real-model",
|
||||
model_seed=42,
|
||||
provider="real-provider",
|
||||
base_url="https://provider.invalid/v1",
|
||||
max_model_tokens=1024,
|
||||
transition_pause=None,
|
||||
skip_device_time=True,
|
||||
console_port=5554,
|
||||
grpc_port=8554,
|
||||
seed=43,
|
||||
)
|
||||
with pytest.raises(RuntimeError, match="Resume seed mismatch"):
|
||||
_validate_resume_evidence(evidence, args)
|
||||
@@ -0,0 +1,652 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-merged-20260804T092058Z`
|
||||
- Generated (UTC): `2026-08-04T09:20:58Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **true**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed policy, step budget, Pixel 6/API-33 device class, upstream setup, and app versions across isolated shards.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 580/580 | 0.045 | 0.134 | 9.672 | 109.861 | 18.998 | 169069.564 | 97384410 / 675937 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`full_candidate_rerun_completed`**
|
||||
- Reason: The direct 116-task x five-trial candidate rerun completed on five independent reference-environment shards. Negative evaluator results are retained.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
The complete 116-task, five-trial candidate rerun gate is satisfied by direct episode evidence.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudio / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudio / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudio / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudio / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / BrowserMultiply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ContactsAddContact / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsAddContact / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsAddContact / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ExpenseAddSingle / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteSingle / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteSingle / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteSingle / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / FilesDeleteFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateFolder / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateFolder / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateFolder / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteAllNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorDeleteNewestNote / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteNewestNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorDeleteNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorEditNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesIsTodo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OpenAppTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OpenAppTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndMarker / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndMarker / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleRecipe / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsReply / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsResend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivityDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMinVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMinVerify / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate rerun on 116 tasks across five independent shards showed a success rate of 4.48% with 26 successes out of 580 episodes. The mean latency was 109.86 seconds, and the mean number of LLM calls was 19. The mean total tokens were 169,069.56, with a mean evaluator reward of 0.133621.
|
||||
- Cost/benefit interpretation: The cost of running the candidate on 116 tasks was relatively high, with a mean latency of 109.86 seconds and a mean of 19 LLM calls per episode. The benefit, in terms of success rate, was minimal, with only 26 successes out of 580 episodes. The low success rate and high cost suggest that the current approach may not be efficient or effective.
|
||||
- Residual pattern: Multiple tasks, such as AudioRecorderRecordAudio, CameraTakePhoto, and ExpenseAddMultiple, consistently failed across multiple episodes.
|
||||
- Residual pattern: The failure rate was particularly high for tasks involving complex interactions with the UI, such as ContactsAddContact and ExpenseDeleteMultiple2.
|
||||
- Residual pattern: The evaluator reward was low for most episodes, indicating that the tasks were not being completed as expected.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 16,384 characters for action selection or 8,192 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- Run stopped by environment/runtime blocker: Generated params do not match checkpoint for MarkorCreateFolder:trial-4:seed-25270
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This user-requested local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,209 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:13:52Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.026 | 0.129 | 9.698 | 110.113 | 19.103 | 174058.216 | 20054856 / 135897 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / FilesDeleteFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OsmAndFavorite / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsResend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMinVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOff / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, achieving a success rate of 2.59% with an estimated cost of $0.00. The mean latency was 110.11 seconds, and the mean number of LLM calls was 19.10. The mean total tokens used were 174,058.22.
|
||||
- Cost/benefit interpretation: The cost of running the 116 tasks was minimal, with an estimated cost of $0.00. However, the low success rate of 2.59% indicates that the current approach is not cost-effective. The high latency and token usage suggest that the model may need optimization to reduce computational overhead.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only 3 out of 116 tasks succeeding.
|
||||
- Residual pattern: Tasks involving complex interactions with the UI, such as 'RetroPlayingQueue' and 'SportsTrackerTotalDistanceForCategoryOverInterval', had the highest failure rates.
|
||||
- Residual pattern: Tasks that required multiple steps or complex conditions, like 'RecipeDeleteMultipleRecipesWithNoise', were particularly challenging.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 16,384 characters for action selection or 8,192 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,206 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:14:33Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.034 | 0.121 | 9.569 | 106.154 | 18.690 | 163049.526 | 18780790 / 132955 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / CameraTakePhoto / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ClockStopWatchRunning / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseAddMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsNewContactDraft / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorDeleteNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorMergeNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleRecipe / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivityDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, and all tasks completed without errors. The success rate was 3.45%, with 4 successes out of 116 episodes. The mean latency was 106.15 seconds, and the mean total tokens were 163,049.53.
|
||||
- Cost/benefit interpretation: The cost in terms of latency and token usage is high, but the benefit in terms of preserving paired success without regression is maintained. However, the low success rate suggests that the current approach may not be effective.
|
||||
- Residual pattern: Most tasks failed to achieve success as defined by the evaluator reward.
|
||||
- Residual pattern: Tasks such as 'SystemBrightnessMax', 'SystemBrightnessMin', and 'SystemBrightnessMinVerify' had high latency and token usage.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,203 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:14:42Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.052 | 0.142 | 9.698 | 108.721 | 19.078 | 166715.009 | 19204766 / 134175 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / FilesDeleteFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarNextEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOn / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksCompletedTasksForDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / NotesIsTodo / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesTodoItemCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, achieving a success rate of 5.17% with 6 successes out of 116 episodes. The mean latency was 108.72 seconds, and the mean total tokens were 166,715. The cost-benefit analysis indicates that while the candidate approach preserved success with no regression, it did not meet the latency and token efficiency guardrails.
|
||||
- Cost/benefit interpretation: The candidate approach preserved success with no regression, but it did not meet the latency and token efficiency guardrails. The higher latency and token usage suggest that the candidate approach may be less efficient than the paired comparison, which could impact the overall cost and performance of the system.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only a few tasks (e.g., OpenAppTaskEval, SimpleCalendarNextMeetingWithPerson) succeeding.
|
||||
- Residual pattern: The mean latency and mean total tokens were higher than the guardrails specified for the paired comparison.
|
||||
- Residual pattern: The candidate approach did not meet the latency and token efficiency requirements, as the mean latency was 1.5 times the paired comparison and the mean tokens were 0.75 times the raw-UIAutomator tokens.
|
||||
- Next hypothesis `H5C_extended` (middle): Further refine the input pipeline by optimizing the element list to reduce latency and token usage while maintaining success. Target: Reduce the mean latency to 1.25 times the paired comparison and the mean tokens to 0.65 times the raw-UIAutomator tokens. Verification: Conduct a full-suite candidate rerun with the optimized element list to validate the hypothesis.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,204 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:19:57Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.052 | 0.129 | 9.716 | 109.254 | 19.069 | 169894.517 | 19570600 / 137164 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseAddMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteNewestNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorEditNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OsmAndFavorite / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / NotesIsTodo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset rerun completed with 116 episodes, achieving a success rate of 5.17% and an estimated cost of $0.00. The mean latency was 109.25 seconds, and the mean number of LLM calls was 19.07.
|
||||
- Cost/benefit interpretation: The cost of running the candidate subset is minimal, with an estimated cost of $0.00. However, the success rate is low, and the latency and number of LLM calls are high, indicating that the current approach may not be efficient or effective.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only a few tasks (e.g., NotesMeetingAttendeeCount, SystemBrightnessMaxVerify) succeeding.
|
||||
- Residual pattern: The majority of tasks took longer than the mean latency of 109.25 seconds, indicating potential inefficiencies.
|
||||
- Residual pattern: The mean number of LLM calls per task was 19.07, which is relatively high, suggesting that the model may be making multiple calls to achieve a task.
|
||||
- Next hypothesis `H5C_extended` (middle): Implement a more aggressive filtering of the UIAutomator hierarchy to reduce the number of LLM calls and improve latency while maintaining success rates. Target: Reduce the mean number of LLM calls per task to 10 and decrease the mean latency to 80 seconds. Verification: Run a full-suite candidate rerun with the new filtering strategy and compare the success rate, latency, and number of LLM calls to the current results.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Run stopped by environment/runtime blocker: Generated params do not match checkpoint for MarkorCreateFolder:trial-4:seed-25270
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,202 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:13:41Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.060 | 0.147 | 9.681 | 115.062 | 19.052 | 171630.552 | 19773398 / 135746 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ContactsAddContact / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsNewContactDraft / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMinVerify / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksCompletedTasksForDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, achieving a success rate of 6.03% with 7 successes out of 116 episodes. The mean latency was 115.06 seconds, and the mean total tokens were 171,630.55. The cost-benefit analysis indicates that while the candidate subset showed some success, the high latency and token usage suggest further optimization is needed.
|
||||
- Cost/benefit interpretation: The candidate subset showed some success but with high latency and token usage. Further optimization is needed to reduce these metrics while maintaining or improving success rates.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only 7 out of 116 tasks completing successfully.
|
||||
- Residual pattern: The mean latency of 115.06 seconds is significantly high, indicating potential inefficiencies in the current implementation.
|
||||
- Residual pattern: The mean total tokens of 171,630.55 are also high, suggesting that the model is using more resources than necessary.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
@@ -0,0 +1,136 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "7-12",
|
||||
"status": "complete",
|
||||
"generated_at_utc": "2026-08-04T09:35:34.274750+00:00",
|
||||
"run_dir": "chapter7/android-world/validation",
|
||||
"git_commit": "0d0df3e758872c7e73a998e57d2e529b16ece095",
|
||||
"command": "Five isolated Pixel 6/API-33 trial shards used run_controlled_experiment.py with local-vLLM credentials supplied only by environment-variable name; merge_candidate_shards.py strictly merged trials 1-5.",
|
||||
"provider_receipt_count": 0,
|
||||
"status_reasons": [
|
||||
"580/580 direct episodes completed across 116 tasks x five trials with zero runtime errors; evaluator failures are retained.",
|
||||
"Every shard completed official Pixel 6/API-33 setup and recorded the same 24/24 required package versions.",
|
||||
"The result is negative: 26/580 strict successes and deployment is not approved.",
|
||||
"The candidate used local Qwen2.5-7B while the paired H5C source used Doubao, so no same-model uplift or noninferiority is claimed.",
|
||||
"All disclosed evaluator, emulator-race, context-limit, resume, and exact-parameter retry compatibility treatments remain in retained evidence."
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"path": "chapter7/android-world/experiment_core.py",
|
||||
"bytes": 24201,
|
||||
"sha256": "752c1f181f0bef3cb0726fcc35eddc8c715624c4804c39b9a2d371518f4b3ac6"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/run_controlled_experiment.py",
|
||||
"bytes": 76108,
|
||||
"sha256": "ebc5417066fadbcd3de15d878d8b95a6ab9bff7f1b8e9eec392d148dbc59817e"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/merge_candidate_shards.py",
|
||||
"bytes": 11368,
|
||||
"sha256": "c7b0519c2246e88750f91859603fd34fea72a67f04c1e8eacb8fc5d2d63cd0f1"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/t3a_summary.md",
|
||||
"bytes": 28800,
|
||||
"sha256": "e5616507e2b0658b64dc5d0e2c08e6207eba12da8345a31d1cb85bf68957f002"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/t3a_failed_analysis.md",
|
||||
"bytes": 14448,
|
||||
"sha256": "7486112a3bf0791a17ca72b81b3745dcd92936416b5c3da937618c8d7a6eb00d"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_qwen_20260804/evidence.json",
|
||||
"bytes": 3932422,
|
||||
"sha256": "526e73e061bbafa3802484934dd29512a3b94a6a066d0efa10fc44c3f8138083"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_qwen_20260804/report.md",
|
||||
"bytes": 70854,
|
||||
"sha256": "4b4222ba4597eb57f35f40a30de180284d00a8fa66824e5492c75c4c8ce8b96a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard1/evidence.json",
|
||||
"bytes": 833946,
|
||||
"sha256": "8c9bad94ee2141c4de2e666e291f41e1f7d7cf7c53a8ad77a73ab17341bb68a3"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard1/report.md",
|
||||
"bytes": 23232,
|
||||
"sha256": "6113f7d935c4c1f085a53a4a2f0be5e9d8053e47c5aa4434d56a6a5d7c56e6a4"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard2/evidence.json",
|
||||
"bytes": 805331,
|
||||
"sha256": "2cf319e6eb7f99758c7dd093a46d445afa29ff35a919afc5f08f01c0a2085f44"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard2/report.md",
|
||||
"bytes": 22455,
|
||||
"sha256": "573f6fa57a846d6b231db2702069afff277434f39518deec8b64bdc0720eac92"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard3/evidence.json",
|
||||
"bytes": 815150,
|
||||
"sha256": "9b4bfed4165696044768c9e05fa00d696469ea99a06e9d3f694ea817a1041911"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard3/report.md",
|
||||
"bytes": 22157,
|
||||
"sha256": "8e37b4fdaf97eaa0bc14a6a161bcf14914ae6512782a94b5901c7bbfc361806a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard4/evidence.json",
|
||||
"bytes": 820872,
|
||||
"sha256": "4da385617b9b2f7a3d684884fd79184970e23d862583ab6b622b48f975e15696"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard4/report.md",
|
||||
"bytes": 21949,
|
||||
"sha256": "509a9224d6ce823c6d9ccff15a962fafdc0ed0b9724c053c2d83754517764f02"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard5/evidence.json",
|
||||
"bytes": 808670,
|
||||
"sha256": "245f99f7a8924d1b8340e510a6b12f345ac35ea779099605755d15d180109ce3"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard5/report.md",
|
||||
"bytes": 21696,
|
||||
"sha256": "4d22f0cdeb7b5f82560241b2f91169b94629c1f9cca2f320de4f54f5f1b99479"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"bytes": 47072,
|
||||
"sha256": "b2276e35e1f58d32775ea70ac04142a7d3075c074b9f31d40b153e1be10f93da"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5_a11y_api35_20260729/report.md",
|
||||
"bytes": 8239,
|
||||
"sha256": "5d5b0db9580020c286096ca53617cbd2d0f7fb811e48c4cf51688212ef638272"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5c_compact_api35_20260729/evidence.json",
|
||||
"bytes": 39643,
|
||||
"sha256": "a60204b6c0ac187b0ae9a68182b5ada302420fc31b2fc22c308e10eb1242e390"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5c_compact_api35_20260729/report.md",
|
||||
"bytes": 8930,
|
||||
"sha256": "55abde0ef9a6d5ed090f26bc99e424346c42a5e98036a278bd50edcf4bf48f2e"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"bytes": 45941,
|
||||
"sha256": "a1665bffafed7b8935db03dc4369435015f784d6f6465fe09c7da86fc0504635"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_wifi_api35_20260729/report.md",
|
||||
"bytes": 4096,
|
||||
"sha256": "067f176e4722082afd45bedbd79a392b26d677d54013a8b14a81cea1fbd70d57"
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,93 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260729T122904Z`
|
||||
- Generated (UTC): `2026-07-29T13:17:47Z`
|
||||
- Upstream commit: `0e95d641e244504c22087cc29b013f3b2428a261`
|
||||
- Device: `sdk_gphone64_arm64`, API `35` (upstream tested reference: API `33`)
|
||||
- Observation method: `varies_by_arm:a11y_forwarder_app_vs_uiautomator`
|
||||
- Provider/model: `ark` / `doubao-seed-1-6-250615`
|
||||
- Scope: 4 task(s), 1 trial(s), mode `paired`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5`
|
||||
- Change: Select AndroidWorld's UIAUTOMATOR observation method in the companion runner without changing upstream source.
|
||||
- Expected measurable result: At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, no regression, and at most 1.5x mean latency and tokens; never treat a subset gain as full-suite success or deployment approval.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_middle` — middle-layer input-pipeline ablation prompted by phase-1 residual traces
|
||||
- Independent variable: accessibility observation pipeline (gRPC forwarder versus UIAutomator)
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| control | 4/4 | 0.250 | 0.500 | 8.000 | 172.849 | 14.000 | 68310.500 | 251587 / 21655 |
|
||||
| treatment | 4/4 | 1.000 | 1.000 | 5.750 | 136.235 | 11.000 | 170673.500 | 671404 / 11290 |
|
||||
|
||||
| Task / trial | Control | Treatment | Δ success | Control→treatment steps |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| SystemWifiTurnOff / 1 | 0 | 1 | +1 | 10→6 |
|
||||
| SystemWifiTurnOffVerify / 1 | 0 | 1 | +1 | 10→6 |
|
||||
| SystemWifiTurnOn / 1 | 0 | 1 | +1 | 10→7 |
|
||||
| SystemWifiTurnOnVerify / 1 | 1 | 1 | +0 | 2→4 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`restrict_candidate_due_to_cost`**
|
||||
- Reason: Treatment improved paired success without regressions, but exceeded the latency/token guardrails (1.50x / 1.50x). Restrict it to targeted follow-up; do not promote it to the full suite yet.
|
||||
- Treatment/control mean latency ratio: 0.788
|
||||
- Treatment/control mean token ratio: 2.498
|
||||
- Treatment/control mean LLM-call ratio: 0.786
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `control / SystemWifiTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `control / SystemWifiTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `control / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: Experiment 7-12 compared control (a11y-forwarder observation) and treatment (UIAutomator observation) arms in a paired setup with 4 Wi-Fi system Settings tasks. Treatment achieved 100% success (4/4) vs control's 25% (1/4), reduced mean latency (136.2s vs 172.8s) and LLM calls (11.0 vs 14.0), but had a mean token ratio (treatment/control) of 2.498, exceeding the 1.5x guardrail. Environment boundaries include API 35 AVD (vs upstream API 33 reference), restriction to Settings tasks, UIAutomator as a compatibility path (not reference config), and skipped device-time setting due to non-root AVD limitations.
|
||||
- Cost/benefit interpretation: Treatment provides substantial benefit via improved success rate (net +3) and reduced latency/LLM calls, but incurs significantly higher token cost (2.498x control), violating token guardrails and limiting deployment despite success gains.
|
||||
- Residual pattern: Treatment mean token ratio (2.498x) exceeds 1.5x guardrail
|
||||
- Residual pattern: Control arm has low success rate (25%, 1/4 completed episodes)
|
||||
- Next hypothesis `H6` (middle): Optimize UIAutomator observation pipeline to reduce token usage while maintaining treatment success rate Target: Mean token ratio (treatment/control) ≤1.5x and success rate ≥1.0 in paired Wi-Fi tasks Verification: Conduct paired run with optimized UIAutomator pipeline vs control, using same 4 Wi-Fi tasks, API 35 AVD environment, and guardrails; measure token ratio and success rate
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.
|
||||
- The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
@@ -0,0 +1,940 @@
|
||||
{
|
||||
"arm_summary": {
|
||||
"control": {
|
||||
"completed_episodes": 4,
|
||||
"episodes": 4,
|
||||
"error_episodes": 0,
|
||||
"mean_evaluator_reward": 1.0,
|
||||
"mean_latency_s": 101.198336,
|
||||
"mean_llm_calls": 8.5,
|
||||
"mean_llm_latency_s": 62.453747,
|
||||
"mean_steps": 4.75,
|
||||
"mean_total_tokens": 139439.5,
|
||||
"success_rate": 1.0,
|
||||
"successes": 4,
|
||||
"total_input_tokens": 549928,
|
||||
"total_output_tokens": 7830,
|
||||
"total_tokens": 557758
|
||||
},
|
||||
"treatment": {
|
||||
"completed_episodes": 4,
|
||||
"episodes": 4,
|
||||
"error_episodes": 0,
|
||||
"mean_evaluator_reward": 1.0,
|
||||
"mean_latency_s": 99.184035,
|
||||
"mean_llm_calls": 8.5,
|
||||
"mean_llm_latency_s": 59.748216,
|
||||
"mean_steps": 4.75,
|
||||
"mean_total_tokens": 70557.5,
|
||||
"success_rate": 1.0,
|
||||
"successes": 4,
|
||||
"total_input_tokens": 274067,
|
||||
"total_output_tokens": 8163,
|
||||
"total_tokens": 282230
|
||||
}
|
||||
},
|
||||
"baseline": {
|
||||
"agent": "t3a_claude4_sonnet",
|
||||
"provenance": "historical bundled report; not generated by this runner",
|
||||
"reported_success_rate_approx": 0.88,
|
||||
"run_date": "2025-07-02",
|
||||
"source": "t3a_summary.md and t3a_failed_analysis.md",
|
||||
"tasks": 116,
|
||||
"trials_per_task": 1
|
||||
},
|
||||
"command": [
|
||||
"run_controlled_experiment.py",
|
||||
"--mode",
|
||||
"paired",
|
||||
"--hypothesis",
|
||||
"H5C",
|
||||
"--source-phase1-evidence",
|
||||
"validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"--source-phase2-evidence",
|
||||
"validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"--tasks",
|
||||
"SystemWifiTurnOff,SystemWifiTurnOffVerify,SystemWifiTurnOn,SystemWifiTurnOnVerify",
|
||||
"--trials",
|
||||
"1",
|
||||
"--seed",
|
||||
"42",
|
||||
"--model-seed",
|
||||
"42",
|
||||
"--max-steps",
|
||||
"10",
|
||||
"--transition-pause",
|
||||
"0.5",
|
||||
"--skip-device-time",
|
||||
"--output-dir",
|
||||
"validation/paired_h5c_compact_api35_20260729"
|
||||
],
|
||||
"credentials_persisted": false,
|
||||
"decision": {
|
||||
"completed_pairs": 4,
|
||||
"deployment_approved": false,
|
||||
"guardrails": {
|
||||
"maximum_latency_ratio": 1.5,
|
||||
"maximum_token_ratio": 0.75,
|
||||
"objective": "success_noninferiority_and_token_reduction",
|
||||
"passed": true,
|
||||
"require_all_treatment_pairs_successful": true
|
||||
},
|
||||
"mean_latency_ratio_treatment_over_control": 0.980096,
|
||||
"mean_llm_call_ratio_treatment_over_control": 1.0,
|
||||
"mean_token_ratio_treatment_over_control": 0.506008,
|
||||
"net_success_delta": 0,
|
||||
"outcome": "promote_efficient_candidate_to_full_suite_rerun",
|
||||
"paired_regressions": 0,
|
||||
"promote_to_full_suite_candidate": true,
|
||||
"reason": "Treatment preserved paired success with no regression and passed the latency/token efficiency guardrails. This is a candidate decision only.",
|
||||
"required_treatment_successes": 4,
|
||||
"scope_recommendation": "full_suite_candidate_only",
|
||||
"success_preservation_passed": true,
|
||||
"treatment_successes": 4
|
||||
},
|
||||
"diagnosis": {
|
||||
"findings": [
|
||||
"The historical run evaluated 116 tasks once each and reports approximately 88% overall success.",
|
||||
"Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.",
|
||||
"The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.",
|
||||
"The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause."
|
||||
],
|
||||
"layered_hypotheses": [
|
||||
{
|
||||
"id": "H1",
|
||||
"idea": "Add Wi-Fi Settings navigation and final-state verification guidance.",
|
||||
"layer": "surface",
|
||||
"status": "tested in source phase 1",
|
||||
"target": "At least one net paired success across the four Wi-Fi tasks, with no regression.",
|
||||
"verification": "Paired upstream-prompt versus task-guideline ablation with matched seeds."
|
||||
},
|
||||
{
|
||||
"id": "H2",
|
||||
"idea": "Add application-specific recognition rules for the non-standard Tasks UI.",
|
||||
"layer": "surface",
|
||||
"status": "not tested",
|
||||
"target": "Improve at least two of the six historical Tasks failures with no regression.",
|
||||
"verification": "Paired Tasks-only prompt/tool-description ablation after app provisioning."
|
||||
},
|
||||
{
|
||||
"id": "H3",
|
||||
"idea": "Repair and validate the multimodal input path for transcription tasks.",
|
||||
"layer": "middle",
|
||||
"status": "not tested",
|
||||
"target": "Raise transcription success above the historical 0% while bounding added tokens and latency.",
|
||||
"verification": "Paired screenshot-disabled versus screenshot-enabled transcription run."
|
||||
},
|
||||
{
|
||||
"id": "H4",
|
||||
"idea": "Conditionally enable deeper thinking for counting tasks.",
|
||||
"layer": "middle",
|
||||
"status": "not tested",
|
||||
"target": "Improve math/counting success without applying the cost to unrelated tasks.",
|
||||
"verification": "Paired tag-routed thinking-mode ablation with latency and token guardrails."
|
||||
},
|
||||
{
|
||||
"id": "H5",
|
||||
"idea": "Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path.",
|
||||
"layer": "middle",
|
||||
"status": "tested in source phase 2",
|
||||
"target": "At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens.",
|
||||
"verification": "Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds."
|
||||
},
|
||||
{
|
||||
"id": "H5C",
|
||||
"idea": "Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost.",
|
||||
"layer": "middle",
|
||||
"status": "tested in this run",
|
||||
"target": "Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.",
|
||||
"verification": "Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator."
|
||||
},
|
||||
{
|
||||
"id": "H6",
|
||||
"idea": "Combine screenshots with the structured UI tree and compare stronger vision-capable models.",
|
||||
"layer": "deep",
|
||||
"status": "not tested",
|
||||
"target": "Improve complex-UI success enough to justify multimodal latency and token cost.",
|
||||
"verification": "Factorial UI-tree/screenshot/model ablation on the full tagged slice."
|
||||
}
|
||||
]
|
||||
},
|
||||
"environment": {
|
||||
"a11y_method": "varies_by_arm:uiautomator_vs_uiautomator_compact",
|
||||
"android_world_checkout": "/Users/boj/book/ai-agent-book/chapter7/android_world",
|
||||
"android_world_checkout_clean": true,
|
||||
"android_world_commit": "0e95d641e244504c22087cc29b013f3b2428a261",
|
||||
"api_level": 35,
|
||||
"avd_name": "Pixel_9_Pro_API_35",
|
||||
"device_model": "sdk_gphone64_arm64",
|
||||
"device_serial": "emulator-5554",
|
||||
"grpc_port": 8554,
|
||||
"perform_emulator_setup": false,
|
||||
"physical_size": "Physical size: 1280x2856",
|
||||
"protobuf_bootstrap": {
|
||||
"generated_protobufs_in_temporary_directory": true,
|
||||
"temporary_directory_persisted": false
|
||||
},
|
||||
"skip_device_time": true,
|
||||
"upstream_tested_api_level": 33,
|
||||
"upstream_tested_device": "Pixel 6"
|
||||
},
|
||||
"environment_boundaries": [
|
||||
"The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.",
|
||||
"The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.",
|
||||
"UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.",
|
||||
"Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.",
|
||||
"Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time."
|
||||
],
|
||||
"episodes": [
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 105.382271,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 132349,
|
||||
"latency_s": 64.872256,
|
||||
"output_tokens": 2008,
|
||||
"reasoning_tokens": 1488,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOff:trial-1:seed-42",
|
||||
"pair_seed": 42,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOff",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 22.742624,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will open the Settings menu where Wi-Fi settings can be accessed.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.091666,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows an error dialog \"com.google.androidenv.accessibilityforwarder keeps stopping\" which is blocking access to the Settings menu. To proceed with turning off Wi-Fi, I need to dismiss this dialog by closing the problematic app. The \"Close app\" button (index 11) is visible and clickable, so clicking it will remove the dialog and allow me to continue navigating to Wi-Fi settings.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "input_text",
|
||||
"done": false,
|
||||
"elapsed_s": 24.93577,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with turning off Wi-Fi, I need to locate the Wi-Fi settings. Since the current screen is the settings search screen, the most efficient way is to search for \"Wi-Fi\" using the search bar. The search bar (UI element 15) is an editable text field and is focused, making it suitable for input.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 25.397996,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows the Wi-Fi settings with a checked (on) switch at index 32. To turn off Wi-Fi, I need to click this switch to toggle it to the off position.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 8.938998,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (index 32) is now unchecked (is_checked=False), indicating that Wi-Fi has been turned off. The user's goal of turning Wi-Fi off has been successfully achieved.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 103.549168,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 75681,
|
||||
"latency_s": 65.635331,
|
||||
"output_tokens": 2101,
|
||||
"reasoning_tokens": 1568,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOff:trial-1:seed-42",
|
||||
"pair_seed": 42,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOff",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.105315,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 12 is the \"Settings\" app, which is visible and clickable. Clicking it will open the Settings menu where Wi-Fi settings can be accessed.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 23.230104,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, I need to access the Wi-Fi settings. The \"Network & internet\" option (UI element 6) is visible and clickable, and its summary mentions \"Wi‑Fi\", indicating it contains Wi-Fi settings. Clicking this will lead to the next screen where Wi-Fi can be toggled off.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.825863,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi settings and turn it off, I need to click on the \"Internet\" option, which is part of the clickable linear layout (UI element 3) on the Network & internet settings page.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 27.327625,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows the Internet settings where the Wi-Fi switch is visible. The Wi-Fi option is contained in UI element 9 (a LinearLayout), which is clickable. This element includes the \"Wi-Fi\" text (UI element 10) and the switch (UI element 11) that is currently on (is_checked=True). Clicking this LinearLayout will toggle the Wi-Fi switch off.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 8.69858,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch is now off (is_checked=False as per UI element 11), so the task of turning Wi-Fi off has been successfully completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 102.209211,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 64462,
|
||||
"latency_s": 62.502274,
|
||||
"output_tokens": 2130,
|
||||
"reasoning_tokens": 1620,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOffVerify:trial-1:seed-1051",
|
||||
"pair_seed": 1051,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOffVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.831033,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows an error message pop-up that is blocking access to the main interface. To proceed with turning off Wi-Fi, I need to close this pop-up first. The \"Close app\" button (UI element 2) is visible and clickable, so clicking it will dismiss the pop-up.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 23.270511,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the next step is to open the Settings app, as Wi-Fi settings are typically located there. The \"Settings\" app is visible on the home screen (UI element 12), which is clickable. Clicking it will allow access to the settings menu where Wi-Fi can be managed.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 23.028096,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, I need to access the network settings. The \"Network & internet\" option (UI element 6, which is a clickable LinearLayout containing the \"Network & internet\" title and summary) is the appropriate next step to navigate to Wi-Fi controls.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.913882,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi controls, I need to navigate to the \"Internet\" settings. The \"Internet\" option is contained within the clickable LinearLayout (UI element 3), which is visible and interactive on the current \"Network & internet\" screen. Clicking this element will likely lead to the page where Wi-Fi can be toggled off.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 11.76325,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 11) is already in the off position (is_checked=False), so the goal of turning Wi-Fi off has been achieved.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 112.724126,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 150834,
|
||||
"latency_s": 67.904524,
|
||||
"output_tokens": 2205,
|
||||
"reasoning_tokens": 1649,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOffVerify:trial-1:seed-1051",
|
||||
"pair_seed": 1051,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOffVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 18.772365,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the first step is to open the Settings app. On the home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will open the Settings menu where Wi-Fi settings can be accessed.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 25.71773,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, I need to access the Wi-Fi settings. The \"Network & internet\" option (UI element 24) in the Settings menu is the appropriate entry point as it mentions \"Wi‑Fi\" in its summary. The parent LinearLayout (UI element 20) is clickable, so I will tap on it to proceed to the Network & internet settings page.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 24.548097,
|
||||
"goal_status": null,
|
||||
"reason": "The goal is to turn off Wi-Fi. Currently, on the Settings home screen, the \"Network & internet\" section (UI element 20) is visible, which is the correct path to access Wi-Fi settings. Since the previous attempt to click this element didn't result in a screen change (possibly due to a delay), retrying the click on UI element 20 is necessary to navigate to the Network & internet settings where Wi-Fi can be toggled off.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.737918,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi settings, I need to navigate into the \"Internet\" section under \"Network & internet\". The \"Internet\" option (UI element 15) is clickable and likely contains Wi-Fi controls, so clicking it will allow me to proceed toward turning off Wi-Fi.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 13.895503,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 31) is already in the off position (is_checked=False), so the task of turning Wi-Fi off has been completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 100.007401,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 146625,
|
||||
"latency_s": 60.454149,
|
||||
"output_tokens": 1997,
|
||||
"reasoning_tokens": 1484,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOn:trial-1:seed-2060",
|
||||
"pair_seed": 2060,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOn",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.700473,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will allow access to the settings menu where Wi-Fi can be enabled.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.450628,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed towards enabling Wi-Fi, I need to access the \"Network & internet\" settings, which is the logical next step as it contains Wi-Fi options. The \"Network & internet\" element (index 24) is visible and clickable on the current screen.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 22.812015,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed towards enabling Wi-Fi, I need to access the Internet settings which typically include Wi-Fi options. The \"Internet\" option (contained in the clickable LinearLayout at index 15) is visible on the screen and is the logical next step to find the Wi-Fi toggle.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.686302,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows the Internet settings page where the Wi-Fi option is present. The Wi-Fi switch (UI element 31) is currently unchecked (is_checked=False), indicating Wi-Fi is off. To turn Wi-Fi on, I need to click this switch to toggle it to the on position.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 10.361169,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 35) is now checked, indicating Wi-Fi is enabled. The screen also shows \"AndroidWifi\" as connected, confirming the task of turning Wi-Fi on has been successfully completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 106.717192,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 74268,
|
||||
"latency_s": 60.267918,
|
||||
"output_tokens": 2150,
|
||||
"reasoning_tokens": 1681,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOn:trial-1:seed-2060",
|
||||
"pair_seed": 2060,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOn",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "open_app",
|
||||
"done": false,
|
||||
"elapsed_s": 23.770885,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app where network settings are located. The current screen shows a \"Settings\" app icon (UI element 12), and using the `open_app` action is the most direct way to launch the app.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 27.515651,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed to the Wi-Fi settings, I need to access the \"Network & internet\" section. The UI element 6 is a clickable LinearLayout that contains the \"Network & internet\" text (elements 7 and 8), making it the appropriate element to interact with.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 22.324583,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi settings, I need to navigate into the \"Internet\" section, which is likely to contain Wi-Fi controls. The \"Internet\" option is part of the clickable LinearLayout (UI element 3) on the current screen.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 19.494808,
|
||||
"goal_status": null,
|
||||
"reason": "The Wi-Fi switch (UI element 11) is currently off (is_checked=False). To turn Wi-Fi on, I need to click this switch to toggle it to the on position.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 11.27913,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 11) is now checked (is_checked=True), indicating Wi-Fi is turned on. Additionally, \"AndroidWifi\" is shown as connected (UI element 12), confirming the task is completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 84.26057,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 7,
|
||||
"input_tokens": 59656,
|
||||
"latency_s": 50.587342,
|
||||
"output_tokens": 1782,
|
||||
"reasoning_tokens": 1390,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOnVerify:trial-1:seed-3069",
|
||||
"pair_seed": 3069,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 4,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOnVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.295946,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 12 is the \"Settings\" app, which is visible and clickable. Clicking on it will allow access to the settings menu where Wi-Fi can be enabled.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.66904,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with turning on Wi-Fi, I need to access the \"Network & internet\" settings, as it contains Wi-Fi options. The \"Network & internet\" section is represented by the clickable LinearLayout (UI element 6) on the current screen. Clicking this will navigate to the relevant submenu where Wi-Fi can be enabled.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.764651,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with turning on Wi-Fi, I need to access the Wi-Fi settings. The \"Internet\" section (contained in UI element 3, a clickable LinearLayout) is the logical next step, as it likely leads to Wi-Fi configuration options.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 8.154307,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 11) is already checked (is_checked=True), indicating that Wi-Fi is enabled. The user's goal of turning Wi-Fi on has been achieved.",
|
||||
"step": 4
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 86.679544,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 7,
|
||||
"input_tokens": 120120,
|
||||
"latency_s": 56.58406,
|
||||
"output_tokens": 1620,
|
||||
"reasoning_tokens": 1195,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOnVerify:trial-1:seed-3069",
|
||||
"pair_seed": 3069,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 4,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOnVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.802337,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app. On the home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will allow access to the settings menu where Wi-Fi can be enabled.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 24.619676,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with enabling Wi-Fi, I need to access the \"Network & internet\" settings, as it contains Wi-Fi options. The \"Network & internet\" section is a clickable element (UI element 20) on the current Settings screen. Clicking this will navigate to the relevant submenu where Wi-Fi can be toggled on.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.561039,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with enabling Wi-Fi, I need to access the Wi-Fi settings. The \"Internet\" option (UI element 19) with \"AndroidWifi\" as the summary (UI element 20) is likely the entry point to Wi-Fi settings. The parent linear layout (UI element 15) is clickable, so clicking it will navigate to the Wi-Fi settings screen where the toggle can be accessed.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 11.348688,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 35) is already in the checked state, indicating that Wi-Fi is turned on. The user's goal of turning Wi-Fi on has been achieved.",
|
||||
"step": 4
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
}
|
||||
],
|
||||
"experiment": "7-12",
|
||||
"experiment_complete": false,
|
||||
"generated_at_utc": "2026-07-29T14:12:27Z",
|
||||
"hypothesis": {
|
||||
"change": "Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.",
|
||||
"expected_result": "Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.",
|
||||
"guardrails": "Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.",
|
||||
"guideline_sha256": null,
|
||||
"guidelines": [],
|
||||
"id": "H5C",
|
||||
"layer": "middle",
|
||||
"verification": "Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator."
|
||||
},
|
||||
"llm_analysis": {
|
||||
"cost_benefit_interpretation": "The treatment provides significant token efficiency (mean token ratio 0.506) with preserved success and marginal latency improvement (mean latency ratio 0.980) in the tested 4-task subset. However, interpretation is bounded by the API/app environment: results are from an API 35 AVD (not upstream API 33 reference), restricted to system Settings tasks (no full third-party app bundle), and use UIAutomator as a compatibility path (not upstream reference configuration), limiting generalizability beyond the tested scope.",
|
||||
"llm": {
|
||||
"calls": 1,
|
||||
"input_tokens": 3257,
|
||||
"latency_s": 34.44844,
|
||||
"output_tokens": 1101,
|
||||
"reasoning_tokens": 647,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"next_hypothesis": {
|
||||
"id": "H5C-full",
|
||||
"idea": "Evaluate compact UIAutomator (semantic-filtered elements) across the full AndroidWorld task suite to verify token efficiency and success preservation beyond the 4-task Settings subset.",
|
||||
"layer": "middle",
|
||||
"target": "Full task suite (all 116 tasks) with upstream reference environment (Pixel 6 / API 33) and provisioned full third-party AndroidWorld app bundle.",
|
||||
"verification": "Paired run comparing compact UIAutomator (treatment) vs raw UIAutomator (control) across all tasks, ensuring guardrails (mean token ratio ≤0.75, mean latency ratio ≤1.5, success non-inferiority) hold in the reference environment."
|
||||
},
|
||||
"observed_failure_pattern": [],
|
||||
"source": "real configured LLM over aggregate direct evidence",
|
||||
"status": "completed",
|
||||
"summary": "A paired experiment comparing raw UIAutomator (control) and compact UIAutomator (treatment) on 4 system Settings tasks (SystemWifiTurnOff, SystemWifiTurnOffVerify, SystemWifiTurnOn, SystemWifiTurnOnVerify) in an API 35 AVD environment. Both arms completed 4 episodes with 100% success rate, identical mean steps (4.75) and LLM calls (8.5). Treatment showed lower mean total tokens (50.6% of control) and slightly lower mean latency (98.0% of control). The decision was to promote the treatment as a full-suite candidate rerun, as it preserved success, passed latency/token guardrails, but remains a subset with environment limitations."
|
||||
},
|
||||
"model": {
|
||||
"base_url": "https://ark.cn-beijing.volces.com/api/v3",
|
||||
"max_tokens": 1024,
|
||||
"model": "doubao-seed-1-6-250615",
|
||||
"provider": "ark",
|
||||
"seed": 42,
|
||||
"temperature": 0
|
||||
},
|
||||
"paired_comparison": [
|
||||
{
|
||||
"control_latency_s": 105.382271,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 5,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOff:trial-1:seed-42",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOff",
|
||||
"treatment_latency_s": 103.549168,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 5,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"control_latency_s": 112.724126,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 5,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOffVerify:trial-1:seed-1051",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOffVerify",
|
||||
"treatment_latency_s": 102.209211,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 5,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"control_latency_s": 100.007401,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 5,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOn:trial-1:seed-2060",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOn",
|
||||
"treatment_latency_s": 106.717192,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 5,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"control_latency_s": 86.679544,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 4,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOnVerify:trial-1:seed-3069",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOnVerify",
|
||||
"treatment_latency_s": 84.26057,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 4,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
}
|
||||
],
|
||||
"phase": {
|
||||
"description": "middle-layer input-pipeline cost refinement after the H5 success/cost result",
|
||||
"id": "phase_2_cost_refinement",
|
||||
"independent_variable": "raw versus semantic-filtered UIAutomator element list",
|
||||
"source_phase1_decision": {
|
||||
"completed_pairs": 4,
|
||||
"mean_latency_ratio_treatment_over_control": 0.672411,
|
||||
"net_success_delta": 0,
|
||||
"outcome": "inconclusive_no_success_gain",
|
||||
"paired_regressions": 0,
|
||||
"promote_to_full_suite_candidate": false,
|
||||
"reason": "Treatment produced no paired success gain; keep the upstream control prompt."
|
||||
},
|
||||
"source_phase1_evidence": "/Users/boj/book/ai-agent-book/chapter7/android-world/validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"source_phase1_run_id": "exp7-12-20260729T114648Z",
|
||||
"source_phase2_decision": {
|
||||
"completed_pairs": 4,
|
||||
"deployment_approved": false,
|
||||
"guardrails": {
|
||||
"maximum_latency_ratio": 1.5,
|
||||
"maximum_token_ratio": 1.5,
|
||||
"passed": false
|
||||
},
|
||||
"mean_latency_ratio_treatment_over_control": 0.788177,
|
||||
"mean_llm_call_ratio_treatment_over_control": 0.785714,
|
||||
"mean_token_ratio_treatment_over_control": 2.498496,
|
||||
"net_success_delta": 3,
|
||||
"outcome": "restrict_candidate_due_to_cost",
|
||||
"paired_regressions": 0,
|
||||
"promote_to_full_suite_candidate": false,
|
||||
"reason": "Treatment improved paired success without regressions, but exceeded the latency/token guardrails (1.50x / 1.50x). Restrict it to targeted follow-up; do not promote it to the full suite yet.",
|
||||
"scope_recommendation": "do_not_deploy"
|
||||
},
|
||||
"source_phase2_evidence": "/Users/boj/book/ai-agent-book/chapter7/android-world/validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"source_phase2_run_id": "exp7-12-20260729T122904Z"
|
||||
},
|
||||
"resume_commands": [
|
||||
[
|
||||
"run_controlled_experiment.py",
|
||||
"--mode",
|
||||
"paired",
|
||||
"--hypothesis",
|
||||
"H5C",
|
||||
"--source-phase1-evidence",
|
||||
"validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"--source-phase2-evidence",
|
||||
"validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"--tasks",
|
||||
"SystemWifiTurnOff,SystemWifiTurnOffVerify,SystemWifiTurnOn,SystemWifiTurnOnVerify",
|
||||
"--trials",
|
||||
"1",
|
||||
"--seed",
|
||||
"42",
|
||||
"--model-seed",
|
||||
"42",
|
||||
"--max-steps",
|
||||
"10",
|
||||
"--transition-pause",
|
||||
"0.5",
|
||||
"--skip-device-time",
|
||||
"--output-dir",
|
||||
"validation/paired_h5c_compact_api35_20260729",
|
||||
"--resume"
|
||||
]
|
||||
],
|
||||
"run_id": "exp7-12-20260729T131911Z",
|
||||
"schema_version": 1,
|
||||
"scope": {
|
||||
"base_pair_seed": 42,
|
||||
"completed_episodes": 8,
|
||||
"direct_episode_gate_completed": false,
|
||||
"error_episodes": 0,
|
||||
"full_suite_completed": false,
|
||||
"manuscript_five_seed_gate_completed": false,
|
||||
"max_steps": 10,
|
||||
"mode": "paired",
|
||||
"tasks": [
|
||||
"SystemWifiTurnOff",
|
||||
"SystemWifiTurnOffVerify",
|
||||
"SystemWifiTurnOn",
|
||||
"SystemWifiTurnOnVerify"
|
||||
],
|
||||
"transition_pause_s": 0.5,
|
||||
"trials_per_task": 1
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260729T131911Z`
|
||||
- Generated (UTC): `2026-07-29T14:12:27Z`
|
||||
- Upstream commit: `0e95d641e244504c22087cc29b013f3b2428a261`
|
||||
- Device: `sdk_gphone64_arm64`, API `35` (upstream tested reference: API `33`)
|
||||
- Observation method: `varies_by_arm:uiautomator_vs_uiautomator_compact`
|
||||
- Provider/model: `ark` / `doubao-seed-1-6-250615`
|
||||
- Scope: 4 task(s), 1 trial(s), mode `paired`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| control | 4/4 | 1.000 | 1.000 | 4.750 | 101.198 | 8.500 | 139439.500 | 549928 / 7830 |
|
||||
| treatment | 4/4 | 1.000 | 1.000 | 4.750 | 99.184 | 8.500 | 70557.500 | 274067 / 8163 |
|
||||
|
||||
| Task / trial | Control | Treatment | Δ success | Control→treatment steps |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| SystemWifiTurnOff / 1 | 1 | 1 | +0 | 5→5 |
|
||||
| SystemWifiTurnOffVerify / 1 | 1 | 1 | +0 | 5→5 |
|
||||
| SystemWifiTurnOn / 1 | 1 | 1 | +0 | 5→5 |
|
||||
| SystemWifiTurnOnVerify / 1 | 1 | 1 | +0 | 4→4 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`promote_efficient_candidate_to_full_suite_rerun`**
|
||||
- Reason: Treatment preserved paired success with no regression and passed the latency/token efficiency guardrails. This is a candidate decision only.
|
||||
- Treatment/control mean latency ratio: 0.980
|
||||
- Treatment/control mean token ratio: 0.506
|
||||
- Treatment/control mean LLM-call ratio: 1.000
|
||||
- Cost guardrails passed: **true**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: A paired experiment comparing raw UIAutomator (control) and compact UIAutomator (treatment) on 4 system Settings tasks (SystemWifiTurnOff, SystemWifiTurnOffVerify, SystemWifiTurnOn, SystemWifiTurnOnVerify) in an API 35 AVD environment. Both arms completed 4 episodes with 100% success rate, identical mean steps (4.75) and LLM calls (8.5). Treatment showed lower mean total tokens (50.6% of control) and slightly lower mean latency (98.0% of control). The decision was to promote the treatment as a full-suite candidate rerun, as it preserved success, passed latency/token guardrails, but remains a subset with environment limitations.
|
||||
- Cost/benefit interpretation: The treatment provides significant token efficiency (mean token ratio 0.506) with preserved success and marginal latency improvement (mean latency ratio 0.980) in the tested 4-task subset. However, interpretation is bounded by the API/app environment: results are from an API 35 AVD (not upstream API 33 reference), restricted to system Settings tasks (no full third-party app bundle), and use UIAutomator as a compatibility path (not upstream reference configuration), limiting generalizability beyond the tested scope.
|
||||
- Next hypothesis `H5C-full` (middle): Evaluate compact UIAutomator (semantic-filtered elements) across the full AndroidWorld task suite to verify token efficiency and success preservation beyond the 4-task Settings subset. Target: Full task suite (all 116 tasks) with upstream reference environment (Pixel 6 / API 33) and provisioned full third-party AndroidWorld app bundle. Verification: Paired run comparing compact UIAutomator (treatment) vs raw UIAutomator (control) across all tasks, ensuring guardrails (mean token ratio ≤0.75, mean latency ratio ≤1.5, success non-inferiority) hold in the reference environment.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.
|
||||
- The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,67 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260729T114648Z`
|
||||
- Generated (UTC): `2026-07-29T12:13:15Z`
|
||||
- Upstream commit: `0e95d641e244504c22087cc29b013f3b2428a261`
|
||||
- Device: `sdk_gphone64_arm64`, API `35` (upstream tested reference: API `33`)
|
||||
- Provider/model: `ark` / `doubao-seed-1-6-250615`
|
||||
- Scope: 4 task(s), 1 trial(s), mode `paired`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
- ID: `H1`
|
||||
- Change: Use upstream T3A.set_task_guidelines to add only Wi-Fi Settings navigation and final-state verification guidance.
|
||||
- Expected measurable result: At least one net paired Wi-Fi success, with no paired regression; record reward, steps, latency, calls, and tokens.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; do not treat a subset gain as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
Control and treatment use the same checkout, model, task parameters, step budget, and emulator. Only the task-specific T3A guidelines differ. Arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Input / output tokens |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| control | 4/4 | 0.250 | 0.500 | 8.500 | 233.465 | 15.750 | 411525 / 31094 |
|
||||
| treatment | 4/4 | 0.250 | 0.500 | 8.000 | 156.985 | 12.500 | 190519 / 19520 |
|
||||
|
||||
| Task / trial | Control | Treatment | Δ success | Control→treatment steps |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| SystemWifiTurnOff / 1 | 0 | 0 | +0 | 10→10 |
|
||||
| SystemWifiTurnOffVerify / 1 | 0 | 0 | +0 | 10→10 |
|
||||
| SystemWifiTurnOn / 1 | 0 | 0 | +0 | 10→10 |
|
||||
| SystemWifiTurnOnVerify / 1 | 1 | 1 | +0 | 4→2 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`inconclusive_no_success_gain`**
|
||||
- Reason: Treatment produced no paired success gain; keep the upstream control prompt.
|
||||
- Treatment/control mean latency ratio: 0.672
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `control / SystemWifiTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `treatment / SystemWifiTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `treatment / SystemWifiTurnOffVerify / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `control / SystemWifiTurnOffVerify / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `control / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `treatment / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.
|
||||
- The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.
|
||||
- Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
Reference in New Issue
Block a user