ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,652 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-merged-20260804T092058Z`
|
||||
- Generated (UTC): `2026-08-04T09:20:58Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **true**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed policy, step budget, Pixel 6/API-33 device class, upstream setup, and app versions across isolated shards.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 580/580 | 0.045 | 0.134 | 9.672 | 109.861 | 18.998 | 169069.564 | 97384410 / 675937 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`full_candidate_rerun_completed`**
|
||||
- Reason: The direct 116-task x five-trial candidate rerun completed on five independent reference-environment shards. Negative evaluator results are retained.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
The complete 116-task, five-trial candidate rerun gate is satisfied by direct episode evidence.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudio / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudio / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudio / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudio / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / BrowserMultiply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ContactsAddContact / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsAddContact / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsAddContact / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ExpenseAddSingle / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteSingle / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteSingle / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteSingle / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / FilesDeleteFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateFolder / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateFolder / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateFolder / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteAllNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorDeleteNewestNote / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteNewestNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorDeleteNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorEditNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesIsTodo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OpenAppTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OpenAppTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndMarker / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndMarker / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleRecipe / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsReply / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsResend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivityDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOn / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMinVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMinVerify / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksCompletedTasksForDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate rerun on 116 tasks across five independent shards showed a success rate of 4.48% with 26 successes out of 580 episodes. The mean latency was 109.86 seconds, and the mean number of LLM calls was 19. The mean total tokens were 169,069.56, with a mean evaluator reward of 0.133621.
|
||||
- Cost/benefit interpretation: The cost of running the candidate on 116 tasks was relatively high, with a mean latency of 109.86 seconds and a mean of 19 LLM calls per episode. The benefit, in terms of success rate, was minimal, with only 26 successes out of 580 episodes. The low success rate and high cost suggest that the current approach may not be efficient or effective.
|
||||
- Residual pattern: Multiple tasks, such as AudioRecorderRecordAudio, CameraTakePhoto, and ExpenseAddMultiple, consistently failed across multiple episodes.
|
||||
- Residual pattern: The failure rate was particularly high for tasks involving complex interactions with the UI, such as ContactsAddContact and ExpenseDeleteMultiple2.
|
||||
- Residual pattern: The evaluator reward was low for most episodes, indicating that the tasks were not being completed as expected.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 16,384 characters for action selection or 8,192 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- Run stopped by environment/runtime blocker: Generated params do not match checkpoint for MarkorCreateFolder:trial-4:seed-25270
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This user-requested local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,209 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:13:52Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.026 | 0.129 | 9.698 | 110.113 | 19.103 | 174058.216 | 20054856 / 135897 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteMultiple / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / FilesDeleteFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OsmAndFavorite / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsResend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMinVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOff / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 1`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, achieving a success rate of 2.59% with an estimated cost of $0.00. The mean latency was 110.11 seconds, and the mean number of LLM calls was 19.10. The mean total tokens used were 174,058.22.
|
||||
- Cost/benefit interpretation: The cost of running the 116 tasks was minimal, with an estimated cost of $0.00. However, the low success rate of 2.59% indicates that the current approach is not cost-effective. The high latency and token usage suggest that the model may need optimization to reduce computational overhead.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only 3 out of 116 tasks succeeding.
|
||||
- Residual pattern: Tasks involving complex interactions with the UI, such as 'RetroPlayingQueue' and 'SportsTrackerTotalDistanceForCategoryOverInterval', had the highest failure rates.
|
||||
- Residual pattern: Tasks that required multiple steps or complex conditions, like 'RecipeDeleteMultipleRecipesWithNoise', were particularly challenging.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 16,384 characters for action selection or 8,192 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,206 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:14:33Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.034 | 0.121 | 9.569 | 106.154 | 18.690 | 163049.526 | 18780790 / 132955 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / CameraTakePhoto / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ClockStopWatchRunning / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseAddMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsNewContactDraft / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorDeleteNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorMergeNotes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleRecipe / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSend / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivityDuration / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMax / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 2`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 2`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 2`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, and all tasks completed without errors. The success rate was 3.45%, with 4 successes out of 116 episodes. The mean latency was 106.15 seconds, and the mean total tokens were 163,049.53.
|
||||
- Cost/benefit interpretation: The cost in terms of latency and token usage is high, but the benefit in terms of preserving paired success without regression is maintained. However, the low success rate suggests that the current approach may not be effective.
|
||||
- Residual pattern: Most tasks failed to achieve success as defined by the evaluator reward.
|
||||
- Residual pattern: Tasks such as 'SystemBrightnessMax', 'SystemBrightnessMin', and 'SystemBrightnessMinVerify' had high latency and token usage.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,203 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:14:42Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.052 | 0.142 | 9.698 | 108.721 | 19.078 | 166715.009 | 19204766 / 134175 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / FilesDeleteFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarNextEvent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMaxVerify / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMin / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemWifiTurnOn / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksCompletedTasksForDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesRecipeIngredientCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 3`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / NotesIsTodo / trial 3`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesTodoItemCount / trial 3`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, achieving a success rate of 5.17% with 6 successes out of 116 episodes. The mean latency was 108.72 seconds, and the mean total tokens were 166,715. The cost-benefit analysis indicates that while the candidate approach preserved success with no regression, it did not meet the latency and token efficiency guardrails.
|
||||
- Cost/benefit interpretation: The candidate approach preserved success with no regression, but it did not meet the latency and token efficiency guardrails. The higher latency and token usage suggest that the candidate approach may be less efficient than the paired comparison, which could impact the overall cost and performance of the system.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only a few tasks (e.g., OpenAppTaskEval, SimpleCalendarNextMeetingWithPerson) succeeding.
|
||||
- Residual pattern: The mean latency and mean total tokens were higher than the guardrails specified for the paired comparison.
|
||||
- Residual pattern: The candidate approach did not meet the latency and token efficiency requirements, as the mean latency was 1.5 times the paired comparison and the mean tokens were 0.75 times the raw-UIAutomator tokens.
|
||||
- Next hypothesis `H5C_extended` (middle): Further refine the input pipeline by optimizing the element list to reduce latency and token usage while maintaining success. Target: Reduce the mean latency to 1.25 times the paired comparison and the mean tokens to 0.65 times the raw-UIAutomator tokens. Verification: Conduct a full-suite candidate rerun with the optimized element list to validate the hypothesis.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: 8,192 characters for action selection or 4,096 from each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,204 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:19:57Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.052 | 0.129 | 9.716 | 109.254 | 19.069 | 169894.517 | 19570600 / 137164 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchRunning / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsAddContact / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseAddMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateFolder / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorCreateNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / MarkorDeleteNewestNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / MarkorEditNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ContactsNewContactDraft / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / OsmAndFavorite / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndTrack / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithConstraint / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOffVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemWifiTurnOn / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOnVerify / trial 4`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / TasksCompletedTasksForDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemCopyToClipboard / trial 4`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / NotesIsTodo / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 4`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset rerun completed with 116 episodes, achieving a success rate of 5.17% and an estimated cost of $0.00. The mean latency was 109.25 seconds, and the mean number of LLM calls was 19.07.
|
||||
- Cost/benefit interpretation: The cost of running the candidate subset is minimal, with an estimated cost of $0.00. However, the success rate is low, and the latency and number of LLM calls are high, indicating that the current approach may not be efficient or effective.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only a few tasks (e.g., NotesMeetingAttendeeCount, SystemBrightnessMaxVerify) succeeding.
|
||||
- Residual pattern: The majority of tasks took longer than the mean latency of 109.25 seconds, indicating potential inefficiencies.
|
||||
- Residual pattern: The mean number of LLM calls per task was 19.07, which is relatively high, suggesting that the model may be making multiple calls to achieve a task.
|
||||
- Next hypothesis `H5C_extended` (middle): Implement a more aggressive filtering of the UIAutomator hierarchy to reduce the number of LLM calls and improve latency while maintaining success rates. Target: Reduce the mean number of LLM calls per task to 10 and decrease the mean latency to 80 seconds. Verification: Run a full-suite candidate rerun with the new filtering strategy and compare the success rate, latency, and number of LLM calls to the current results.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Run stopped by environment/runtime blocker: Generated params do not match checkpoint for MarkorCreateFolder:trial-4:seed-25270
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,202 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260804T045559Z`
|
||||
- Generated (UTC): `2026-08-04T09:13:41Z`
|
||||
- Upstream commit: `d9c569f764b3a5629321858de03ff653d0f24056`
|
||||
- Device: `sdk_gphone64_x86_64`, API `33` (upstream tested reference: API `33`)
|
||||
- Observation method: `uiautomator_compact`
|
||||
- Provider/model: `local-vllm` / `qwen2.5-7b-instruct-local`
|
||||
- Model source/runtime: `local_gpu` / `vllm-0.19.0`
|
||||
- Accelerator: `NVIDIA_RTX_PRO_6000_Blackwell_96GB`
|
||||
- Required apps: `24/24`
|
||||
- Scope: 116 task(s), 5 trial(s), mode `candidate_rerun`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens | Est. cost (USD) |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| candidate | 116/116 | 0.060 | 0.147 | 9.681 | 115.062 | 19.052 | 171630.552 | 19773398 / 135746 | 0.000000 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`candidate_subset_rerun_completed`**
|
||||
- Reason: A real modified candidate subset rerun completed, but it is not the 116-task × five-trial gate and cannot approve deployment.
|
||||
- Treatment/control mean latency ratio: n/a
|
||||
- Treatment/control mean token ratio: n/a
|
||||
- Treatment/control mean LLM-call ratio: n/a
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `candidate / AudioRecorderRecordAudio / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / AudioRecorderRecordAudioWithFileName / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserDraw / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMaze / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / BrowserMultiply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakePhoto / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / CameraTakeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockStopWatchPausedVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ClockStopWatchRunning / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ClockTimerEntry / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / ContactsAddContact / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultiple / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromGallery / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddSingle / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteDuplicates / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteDuplicates2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteMultiple / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ExpenseDeleteMultiple2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseDeleteSingle / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / ContactsNewContactDraft / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / ExpenseAddMultipleFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesDeleteFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / FilesMoveFile / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorAddNoteHeader / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorChangeNoteContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteAndSms / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorCreateNoteFromClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteAllNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNewestNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorDeleteNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorEditNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMergeNotes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorMoveNote / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeReceipt / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / MarkorTranscribeVideo / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesTodoItemCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OpenAppTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndFavorite / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / OsmAndMarker / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / OsmAndTrack / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromImage / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddMultipleRecipesFromMarkor2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeAddSingleRecipe / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes2 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteDuplicateRecipes3 / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipes / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteMultipleRecipesWithNoise / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RecipeDeleteSingleWithRecipeWithNoise / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / RetroCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlaylistDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroSavePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SaveCopyOfReceiptTaskEval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventInTwoWeeks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddOneEventTomorrow / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAddRepeatingEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarAnyEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEvents / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteEventsOnRelativeDay / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarDeleteOneEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventOnDateAtTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsInTimeRange / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarEventsOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarFirstEventAfterStartTime / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarLocationOfEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextEvent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleCalendarNextMeetingWithPerson / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SimpleDrawProCreateDrawing / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReply / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsResend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSend / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendClipboardContent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsSendReceivedAddress / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SportsTrackerActivitiesCountForWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivitiesOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerActivityDuration / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerLongestDistanceActivity / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDistanceForCategoryOverInterval / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SportsTrackerTotalDurationForCategoryThisWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOffVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBluetoothTurnOn / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBluetoothTurnOnVerify / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / SystemBrightnessMax / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMin / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemBrightnessMinVerify / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / SystemCopyToClipboard / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOff / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SystemWifiTurnOn / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TasksCompletedTasksForDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueNextWeek / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasks / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksHighPriorityTasksDueOnDate / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TasksIncompleteTasksOnDate / trial 5`: agent declared completion, but the real evaluator state failed
|
||||
- `candidate / TurnOffWifiAndTurnOnBluetooth / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / TurnOnWifiAndOpenApp / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreatePlaylist / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / VlcCreateTwoPlaylists / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / RetroPlayingQueue / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / SimpleSmsReplyMostRecent / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
- `candidate / NotesIsTodo / trial 5`: final evaluator state passed, but the agent never declared completion
|
||||
- `candidate / NotesMeetingAttendeeCount / trial 5`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: The candidate subset of 116 tasks was rerun, achieving a success rate of 6.03% with 7 successes out of 116 episodes. The mean latency was 115.06 seconds, and the mean total tokens were 171,630.55. The cost-benefit analysis indicates that while the candidate subset showed some success, the high latency and token usage suggest further optimization is needed.
|
||||
- Cost/benefit interpretation: The candidate subset showed some success but with high latency and token usage. Further optimization is needed to reduce these metrics while maintaining or improving success rates.
|
||||
- Residual pattern: Most tasks failed to achieve success, with only 7 out of 116 tasks completing successfully.
|
||||
- Residual pattern: The mean latency of 115.06 seconds is significantly high, indicating potential inefficiencies in the current implementation.
|
||||
- Residual pattern: The mean total tokens of 171,630.55 are also high, suggesting that the model is using more resources than necessary.
|
||||
- Next hypothesis `H5C` (middle): Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements. Target: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. Verification: Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- The full-suite candidate uses model qwen2.5-7b-instruct-local, while the promoted paired H5C source used doubao-seed-1-6-250615. This local-GPU campaign evaluates the promoted observation treatment but is not a same-model extension of the paired result.
|
||||
- ContactsNewContactDraft's official success predicate was fed the upstream UIAutomator state.ui_elements because that observation mode does not populate state.forest; the predicate and requested contact fields were not changed.
|
||||
- Clipboard get/set retries once after the exact Clipper foreground-access runtime error; the operation, content, task, and evaluator are unchanged.
|
||||
- SimpleSmsReplyMostRecent polls the unchanged inbox query for up to five additional seconds because emulator-injected SMS delivery can lag past upstream's fixed wait. If the inbox remains empty, the exact last injected address/body is inserted into the same SMS database that upstream clears directly; task data and the evaluator are unchanged.
|
||||
- The pinned official Retro Music APK omits the playing_queue table, a known upstream runtime error. Only that exact missing-table condition was mapped to an empty observed queue so the unchanged exact queue predicate records an evaluator failure instead of losing the episode.
|
||||
- Runtime-error retries reuse the exact task parameters retained in the discarded error checkpoints; upstream parameter-generator drift cannot silently change the retried task.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes 8,192 characters only from the middle of the current-screen indexed UI section. Prompt prefix, goal, history, leading and trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
- If compact UIAutomator still exceeds the pinned model's native 32,768-token context, the retry removes a bounded middle span only from indexed UI descriptions: at most 12,000 retained characters for action selection or 6,000 for each before/after summary screen. Prompt prefix, goal, history, action, reason, leading/trailing UI elements and indices, guidance, and output format remain; per-episode removal counters are retained.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
@@ -0,0 +1,136 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"experiment": "7-12",
|
||||
"status": "complete",
|
||||
"generated_at_utc": "2026-08-04T09:35:34.274750+00:00",
|
||||
"run_dir": "chapter7/android-world/validation",
|
||||
"git_commit": "0d0df3e758872c7e73a998e57d2e529b16ece095",
|
||||
"command": "Five isolated Pixel 6/API-33 trial shards used run_controlled_experiment.py with local-vLLM credentials supplied only by environment-variable name; merge_candidate_shards.py strictly merged trials 1-5.",
|
||||
"provider_receipt_count": 0,
|
||||
"status_reasons": [
|
||||
"580/580 direct episodes completed across 116 tasks x five trials with zero runtime errors; evaluator failures are retained.",
|
||||
"Every shard completed official Pixel 6/API-33 setup and recorded the same 24/24 required package versions.",
|
||||
"The result is negative: 26/580 strict successes and deployment is not approved.",
|
||||
"The candidate used local Qwen2.5-7B while the paired H5C source used Doubao, so no same-model uplift or noninferiority is claimed.",
|
||||
"All disclosed evaluator, emulator-race, context-limit, resume, and exact-parameter retry compatibility treatments remain in retained evidence."
|
||||
],
|
||||
"inputs": [
|
||||
{
|
||||
"path": "chapter7/android-world/experiment_core.py",
|
||||
"bytes": 24201,
|
||||
"sha256": "752c1f181f0bef3cb0726fcc35eddc8c715624c4804c39b9a2d371518f4b3ac6"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/run_controlled_experiment.py",
|
||||
"bytes": 76108,
|
||||
"sha256": "ebc5417066fadbcd3de15d878d8b95a6ab9bff7f1b8e9eec392d148dbc59817e"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/merge_candidate_shards.py",
|
||||
"bytes": 11368,
|
||||
"sha256": "c7b0519c2246e88750f91859603fd34fea72a67f04c1e8eacb8fc5d2d63cd0f1"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/t3a_summary.md",
|
||||
"bytes": 28800,
|
||||
"sha256": "e5616507e2b0658b64dc5d0e2c08e6207eba12da8345a31d1cb85bf68957f002"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/t3a_failed_analysis.md",
|
||||
"bytes": 14448,
|
||||
"sha256": "7486112a3bf0791a17ca72b81b3745dcd92936416b5c3da937618c8d7a6eb00d"
|
||||
}
|
||||
],
|
||||
"artifacts": [
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_qwen_20260804/evidence.json",
|
||||
"bytes": 3932422,
|
||||
"sha256": "526e73e061bbafa3802484934dd29512a3b94a6a066d0efa10fc44c3f8138083"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_qwen_20260804/report.md",
|
||||
"bytes": 70854,
|
||||
"sha256": "4b4222ba4597eb57f35f40a30de180284d00a8fa66824e5492c75c4c8ce8b96a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard1/evidence.json",
|
||||
"bytes": 833946,
|
||||
"sha256": "8c9bad94ee2141c4de2e666e291f41e1f7d7cf7c53a8ad77a73ab17341bb68a3"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard1/report.md",
|
||||
"bytes": 23232,
|
||||
"sha256": "6113f7d935c4c1f085a53a4a2f0be5e9d8053e47c5aa4434d56a6a5d7c56e6a4"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard2/evidence.json",
|
||||
"bytes": 805331,
|
||||
"sha256": "2cf319e6eb7f99758c7dd093a46d445afa29ff35a919afc5f08f01c0a2085f44"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard2/report.md",
|
||||
"bytes": 22455,
|
||||
"sha256": "573f6fa57a846d6b231db2702069afff277434f39518deec8b64bdc0720eac92"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard3/evidence.json",
|
||||
"bytes": 815150,
|
||||
"sha256": "9b4bfed4165696044768c9e05fa00d696469ea99a06e9d3f694ea817a1041911"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard3/report.md",
|
||||
"bytes": 22157,
|
||||
"sha256": "8e37b4fdaf97eaa0bc14a6a161bcf14914ae6512782a94b5901c7bbfc361806a"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard4/evidence.json",
|
||||
"bytes": 820872,
|
||||
"sha256": "4da385617b9b2f7a3d684884fd79184970e23d862583ab6b622b48f975e15696"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard4/report.md",
|
||||
"bytes": 21949,
|
||||
"sha256": "509a9224d6ce823c6d9ccff15a962fafdc0ed0b9724c053c2d83754517764f02"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard5/evidence.json",
|
||||
"bytes": 808670,
|
||||
"sha256": "245f99f7a8924d1b8340e510a6b12f345ac35ea779099605755d15d180109ce3"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/candidate_h5c_api33_local_shard5/report.md",
|
||||
"bytes": 21696,
|
||||
"sha256": "4d22f0cdeb7b5f82560241b2f91169b94629c1f9cca2f320de4f54f5f1b99479"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"bytes": 47072,
|
||||
"sha256": "b2276e35e1f58d32775ea70ac04142a7d3075c074b9f31d40b153e1be10f93da"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5_a11y_api35_20260729/report.md",
|
||||
"bytes": 8239,
|
||||
"sha256": "5d5b0db9580020c286096ca53617cbd2d0f7fb811e48c4cf51688212ef638272"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5c_compact_api35_20260729/evidence.json",
|
||||
"bytes": 39643,
|
||||
"sha256": "a60204b6c0ac187b0ae9a68182b5ada302420fc31b2fc22c308e10eb1242e390"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_h5c_compact_api35_20260729/report.md",
|
||||
"bytes": 8930,
|
||||
"sha256": "55abde0ef9a6d5ed090f26bc99e424346c42a5e98036a278bd50edcf4bf48f2e"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"bytes": 45941,
|
||||
"sha256": "a1665bffafed7b8935db03dc4369435015f784d6f6465fe09c7da86fc0504635"
|
||||
},
|
||||
{
|
||||
"path": "chapter7/android-world/validation/paired_wifi_api35_20260729/report.md",
|
||||
"bytes": 4096,
|
||||
"sha256": "067f176e4722082afd45bedbd79a392b26d677d54013a8b14a81cea1fbd70d57"
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,93 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260729T122904Z`
|
||||
- Generated (UTC): `2026-07-29T13:17:47Z`
|
||||
- Upstream commit: `0e95d641e244504c22087cc29b013f3b2428a261`
|
||||
- Device: `sdk_gphone64_arm64`, API `35` (upstream tested reference: API `33`)
|
||||
- Observation method: `varies_by_arm:a11y_forwarder_app_vs_uiautomator`
|
||||
- Provider/model: `ark` / `doubao-seed-1-6-250615`
|
||||
- Scope: 4 task(s), 1 trial(s), mode `paired`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5`
|
||||
- Change: Select AndroidWorld's UIAUTOMATOR observation method in the companion runner without changing upstream source.
|
||||
- Expected measurable result: At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, no regression, and at most 1.5x mean latency and tokens; never treat a subset gain as full-suite success or deployment approval.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_middle` — middle-layer input-pipeline ablation prompted by phase-1 residual traces
|
||||
- Independent variable: accessibility observation pipeline (gRPC forwarder versus UIAutomator)
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| control | 4/4 | 0.250 | 0.500 | 8.000 | 172.849 | 14.000 | 68310.500 | 251587 / 21655 |
|
||||
| treatment | 4/4 | 1.000 | 1.000 | 5.750 | 136.235 | 11.000 | 170673.500 | 671404 / 11290 |
|
||||
|
||||
| Task / trial | Control | Treatment | Δ success | Control→treatment steps |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| SystemWifiTurnOff / 1 | 0 | 1 | +1 | 10→6 |
|
||||
| SystemWifiTurnOffVerify / 1 | 0 | 1 | +1 | 10→6 |
|
||||
| SystemWifiTurnOn / 1 | 0 | 1 | +1 | 10→7 |
|
||||
| SystemWifiTurnOnVerify / 1 | 1 | 1 | +0 | 2→4 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`restrict_candidate_due_to_cost`**
|
||||
- Reason: Treatment improved paired success without regressions, but exceeded the latency/token guardrails (1.50x / 1.50x). Restrict it to targeted follow-up; do not promote it to the full suite yet.
|
||||
- Treatment/control mean latency ratio: 0.788
|
||||
- Treatment/control mean token ratio: 2.498
|
||||
- Treatment/control mean LLM-call ratio: 0.786
|
||||
- Cost guardrails passed: **false**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `control / SystemWifiTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `control / SystemWifiTurnOffVerify / trial 1`: final evaluator state passed, but the agent never declared completion
|
||||
- `control / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: Experiment 7-12 compared control (a11y-forwarder observation) and treatment (UIAutomator observation) arms in a paired setup with 4 Wi-Fi system Settings tasks. Treatment achieved 100% success (4/4) vs control's 25% (1/4), reduced mean latency (136.2s vs 172.8s) and LLM calls (11.0 vs 14.0), but had a mean token ratio (treatment/control) of 2.498, exceeding the 1.5x guardrail. Environment boundaries include API 35 AVD (vs upstream API 33 reference), restriction to Settings tasks, UIAutomator as a compatibility path (not reference config), and skipped device-time setting due to non-root AVD limitations.
|
||||
- Cost/benefit interpretation: Treatment provides substantial benefit via improved success rate (net +3) and reduced latency/LLM calls, but incurs significantly higher token cost (2.498x control), violating token guardrails and limiting deployment despite success gains.
|
||||
- Residual pattern: Treatment mean token ratio (2.498x) exceeds 1.5x guardrail
|
||||
- Residual pattern: Control arm has low success rate (25%, 1/4 completed episodes)
|
||||
- Next hypothesis `H6` (middle): Optimize UIAutomator observation pipeline to reduce token usage while maintaining treatment success rate Target: Mean token ratio (treatment/control) ≤1.5x and success rate ≥1.0 in paired Wi-Fi tasks Verification: Conduct paired run with optimized UIAutomator pipeline vs control, using same 4 Wi-Fi tasks, API 35 AVD environment, and guardrails; measure token ratio and success rate
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.
|
||||
- The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
@@ -0,0 +1,940 @@
|
||||
{
|
||||
"arm_summary": {
|
||||
"control": {
|
||||
"completed_episodes": 4,
|
||||
"episodes": 4,
|
||||
"error_episodes": 0,
|
||||
"mean_evaluator_reward": 1.0,
|
||||
"mean_latency_s": 101.198336,
|
||||
"mean_llm_calls": 8.5,
|
||||
"mean_llm_latency_s": 62.453747,
|
||||
"mean_steps": 4.75,
|
||||
"mean_total_tokens": 139439.5,
|
||||
"success_rate": 1.0,
|
||||
"successes": 4,
|
||||
"total_input_tokens": 549928,
|
||||
"total_output_tokens": 7830,
|
||||
"total_tokens": 557758
|
||||
},
|
||||
"treatment": {
|
||||
"completed_episodes": 4,
|
||||
"episodes": 4,
|
||||
"error_episodes": 0,
|
||||
"mean_evaluator_reward": 1.0,
|
||||
"mean_latency_s": 99.184035,
|
||||
"mean_llm_calls": 8.5,
|
||||
"mean_llm_latency_s": 59.748216,
|
||||
"mean_steps": 4.75,
|
||||
"mean_total_tokens": 70557.5,
|
||||
"success_rate": 1.0,
|
||||
"successes": 4,
|
||||
"total_input_tokens": 274067,
|
||||
"total_output_tokens": 8163,
|
||||
"total_tokens": 282230
|
||||
}
|
||||
},
|
||||
"baseline": {
|
||||
"agent": "t3a_claude4_sonnet",
|
||||
"provenance": "historical bundled report; not generated by this runner",
|
||||
"reported_success_rate_approx": 0.88,
|
||||
"run_date": "2025-07-02",
|
||||
"source": "t3a_summary.md and t3a_failed_analysis.md",
|
||||
"tasks": 116,
|
||||
"trials_per_task": 1
|
||||
},
|
||||
"command": [
|
||||
"run_controlled_experiment.py",
|
||||
"--mode",
|
||||
"paired",
|
||||
"--hypothesis",
|
||||
"H5C",
|
||||
"--source-phase1-evidence",
|
||||
"validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"--source-phase2-evidence",
|
||||
"validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"--tasks",
|
||||
"SystemWifiTurnOff,SystemWifiTurnOffVerify,SystemWifiTurnOn,SystemWifiTurnOnVerify",
|
||||
"--trials",
|
||||
"1",
|
||||
"--seed",
|
||||
"42",
|
||||
"--model-seed",
|
||||
"42",
|
||||
"--max-steps",
|
||||
"10",
|
||||
"--transition-pause",
|
||||
"0.5",
|
||||
"--skip-device-time",
|
||||
"--output-dir",
|
||||
"validation/paired_h5c_compact_api35_20260729"
|
||||
],
|
||||
"credentials_persisted": false,
|
||||
"decision": {
|
||||
"completed_pairs": 4,
|
||||
"deployment_approved": false,
|
||||
"guardrails": {
|
||||
"maximum_latency_ratio": 1.5,
|
||||
"maximum_token_ratio": 0.75,
|
||||
"objective": "success_noninferiority_and_token_reduction",
|
||||
"passed": true,
|
||||
"require_all_treatment_pairs_successful": true
|
||||
},
|
||||
"mean_latency_ratio_treatment_over_control": 0.980096,
|
||||
"mean_llm_call_ratio_treatment_over_control": 1.0,
|
||||
"mean_token_ratio_treatment_over_control": 0.506008,
|
||||
"net_success_delta": 0,
|
||||
"outcome": "promote_efficient_candidate_to_full_suite_rerun",
|
||||
"paired_regressions": 0,
|
||||
"promote_to_full_suite_candidate": true,
|
||||
"reason": "Treatment preserved paired success with no regression and passed the latency/token efficiency guardrails. This is a candidate decision only.",
|
||||
"required_treatment_successes": 4,
|
||||
"scope_recommendation": "full_suite_candidate_only",
|
||||
"success_preservation_passed": true,
|
||||
"treatment_successes": 4
|
||||
},
|
||||
"diagnosis": {
|
||||
"findings": [
|
||||
"The historical run evaluated 116 tasks once each and reports approximately 88% overall success.",
|
||||
"Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.",
|
||||
"The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.",
|
||||
"The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause."
|
||||
],
|
||||
"layered_hypotheses": [
|
||||
{
|
||||
"id": "H1",
|
||||
"idea": "Add Wi-Fi Settings navigation and final-state verification guidance.",
|
||||
"layer": "surface",
|
||||
"status": "tested in source phase 1",
|
||||
"target": "At least one net paired success across the four Wi-Fi tasks, with no regression.",
|
||||
"verification": "Paired upstream-prompt versus task-guideline ablation with matched seeds."
|
||||
},
|
||||
{
|
||||
"id": "H2",
|
||||
"idea": "Add application-specific recognition rules for the non-standard Tasks UI.",
|
||||
"layer": "surface",
|
||||
"status": "not tested",
|
||||
"target": "Improve at least two of the six historical Tasks failures with no regression.",
|
||||
"verification": "Paired Tasks-only prompt/tool-description ablation after app provisioning."
|
||||
},
|
||||
{
|
||||
"id": "H3",
|
||||
"idea": "Repair and validate the multimodal input path for transcription tasks.",
|
||||
"layer": "middle",
|
||||
"status": "not tested",
|
||||
"target": "Raise transcription success above the historical 0% while bounding added tokens and latency.",
|
||||
"verification": "Paired screenshot-disabled versus screenshot-enabled transcription run."
|
||||
},
|
||||
{
|
||||
"id": "H4",
|
||||
"idea": "Conditionally enable deeper thinking for counting tasks.",
|
||||
"layer": "middle",
|
||||
"status": "not tested",
|
||||
"target": "Improve math/counting success without applying the cost to unrelated tasks.",
|
||||
"verification": "Paired tag-routed thinking-mode ablation with latency and token guardrails."
|
||||
},
|
||||
{
|
||||
"id": "H5",
|
||||
"idea": "Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path.",
|
||||
"layer": "middle",
|
||||
"status": "tested in source phase 2",
|
||||
"target": "At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens.",
|
||||
"verification": "Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds."
|
||||
},
|
||||
{
|
||||
"id": "H5C",
|
||||
"idea": "Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost.",
|
||||
"layer": "middle",
|
||||
"status": "tested in this run",
|
||||
"target": "Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.",
|
||||
"verification": "Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator."
|
||||
},
|
||||
{
|
||||
"id": "H6",
|
||||
"idea": "Combine screenshots with the structured UI tree and compare stronger vision-capable models.",
|
||||
"layer": "deep",
|
||||
"status": "not tested",
|
||||
"target": "Improve complex-UI success enough to justify multimodal latency and token cost.",
|
||||
"verification": "Factorial UI-tree/screenshot/model ablation on the full tagged slice."
|
||||
}
|
||||
]
|
||||
},
|
||||
"environment": {
|
||||
"a11y_method": "varies_by_arm:uiautomator_vs_uiautomator_compact",
|
||||
"android_world_checkout": "/Users/boj/book/ai-agent-book/chapter7/android_world",
|
||||
"android_world_checkout_clean": true,
|
||||
"android_world_commit": "0e95d641e244504c22087cc29b013f3b2428a261",
|
||||
"api_level": 35,
|
||||
"avd_name": "Pixel_9_Pro_API_35",
|
||||
"device_model": "sdk_gphone64_arm64",
|
||||
"device_serial": "emulator-5554",
|
||||
"grpc_port": 8554,
|
||||
"perform_emulator_setup": false,
|
||||
"physical_size": "Physical size: 1280x2856",
|
||||
"protobuf_bootstrap": {
|
||||
"generated_protobufs_in_temporary_directory": true,
|
||||
"temporary_directory_persisted": false
|
||||
},
|
||||
"skip_device_time": true,
|
||||
"upstream_tested_api_level": 33,
|
||||
"upstream_tested_device": "Pixel 6"
|
||||
},
|
||||
"environment_boundaries": [
|
||||
"The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.",
|
||||
"The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.",
|
||||
"UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.",
|
||||
"Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.",
|
||||
"Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time."
|
||||
],
|
||||
"episodes": [
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 105.382271,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 132349,
|
||||
"latency_s": 64.872256,
|
||||
"output_tokens": 2008,
|
||||
"reasoning_tokens": 1488,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOff:trial-1:seed-42",
|
||||
"pair_seed": 42,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOff",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 22.742624,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will open the Settings menu where Wi-Fi settings can be accessed.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.091666,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows an error dialog \"com.google.androidenv.accessibilityforwarder keeps stopping\" which is blocking access to the Settings menu. To proceed with turning off Wi-Fi, I need to dismiss this dialog by closing the problematic app. The \"Close app\" button (index 11) is visible and clickable, so clicking it will remove the dialog and allow me to continue navigating to Wi-Fi settings.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "input_text",
|
||||
"done": false,
|
||||
"elapsed_s": 24.93577,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with turning off Wi-Fi, I need to locate the Wi-Fi settings. Since the current screen is the settings search screen, the most efficient way is to search for \"Wi-Fi\" using the search bar. The search bar (UI element 15) is an editable text field and is focused, making it suitable for input.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 25.397996,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows the Wi-Fi settings with a checked (on) switch at index 32. To turn off Wi-Fi, I need to click this switch to toggle it to the off position.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 8.938998,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (index 32) is now unchecked (is_checked=False), indicating that Wi-Fi has been turned off. The user's goal of turning Wi-Fi off has been successfully achieved.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 103.549168,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 75681,
|
||||
"latency_s": 65.635331,
|
||||
"output_tokens": 2101,
|
||||
"reasoning_tokens": 1568,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOff:trial-1:seed-42",
|
||||
"pair_seed": 42,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOff",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.105315,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 12 is the \"Settings\" app, which is visible and clickable. Clicking it will open the Settings menu where Wi-Fi settings can be accessed.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 23.230104,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, I need to access the Wi-Fi settings. The \"Network & internet\" option (UI element 6) is visible and clickable, and its summary mentions \"Wi‑Fi\", indicating it contains Wi-Fi settings. Clicking this will lead to the next screen where Wi-Fi can be toggled off.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.825863,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi settings and turn it off, I need to click on the \"Internet\" option, which is part of the clickable linear layout (UI element 3) on the Network & internet settings page.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 27.327625,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows the Internet settings where the Wi-Fi switch is visible. The Wi-Fi option is contained in UI element 9 (a LinearLayout), which is clickable. This element includes the \"Wi-Fi\" text (UI element 10) and the switch (UI element 11) that is currently on (is_checked=True). Clicking this LinearLayout will toggle the Wi-Fi switch off.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 8.69858,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch is now off (is_checked=False as per UI element 11), so the task of turning Wi-Fi off has been successfully completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 102.209211,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 64462,
|
||||
"latency_s": 62.502274,
|
||||
"output_tokens": 2130,
|
||||
"reasoning_tokens": 1620,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOffVerify:trial-1:seed-1051",
|
||||
"pair_seed": 1051,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOffVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.831033,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows an error message pop-up that is blocking access to the main interface. To proceed with turning off Wi-Fi, I need to close this pop-up first. The \"Close app\" button (UI element 2) is visible and clickable, so clicking it will dismiss the pop-up.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 23.270511,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the next step is to open the Settings app, as Wi-Fi settings are typically located there. The \"Settings\" app is visible on the home screen (UI element 12), which is clickable. Clicking it will allow access to the settings menu where Wi-Fi can be managed.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 23.028096,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, I need to access the network settings. The \"Network & internet\" option (UI element 6, which is a clickable LinearLayout containing the \"Network & internet\" title and summary) is the appropriate next step to navigate to Wi-Fi controls.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.913882,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi controls, I need to navigate to the \"Internet\" settings. The \"Internet\" option is contained within the clickable LinearLayout (UI element 3), which is visible and interactive on the current \"Network & internet\" screen. Clicking this element will likely lead to the page where Wi-Fi can be toggled off.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 11.76325,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 11) is already in the off position (is_checked=False), so the goal of turning Wi-Fi off has been achieved.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 112.724126,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi off.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 150834,
|
||||
"latency_s": 67.904524,
|
||||
"output_tokens": 2205,
|
||||
"reasoning_tokens": 1649,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOffVerify:trial-1:seed-1051",
|
||||
"pair_seed": 1051,
|
||||
"params": {
|
||||
"on_or_off": "off"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOffVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 18.772365,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, the first step is to open the Settings app. On the home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will open the Settings menu where Wi-Fi settings can be accessed.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 25.71773,
|
||||
"goal_status": null,
|
||||
"reason": "To turn off Wi-Fi, I need to access the Wi-Fi settings. The \"Network & internet\" option (UI element 24) in the Settings menu is the appropriate entry point as it mentions \"Wi‑Fi\" in its summary. The parent LinearLayout (UI element 20) is clickable, so I will tap on it to proceed to the Network & internet settings page.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 24.548097,
|
||||
"goal_status": null,
|
||||
"reason": "The goal is to turn off Wi-Fi. Currently, on the Settings home screen, the \"Network & internet\" section (UI element 20) is visible, which is the correct path to access Wi-Fi settings. Since the previous attempt to click this element didn't result in a screen change (possibly due to a delay), retrying the click on UI element 20 is necessary to navigate to the Network & internet settings where Wi-Fi can be toggled off.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.737918,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi settings, I need to navigate into the \"Internet\" section under \"Network & internet\". The \"Internet\" option (UI element 15) is clickable and likely contains Wi-Fi controls, so clicking it will allow me to proceed toward turning off Wi-Fi.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 13.895503,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 31) is already in the off position (is_checked=False), so the task of turning Wi-Fi off has been completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 100.007401,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 146625,
|
||||
"latency_s": 60.454149,
|
||||
"output_tokens": 1997,
|
||||
"reasoning_tokens": 1484,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOn:trial-1:seed-2060",
|
||||
"pair_seed": 2060,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOn",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.700473,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will allow access to the settings menu where Wi-Fi can be enabled.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.450628,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed towards enabling Wi-Fi, I need to access the \"Network & internet\" settings, which is the logical next step as it contains Wi-Fi options. The \"Network & internet\" element (index 24) is visible and clickable on the current screen.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 22.812015,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed towards enabling Wi-Fi, I need to access the Internet settings which typically include Wi-Fi options. The \"Internet\" option (contained in the clickable LinearLayout at index 15) is visible on the screen and is the logical next step to find the Wi-Fi toggle.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.686302,
|
||||
"goal_status": null,
|
||||
"reason": "The current screen shows the Internet settings page where the Wi-Fi option is present. The Wi-Fi switch (UI element 31) is currently unchecked (is_checked=False), indicating Wi-Fi is off. To turn Wi-Fi on, I need to click this switch to toggle it to the on position.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 10.361169,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 35) is now checked, indicating Wi-Fi is enabled. The screen also shows \"AndroidWifi\" as connected, confirming the task of turning Wi-Fi on has been successfully completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 106.717192,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 0.0,
|
||||
"llm": {
|
||||
"calls": 9,
|
||||
"input_tokens": 74268,
|
||||
"latency_s": 60.267918,
|
||||
"output_tokens": 2150,
|
||||
"reasoning_tokens": 1681,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOn:trial-1:seed-2060",
|
||||
"pair_seed": 2060,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 5,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOn",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "open_app",
|
||||
"done": false,
|
||||
"elapsed_s": 23.770885,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app where network settings are located. The current screen shows a \"Settings\" app icon (UI element 12), and using the `open_app` action is the most direct way to launch the app.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 27.515651,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed to the Wi-Fi settings, I need to access the \"Network & internet\" section. The UI element 6 is a clickable LinearLayout that contains the \"Network & internet\" text (elements 7 and 8), making it the appropriate element to interact with.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 22.324583,
|
||||
"goal_status": null,
|
||||
"reason": "To access the Wi-Fi settings, I need to navigate into the \"Internet\" section, which is likely to contain Wi-Fi controls. The \"Internet\" option is part of the clickable LinearLayout (UI element 3) on the current screen.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 19.494808,
|
||||
"goal_status": null,
|
||||
"reason": "The Wi-Fi switch (UI element 11) is currently off (is_checked=False). To turn Wi-Fi on, I need to click this switch to toggle it to the on position.",
|
||||
"step": 4
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 11.27913,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 11) is now checked (is_checked=True), indicating Wi-Fi is turned on. Additionally, \"AndroidWifi\" is shown as connected (UI element 12), confirming the task is completed.",
|
||||
"step": 5
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "treatment",
|
||||
"elapsed_s": 84.26057,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 7,
|
||||
"input_tokens": 59656,
|
||||
"latency_s": 50.587342,
|
||||
"output_tokens": 1782,
|
||||
"reasoning_tokens": 1390,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator_compact",
|
||||
"order_position": 1,
|
||||
"pair_id": "SystemWifiTurnOnVerify:trial-1:seed-3069",
|
||||
"pair_seed": 3069,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 4,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOnVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 20.295946,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app. On the current home screen, UI element 12 is the \"Settings\" app, which is visible and clickable. Clicking on it will allow access to the settings menu where Wi-Fi can be enabled.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.66904,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with turning on Wi-Fi, I need to access the \"Network & internet\" settings, as it contains Wi-Fi options. The \"Network & internet\" section is represented by the clickable LinearLayout (UI element 6) on the current screen. Clicking this will navigate to the relevant submenu where Wi-Fi can be enabled.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.764651,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with turning on Wi-Fi, I need to access the Wi-Fi settings. The \"Internet\" section (contained in UI element 3, a clickable LinearLayout) is the logical next step, as it likely leads to Wi-Fi configuration options.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 8.154307,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 11) is already checked (is_checked=True), indicating that Wi-Fi is enabled. The user's goal of turning Wi-Fi on has been achieved.",
|
||||
"step": 4
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"agent_declared_done": true,
|
||||
"arm": "control",
|
||||
"elapsed_s": 86.679544,
|
||||
"error": null,
|
||||
"evaluator_reward": 1.0,
|
||||
"goal": "Turn wifi on.",
|
||||
"initial_evaluator_reward": 1.0,
|
||||
"llm": {
|
||||
"calls": 7,
|
||||
"input_tokens": 120120,
|
||||
"latency_s": 56.58406,
|
||||
"output_tokens": 1620,
|
||||
"reasoning_tokens": 1195,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"observation_method": "uiautomator",
|
||||
"order_position": 2,
|
||||
"pair_id": "SystemWifiTurnOnVerify:trial-1:seed-3069",
|
||||
"pair_seed": 3069,
|
||||
"params": {
|
||||
"on_or_off": "on"
|
||||
},
|
||||
"status": "completed",
|
||||
"steps": 4,
|
||||
"success": true,
|
||||
"task": "SystemWifiTurnOnVerify",
|
||||
"task_guidelines_applied": false,
|
||||
"trace": [
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 21.802337,
|
||||
"goal_status": null,
|
||||
"reason": "To turn on Wi-Fi, the first step is to open the Settings app. On the home screen, UI element 26 is the \"Settings\" app, which is clickable. Clicking it will allow access to the settings menu where Wi-Fi can be enabled.",
|
||||
"step": 1
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 24.619676,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with enabling Wi-Fi, I need to access the \"Network & internet\" settings, as it contains Wi-Fi options. The \"Network & internet\" section is a clickable element (UI element 20) on the current Settings screen. Clicking this will navigate to the relevant submenu where Wi-Fi can be toggled on.",
|
||||
"step": 2
|
||||
},
|
||||
{
|
||||
"action_type": "click",
|
||||
"done": false,
|
||||
"elapsed_s": 26.561039,
|
||||
"goal_status": null,
|
||||
"reason": "To proceed with enabling Wi-Fi, I need to access the Wi-Fi settings. The \"Internet\" option (UI element 19) with \"AndroidWifi\" as the summary (UI element 20) is likely the entry point to Wi-Fi settings. The parent linear layout (UI element 15) is clickable, so clicking it will navigate to the Wi-Fi settings screen where the toggle can be accessed.",
|
||||
"step": 3
|
||||
},
|
||||
{
|
||||
"action_type": "status",
|
||||
"done": true,
|
||||
"elapsed_s": 11.348688,
|
||||
"goal_status": "complete",
|
||||
"reason": "The Wi-Fi switch (UI element 35) is already in the checked state, indicating that Wi-Fi is turned on. The user's goal of turning Wi-Fi on has been achieved.",
|
||||
"step": 4
|
||||
}
|
||||
],
|
||||
"trial": 1
|
||||
}
|
||||
],
|
||||
"experiment": "7-12",
|
||||
"experiment_complete": false,
|
||||
"generated_at_utc": "2026-07-29T14:12:27Z",
|
||||
"hypothesis": {
|
||||
"change": "Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.",
|
||||
"expected_result": "Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.",
|
||||
"guardrails": "Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.",
|
||||
"guideline_sha256": null,
|
||||
"guidelines": [],
|
||||
"id": "H5C",
|
||||
"layer": "middle",
|
||||
"verification": "Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator."
|
||||
},
|
||||
"llm_analysis": {
|
||||
"cost_benefit_interpretation": "The treatment provides significant token efficiency (mean token ratio 0.506) with preserved success and marginal latency improvement (mean latency ratio 0.980) in the tested 4-task subset. However, interpretation is bounded by the API/app environment: results are from an API 35 AVD (not upstream API 33 reference), restricted to system Settings tasks (no full third-party app bundle), and use UIAutomator as a compatibility path (not upstream reference configuration), limiting generalizability beyond the tested scope.",
|
||||
"llm": {
|
||||
"calls": 1,
|
||||
"input_tokens": 3257,
|
||||
"latency_s": 34.44844,
|
||||
"output_tokens": 1101,
|
||||
"reasoning_tokens": 647,
|
||||
"system_fingerprints": []
|
||||
},
|
||||
"next_hypothesis": {
|
||||
"id": "H5C-full",
|
||||
"idea": "Evaluate compact UIAutomator (semantic-filtered elements) across the full AndroidWorld task suite to verify token efficiency and success preservation beyond the 4-task Settings subset.",
|
||||
"layer": "middle",
|
||||
"target": "Full task suite (all 116 tasks) with upstream reference environment (Pixel 6 / API 33) and provisioned full third-party AndroidWorld app bundle.",
|
||||
"verification": "Paired run comparing compact UIAutomator (treatment) vs raw UIAutomator (control) across all tasks, ensuring guardrails (mean token ratio ≤0.75, mean latency ratio ≤1.5, success non-inferiority) hold in the reference environment."
|
||||
},
|
||||
"observed_failure_pattern": [],
|
||||
"source": "real configured LLM over aggregate direct evidence",
|
||||
"status": "completed",
|
||||
"summary": "A paired experiment comparing raw UIAutomator (control) and compact UIAutomator (treatment) on 4 system Settings tasks (SystemWifiTurnOff, SystemWifiTurnOffVerify, SystemWifiTurnOn, SystemWifiTurnOnVerify) in an API 35 AVD environment. Both arms completed 4 episodes with 100% success rate, identical mean steps (4.75) and LLM calls (8.5). Treatment showed lower mean total tokens (50.6% of control) and slightly lower mean latency (98.0% of control). The decision was to promote the treatment as a full-suite candidate rerun, as it preserved success, passed latency/token guardrails, but remains a subset with environment limitations."
|
||||
},
|
||||
"model": {
|
||||
"base_url": "https://ark.cn-beijing.volces.com/api/v3",
|
||||
"max_tokens": 1024,
|
||||
"model": "doubao-seed-1-6-250615",
|
||||
"provider": "ark",
|
||||
"seed": 42,
|
||||
"temperature": 0
|
||||
},
|
||||
"paired_comparison": [
|
||||
{
|
||||
"control_latency_s": 105.382271,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 5,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOff:trial-1:seed-42",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOff",
|
||||
"treatment_latency_s": 103.549168,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 5,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"control_latency_s": 112.724126,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 5,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOffVerify:trial-1:seed-1051",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOffVerify",
|
||||
"treatment_latency_s": 102.209211,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 5,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"control_latency_s": 100.007401,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 5,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOn:trial-1:seed-2060",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOn",
|
||||
"treatment_latency_s": 106.717192,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 5,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
},
|
||||
{
|
||||
"control_latency_s": 86.679544,
|
||||
"control_reward": 1.0,
|
||||
"control_steps": 4,
|
||||
"control_success": true,
|
||||
"pair_id": "SystemWifiTurnOnVerify:trial-1:seed-3069",
|
||||
"reward_delta": 0.0,
|
||||
"success_delta": 0,
|
||||
"task": "SystemWifiTurnOnVerify",
|
||||
"treatment_latency_s": 84.26057,
|
||||
"treatment_reward": 1.0,
|
||||
"treatment_steps": 4,
|
||||
"treatment_success": true,
|
||||
"trial": 1
|
||||
}
|
||||
],
|
||||
"phase": {
|
||||
"description": "middle-layer input-pipeline cost refinement after the H5 success/cost result",
|
||||
"id": "phase_2_cost_refinement",
|
||||
"independent_variable": "raw versus semantic-filtered UIAutomator element list",
|
||||
"source_phase1_decision": {
|
||||
"completed_pairs": 4,
|
||||
"mean_latency_ratio_treatment_over_control": 0.672411,
|
||||
"net_success_delta": 0,
|
||||
"outcome": "inconclusive_no_success_gain",
|
||||
"paired_regressions": 0,
|
||||
"promote_to_full_suite_candidate": false,
|
||||
"reason": "Treatment produced no paired success gain; keep the upstream control prompt."
|
||||
},
|
||||
"source_phase1_evidence": "/Users/boj/book/ai-agent-book/chapter7/android-world/validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"source_phase1_run_id": "exp7-12-20260729T114648Z",
|
||||
"source_phase2_decision": {
|
||||
"completed_pairs": 4,
|
||||
"deployment_approved": false,
|
||||
"guardrails": {
|
||||
"maximum_latency_ratio": 1.5,
|
||||
"maximum_token_ratio": 1.5,
|
||||
"passed": false
|
||||
},
|
||||
"mean_latency_ratio_treatment_over_control": 0.788177,
|
||||
"mean_llm_call_ratio_treatment_over_control": 0.785714,
|
||||
"mean_token_ratio_treatment_over_control": 2.498496,
|
||||
"net_success_delta": 3,
|
||||
"outcome": "restrict_candidate_due_to_cost",
|
||||
"paired_regressions": 0,
|
||||
"promote_to_full_suite_candidate": false,
|
||||
"reason": "Treatment improved paired success without regressions, but exceeded the latency/token guardrails (1.50x / 1.50x). Restrict it to targeted follow-up; do not promote it to the full suite yet.",
|
||||
"scope_recommendation": "do_not_deploy"
|
||||
},
|
||||
"source_phase2_evidence": "/Users/boj/book/ai-agent-book/chapter7/android-world/validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"source_phase2_run_id": "exp7-12-20260729T122904Z"
|
||||
},
|
||||
"resume_commands": [
|
||||
[
|
||||
"run_controlled_experiment.py",
|
||||
"--mode",
|
||||
"paired",
|
||||
"--hypothesis",
|
||||
"H5C",
|
||||
"--source-phase1-evidence",
|
||||
"validation/paired_wifi_api35_20260729/evidence.json",
|
||||
"--source-phase2-evidence",
|
||||
"validation/paired_h5_a11y_api35_20260729/evidence.json",
|
||||
"--tasks",
|
||||
"SystemWifiTurnOff,SystemWifiTurnOffVerify,SystemWifiTurnOn,SystemWifiTurnOnVerify",
|
||||
"--trials",
|
||||
"1",
|
||||
"--seed",
|
||||
"42",
|
||||
"--model-seed",
|
||||
"42",
|
||||
"--max-steps",
|
||||
"10",
|
||||
"--transition-pause",
|
||||
"0.5",
|
||||
"--skip-device-time",
|
||||
"--output-dir",
|
||||
"validation/paired_h5c_compact_api35_20260729",
|
||||
"--resume"
|
||||
]
|
||||
],
|
||||
"run_id": "exp7-12-20260729T131911Z",
|
||||
"schema_version": 1,
|
||||
"scope": {
|
||||
"base_pair_seed": 42,
|
||||
"completed_episodes": 8,
|
||||
"direct_episode_gate_completed": false,
|
||||
"error_episodes": 0,
|
||||
"full_suite_completed": false,
|
||||
"manuscript_five_seed_gate_completed": false,
|
||||
"max_steps": 10,
|
||||
"mode": "paired",
|
||||
"tasks": [
|
||||
"SystemWifiTurnOff",
|
||||
"SystemWifiTurnOffVerify",
|
||||
"SystemWifiTurnOn",
|
||||
"SystemWifiTurnOnVerify"
|
||||
],
|
||||
"transition_pause_s": 0.5,
|
||||
"trials_per_task": 1
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260729T131911Z`
|
||||
- Generated (UTC): `2026-07-29T14:12:27Z`
|
||||
- Upstream commit: `0e95d641e244504c22087cc29b013f3b2428a261`
|
||||
- Device: `sdk_gphone64_arm64`, API `35` (upstream tested reference: API `33`)
|
||||
- Observation method: `varies_by_arm:uiautomator_vs_uiautomator_compact`
|
||||
- Provider/model: `ark` / `doubao-seed-1-6-250615`
|
||||
- Scope: 4 task(s), 1 trial(s), mode `paired`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
The diagnosis produced explicit surface, middle, and deep hypotheses. Only one variable is changed in this run; the other hypotheses remain untested.
|
||||
|
||||
| Layer / ID | Proposed change | Target | Verification | Status |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| surface / `H1` | Add Wi-Fi Settings navigation and final-state verification guidance. | At least one net paired success across the four Wi-Fi tasks, with no regression. | Paired upstream-prompt versus task-guideline ablation with matched seeds. | tested in source phase 1 |
|
||||
| surface / `H2` | Add application-specific recognition rules for the non-standard Tasks UI. | Improve at least two of the six historical Tasks failures with no regression. | Paired Tasks-only prompt/tool-description ablation after app provisioning. | not tested |
|
||||
| middle / `H3` | Repair and validate the multimodal input path for transcription tasks. | Raise transcription success above the historical 0% while bounding added tokens and latency. | Paired screenshot-disabled versus screenshot-enabled transcription run. | not tested |
|
||||
| middle / `H4` | Conditionally enable deeper thinking for counting tasks. | Improve math/counting success without applying the cost to unrelated tasks. | Paired tag-routed thinking-mode ablation with latency and token guardrails. | not tested |
|
||||
| middle / `H5` | Replace the API-35-incompatible gRPC accessibility feed with upstream's UIAutomator observation path. | At least one net paired Wi-Fi success with no regression and at most 1.5x latency/tokens. | Paired a11y-forwarder versus UIAutomator run with the same upstream T3A prompt and matched seeds. | tested in source phase 2 |
|
||||
| middle / `H5C` | Filter non-semantic UIAutomator container nodes after H5 exposed excessive prompt-token cost. | Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency. | Paired raw-UIAutomator versus compact-UIAutomator run with matched tasks, seeds, prompt, and evaluator. | tested in this run |
|
||||
| deep / `H6` | Combine screenshots with the structured UI tree and compare stronger vision-capable models. | Improve complex-UI success enough to justify multimodal latency and token cost. | Factorial UI-tree/screenshot/model ablation on the full tagged slice. | not tested |
|
||||
|
||||
Selected hypothesis: `H5C`
|
||||
- Change: Use the real upstream UIAutomator hierarchy but retain only visible text, descriptions, and actionable/scrollable elements.
|
||||
- Expected measurable result: Preserve H5 paired success with no regression while using at most 0.75x raw-UIAutomator tokens and 1.5x latency.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; require at least four completed pairs, every compact-UIAutomator treatment pair successful, no paired regression, at most 1.5x mean latency, and at most 0.75x raw-UIAutomator mean tokens. Passing a paired gate permits only a full-suite candidate rerun; it is not deployment approval, and a subset must never be reported as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
- Phase: `phase_2_cost_refinement` — middle-layer input-pipeline cost refinement after the H5 success/cost result
|
||||
- Independent variable: raw versus semantic-filtered UIAutomator element list
|
||||
- Controls: same checkout, model, task parameters, generated seed, step budget, and emulator; arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Mean tokens | Input / output tokens |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| control | 4/4 | 1.000 | 1.000 | 4.750 | 101.198 | 8.500 | 139439.500 | 549928 / 7830 |
|
||||
| treatment | 4/4 | 1.000 | 1.000 | 4.750 | 99.184 | 8.500 | 70557.500 | 274067 / 8163 |
|
||||
|
||||
| Task / trial | Control | Treatment | Δ success | Control→treatment steps |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| SystemWifiTurnOff / 1 | 1 | 1 | +0 | 5→5 |
|
||||
| SystemWifiTurnOffVerify / 1 | 1 | 1 | +0 | 5→5 |
|
||||
| SystemWifiTurnOn / 1 | 1 | 1 | +0 | 5→5 |
|
||||
| SystemWifiTurnOnVerify / 1 | 1 | 1 | +0 | 4→4 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`promote_efficient_candidate_to_full_suite_rerun`**
|
||||
- Reason: Treatment preserved paired success with no regression and passed the latency/token efficiency guardrails. This is a candidate decision only.
|
||||
- Treatment/control mean latency ratio: 0.980
|
||||
- Treatment/control mean token ratio: 0.506
|
||||
- Treatment/control mean LLM-call ratio: 1.000
|
||||
- Cost guardrails passed: **true**
|
||||
- Deployment approved: **false**
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
### LLM analysis of this run
|
||||
|
||||
The following bounded interpretation was produced by the configured real LLM from the aggregate evidence (the JSON remains authoritative):
|
||||
|
||||
- Summary: A paired experiment comparing raw UIAutomator (control) and compact UIAutomator (treatment) on 4 system Settings tasks (SystemWifiTurnOff, SystemWifiTurnOffVerify, SystemWifiTurnOn, SystemWifiTurnOnVerify) in an API 35 AVD environment. Both arms completed 4 episodes with 100% success rate, identical mean steps (4.75) and LLM calls (8.5). Treatment showed lower mean total tokens (50.6% of control) and slightly lower mean latency (98.0% of control). The decision was to promote the treatment as a full-suite candidate rerun, as it preserved success, passed latency/token guardrails, but remains a subset with environment limitations.
|
||||
- Cost/benefit interpretation: The treatment provides significant token efficiency (mean token ratio 0.506) with preserved success and marginal latency improvement (mean latency ratio 0.980) in the tested 4-task subset. However, interpretation is bounded by the API/app environment: results are from an API 35 AVD (not upstream API 33 reference), restricted to system Settings tasks (no full third-party app bundle), and use UIAutomator as a compatibility path (not upstream reference configuration), limiting generalizability beyond the tested scope.
|
||||
- Next hypothesis `H5C-full` (middle): Evaluate compact UIAutomator (semantic-filtered elements) across the full AndroidWorld task suite to verify token efficiency and success preservation beyond the 4-task Settings subset. Target: Full task suite (all 116 tasks) with upstream reference environment (Pixel 6 / API 33) and provisioned full third-party AndroidWorld app bundle. Verification: Paired run comparing compact UIAutomator (treatment) vs raw UIAutomator (control) across all tasks, ensuring guardrails (mean token ratio ≤0.75, mean latency ratio ≤1.5, success non-inferiority) hold in the reference environment.
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.
|
||||
- The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.
|
||||
- UIAutomator is an upstream AndroidWorld observation option selected by the companion runner; it preserves real UI actions/evaluators but is a compatibility path, not the upstream API-33 reference configuration.
|
||||
- Compact UIAutomator removes only non-semantic container nodes; observations, coordinates, Android actions, and AndroidWorld evaluators remain real.
|
||||
- Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,67 @@
|
||||
# Experiment 7-12 AndroidWorld iteration report
|
||||
|
||||
- Run ID: `exp7-12-20260729T114648Z`
|
||||
- Generated (UTC): `2026-07-29T12:13:15Z`
|
||||
- Upstream commit: `0e95d641e244504c22087cc29b013f3b2428a261`
|
||||
- Device: `sdk_gphone64_arm64`, API `35` (upstream tested reference: API `33`)
|
||||
- Provider/model: `ark` / `doubao-seed-1-6-250615`
|
||||
- Scope: 4 task(s), 1 trial(s), mode `paired`
|
||||
- Full 116-task × 5-seed suite completed: **false**
|
||||
|
||||
The bundled ~88% baseline is historical input evidence. The manuscript's 88%→94% numbers are explicitly hypothetical and are not used as rerun results here.
|
||||
|
||||
## 1. Diagnose
|
||||
|
||||
- The historical run evaluated 116 tasks once each and reports approximately 88% overall success.
|
||||
- Wi-Fi is a concentrated failure cluster: three of the four SystemWifiTurn* rows failed in the bundled per-task table.
|
||||
- The capability matrix links the cluster to weak complex_ui_understanding, information_retrieval, and requires_setup behavior.
|
||||
- The failed traces show navigation/state-verification loops; increasing the step cap alone would treat a symptom rather than the cause.
|
||||
|
||||
## 2. Hypothesis
|
||||
|
||||
- ID: `H1`
|
||||
- Change: Use upstream T3A.set_task_guidelines to add only Wi-Fi Settings navigation and final-state verification guidance.
|
||||
- Expected measurable result: At least one net paired Wi-Fi success, with no paired regression; record reward, steps, latency, calls, and tokens.
|
||||
- Guardrails: Same model, seed, task parameters, emulator, checkout, and step budget; do not treat a subset gain as full-suite success.
|
||||
|
||||
## 3. Controlled experiment
|
||||
|
||||
Control and treatment use the same checkout, model, task parameters, step budget, and emulator. Only the task-specific T3A guidelines differ. Arm order alternates by pair.
|
||||
|
||||
| Arm | Episodes | Success | Reward | Steps | Latency (s) | LLM calls | Input / output tokens |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| control | 4/4 | 0.250 | 0.500 | 8.500 | 233.465 | 15.750 | 411525 / 31094 |
|
||||
| treatment | 4/4 | 0.250 | 0.500 | 8.000 | 156.985 | 12.500 | 190519 / 19520 |
|
||||
|
||||
| Task / trial | Control | Treatment | Δ success | Control→treatment steps |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| SystemWifiTurnOff / 1 | 0 | 0 | +0 | 10→10 |
|
||||
| SystemWifiTurnOffVerify / 1 | 0 | 0 | +0 | 10→10 |
|
||||
| SystemWifiTurnOn / 1 | 0 | 0 | +0 | 10→10 |
|
||||
| SystemWifiTurnOnVerify / 1 | 1 | 1 | +0 | 4→2 |
|
||||
|
||||
## 4. Data-driven decision
|
||||
|
||||
- Outcome: **`inconclusive_no_success_gain`**
|
||||
- Reason: Treatment produced no paired success gain; keep the upstream control prompt.
|
||||
- Treatment/control mean latency ratio: 0.672
|
||||
|
||||
## 5. Rerun and next report
|
||||
|
||||
This run is a real controlled subset/smoke rerun, not the complete AndroidWorld benchmark. The next gate is a conditionally enabled candidate rerun over all 116 tasks with five seeds after provisioning the upstream API-33 app environment.
|
||||
|
||||
Observed residual failures:
|
||||
- `control / SystemWifiTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `treatment / SystemWifiTurnOff / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `treatment / SystemWifiTurnOffVerify / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `control / SystemWifiTurnOffVerify / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `control / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
- `treatment / SystemWifiTurnOn / trial 1`: evaluator reward / completion gate was not satisfied
|
||||
|
||||
## Environment boundaries
|
||||
|
||||
- The available AVD is API 35, while upstream is tested on Pixel 6 / API 33. Results are real but not reference-environment comparable.
|
||||
- The full third-party AndroidWorld app bundle was not provisioned. This run is restricted to system Settings tasks.
|
||||
- Per-task device-time setting was skipped because the non-root API-35 AVD rejects `adb shell date`; Wi-Fi evaluators do not depend on time.
|
||||
|
||||
The JSON beside this report is the authoritative evidence. It contains episode-level evaluator rewards, actions, timing, token counts, configuration, and explicit completion gates; credentials and raw prompts are not stored.
|
||||
Reference in New Issue
Block a user