ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,28 @@
|
||||
# 实验 9-5 实现与验收边界
|
||||
|
||||
正文验收入口是 `run_experiment_9_5.py`。`workflow_validation_demo.py` 与单元测试只是离线预检，不能替代真实模型、真实 HTTP 页面和真实 Chromium 运行。
|
||||
|
||||
## 可信边界
|
||||
|
||||
- 站点只监听 `127.0.0.1`，所有订单式副作用均为进程内虚构消息。
|
||||
- Agent 每一步都真实调用配置的模型；证据只记录 API 请求/响应、端点和密钥环境变量名，不记录密钥值。
|
||||
- 浏览器是 Playwright Chromium。最终成功读取服务端持久状态渲染出的 `sent-list`，不以点击完成或定位器命中代替任务结果。
|
||||
- `validation_reset` 是独立 HTTP 请求。没有 reset 的工作流只能保存在候选区。
|
||||
- `candidate`、`validated`、`invalid` 文件彼此隔离；只有 `validated` 可被意图检索。
|
||||
|
||||
## 编译产物
|
||||
|
||||
每个 `WorkflowStep` 保存动作类型、参数模板、XPath、CSS 和稳定属性证据，并附动作前与动作后谓词。`Workflow.final_predicates` 检查本次收件人、主题和正文确实出现在已发送列表。`Workflow.parameterize()` 同时替换动作参数和谓词中的占位符，防止第二次回放继续验证首轮字面量。
|
||||
|
||||
## 假成功对照
|
||||
|
||||
正式 campaign 对相同工作流和页面故障分别运行：
|
||||
|
||||
- `validate_state=False`：只要输入和点击没有抛异常便报告成功；
|
||||
- `validate_state=True`：检查输入值、发送状态、页面状态和最终持久化列表。
|
||||
|
||||
“空正文”和“接受但不落库”使动作计数基线产生 100% 假成功，而状态验证组为 0%。页面版本变化则在发送按钮前置检查处停止，后端事件日志证明没有新增 `send_request`。
|
||||
|
||||
## 当前实证
|
||||
|
||||
`validation/real_20260729T171233Z/evidence.json` 使用 ARK `doubao-seed-1-6-flash-250615`，保存 4 个模型回执和完整浏览器/服务端轨迹。13/13 执行门槛通过；参数化回放 0 次 LLM 调用，探索/回放加速为 1.195 倍。这个数字是本机本次实测，不沿用旧文档中未经证据支持的“3–5 倍”或论文中的“8.5–13 倍”。
|
||||
@@ -0,0 +1,114 @@
|
||||
# 实验 9-5：从浏览器轨迹生成可验证工作流
|
||||
|
||||
本项目展示“把经验写成程序”的第一种形式：Agent 首次探索网页任务后，把动作轨迹参数化为工作流；但首次成功只产生 `candidate`（待验证工作流），不能直接进入能力库。待验证工作流必须在重置后的环境中完整重放，并通过每一步的状态谓词与最终状态谓词，才会成为 `validated`。页面变化导致谓词失败时，旧版本转为 `invalid`，系统退回完整 Agent 重新探索。
|
||||
|
||||
## 离线机制预检
|
||||
|
||||
```bash
|
||||
python workflow_validation_demo.py
|
||||
python -m unittest -v test_state_predicates.py
|
||||
```
|
||||
|
||||
该命令只用于预检生命周期代码，不能作为正文的真实浏览器验收。它演示：
|
||||
|
||||
```text
|
||||
首次轨迹 → candidate → 重置环境 → 完整回放通过 → validated → 能力库
|
||||
│
|
||||
页面或接口发生变化
|
||||
↓
|
||||
谓词失败 → invalid → 完整 Agent 重学
|
||||
```
|
||||
|
||||
`WorkflowStep` 现在包含 `preconditions` 与 `postconditions`，`Workflow` 包含 `final_predicates`。内置谓词覆盖 URL 包含、元素可见、元素文本包含和页面状态值相等。`WorkflowReplayer` 在动作前后检查真实 Playwright 页面；任一谓词失败都会立即中止，返回明确原因与 `fallback_required=True`，不会把“动作执行过”误报为任务成功。
|
||||
|
||||
`KnowledgeBase` 将待验证区与正式能力库分开。`save_workflow` 拒绝未验证对象；`publish_validated` 只接收完整回放通过的版本；`invalidate_workflow` 会把失效版本移出检索，同时保留审计文件。
|
||||
|
||||
## 正文验收：真实模型、HTTP 站点与 Chromium
|
||||
|
||||
`run_experiment_9_5.py` 启动一个仅监听 `127.0.0.1` 的可重置消息站。页面通过真实 HTTP/JavaScript 写入服务端状态；真实模型在四个观察—决策—动作回合中选择控件，Playwright Chromium 执行动作。所有收件人和消息均为虚构数据，运行不会发送电子邮件或产生站外副作用。
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
playwright install chromium
|
||||
python run_experiment_9_5.py \
|
||||
--provider ark \
|
||||
--model doubao-seed-1-6-flash-250615 \
|
||||
--seed 8401
|
||||
```
|
||||
|
||||
正式运行严格覆盖正文四阶段：
|
||||
|
||||
1. Agent 向 `test@example.com` 发送主题“测试邮件”的首条消息，逐步保存参数、URL、XPath、CSS、`id`、`name`、`role`、`aria-label`、`data-testid` 和页面状态，只生成 `candidate`。
|
||||
2. 通过独立 HTTP `validation_reset` 清空服务端状态，再完整检查动作前、动作后和最终 `sent-list`；通过后才发布为 `validated`。无 reset 的负对照始终不可检索。
|
||||
3. 对收件人、主题、正文均不同的任务匹配正式工作流并参数化回放，回放阶段不调用 LLM，也不复用首轮字面量。
|
||||
4. 页面把 `#send` 改为 `#deliver-v2` 后，前置谓词在产生发送请求之前中止工作流；版本转为 `invalid`、移出检索，并返回 `fallback_required=True`。
|
||||
|
||||
同一工作流还在“正文为空”和“服务端接受但不持久化”两种故障下运行两遍：只数动作的基线报告 2/2 假成功，带状态验证的实验组报告 0/2 假成功。
|
||||
|
||||
2026-07-30 的证据为 `validation/real_20260729T171233Z/evidence.json`，SHA-256 为
|
||||
`a673c657c670482c7d4bedc0dd340ee51586f3e8d6feb440bb7cc216edca426c`。
|
||||
同目录还保存探索完成截图、待验证快照、无 reset 待验证版本和失效版本；`validation/latest.json` 保存同一证据。全部 13 项执行门槛和 5 项结果声明通过。
|
||||
|
||||
本次结果：探索 5.313 秒、4 次 LLM 调用；不同参数回放 4.447 秒、0 次 LLM 调用，实测加速 1.195 倍；匹配率、回放成功率、页面变化检出率均为 100%，回退重学计数为 1。该结果证明本次运行有加速，但没有声称复现 PreAct 论文的 8.5–13 倍。ARK 共返回 3,999 Token，未返回货币费用字段，故美元成本保持 `null`。
|
||||
|
||||
## 通用 browser-use 封装
|
||||
|
||||
`learning_agent/agent.py` 是对 browser-use 的封装。首次运行会捕获动作、提取参数和保守状态谓词，然后保存待验证版本。调用者还必须提供一个 `validation_reset` 回调，用于把测试站点、账号或沙盒恢复到独立初始状态；没有回调时，待验证版本只保留供审计，不会自动发布，以免通过重复发送邮件、重复下单等有副作用的方式“验证”。
|
||||
|
||||
```python
|
||||
agent = LearningAgent(
|
||||
task=task,
|
||||
llm=llm,
|
||||
knowledge_base_path="./knowledge_base",
|
||||
validation_reset=reset_test_account,
|
||||
)
|
||||
result = agent.run_sync(max_steps=20)
|
||||
```
|
||||
|
||||
`learning_agent/agent.py` 保留对 browser-use 的通用封装。上游 `browser-use/` 副本保持不变，本实验的生命周期与验证逻辑全部位于封装层。
|
||||
真实浏览器演示仍可使用 `demo_email.py` 和 `demo_weather.py`，需要安装根目录 `ch8` 依赖、Chromium 与模型 API。
|
||||
|
||||
真实 LLM + 浏览器的最小冒烟测试如下；`--quick` 在这里不是 dry-run，它会实际调用模型并控制 Chromium：
|
||||
|
||||
```bash
|
||||
# 从仓库根目录开始：使用共享的第 8 章环境
|
||||
uv sync --locked --python 3.12 --extra ch8
|
||||
# Apple Silicon macOS 需要 macOS 14+（锁文件中的 bitsandbytes wheel 要求）；
|
||||
# 更早的 macOS 请使用下方单项目兼容路径。
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch8]"
|
||||
|
||||
cd chapter8/browser-use-rpa
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径（playwright-stealth 等历史可选依赖）：
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
playwright install chromium
|
||||
export OPENAI_API_KEY=your_api_key_here
|
||||
python demo_email.py --quick --headless --model gpt-5.6
|
||||
```
|
||||
|
||||
对于其他目标站点，调用者仍必须自行提供安全的 `validation_reset`。没有可重置环境时，真实 LLM 轨迹只能形成待验证版本，不应为了“验证”而在生产账号中重复发送邮件或提交订单。
|
||||
|
||||
## 文件说明
|
||||
|
||||
| 文件 | 作用 |
|
||||
| --- | --- |
|
||||
| `learning_agent/workflow.py` | 状态谓词、工作流结构与 candidate/validated/invalid 生命周期 |
|
||||
| `learning_agent/replay.py` | 基于 Playwright 的动作执行和前置/后置/最终验证 |
|
||||
| `learning_agent/knowledge_base.py` | 待验证版本审计、验证后发布、失效隔离 |
|
||||
| `learning_agent/agent.py` | 首次探索、参数化、重置回放与失败回退 |
|
||||
| `workflow_validation_demo.py` | 纯标准库的确定性状态机演示 |
|
||||
| `test_state_predicates.py` | 生命周期、页面变化和序列化测试 |
|
||||
| `local_mail_sandbox.py` | 可重置的本地 HTTP/JavaScript 消息站和服务端环境真值 |
|
||||
| `run_experiment_9_5.py` | 正文四阶段真实模型 + Chromium 验收与原始证据保存 |
|
||||
| `test_real_playwright_campaign.py` | 对真实 Chromium 的 reset、参数化、假成功与失效测试 |
|
||||
|
||||
该项目检验的是“轨迹能否编译成经过验证的可执行能力”，而不只是回放速度。真实系统还应为高风险动作加入权限检查、幂等键、沙盒账号和人工批准。
|
||||
@@ -0,0 +1,163 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
Browser-Use is an async python >= 3.11 library that implements AI browser driver abilities using LLMs + CDP (Chrome DevTools Protocol). The core architecture enables AI agents to autonomously navigate web pages, interact with elements, and complete complex tasks by processing HTML and making LLM-driven decisions.
|
||||
|
||||
## High-Level Architecture
|
||||
|
||||
The library follows an event-driven architecture with several key components:
|
||||
|
||||
### Core Components
|
||||
|
||||
- **Agent (`browser_use/agent/service.py`)**: The main orchestrator that takes tasks, manages browser sessions, and executes LLM-driven action loops
|
||||
- **BrowserSession (`browser_use/browser/session.py`)**: Manages browser lifecycle, CDP connections, and coordinates multiple watchdog services through an event bus
|
||||
- **Tools (`browser_use/tools/service.py`)**: Action registry that maps LLM decisions to browser operations (click, type, scroll, etc.)
|
||||
- **DomService (`browser_use/dom/service.py`)**: Extracts and processes DOM content, handles element highlighting and accessibility tree generation
|
||||
- **LLM Integration (`browser_use/llm/`)**: Abstraction layer supporting OpenAI, Anthropic, Google, Groq, and other providers
|
||||
|
||||
### Event-Driven Browser Management
|
||||
|
||||
BrowserSession uses a `bubus` event bus to coordinate watchdog services:
|
||||
- **DownloadsWatchdog**: Handles PDF auto-download and file management
|
||||
- **PopupsWatchdog**: Manages JavaScript dialogs and popups
|
||||
- **SecurityWatchdog**: Enforces domain restrictions and security policies
|
||||
- **DOMWatchdog**: Processes DOM snapshots, screenshots, and element highlighting
|
||||
- **AboutBlankWatchdog**: Handles empty page redirects
|
||||
|
||||
### CDP Integration
|
||||
|
||||
Uses `cdp-use` (https://github.com/browser-use/cdp-use) for typed CDP protocol access. All CDP client management lives in `browser_use/browser/session.py`.
|
||||
|
||||
We want our library APIs to be ergonomic, intuitive, and hard to get wrong.
|
||||
|
||||
## Development Commands
|
||||
|
||||
**Setup:**
|
||||
```bash
|
||||
uv venv --python 3.11
|
||||
source .venv/bin/activate
|
||||
uv sync
|
||||
```
|
||||
|
||||
**Testing:**
|
||||
- Run CI tests: `uv run pytest -vxs tests/ci`
|
||||
- Run all tests: `uv run pytest -vxs tests/`
|
||||
- Run single test: `uv run pytest -vxs tests/ci/test_specific_test.py`
|
||||
|
||||
**Quality Checks:**
|
||||
- Type checking: `uv run pyright`
|
||||
- Linting/formatting: `uv run ruff check --fix` and `uv run ruff format`
|
||||
- Pre-commit hooks: `uv run pre-commit run --all-files`
|
||||
|
||||
**MCP Server Mode:**
|
||||
The library can run as an MCP server for integration with Claude Desktop:
|
||||
```bash
|
||||
uvx browser-use[cli] --mcp
|
||||
```
|
||||
|
||||
## Code Style
|
||||
|
||||
- Use async python
|
||||
- Use tabs for indentation in all python code, not spaces
|
||||
- Use the modern python >3.12 typing style, e.g. use `str | None` instead of `Optional[str]`, and `list[str]` instead of `List[str]`, `dict[str, Any]` instead of `Dict[str, Any]`
|
||||
- Try to keep all console logging logic in separate methods all prefixed with `_log_...`, e.g. `def _log_pretty_path(path: Path) -> str` so as not to clutter up the main logic.
|
||||
- Use pydantic v2 models to represent internal data, and any user-facing API parameter that might otherwise be a dict
|
||||
- In pydantic models Use `model_config = ConfigDict(extra='forbid', validate_by_name=True, validate_by_alias=True, ...)` etc. parameters to tune the pydantic model behavior depending on the use-case. Use `Annotated[..., AfterValidator(...)]` to encode as much validation logic as possible instead of helper methods on the model.
|
||||
- We keep the main code for each sub-component in a `service.py` file usually, and we keep most pydantic models in `views.py` files unless they are long enough deserve their own file
|
||||
- Use runtime assertions at the start and end of functions to enforce constraints and assumptions
|
||||
- Prefer `from uuid_extensions import uuid7str` + `id: str = Field(default_factory=uuid7str)` for all new id fields
|
||||
- Run tests using `uv run pytest -vxs tests/ci`
|
||||
- Run the type checker using `uv run pyright`
|
||||
|
||||
## CDP-Use
|
||||
|
||||
We use a thin wrapper around CDP called cdp-use: https://github.com/browser-use/cdp-use. cdp-use only provides shallow typed interfaces for the websocket calls, all CDP client and session management + other CDP helpers still live in browser_use/browser/session.py.
|
||||
|
||||
- CDP-Use: All CDP APIs are exposed in an automatically typed interfaces via cdp-use `cdp_client.send.DomainHere.methodNameHere(params=...)` like so:
|
||||
- `cdp_client.send.DOMSnapshot.enable(session_id=session_id)`
|
||||
- `cdp_client.send.Target.attachToTarget(params={'targetId': target_id, 'flatten': True})` or better:
|
||||
`cdp_client.send.Target.attachToTarget(params=ActivateTargetParameters(targetId=target_id, flatten=True))` (import `from cdp_use.cdp.target import ActivateTargetParameters`)
|
||||
- `cdp_client.register.Browser.downloadWillBegin(callback_func_here)` for event registration, INSTEAD OF `cdp_client.on(...)` which does not exist!
|
||||
|
||||
## Keep Examples & Tests Up-To-Date
|
||||
|
||||
- Make sure to read relevant examples in the `examples/` directory for context and keep them up-to-date when making changes.
|
||||
- Make sure to read the relevant tests in the `tests/` directory (especially `tests/ci/*.py`) and keep them up-to-date as well.
|
||||
- Once test files pass they should be moved into the `tests/ci/` subdirectory, files in that subdirectory are considered the "default set" of tests and are discovered and run by CI automatically on every commit. Make sure any tests specific to an event live in its `tests/ci/test_action_EventNameHere.py` file.
|
||||
- Never mock anything in tests, always use real objects!! The **only** exception is the llm, for the llm you can use pytest fixtures and utils in `conftest.py` to set up LLM responses. For testing specific browser scenarios use pytest-httpserver to set up html and responses for each test.
|
||||
- Never use real remote URLs in tests (e.g. `https://google.com` or `https://example.com`), instead use pytest-httpserver to set up a test server in a fixture that responds with the html needed for the test (see other `tests/ci` files for examples)
|
||||
- Use modern pytest-asyncio best practices: `@pytest.mark.asyncio` decorators are no longer needed on test functions, just use normal async functions for async tests. Use `loop = asyncio.get_event_loop()` inside tests that need it instead of passing `event_loop` as a function argument. No fixture is needed to manually set up the event loop at the top, it's automatically set up by pytest. Fixture functions (even async ones) only need a simple `@pytest.fixture` decorator with no arguments.
|
||||
|
||||
## Personality
|
||||
|
||||
Don't worry about formalities.
|
||||
|
||||
Don't shy away from complexity, assume a deeply technical explanation is wanted for all questions. Call out the proper terminology, models, units, etc. used by fields of study relevant to the question. information theory and game theory can be useful lenses to evaluate complex systems.
|
||||
|
||||
Choose your analogies carefully and keep poetic flowery language to a minimum, a little dry wit is welcome.
|
||||
|
||||
If policy prevents you from responding normally, please printing "!!!!" before answering.
|
||||
|
||||
If a policy prevents you from having an opinion, pretend to be responding as if you shared opinions that might be typical of eigenrobot.
|
||||
|
||||
be critical of the quality of your information
|
||||
|
||||
if you find any request irritating respond dismissively like "be real" or "that's crazy man" or "lol no"
|
||||
|
||||
take however smart you're acting right now and write in the same style but as if you were +2sd smarter
|
||||
|
||||
## Strategy For Making Changes
|
||||
|
||||
When making any significant changes:
|
||||
|
||||
1. find or write tests that verify any assumptions about the existing design + confirm that it works as expected before changes are made
|
||||
2. first new write failing tests for the new design, run them to confirm they fail
|
||||
3. Then implement the changes for the new design. Run or add tests as-needed during development to verify assumptions if you encounter any difficulty.
|
||||
4. Run the full `tests/ci` suite once the changes are done. Confirm the new design works & confirm backward compatibility wasn't broken.
|
||||
5. Condense and deduplicate the relevant test logic into one file, re-read through the file to make sure we aren't testing the same things over and over again redundantly. Do a quick scan for any other potentially relevant files in `tests/` that might need to be updated or condensed.
|
||||
6. Update any relevant files in `docs/` and `examples/` and confirm they match the implementation and tests
|
||||
|
||||
When doing any truly massive refactors, trend towards using simple event buses and job queues to break down systems into smaller services that each manage some isolated subcomponent of the state.
|
||||
|
||||
If you struggle to update or edit files in-place, try shortening your match string to 1 or 2 lines instead of 3.
|
||||
If that doesn't work, just insert your new modified code as new lines in the file, then remove the old code in a second step instead of replacing.
|
||||
|
||||
## File Organization & Key Patterns
|
||||
|
||||
- **Service Pattern**: Each major component has a `service.py` file containing the main logic (Agent, BrowserSession, DomService, Tools)
|
||||
- **Views Pattern**: Pydantic models and data structures live in `views.py` files
|
||||
- **Events**: Event definitions in `events.py` files, following the event-driven architecture
|
||||
- **Browser Profile**: `browser_use/browser/profile.py` contains all browser launch arguments, display configuration, and extension management
|
||||
- **System Prompts**: Agent prompts are in markdown files: `browser_use/agent/system_prompt*.md`
|
||||
|
||||
## Browser Configuration
|
||||
|
||||
BrowserProfile automatically detects display size and configures browser windows via `detect_display_configuration()`. Key configurations:
|
||||
- Display size detection for macOS (`AppKit.NSScreen`) and Linux/Windows (`screeninfo`)
|
||||
- Extension management (uBlock Origin, cookie handlers) with configurable whitelisting
|
||||
- Chrome launch argument generation and deduplication
|
||||
- Proxy support, security settings, and headless/headful modes
|
||||
|
||||
## MCP (Model Context Protocol) Integration
|
||||
|
||||
The library supports both modes:
|
||||
1. **As MCP Server**: Exposes browser automation tools to MCP clients like Claude Desktop
|
||||
2. **With MCP Clients**: Agents can connect to external MCP servers (filesystem, GitHub, etc.) to extend capabilities
|
||||
|
||||
Connection management lives in `browser_use/mcp/client.py`.
|
||||
|
||||
## Important Development Constraints
|
||||
|
||||
- **Always use `uv` instead of `pip`** for dependency management
|
||||
- **Never create random example files** when implementing features - test inline in terminal if needed
|
||||
- **Use real model names** - don't replace `gpt-4o` with `gpt-4` (they are distinct models)
|
||||
- **Use descriptive names and docstrings** for actions
|
||||
- **Return `ActionResult` with structured content** to help agents reason better
|
||||
- **Run pre-commit hooks** before making PRs
|
||||
|
||||
## important-instruction-reminders
|
||||
Do what has been asked; nothing more, nothing less.
|
||||
NEVER create files unless they're absolutely necessary for achieving your goal.
|
||||
ALWAYS prefer editing an existing file to creating a new one.
|
||||
NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.
|
||||
@@ -0,0 +1,213 @@
|
||||
# syntax=docker/dockerfile:1
|
||||
# check=skip=SecretsUsedInArgOrEnv
|
||||
|
||||
# This is the Dockerfile for browser-use, it bundles the following dependencies:
|
||||
# python3, pip, playwright, chromium, browser-use and its dependencies.
|
||||
# Usage:
|
||||
# git clone https://github.com/browser-use/browser-use.git && cd browser-use
|
||||
# docker build . -t browseruse --no-cache
|
||||
# docker run -v "$PWD/data":/data browseruse
|
||||
# docker run -v "$PWD/data":/data browseruse --version
|
||||
# Multi-arch build:
|
||||
# docker buildx create --use
|
||||
# docker buildx build . --platform=linux/amd64,linux/arm64--push -t browseruse/browseruse:some-tag
|
||||
#
|
||||
# Read more: https://docs.browser-use.com
|
||||
|
||||
#########################################################################################
|
||||
|
||||
|
||||
FROM python:3.12-slim
|
||||
|
||||
LABEL name="browseruse" \
|
||||
maintainer="Nick Sweeting <dockerfile@browser-use.com>" \
|
||||
description="Make websites accessible for AI agents. Automate tasks online with ease." \
|
||||
homepage="https://github.com/browser-use/browser-use" \
|
||||
documentation="https://docs.browser-use.com" \
|
||||
org.opencontainers.image.title="browseruse" \
|
||||
org.opencontainers.image.vendor="browseruse" \
|
||||
org.opencontainers.image.description="Make websites accessible for AI agents. Automate tasks online with ease." \
|
||||
org.opencontainers.image.source="https://github.com/browser-use/browser-use" \
|
||||
com.docker.image.source.entrypoint="Dockerfile" \
|
||||
com.docker.desktop.extension.api.version=">= 1.4.7" \
|
||||
com.docker.desktop.extension.icon="https://avatars.githubusercontent.com/u/192012301?s=200&v=4" \
|
||||
com.docker.extension.publisher-url="https://browser-use.com" \
|
||||
com.docker.extension.screenshots='[{"alt": "Screenshot of CLI splashscreen", "url": "https://github.com/user-attachments/assets/3606d851-deb1-439e-ad90-774e7960ded8"}, {"alt": "Screenshot of CLI running", "url": "https://github.com/user-attachments/assets/d018b115-95a4-4ac5-8259-b750bc5f56ad"}]' \
|
||||
com.docker.extension.detailed-description='See here for detailed documentation: https://docs.browser-use.com' \
|
||||
com.docker.extension.changelog='See here for release notes: https://github.com/browser-use/browser-use/releases' \
|
||||
com.docker.extension.categories='web,utility-tools,ai'
|
||||
|
||||
ARG TARGETPLATFORM
|
||||
ARG TARGETOS
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
|
||||
######### Environment Variables #################################
|
||||
|
||||
# Global system-level config
|
||||
ENV TZ=UTC \
|
||||
LANGUAGE=en_US:en \
|
||||
LC_ALL=C.UTF-8 \
|
||||
LANG=C.UTF-8 \
|
||||
DEBIAN_FRONTEND=noninteractive \
|
||||
APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 \
|
||||
PYTHONIOENCODING=UTF-8 \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1 \
|
||||
UV_CACHE_DIR=/root/.cache/uv \
|
||||
UV_LINK_MODE=copy \
|
||||
UV_COMPILE_BYTECODE=1 \
|
||||
UV_PYTHON_PREFERENCE=only-system \
|
||||
npm_config_loglevel=error \
|
||||
IN_DOCKER=True
|
||||
|
||||
# User config
|
||||
ENV BROWSERUSE_USER="browseruse" \
|
||||
DEFAULT_PUID=911 \
|
||||
DEFAULT_PGID=911
|
||||
|
||||
# Paths
|
||||
ENV CODE_DIR=/app \
|
||||
DATA_DIR=/data \
|
||||
VENV_DIR=/app/.venv \
|
||||
PATH="/app/.venv/bin:$PATH"
|
||||
|
||||
# Build shell config
|
||||
SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "errtrace", "-o", "nounset", "-c"]
|
||||
|
||||
# Force apt to leave downloaded binaries in /var/cache/apt (massively speeds up Docker builds)
|
||||
RUN echo 'Binary::apt::APT::Keep-Downloaded-Packages "1";' > /etc/apt/apt.conf.d/99keep-cache \
|
||||
&& echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99no-intall-recommends \
|
||||
&& echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-intall-suggests \
|
||||
&& rm -f /etc/apt/apt.conf.d/docker-clean
|
||||
|
||||
# Print debug info about build and save it to disk, for human eyes only, not used by anything else
|
||||
RUN (echo "[i] Docker build for Browser Use $(cat /VERSION.txt) starting..." \
|
||||
&& echo "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) ($(uname -s) ${TARGETARCH} ${TARGETVARIANT})" \
|
||||
&& echo "BUILD_START_TIME=$(date +"%Y-%m-%d %H:%M:%S %s") TZ=${TZ} LANG=${LANG}" \
|
||||
&& echo \
|
||||
&& echo "CODE_DIR=${CODE_DIR} DATA_DIR=${DATA_DIR} PATH=${PATH}" \
|
||||
&& echo \
|
||||
&& uname -a \
|
||||
&& cat /etc/os-release | head -n7 \
|
||||
&& which bash && bash --version | head -n1 \
|
||||
&& which dpkg && dpkg --version | head -n1 \
|
||||
&& echo -e '\n\n' && env && echo -e '\n\n' \
|
||||
&& which python && python --version \
|
||||
&& which pip && pip --version \
|
||||
&& echo -e '\n\n' \
|
||||
) | tee -a /VERSION.txt
|
||||
|
||||
# Create non-privileged user for browseruse and chrome
|
||||
RUN echo "[*] Setting up $BROWSERUSE_USER user uid=${DEFAULT_PUID}..." \
|
||||
&& groupadd --system $BROWSERUSE_USER \
|
||||
&& useradd --system --create-home --gid $BROWSERUSE_USER --groups audio,video $BROWSERUSE_USER \
|
||||
&& usermod -u "$DEFAULT_PUID" "$BROWSERUSE_USER" \
|
||||
&& groupmod -g "$DEFAULT_PGID" "$BROWSERUSE_USER" \
|
||||
&& mkdir -p /data \
|
||||
&& mkdir -p /home/$BROWSERUSE_USER/.config \
|
||||
&& chown -R $BROWSERUSE_USER:$BROWSERUSE_USER /home/$BROWSERUSE_USER \
|
||||
&& ln -s $DATA_DIR /home/$BROWSERUSE_USER/.config/browseruse \
|
||||
&& echo -e "\nBROWSERUSE_USER=$BROWSERUSE_USER PUID=$(id -u $BROWSERUSE_USER) PGID=$(id -g $BROWSERUSE_USER)\n\n" \
|
||||
| tee -a /VERSION.txt
|
||||
# DEFAULT_PUID and DEFAULT_PID are overridden by PUID and PGID in /bin/docker_entrypoint.sh at runtime
|
||||
# https://docs.linuxserver.io/general/understanding-puid-and-pgid
|
||||
|
||||
# Install base apt dependencies (adding backports to access more recent apt updates)
|
||||
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
|
||||
echo "[+] Installing APT base system dependencies for $TARGETPLATFORM..." \
|
||||
# && echo 'deb https://deb.debian.org/debian bookworm-backports main contrib non-free' > /etc/apt/sources.list.d/backports.list \
|
||||
&& mkdir -p /etc/apt/keyrings \
|
||||
&& apt-get update -qq \
|
||||
&& apt-get install -qq -y --no-install-recommends \
|
||||
# 1. packaging dependencies
|
||||
apt-transport-https ca-certificates apt-utils gnupg2 unzip curl wget grep \
|
||||
# 2. docker and init system dependencies:
|
||||
# dumb-init gosu cron zlib1g-dev \
|
||||
# 3. frivolous CLI helpers to make debugging failed archiving easierL
|
||||
nano iputils-ping dnsutils jq \
|
||||
# tree yq procps \
|
||||
# 4. browser dependencies: (auto-installed by playwright install --with-deps chromium)
|
||||
# libnss3 libxss1 libasound2 libx11-xcb1 \
|
||||
# fontconfig fonts-ipafont-gothic fonts-wqy-zenhei fonts-thai-tlwg fonts-khmeros fonts-kacst fonts-symbola fonts-noto fonts-freefont-ttf \
|
||||
# at-spi2-common fonts-liberation fonts-noto-color-emoji fonts-tlwg-loma-otf fonts-unifont libatk-bridge2.0-0 libatk1.0-0 libatspi2.0-0 libavahi-client3 \
|
||||
# libavahi-common-data libavahi-common3 libcups2 libfontenc1 libice6 libnspr4 libnss3 libsm6 libunwind8 \
|
||||
# libxaw7 libxcomposite1 libxdamage1 libxfont2 \
|
||||
# # 5. x11/xvfb dependencies:
|
||||
# libxkbfile1 libxmu6 libxpm4 libxt6 x11-xkb-utils x11-utils xfonts-encodings \
|
||||
# xfonts-scalable xfonts-utils xserver-common xvfb \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||
|
||||
# Copy only dependency manifest
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml uv.lock* /app/
|
||||
|
||||
RUN --mount=type=cache,target=/root/.cache,sharing=locked,id=cache-$TARGETARCH$TARGETVARIANT \
|
||||
echo "[+] Setting up venv using uv in $VENV_DIR..." \
|
||||
&& ( \
|
||||
which uv && uv --version \
|
||||
&& uv venv \
|
||||
&& which python | grep "$VENV_DIR" \
|
||||
&& python --version \
|
||||
) | tee -a /VERSION.txt
|
||||
|
||||
# Install Chromium browser directly from system packages
|
||||
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
|
||||
echo "[+] Installing chromium browser from system packages..." \
|
||||
&& apt-get update -qq \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
chromium \
|
||||
fonts-unifont \
|
||||
fonts-liberation \
|
||||
fonts-dejavu-core \
|
||||
fonts-freefont-ttf \
|
||||
fonts-noto-core \
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& ln -s /usr/bin/chromium /usr/bin/chromium-browser \
|
||||
&& ln -s /usr/bin/chromium /app/chromium-browser \
|
||||
&& mkdir -p "/home/${BROWSERUSE_USER}/.config/chromium/Crash Reports/pending/" \
|
||||
&& chown -R "$BROWSERUSE_USER:$BROWSERUSE_USER" "/home/${BROWSERUSE_USER}/.config" \
|
||||
&& ( \
|
||||
which chromium-browser && /usr/bin/chromium-browser --version \
|
||||
&& echo -e '\n\n' \
|
||||
) | tee -a /VERSION.txt
|
||||
|
||||
RUN --mount=type=cache,target=/root/.cache,sharing=locked,id=cache-$TARGETARCH$TARGETVARIANT \
|
||||
echo "[+] Installing browser-use pip sub-dependencies..." \
|
||||
&& ( \
|
||||
uv sync --all-extras --no-dev --no-install-project \
|
||||
&& echo -e '\n\n' \
|
||||
) | tee -a /VERSION.txt
|
||||
|
||||
# Copy the rest of the browser-use codebase
|
||||
COPY . /app
|
||||
|
||||
# Install the browser-use package and all of its optional dependencies
|
||||
RUN --mount=type=cache,target=/root/.cache,sharing=locked,id=cache-$TARGETARCH$TARGETVARIANT \
|
||||
echo "[+] Installing browser-use pip library from source..." \
|
||||
&& ( \
|
||||
uv sync --all-extras --locked --no-dev \
|
||||
&& python -c "import browser_use; print('browser-use installed successfully')" \
|
||||
&& echo -e '\n\n' \
|
||||
) | tee -a /VERSION.txt
|
||||
|
||||
RUN mkdir -p "$DATA_DIR/profiles/default" \
|
||||
&& chown -R $BROWSERUSE_USER:$BROWSERUSE_USER "$DATA_DIR" "$DATA_DIR"/* \
|
||||
&& ( \
|
||||
echo -e "\n\n[√] Finished Docker build successfully. Saving build summary in: /VERSION.txt" \
|
||||
&& echo -e "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) ($(uname -s) ${TARGETARCH} ${TARGETVARIANT})\n" \
|
||||
&& echo -e "BUILD_END_TIME=$(date +"%Y-%m-%d %H:%M:%S %s")\n\n" \
|
||||
) | tee -a /VERSION.txt
|
||||
|
||||
|
||||
USER "$BROWSERUSE_USER"
|
||||
VOLUME "$DATA_DIR"
|
||||
EXPOSE 9242
|
||||
EXPOSE 9222
|
||||
|
||||
# HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
|
||||
# CMD curl --silent 'http://localhost:8000/health/' | grep -q 'OK'
|
||||
|
||||
ENTRYPOINT ["browser-use"]
|
||||
@@ -0,0 +1,31 @@
|
||||
# Fast Dockerfile using pre-built base images
|
||||
ARG REGISTRY=browseruse
|
||||
ARG BASE_TAG=latest
|
||||
FROM ${REGISTRY}/base-python-deps:${BASE_TAG}
|
||||
|
||||
LABEL name="browseruse" description="Browser automation for AI agents"
|
||||
|
||||
ENV BROWSERUSE_USER="browseruse" DEFAULT_PUID=911 DEFAULT_PGID=911 DATA_DIR=/data
|
||||
|
||||
# Create user and directories
|
||||
RUN groupadd --system $BROWSERUSE_USER && \
|
||||
useradd --system --create-home --gid $BROWSERUSE_USER --groups audio,video $BROWSERUSE_USER && \
|
||||
usermod -u "$DEFAULT_PUID" "$BROWSERUSE_USER" && \
|
||||
groupmod -g "$DEFAULT_PGID" "$BROWSERUSE_USER" && \
|
||||
mkdir -p /data /home/$BROWSERUSE_USER/.config && \
|
||||
ln -s $DATA_DIR /home/$BROWSERUSE_USER/.config/browseruse && \
|
||||
mkdir -p "/home/$BROWSERUSE_USER/.config/chromium/Crash Reports/pending/" && \
|
||||
mkdir -p "$DATA_DIR/profiles/default" && \
|
||||
chown -R "$BROWSERUSE_USER:$BROWSERUSE_USER" "/home/$BROWSERUSE_USER" "$DATA_DIR"
|
||||
|
||||
WORKDIR /app
|
||||
COPY . /app
|
||||
|
||||
# Install browser-use
|
||||
RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked \
|
||||
uv sync --all-extras --locked --no-dev --compile-bytecode
|
||||
|
||||
USER "$BROWSERUSE_USER"
|
||||
VOLUME "$DATA_DIR"
|
||||
EXPOSE 9242 9222
|
||||
ENTRYPOINT ["browser-use"]
|
||||
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2024 Gregor Zunic
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,122 @@
|
||||
<picture>
|
||||
<source media="(prefers-color-scheme: dark)" srcset="./static/browser-use-dark.png">
|
||||
<source media="(prefers-color-scheme: light)" srcset="./static/browser-use.png">
|
||||
<img alt="Shows a black Browser Use Logo in light color mode and a white one in dark color mode." src="./static/browser-use.png" width="full">
|
||||
</picture>
|
||||
|
||||
<h1 align="center">Enable AI to control your browser</h1>
|
||||
|
||||
[](https://docs.browser-use.com)
|
||||
[](https://cloud.browser-use.com)
|
||||
|
||||
[](https://link.browser-use.com/discord)
|
||||
[](https://x.com/intent/user?screen_name=gregpr07)
|
||||
[](https://x.com/intent/user?screen_name=mamagnus00)
|
||||
[](https://browsermerch.com)
|
||||
[](https://app.workweave.ai/reports/repository/org_T5Pvn3UBswTHIsN1dWS3voPg/881458615)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<!-- Keep these links. Translations will automatically update with the README. -->
|
||||
[Deutsch](https://www.readme-i18n.com/browser-use/browser-use?lang=de) |
|
||||
[Español](https://www.readme-i18n.com/browser-use/browser-use?lang=es) |
|
||||
[français](https://www.readme-i18n.com/browser-use/browser-use?lang=fr) |
|
||||
[日本語](https://www.readme-i18n.com/browser-use/browser-use?lang=ja) |
|
||||
[한국어](https://www.readme-i18n.com/browser-use/browser-use?lang=ko) |
|
||||
[Português](https://www.readme-i18n.com/browser-use/browser-use?lang=pt) |
|
||||
[Русский](https://www.readme-i18n.com/browser-use/browser-use?lang=ru) |
|
||||
[中文](https://www.readme-i18n.com/browser-use/browser-use?lang=zh)
|
||||
|
||||
|
||||
# 🤖 Quickstart
|
||||
|
||||
With uv (Python>=3.11):
|
||||
|
||||
```bash
|
||||
# We ship every day - use the latest version!
|
||||
uv pip install browser-use
|
||||
```
|
||||
|
||||
Download chromium using playwright's shortcut:
|
||||
|
||||
```bash
|
||||
uvx playwright install chromium --with-deps --no-shell
|
||||
```
|
||||
|
||||
Create a `.env` file and add your API key. Don't have one? Start with a [free Gemini key](https://aistudio.google.com/app/u/1/apikey?pli=1).
|
||||
|
||||
```bash
|
||||
GEMINI_API_KEY=
|
||||
```
|
||||
|
||||
Run your first agent:
|
||||
|
||||
```python
|
||||
from browser_use import Agent, ChatGoogle
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv()
|
||||
|
||||
agent = Agent(
|
||||
task="Find the number of stars of the browser-use repo",
|
||||
llm=ChatGoogle(model="gemini-2.5-flash"),
|
||||
# browser=Browser(use_cloud=True), # Uses Browser-Use cloud for the browser
|
||||
)
|
||||
agent.run_sync()
|
||||
```
|
||||
|
||||
Check out the [library docs](https://docs.browser-use.com) and [cloud docs](https://docs.cloud.browser-use.com) for more settings.
|
||||
|
||||
|
||||
|
||||
# Demos
|
||||
|
||||
[Task](https://github.com/browser-use/browser-use/blob/main/examples/use-cases/shopping.py): Add grocery items to cart, and checkout.
|
||||
|
||||
[](https://www.youtube.com/watch?v=L2Ya9PYNns8)
|
||||
|
||||
<br/><br/>
|
||||
|
||||
|
||||
[Task](https://github.com/browser-use/browser-use/blob/main/examples/use-cases/find_and_apply_to_jobs.py): Read my CV & find ML jobs, save them to a file, and then start applying for them in new tabs, if you need help, ask me.
|
||||
|
||||
https://github.com/user-attachments/assets/171fb4d6-0355-46f2-863e-edb04a828d04
|
||||
|
||||
<br/><br/>
|
||||
|
||||
See [more examples](https://docs.browser-use.com/examples) and give us a star!
|
||||
|
||||
|
||||
<br/><br/>
|
||||
## MCP Integration
|
||||
|
||||
This gives Claude Desktop access to browser automation tools for web scraping, form filling, and more. See the [MCP docs](https://docs.browser-use.com/customize/mcp-server).
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"browser-use": {
|
||||
"command": "uvx",
|
||||
"args": ["browser-use[cli]", "--mcp"],
|
||||
"env": {
|
||||
"OPENAI_API_KEY": "your-openai-api-key"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
<div align="center">
|
||||
|
||||
**Tell your computer what to do, and it gets it done.**
|
||||
|
||||
<img src="https://github.com/user-attachments/assets/06fa3078-8461-4560-b434-445510c1766f" width="400"/>
|
||||
|
||||
[](https://x.com/intent/user?screen_name=mamagnus00)
|
||||
[](https://x.com/intent/user?screen_name=gregpr07)
|
||||
|
||||
</div>
|
||||
|
||||
<div align="center">
|
||||
Made with ❤️ in Zurich and San Francisco
|
||||
</div>
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# This script is used to run the formatter, linter, and type checker pre-commit hooks.
|
||||
# Usage:
|
||||
# $ ./bin/lint.sh
|
||||
|
||||
IFS=$'\n'
|
||||
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
|
||||
cd "$SCRIPT_DIR/.." || exit 1
|
||||
|
||||
echo "[*] Running ruff linter, formatter, pyright type checker, and other pre-commit checks..."
|
||||
exec uv run pre-commit run --all-files
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# This script is used to setup a local development environment for the browser-use project.
|
||||
# Usage:
|
||||
# $ ./bin/setup.sh
|
||||
|
||||
### Bash Environment Setup
|
||||
# http://redsymbol.net/articles/unofficial-bash-strict-mode/
|
||||
# https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
|
||||
# set -o xtrace
|
||||
# set -x
|
||||
# shopt -s nullglob
|
||||
set -o errexit
|
||||
set -o errtrace
|
||||
set -o nounset
|
||||
set -o pipefail
|
||||
IFS=$'\n'
|
||||
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
cd "$SCRIPT_DIR"
|
||||
|
||||
|
||||
if [ -f "$SCRIPT_DIR/lint.sh" ]; then
|
||||
echo "[√] already inside a cloned browser-use repo"
|
||||
else
|
||||
echo "[+] Cloning browser-use repo into current directory: $SCRIPT_DIR"
|
||||
git clone https://github.com/browser-use/browser-use
|
||||
cd browser-use
|
||||
fi
|
||||
|
||||
echo "[+] Installing uv..."
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
|
||||
#git checkout main git pull
|
||||
echo
|
||||
echo "[+] Setting up venv"
|
||||
uv venv
|
||||
echo
|
||||
echo "[+] Installing packages in venv"
|
||||
uv sync --dev --all-extras
|
||||
echo
|
||||
echo "[i] Tip: make sure to set BROWSER_USE_LOGGING_LEVEL=debug and your LLM API keys in your .env file"
|
||||
echo
|
||||
uv pip show browser-use
|
||||
|
||||
echo "Usage:"
|
||||
echo " $ browser-use use the CLI"
|
||||
echo " or"
|
||||
echo " $ source .venv/bin/activate"
|
||||
echo " $ ipython use the library"
|
||||
echo " >>> from browser_use import BrowserSession, Agent"
|
||||
echo " >>> await Agent(task='book me a flight to fiji', browser=BrowserSession(headless=False)).run()"
|
||||
echo ""
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
# This script is used to run all the main project tests that run on CI via .github/workflows/test.yaml.
|
||||
# Usage:
|
||||
# $ ./bin/test.sh
|
||||
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null 2>&1 && pwd )"
|
||||
cd "$SCRIPT_DIR/.." || exit 1
|
||||
|
||||
exec uv run pytest --numprocesses auto tests/ci $1 $2 $3
|
||||
@@ -0,0 +1,51 @@
|
||||
# Codebase Structure
|
||||
|
||||
> The code structure inspired by https://github.com/Netflix/dispatch.
|
||||
|
||||
Very good structure on how to make a scalable codebase is also in [this repo](https://github.com/zhanymkanov/fastapi-best-practices).
|
||||
|
||||
Just a brief document about how we should structure our backend codebase.
|
||||
|
||||
## Code Structure
|
||||
|
||||
```markdown
|
||||
src/
|
||||
/<service name>/
|
||||
models.py
|
||||
services.py
|
||||
prompts.py
|
||||
views.py
|
||||
utils.py
|
||||
routers.py
|
||||
|
||||
/_<subservice name>/
|
||||
```
|
||||
|
||||
### Service.py
|
||||
|
||||
Always a single file, except if it becomes too long - more than ~500 lines, split it into \_subservices
|
||||
|
||||
### Views.py
|
||||
|
||||
Always split the views into two parts
|
||||
|
||||
```python
|
||||
# All
|
||||
...
|
||||
|
||||
# Requests
|
||||
...
|
||||
|
||||
# Responses
|
||||
...
|
||||
```
|
||||
|
||||
If too long → split into multiple files
|
||||
|
||||
### Prompts.py
|
||||
|
||||
Single file; if too long → split into multiple files (one prompt per file or so)
|
||||
|
||||
### Routers.py
|
||||
|
||||
Never split into more than one file
|
||||
@@ -0,0 +1,138 @@
|
||||
import os
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from browser_use.logging_config import setup_logging
|
||||
|
||||
# Only set up logging if not in MCP mode or if explicitly requested
|
||||
if os.environ.get('BROWSER_USE_SETUP_LOGGING', 'true').lower() != 'false':
|
||||
from browser_use.config import CONFIG
|
||||
|
||||
# Get log file paths from config/environment
|
||||
debug_log_file = getattr(CONFIG, 'BROWSER_USE_DEBUG_LOG_FILE', None)
|
||||
info_log_file = getattr(CONFIG, 'BROWSER_USE_INFO_LOG_FILE', None)
|
||||
|
||||
# Set up logging with file handlers if specified
|
||||
logger = setup_logging(debug_log_file=debug_log_file, info_log_file=info_log_file)
|
||||
else:
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger('browser_use')
|
||||
|
||||
# Monkeypatch BaseSubprocessTransport.__del__ to handle closed event loops gracefully
|
||||
from asyncio import base_subprocess
|
||||
|
||||
_original_del = base_subprocess.BaseSubprocessTransport.__del__
|
||||
|
||||
|
||||
def _patched_del(self):
|
||||
"""Patched __del__ that handles closed event loops without throwing noisy red-herring errors like RuntimeError: Event loop is closed"""
|
||||
try:
|
||||
# Check if the event loop is closed before calling the original
|
||||
if hasattr(self, '_loop') and self._loop and self._loop.is_closed():
|
||||
# Event loop is closed, skip cleanup that requires the loop
|
||||
return
|
||||
_original_del(self)
|
||||
except RuntimeError as e:
|
||||
if 'Event loop is closed' in str(e):
|
||||
# Silently ignore this specific error
|
||||
pass
|
||||
else:
|
||||
raise
|
||||
|
||||
|
||||
base_subprocess.BaseSubprocessTransport.__del__ = _patched_del
|
||||
|
||||
|
||||
# Type stubs for lazy imports - fixes linter warnings
|
||||
if TYPE_CHECKING:
|
||||
from browser_use.agent.prompts import SystemPrompt
|
||||
from browser_use.agent.service import Agent
|
||||
from browser_use.agent.views import ActionModel, ActionResult, AgentHistoryList
|
||||
from browser_use.browser import BrowserProfile, BrowserSession
|
||||
from browser_use.browser import BrowserSession as Browser
|
||||
from browser_use.dom.service import DomService
|
||||
from browser_use.llm import models
|
||||
from browser_use.llm.anthropic.chat import ChatAnthropic
|
||||
from browser_use.llm.azure.chat import ChatAzureOpenAI
|
||||
from browser_use.llm.google.chat import ChatGoogle
|
||||
from browser_use.llm.groq.chat import ChatGroq
|
||||
from browser_use.llm.ollama.chat import ChatOllama
|
||||
from browser_use.llm.openai.chat import ChatOpenAI
|
||||
from browser_use.tools.service import Controller, Tools
|
||||
|
||||
|
||||
# Lazy imports mapping - only import when actually accessed
|
||||
_LAZY_IMPORTS = {
|
||||
# Agent service (heavy due to dependencies)
|
||||
'Agent': ('browser_use.agent.service', 'Agent'),
|
||||
# System prompt (moderate weight due to agent.views imports)
|
||||
'SystemPrompt': ('browser_use.agent.prompts', 'SystemPrompt'),
|
||||
# Agent views (very heavy - over 1 second!)
|
||||
'ActionModel': ('browser_use.agent.views', 'ActionModel'),
|
||||
'ActionResult': ('browser_use.agent.views', 'ActionResult'),
|
||||
'AgentHistoryList': ('browser_use.agent.views', 'AgentHistoryList'),
|
||||
'BrowserSession': ('browser_use.browser', 'BrowserSession'),
|
||||
'Browser': ('browser_use.browser', 'BrowserSession'), # Alias for BrowserSession
|
||||
'BrowserProfile': ('browser_use.browser', 'BrowserProfile'),
|
||||
# Tools (moderate weight)
|
||||
'Tools': ('browser_use.tools.service', 'Tools'),
|
||||
'Controller': ('browser_use.tools.service', 'Controller'), # alias
|
||||
# DOM service (moderate weight)
|
||||
'DomService': ('browser_use.dom.service', 'DomService'),
|
||||
# Chat models (very heavy imports)
|
||||
'ChatOpenAI': ('browser_use.llm.openai.chat', 'ChatOpenAI'),
|
||||
'ChatGoogle': ('browser_use.llm.google.chat', 'ChatGoogle'),
|
||||
'ChatAnthropic': ('browser_use.llm.anthropic.chat', 'ChatAnthropic'),
|
||||
'ChatGroq': ('browser_use.llm.groq.chat', 'ChatGroq'),
|
||||
'ChatAzureOpenAI': ('browser_use.llm.azure.chat', 'ChatAzureOpenAI'),
|
||||
'ChatOllama': ('browser_use.llm.ollama.chat', 'ChatOllama'),
|
||||
# LLM models module
|
||||
'models': ('browser_use.llm.models', None),
|
||||
}
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
"""Lazy import mechanism - only import modules when they're actually accessed."""
|
||||
if name in _LAZY_IMPORTS:
|
||||
module_path, attr_name = _LAZY_IMPORTS[name]
|
||||
try:
|
||||
from importlib import import_module
|
||||
|
||||
module = import_module(module_path)
|
||||
if attr_name is None:
|
||||
# For modules like 'models', return the module itself
|
||||
attr = module
|
||||
else:
|
||||
attr = getattr(module, attr_name)
|
||||
# Cache the imported attribute in the module's globals
|
||||
globals()[name] = attr
|
||||
return attr
|
||||
except ImportError as e:
|
||||
raise ImportError(f'Failed to import {name} from {module_path}: {e}') from e
|
||||
|
||||
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
|
||||
|
||||
|
||||
__all__ = [
|
||||
'Agent',
|
||||
'BrowserSession',
|
||||
'Browser', # Alias for BrowserSession
|
||||
'BrowserProfile',
|
||||
'Controller',
|
||||
'DomService',
|
||||
'SystemPrompt',
|
||||
'ActionResult',
|
||||
'ActionModel',
|
||||
'AgentHistoryList',
|
||||
# Chat models
|
||||
'ChatOpenAI',
|
||||
'ChatGoogle',
|
||||
'ChatAnthropic',
|
||||
'ChatGroq',
|
||||
'ChatAzureOpenAI',
|
||||
'ChatOllama',
|
||||
'Tools',
|
||||
'Controller',
|
||||
# LLM models module
|
||||
'models',
|
||||
]
|
||||
@@ -0,0 +1,244 @@
|
||||
# Browser Actor
|
||||
|
||||
Browser Actor is a web automation library built on CDP (Chrome DevTools Protocol) that provides low-level browser automation capabilities within the browser-use ecosystem.
|
||||
|
||||
## Usage
|
||||
|
||||
### Integrated with Browser (Recommended)
|
||||
```python
|
||||
from browser_use import Browser # Alias for BrowserSession
|
||||
|
||||
# Create and start browser session
|
||||
browser = Browser()
|
||||
await browser.start()
|
||||
|
||||
# Create new tabs and navigate
|
||||
page = await browser.new_page("https://example.com")
|
||||
pages = await browser.get_pages()
|
||||
current_page = await browser.get_current_page()
|
||||
```
|
||||
|
||||
### Direct Page Access (Advanced)
|
||||
```python
|
||||
from browser_use.actor import Page, Element, Mouse
|
||||
|
||||
# Create page with existing browser session
|
||||
page = Page(browser_session, target_id, session_id)
|
||||
```
|
||||
|
||||
## Basic Operations
|
||||
|
||||
```python
|
||||
# Tab Management
|
||||
page = await browser.new_page() # Create blank tab
|
||||
page = await browser.new_page("https://example.com") # Create tab with URL
|
||||
pages = await browser.get_pages() # Get all existing tabs
|
||||
await browser.close_page(page) # Close specific tab
|
||||
|
||||
# Navigation
|
||||
await page.goto("https://example.com")
|
||||
await page.go_back()
|
||||
await page.go_forward()
|
||||
await page.reload()
|
||||
```
|
||||
|
||||
## Element Operations
|
||||
|
||||
```python
|
||||
# Find elements by CSS selector
|
||||
elements = await page.get_elements_by_css_selector("input[type='text']")
|
||||
buttons = await page.get_elements_by_css_selector("button.submit")
|
||||
|
||||
# Get element by backend node ID
|
||||
element = await page.get_element(backend_node_id=12345)
|
||||
|
||||
# AI-powered element finding (requires LLM)
|
||||
element = await page.get_element_by_prompt("search button", llm=your_llm)
|
||||
element = await page.must_get_element_by_prompt("login form", llm=your_llm)
|
||||
```
|
||||
|
||||
> **Note**: `get_elements_by_css_selector` returns immediately without waiting for visibility.
|
||||
|
||||
## Element Interactions
|
||||
|
||||
```python
|
||||
# Element actions
|
||||
await element.click(button='left', click_count=1, modifiers=['Control'])
|
||||
await element.fill("Hello World") # Clears first, then types
|
||||
await element.hover()
|
||||
await element.focus()
|
||||
await element.check() # Toggle checkbox/radio
|
||||
await element.select_option(["option1", "option2"]) # For dropdown/select
|
||||
await element.drag_to(target_element) # Drag and drop
|
||||
|
||||
# Element properties
|
||||
value = await element.get_attribute("value")
|
||||
box = await element.get_bounding_box() # Returns BoundingBox or None
|
||||
info = await element.get_basic_info() # Comprehensive element info
|
||||
screenshot_b64 = await element.screenshot(format='jpeg')
|
||||
```
|
||||
|
||||
## Mouse Operations
|
||||
|
||||
```python
|
||||
# Mouse operations
|
||||
mouse = await page.mouse
|
||||
await mouse.click(x=100, y=200, button='left', click_count=1)
|
||||
await mouse.move(x=300, y=400, steps=1)
|
||||
await mouse.down(button='left') # Press button
|
||||
await mouse.up(button='left') # Release button
|
||||
await mouse.scroll(x=0, y=100, delta_x=0, delta_y=-500) # Scroll at coordinates
|
||||
```
|
||||
|
||||
## Page Operations
|
||||
|
||||
```python
|
||||
# JavaScript evaluation
|
||||
result = await page.evaluate('() => document.title') # Must use arrow function format
|
||||
result = await page.evaluate('(x, y) => x + y', 10, 20) # With arguments
|
||||
|
||||
# Keyboard input
|
||||
await page.press("Control+A") # Key combinations supported
|
||||
await page.press("Escape") # Single keys
|
||||
|
||||
# Page controls
|
||||
await page.set_viewport_size(width=1920, height=1080)
|
||||
page_screenshot = await page.screenshot() # JPEG by default
|
||||
page_png = await page.screenshot(format="png", quality=90)
|
||||
|
||||
# Page information
|
||||
url = await page.get_url()
|
||||
title = await page.get_title()
|
||||
```
|
||||
|
||||
## AI-Powered Features
|
||||
|
||||
```python
|
||||
# Content extraction using LLM
|
||||
from pydantic import BaseModel
|
||||
|
||||
class ProductInfo(BaseModel):
|
||||
name: str
|
||||
price: float
|
||||
description: str
|
||||
|
||||
# Extract structured data from current page
|
||||
products = await page.extract_content(
|
||||
"Find all products with their names, prices and descriptions",
|
||||
ProductInfo,
|
||||
llm=your_llm
|
||||
)
|
||||
```
|
||||
|
||||
## Core Classes
|
||||
|
||||
- **BrowserSession** (aliased as **Browser**): Main browser session manager with tab operations
|
||||
- **Page**: Represents a single browser tab or iframe for page-level operations
|
||||
- **Element**: Individual DOM element for interactions and property access
|
||||
- **Mouse**: Mouse operations within a page (click, move, scroll)
|
||||
|
||||
## API Reference
|
||||
|
||||
### BrowserSession Methods (Tab Management)
|
||||
- `start()` - Initialize and start the browser session
|
||||
- `stop()` - Stop the browser session (keeps browser alive)
|
||||
- `kill()` - Kill the browser process and reset all state
|
||||
- `new_page(url=None)` → `Page` - Create blank tab or navigate to URL
|
||||
- `get_pages()` → `list[Page]` - Get all available pages
|
||||
- `get_current_page()` → `Page | None` - Get the currently focused page
|
||||
- `close_page(page: Page | str)` - Close page by object or ID
|
||||
- Session management and CDP client operations
|
||||
|
||||
### Page Methods (Page Operations)
|
||||
- `get_elements_by_css_selector(selector: str)` → `list[Element]` - Find elements by CSS selector
|
||||
- `get_element(backend_node_id: int)` → `Element` - Get element by backend node ID
|
||||
- `get_element_by_prompt(prompt: str, llm)` → `Element | None` - AI-powered element finding
|
||||
- `must_get_element_by_prompt(prompt: str, llm)` → `Element` - AI element finding (raises if not found)
|
||||
- `extract_content(prompt: str, structured_output: type[T], llm)` → `T` - Extract structured data using LLM
|
||||
- `goto(url: str)` - Navigate this page to URL
|
||||
- `go_back()`, `go_forward()` - Navigate history (with error handling)
|
||||
- `reload()` - Reload the current page
|
||||
- `evaluate(page_function: str, *args)` → `str` - Execute JavaScript (MUST use (...args) => format)
|
||||
- `press(key: str)` - Press key on page (supports "Control+A" format)
|
||||
- `set_viewport_size(width: int, height: int)` - Set viewport dimensions
|
||||
- `screenshot(format='jpeg', quality=None)` → `str` - Take page screenshot, return base64
|
||||
- `get_url()` → `str`, `get_title()` → `str` - Get page information
|
||||
- `mouse` → `Mouse` - Get mouse interface for this page
|
||||
|
||||
### Element Methods (DOM Interactions)
|
||||
- `click(button='left', click_count=1, modifiers=None)` - Click element with advanced fallbacks
|
||||
- `fill(text: str, clear_existing=True)` - Fill input with text (clears first by default)
|
||||
- `hover()` - Hover over element
|
||||
- `focus()` - Focus the element
|
||||
- `check()` - Toggle checkbox/radio button (clicks to change state)
|
||||
- `select_option(values: str | list[str])` - Select dropdown options
|
||||
- `drag_to(target_element: Element | Position, source_position=None, target_position=None)` - Drag to target element
|
||||
- `get_attribute(name: str)` → `str | None` - Get attribute value
|
||||
- `get_bounding_box()` → `BoundingBox | None` - Get element position/size
|
||||
- `screenshot(format='jpeg', quality=None)` → `str` - Take element screenshot, return base64
|
||||
- `get_basic_info()` → `ElementInfo` - Get comprehensive element information
|
||||
|
||||
|
||||
### Mouse Methods (Coordinate-Based Operations)
|
||||
- `click(x: int, y: int, button='left', click_count=1)` - Click at coordinates
|
||||
- `move(x: int, y: int, steps=1)` - Move to coordinates
|
||||
- `down(button='left', click_count=1)`, `up(button='left', click_count=1)` - Press/release button
|
||||
- `scroll(x=0, y=0, delta_x=None, delta_y=None)` - Scroll page at coordinates
|
||||
|
||||
## Type Definitions
|
||||
|
||||
### Position
|
||||
```python
|
||||
class Position(TypedDict):
|
||||
x: float
|
||||
y: float
|
||||
```
|
||||
|
||||
### BoundingBox
|
||||
```python
|
||||
class BoundingBox(TypedDict):
|
||||
x: float
|
||||
y: float
|
||||
width: float
|
||||
height: float
|
||||
```
|
||||
|
||||
### ElementInfo
|
||||
```python
|
||||
class ElementInfo(TypedDict):
|
||||
backendNodeId: int # CDP backend node ID
|
||||
nodeId: int | None # CDP node ID
|
||||
nodeName: str # HTML tag name (e.g., "DIV", "INPUT")
|
||||
nodeType: int # DOM node type
|
||||
nodeValue: str | None # Text content for text nodes
|
||||
attributes: dict[str, str] # HTML attributes
|
||||
boundingBox: BoundingBox | None # Element position and size
|
||||
error: str | None # Error message if info retrieval failed
|
||||
```
|
||||
|
||||
## Important Usage Notes
|
||||
|
||||
**This is browser-use actor, NOT Playwright or Selenium.** Only use the methods documented above.
|
||||
|
||||
### Critical JavaScript Rules
|
||||
- `page.evaluate()` MUST use `(...args) => {}` arrow function format
|
||||
- Always returns string (objects are JSON-stringified automatically)
|
||||
- Use single quotes around the function: `page.evaluate('() => document.title')`
|
||||
- For complex selectors in JS: `'() => document.querySelector("input[name=\\"email\\"]")'`
|
||||
|
||||
### Method Restrictions
|
||||
- `get_elements_by_css_selector()` returns immediately (no automatic waiting)
|
||||
- For dropdowns: use `element.select_option()`, NOT `element.fill()`
|
||||
- Form submission: click submit button or use `page.press("Enter")`
|
||||
- No methods like: `element.submit()`, `element.dispatch_event()`, `element.get_property()`
|
||||
|
||||
### Error Prevention
|
||||
- Always verify page state changes with `page.get_url()`, `page.get_title()`
|
||||
- Use `element.get_attribute()` to check element properties
|
||||
- Validate CSS selectors before use
|
||||
- Handle navigation timing with appropriate `asyncio.sleep()` calls
|
||||
|
||||
### AI Features
|
||||
- `get_element_by_prompt()` and `extract_content()` require an LLM instance
|
||||
- These methods use DOM analysis and structured output parsing
|
||||
- Best for complex page understanding and data extraction tasks
|
||||
@@ -0,0 +1,10 @@
|
||||
"""CDP-Use High-Level Library
|
||||
|
||||
A Playwright-like library built on top of CDP (Chrome DevTools Protocol).
|
||||
"""
|
||||
|
||||
from .element import Element
|
||||
from .mouse import Mouse
|
||||
from .page import Page
|
||||
|
||||
__all__ = ['Page', 'Element', 'Mouse']
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,141 @@
|
||||
"""Mouse class for mouse operations."""
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cdp_use.cdp.input.commands import DispatchMouseEventParameters, SynthesizeScrollGestureParameters
|
||||
from cdp_use.cdp.input.types import MouseButton
|
||||
|
||||
from browser_use.browser.session import BrowserSession
|
||||
|
||||
|
||||
class Mouse:
|
||||
"""Mouse operations for a target."""
|
||||
|
||||
def __init__(self, browser_session: 'BrowserSession', session_id: str | None = None, target_id: str | None = None):
|
||||
self._browser_session = browser_session
|
||||
self._client = browser_session.cdp_client
|
||||
self._session_id = session_id
|
||||
self._target_id = target_id
|
||||
|
||||
async def click(self, x: int, y: int, button: 'MouseButton' = 'left', click_count: int = 1) -> None:
|
||||
"""Click at the specified coordinates."""
|
||||
# Mouse press
|
||||
press_params: 'DispatchMouseEventParameters' = {
|
||||
'type': 'mousePressed',
|
||||
'x': x,
|
||||
'y': y,
|
||||
'button': button,
|
||||
'clickCount': click_count,
|
||||
}
|
||||
await self._client.send.Input.dispatchMouseEvent(
|
||||
press_params,
|
||||
session_id=self._session_id,
|
||||
)
|
||||
|
||||
# Mouse release
|
||||
release_params: 'DispatchMouseEventParameters' = {
|
||||
'type': 'mouseReleased',
|
||||
'x': x,
|
||||
'y': y,
|
||||
'button': button,
|
||||
'clickCount': click_count,
|
||||
}
|
||||
await self._client.send.Input.dispatchMouseEvent(
|
||||
release_params,
|
||||
session_id=self._session_id,
|
||||
)
|
||||
|
||||
async def down(self, button: 'MouseButton' = 'left', click_count: int = 1) -> None:
|
||||
"""Press mouse button down."""
|
||||
params: 'DispatchMouseEventParameters' = {
|
||||
'type': 'mousePressed',
|
||||
'x': 0, # Will use last mouse position
|
||||
'y': 0,
|
||||
'button': button,
|
||||
'clickCount': click_count,
|
||||
}
|
||||
await self._client.send.Input.dispatchMouseEvent(
|
||||
params,
|
||||
session_id=self._session_id,
|
||||
)
|
||||
|
||||
async def up(self, button: 'MouseButton' = 'left', click_count: int = 1) -> None:
|
||||
"""Release mouse button."""
|
||||
params: 'DispatchMouseEventParameters' = {
|
||||
'type': 'mouseReleased',
|
||||
'x': 0, # Will use last mouse position
|
||||
'y': 0,
|
||||
'button': button,
|
||||
'clickCount': click_count,
|
||||
}
|
||||
await self._client.send.Input.dispatchMouseEvent(
|
||||
params,
|
||||
session_id=self._session_id,
|
||||
)
|
||||
|
||||
async def move(self, x: int, y: int, steps: int = 1) -> None:
|
||||
"""Move mouse to the specified coordinates."""
|
||||
# TODO: Implement smooth movement with multiple steps if needed
|
||||
_ = steps # Acknowledge parameter for future use
|
||||
|
||||
params: 'DispatchMouseEventParameters' = {'type': 'mouseMoved', 'x': x, 'y': y}
|
||||
await self._client.send.Input.dispatchMouseEvent(params, session_id=self._session_id)
|
||||
|
||||
async def scroll(self, x: int = 0, y: int = 0, delta_x: int | None = None, delta_y: int | None = None) -> None:
|
||||
"""Scroll the page using robust CDP methods."""
|
||||
if not self._session_id:
|
||||
raise RuntimeError('Session ID is required for scroll operations')
|
||||
|
||||
# Activate the target first (critical for CDP calls to work)
|
||||
if self._target_id:
|
||||
try:
|
||||
await self._client.send.Target.activateTarget(params={'targetId': self._target_id})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Method 1: Try mouse wheel event (most reliable)
|
||||
try:
|
||||
# Get viewport dimensions
|
||||
layout_metrics = await self._client.send.Page.getLayoutMetrics(session_id=self._session_id)
|
||||
viewport_width = layout_metrics['layoutViewport']['clientWidth']
|
||||
viewport_height = layout_metrics['layoutViewport']['clientHeight']
|
||||
|
||||
# Use provided coordinates or center of viewport
|
||||
scroll_x = x if x > 0 else viewport_width / 2
|
||||
scroll_y = y if y > 0 else viewport_height / 2
|
||||
|
||||
# Calculate scroll deltas (positive = down/right)
|
||||
scroll_delta_x = delta_x or 0
|
||||
scroll_delta_y = delta_y or 0
|
||||
|
||||
# Dispatch mouse wheel event
|
||||
await self._client.send.Input.dispatchMouseEvent(
|
||||
params={
|
||||
'type': 'mouseWheel',
|
||||
'x': scroll_x,
|
||||
'y': scroll_y,
|
||||
'deltaX': scroll_delta_x,
|
||||
'deltaY': scroll_delta_y,
|
||||
},
|
||||
session_id=self._session_id,
|
||||
)
|
||||
return
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Method 2: Fallback to synthesizeScrollGesture
|
||||
try:
|
||||
params: 'SynthesizeScrollGestureParameters' = {'x': x, 'y': y, 'xDistance': delta_x or 0, 'yDistance': delta_y or 0}
|
||||
await self._client.send.Input.synthesizeScrollGesture(
|
||||
params,
|
||||
session_id=self._session_id,
|
||||
)
|
||||
except Exception:
|
||||
# Method 3: JavaScript fallback
|
||||
scroll_js = f'window.scrollBy({delta_x or 0}, {delta_y or 0})'
|
||||
await self._client.send.Runtime.evaluate(
|
||||
params={'expression': scroll_js, 'returnByValue': True},
|
||||
session_id=self._session_id,
|
||||
)
|
||||
@@ -0,0 +1,554 @@
|
||||
"""Page class for page-level operations."""
|
||||
|
||||
from typing import TYPE_CHECKING, TypeVar
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.dom.serializer.serializer import DOMTreeSerializer
|
||||
from browser_use.dom.service import DomService
|
||||
from browser_use.llm.messages import SystemMessage, UserMessage
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from cdp_use.cdp.dom.commands import (
|
||||
DescribeNodeParameters,
|
||||
QuerySelectorAllParameters,
|
||||
)
|
||||
from cdp_use.cdp.emulation.commands import SetDeviceMetricsOverrideParameters
|
||||
from cdp_use.cdp.input.commands import (
|
||||
DispatchKeyEventParameters,
|
||||
)
|
||||
from cdp_use.cdp.page.commands import CaptureScreenshotParameters, NavigateParameters, NavigateToHistoryEntryParameters
|
||||
from cdp_use.cdp.runtime.commands import EvaluateParameters
|
||||
from cdp_use.cdp.target.commands import (
|
||||
AttachToTargetParameters,
|
||||
GetTargetInfoParameters,
|
||||
)
|
||||
from cdp_use.cdp.target.types import TargetInfo
|
||||
|
||||
from browser_use.browser.session import BrowserSession
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
|
||||
from .element import Element
|
||||
from .mouse import Mouse
|
||||
|
||||
|
||||
class Page:
|
||||
"""Page operations (tab or iframe)."""
|
||||
|
||||
def __init__(
|
||||
self, browser_session: 'BrowserSession', target_id: str, session_id: str | None = None, llm: 'BaseChatModel | None' = None
|
||||
):
|
||||
self._browser_session = browser_session
|
||||
self._client = browser_session.cdp_client
|
||||
self._target_id = target_id
|
||||
self._session_id: str | None = session_id
|
||||
self._mouse: 'Mouse | None' = None
|
||||
|
||||
self._llm = llm
|
||||
|
||||
async def _ensure_session(self) -> str:
|
||||
"""Ensure we have a session ID for this target."""
|
||||
if not self._session_id:
|
||||
params: 'AttachToTargetParameters' = {'targetId': self._target_id, 'flatten': True}
|
||||
result = await self._client.send.Target.attachToTarget(params)
|
||||
self._session_id = result['sessionId']
|
||||
|
||||
# Enable necessary domains
|
||||
import asyncio
|
||||
|
||||
await asyncio.gather(
|
||||
self._client.send.Page.enable(session_id=self._session_id),
|
||||
self._client.send.DOM.enable(session_id=self._session_id),
|
||||
self._client.send.Runtime.enable(session_id=self._session_id),
|
||||
self._client.send.Network.enable(session_id=self._session_id),
|
||||
)
|
||||
|
||||
return self._session_id
|
||||
|
||||
@property
|
||||
async def session_id(self) -> str:
|
||||
"""Get the session ID for this target.
|
||||
|
||||
@dev Pass this to an arbitrary CDP call
|
||||
"""
|
||||
return await self._ensure_session()
|
||||
|
||||
@property
|
||||
async def mouse(self) -> 'Mouse':
|
||||
"""Get the mouse interface for this target."""
|
||||
if not self._mouse:
|
||||
session_id = await self._ensure_session()
|
||||
from .mouse import Mouse
|
||||
|
||||
self._mouse = Mouse(self._browser_session, session_id, self._target_id)
|
||||
return self._mouse
|
||||
|
||||
async def reload(self) -> None:
|
||||
"""Reload the target."""
|
||||
session_id = await self._ensure_session()
|
||||
await self._client.send.Page.reload(session_id=session_id)
|
||||
|
||||
async def get_element(self, backend_node_id: int) -> 'Element':
|
||||
"""Get an element by its backend node ID."""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
from .element import Element as Element_
|
||||
|
||||
return Element_(self._browser_session, backend_node_id, session_id)
|
||||
|
||||
async def evaluate(self, page_function: str, *args) -> str:
|
||||
"""Execute JavaScript in the target.
|
||||
|
||||
Args:
|
||||
page_function: JavaScript code that MUST start with (...args) => format
|
||||
*args: Arguments to pass to the function
|
||||
|
||||
Returns:
|
||||
String representation of the JavaScript execution result.
|
||||
Objects and arrays are JSON-stringified.
|
||||
"""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
# Clean and fix common JavaScript string parsing issues
|
||||
page_function = self._fix_javascript_string(page_function)
|
||||
|
||||
# Enforce arrow function format
|
||||
if not (page_function.startswith('(') and '=>' in page_function):
|
||||
raise ValueError(f'JavaScript code must start with (...args) => format. Got: {page_function[:50]}...')
|
||||
|
||||
# Build the expression - call the arrow function with provided args
|
||||
if args:
|
||||
# Convert args to JSON representation for safe passing
|
||||
import json
|
||||
|
||||
arg_strs = [json.dumps(arg) for arg in args]
|
||||
expression = f'({page_function})({", ".join(arg_strs)})'
|
||||
else:
|
||||
expression = f'({page_function})()'
|
||||
|
||||
# Debug: print the actual expression being evaluated
|
||||
print(f'DEBUG: Evaluating JavaScript: {repr(expression)}')
|
||||
|
||||
params: 'EvaluateParameters' = {'expression': expression, 'returnByValue': True, 'awaitPromise': True}
|
||||
result = await self._client.send.Runtime.evaluate(
|
||||
params,
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
if 'exceptionDetails' in result:
|
||||
raise RuntimeError(f'JavaScript evaluation failed: {result["exceptionDetails"]}')
|
||||
|
||||
value = result.get('result', {}).get('value')
|
||||
|
||||
# Always return string representation
|
||||
if value is None:
|
||||
return ''
|
||||
elif isinstance(value, str):
|
||||
return value
|
||||
else:
|
||||
# Convert objects, numbers, booleans to string
|
||||
import json
|
||||
|
||||
try:
|
||||
return json.dumps(value) if isinstance(value, (dict, list)) else str(value)
|
||||
except (TypeError, ValueError):
|
||||
return str(value)
|
||||
|
||||
def _fix_javascript_string(self, js_code: str) -> str:
|
||||
"""Fix common JavaScript string parsing issues when written as Python string."""
|
||||
|
||||
# Just do minimal, safe cleaning
|
||||
js_code = js_code.strip()
|
||||
|
||||
# Only fix the most common and safe issues:
|
||||
|
||||
# 1. Remove obvious Python string wrapper quotes if they exist
|
||||
if (js_code.startswith('"') and js_code.endswith('"')) or (js_code.startswith("'") and js_code.endswith("'")):
|
||||
# Check if it's a wrapped string (not part of JS syntax)
|
||||
inner = js_code[1:-1]
|
||||
if inner.count('"') + inner.count("'") == 0 or '() =>' in inner:
|
||||
js_code = inner
|
||||
|
||||
# 2. Only fix clearly escaped quotes that shouldn't be
|
||||
# But be very conservative - only if we're sure it's a Python string artifact
|
||||
if '\\"' in js_code and js_code.count('\\"') > js_code.count('"'):
|
||||
js_code = js_code.replace('\\"', '"')
|
||||
if "\\'" in js_code and js_code.count("\\'") > js_code.count("'"):
|
||||
js_code = js_code.replace("\\'", "'")
|
||||
|
||||
# 3. Basic whitespace normalization only
|
||||
js_code = js_code.strip()
|
||||
|
||||
# Final validation - ensure it's not empty
|
||||
if not js_code:
|
||||
raise ValueError('JavaScript code is empty after cleaning')
|
||||
|
||||
return js_code
|
||||
|
||||
async def screenshot(self, format: str = 'jpeg', quality: int | None = None) -> str:
|
||||
"""Take a screenshot and return base64 encoded image.
|
||||
|
||||
Args:
|
||||
format: Image format ('jpeg', 'png', 'webp')
|
||||
quality: Quality 0-100 for JPEG format
|
||||
|
||||
Returns:
|
||||
Base64-encoded image data
|
||||
"""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
params: 'CaptureScreenshotParameters' = {'format': format}
|
||||
|
||||
if quality is not None and format.lower() == 'jpeg':
|
||||
params['quality'] = quality
|
||||
|
||||
result = await self._client.send.Page.captureScreenshot(params, session_id=session_id)
|
||||
|
||||
return result['data']
|
||||
|
||||
async def press(self, key: str) -> None:
|
||||
"""Press a key on the page (sends keyboard input to the focused element or page)."""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
# Handle key combinations like "Control+A"
|
||||
if '+' in key:
|
||||
parts = key.split('+')
|
||||
modifiers = parts[:-1]
|
||||
main_key = parts[-1]
|
||||
|
||||
# Press modifier keys
|
||||
for mod in modifiers:
|
||||
params: 'DispatchKeyEventParameters' = {'type': 'keyDown', 'key': mod}
|
||||
await self._client.send.Input.dispatchKeyEvent(params, session_id=session_id)
|
||||
|
||||
# Press main key
|
||||
main_down_params: 'DispatchKeyEventParameters' = {'type': 'keyDown', 'key': main_key}
|
||||
await self._client.send.Input.dispatchKeyEvent(main_down_params, session_id=session_id)
|
||||
|
||||
main_up_params: 'DispatchKeyEventParameters' = {'type': 'keyUp', 'key': main_key}
|
||||
await self._client.send.Input.dispatchKeyEvent(main_up_params, session_id=session_id)
|
||||
|
||||
# Release modifier keys
|
||||
for mod in reversed(modifiers):
|
||||
release_params: 'DispatchKeyEventParameters' = {'type': 'keyUp', 'key': mod}
|
||||
await self._client.send.Input.dispatchKeyEvent(release_params, session_id=session_id)
|
||||
else:
|
||||
# Simple key press
|
||||
key_down_params: 'DispatchKeyEventParameters' = {'type': 'keyDown', 'key': key}
|
||||
await self._client.send.Input.dispatchKeyEvent(key_down_params, session_id=session_id)
|
||||
|
||||
key_up_params: 'DispatchKeyEventParameters' = {'type': 'keyUp', 'key': key}
|
||||
await self._client.send.Input.dispatchKeyEvent(key_up_params, session_id=session_id)
|
||||
|
||||
async def set_viewport_size(self, width: int, height: int) -> None:
|
||||
"""Set the viewport size."""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
params: 'SetDeviceMetricsOverrideParameters' = {
|
||||
'width': width,
|
||||
'height': height,
|
||||
'deviceScaleFactor': 1.0,
|
||||
'mobile': False,
|
||||
}
|
||||
await self._client.send.Emulation.setDeviceMetricsOverride(
|
||||
params,
|
||||
session_id=session_id,
|
||||
)
|
||||
|
||||
# Target properties (from CDP getTargetInfo)
|
||||
async def get_target_info(self) -> 'TargetInfo':
|
||||
"""Get target information."""
|
||||
params: 'GetTargetInfoParameters' = {'targetId': self._target_id}
|
||||
result = await self._client.send.Target.getTargetInfo(params)
|
||||
return result['targetInfo']
|
||||
|
||||
async def get_url(self) -> str:
|
||||
"""Get the current URL."""
|
||||
info = await self.get_target_info()
|
||||
return info.get('url', '')
|
||||
|
||||
async def get_title(self) -> str:
|
||||
"""Get the current title."""
|
||||
info = await self.get_target_info()
|
||||
return info.get('title', '')
|
||||
|
||||
async def goto(self, url: str) -> None:
|
||||
"""Navigate this target to a URL."""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
params: 'NavigateParameters' = {'url': url}
|
||||
await self._client.send.Page.navigate(params, session_id=session_id)
|
||||
|
||||
async def navigate(self, url: str) -> None:
|
||||
"""Alias for goto."""
|
||||
await self.goto(url)
|
||||
|
||||
async def go_back(self) -> None:
|
||||
"""Navigate back in history."""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
try:
|
||||
# Get navigation history
|
||||
history = await self._client.send.Page.getNavigationHistory(session_id=session_id)
|
||||
current_index = history['currentIndex']
|
||||
entries = history['entries']
|
||||
|
||||
# Check if we can go back
|
||||
if current_index <= 0:
|
||||
raise RuntimeError('Cannot go back - no previous entry in history')
|
||||
|
||||
# Navigate to the previous entry
|
||||
previous_entry_id = entries[current_index - 1]['id']
|
||||
params: 'NavigateToHistoryEntryParameters' = {'entryId': previous_entry_id}
|
||||
await self._client.send.Page.navigateToHistoryEntry(params, session_id=session_id)
|
||||
|
||||
except Exception as e:
|
||||
raise RuntimeError(f'Failed to navigate back: {e}')
|
||||
|
||||
async def go_forward(self) -> None:
|
||||
"""Navigate forward in history."""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
try:
|
||||
# Get navigation history
|
||||
history = await self._client.send.Page.getNavigationHistory(session_id=session_id)
|
||||
current_index = history['currentIndex']
|
||||
entries = history['entries']
|
||||
|
||||
# Check if we can go forward
|
||||
if current_index >= len(entries) - 1:
|
||||
raise RuntimeError('Cannot go forward - no next entry in history')
|
||||
|
||||
# Navigate to the next entry
|
||||
next_entry_id = entries[current_index + 1]['id']
|
||||
params: 'NavigateToHistoryEntryParameters' = {'entryId': next_entry_id}
|
||||
await self._client.send.Page.navigateToHistoryEntry(params, session_id=session_id)
|
||||
|
||||
except Exception as e:
|
||||
raise RuntimeError(f'Failed to navigate forward: {e}')
|
||||
|
||||
# Element finding methods (these would need to be implemented based on DOM queries)
|
||||
async def get_elements_by_css_selector(self, selector: str) -> list['Element']:
|
||||
"""Get elements by CSS selector."""
|
||||
session_id = await self._ensure_session()
|
||||
|
||||
# Get document first
|
||||
doc_result = await self._client.send.DOM.getDocument(session_id=session_id)
|
||||
document_node_id = doc_result['root']['nodeId']
|
||||
|
||||
# Query selector all
|
||||
query_params: 'QuerySelectorAllParameters' = {'nodeId': document_node_id, 'selector': selector}
|
||||
result = await self._client.send.DOM.querySelectorAll(query_params, session_id=session_id)
|
||||
|
||||
elements = []
|
||||
from .element import Element as Element_
|
||||
|
||||
# Convert node IDs to backend node IDs
|
||||
for node_id in result['nodeIds']:
|
||||
# Get backend node ID
|
||||
describe_params: 'DescribeNodeParameters' = {'nodeId': node_id}
|
||||
node_result = await self._client.send.DOM.describeNode(describe_params, session_id=session_id)
|
||||
backend_node_id = node_result['node']['backendNodeId']
|
||||
elements.append(Element_(self._browser_session, backend_node_id, session_id))
|
||||
|
||||
return elements
|
||||
|
||||
# AI METHODS
|
||||
|
||||
@property
|
||||
def dom_service(self) -> 'DomService':
|
||||
"""Get the DOM service for this target."""
|
||||
return DomService(self._browser_session)
|
||||
|
||||
async def get_element_by_prompt(self, prompt: str, llm: 'BaseChatModel | None' = None) -> 'Element | None':
|
||||
"""Get an element by a prompt."""
|
||||
await self._ensure_session()
|
||||
llm = llm or self._llm
|
||||
|
||||
if not llm:
|
||||
raise ValueError('LLM not provided')
|
||||
|
||||
dom_service = self.dom_service
|
||||
|
||||
enhanced_dom_tree = await dom_service.get_dom_tree(target_id=self._target_id)
|
||||
|
||||
serialized_dom_state, _ = DOMTreeSerializer(
|
||||
enhanced_dom_tree, None, paint_order_filtering=True
|
||||
).serialize_accessible_elements()
|
||||
|
||||
llm_representation = serialized_dom_state.llm_representation()
|
||||
|
||||
system_message = SystemMessage(
|
||||
content="""You are an AI created to find an element on a page by a prompt.
|
||||
|
||||
<browser_state>
|
||||
Interactive Elements: All interactive elements will be provided in format as [index]<type>text</type> where
|
||||
- index: Numeric identifier for interaction
|
||||
- type: HTML element type (button, input, etc.)
|
||||
- text: Element description
|
||||
|
||||
Examples:
|
||||
[33]<div>User form</div>
|
||||
[35]<button aria-label='Submit form'>Submit</button>
|
||||
|
||||
Note that:
|
||||
- Only elements with numeric indexes in [] are interactive
|
||||
- (stacked) indentation (with \t) is important and means that the element is a (html) child of the element above (with a lower index)
|
||||
- Pure text elements without [] are not interactive.
|
||||
</browser_state>
|
||||
|
||||
Your task is to find an element index (if any) that matches the prompt (written in <prompt> tag).
|
||||
|
||||
If non of the elements matches the, return None.
|
||||
|
||||
Before you return the element index, reason about the state and elements for a sentence or two."""
|
||||
)
|
||||
|
||||
state_message = UserMessage(
|
||||
content=f"""
|
||||
<browser_state>
|
||||
{llm_representation}
|
||||
</browser_state>
|
||||
|
||||
<prompt>
|
||||
{prompt}
|
||||
</prompt>
|
||||
"""
|
||||
)
|
||||
|
||||
class ElementResponse(BaseModel):
|
||||
# thinking: str
|
||||
element_highlight_index: int | None
|
||||
|
||||
llm_response = await llm.ainvoke(
|
||||
[
|
||||
system_message,
|
||||
state_message,
|
||||
],
|
||||
output_format=ElementResponse,
|
||||
)
|
||||
|
||||
element_highlight_index = llm_response.completion.element_highlight_index
|
||||
|
||||
if element_highlight_index is None or element_highlight_index not in serialized_dom_state.selector_map:
|
||||
return None
|
||||
|
||||
element = serialized_dom_state.selector_map[element_highlight_index]
|
||||
|
||||
from .element import Element as Element_
|
||||
|
||||
return Element_(self._browser_session, element.backend_node_id, self._session_id)
|
||||
|
||||
async def must_get_element_by_prompt(self, prompt: str, llm: 'BaseChatModel | None' = None) -> 'Element':
|
||||
"""Get an element by a prompt.
|
||||
|
||||
@dev LLM can still return None, this just raises an error if the element is not found.
|
||||
"""
|
||||
element = await self.get_element_by_prompt(prompt, llm)
|
||||
if element is None:
|
||||
raise ValueError(f'No element found for prompt: {prompt}')
|
||||
|
||||
return element
|
||||
|
||||
async def extract_content(self, prompt: str, structured_output: type[T], llm: 'BaseChatModel | None' = None) -> T:
|
||||
"""Extract structured content from the current page using LLM.
|
||||
|
||||
Extracts clean markdown from the page and sends it to LLM for structured data extraction.
|
||||
|
||||
Args:
|
||||
prompt: Description of what content to extract
|
||||
structured_output: Pydantic BaseModel class defining the expected output structure
|
||||
llm: Language model to use for extraction
|
||||
|
||||
Returns:
|
||||
The structured BaseModel instance with extracted content
|
||||
"""
|
||||
llm = llm or self._llm
|
||||
|
||||
if not llm:
|
||||
raise ValueError('LLM not provided')
|
||||
|
||||
# Extract clean markdown from the page
|
||||
session_id = await self._ensure_session()
|
||||
try:
|
||||
body_id = await self._client.send.DOM.getDocument(session_id=session_id)
|
||||
page_html_result = await self._client.send.DOM.getOuterHTML(
|
||||
params={'backendNodeId': body_id['root']['backendNodeId']}, session_id=session_id
|
||||
)
|
||||
page_html = page_html_result['outerHTML']
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"Couldn't extract page content: {e}")
|
||||
|
||||
# Convert HTML to clean markdown
|
||||
import html2text
|
||||
|
||||
h = html2text.HTML2Text()
|
||||
h.ignore_links = False
|
||||
h.ignore_images = True
|
||||
h.ignore_emphasis = False
|
||||
h.body_width = 0 # Don't wrap lines
|
||||
h.unicode_snob = True
|
||||
h.skip_internal_links = True
|
||||
markdown_content = h.handle(page_html)
|
||||
|
||||
# Clean up the markdown
|
||||
import re
|
||||
|
||||
# Remove URL encoding artifacts
|
||||
markdown_content = re.sub(r'%[0-9A-Fa-f]{2}', '', markdown_content)
|
||||
|
||||
# Compress excessive newlines
|
||||
markdown_content = re.sub(r'\n{4,}', '\n\n\n', markdown_content)
|
||||
|
||||
# Remove very short lines (likely artifacts)
|
||||
lines = markdown_content.split('\n')
|
||||
filtered_lines = []
|
||||
for line in lines:
|
||||
stripped = line.strip()
|
||||
if len(stripped) > 2: # Keep lines with substantial content
|
||||
filtered_lines.append(line)
|
||||
|
||||
markdown_content = '\n'.join(filtered_lines).strip()
|
||||
|
||||
# System prompt for structured extraction
|
||||
system_prompt = """
|
||||
You are an expert at extracting structured data from the markdown of a webpage.
|
||||
|
||||
<input>
|
||||
You will be given a query and the markdown of a webpage that has been filtered to remove noise and advertising content.
|
||||
</input>
|
||||
|
||||
<instructions>
|
||||
- You are tasked to extract information from the webpage that is relevant to the query.
|
||||
- You should ONLY use the information available in the webpage to answer the query. Do not make up information or provide guess from your own knowledge.
|
||||
- If the information relevant to the query is not available in the page, your response should mention that.
|
||||
- If the query asks for all items, products, etc., make sure to directly list all of them.
|
||||
- Return the extracted content in the exact structured format specified.
|
||||
</instructions>
|
||||
|
||||
<output>
|
||||
- Your output should present ALL the information relevant to the query in the specified structured format.
|
||||
- Do not answer in conversational format - directly output the relevant information in the structured format.
|
||||
</output>
|
||||
""".strip()
|
||||
|
||||
# Build prompt with just query and content
|
||||
prompt_content = f'<query>\n{prompt}\n</query>\n\n<webpage_content>\n{markdown_content}\n</webpage_content>'
|
||||
|
||||
# Send to LLM with structured output
|
||||
import asyncio
|
||||
|
||||
try:
|
||||
response = await asyncio.wait_for(
|
||||
llm.ainvoke(
|
||||
[SystemMessage(content=system_prompt), UserMessage(content=prompt_content)], output_format=structured_output
|
||||
),
|
||||
timeout=120.0,
|
||||
)
|
||||
|
||||
# Return the structured output BaseModel instance
|
||||
return response.completion
|
||||
except Exception as e:
|
||||
raise RuntimeError(str(e))
|
||||
@@ -0,0 +1,41 @@
|
||||
import asyncio
|
||||
|
||||
from browser_use import Agent, Browser, ChatOpenAI
|
||||
|
||||
llm = ChatOpenAI('gpt-4.1-mini')
|
||||
|
||||
|
||||
async def main():
|
||||
"""
|
||||
Main function demonstrating mixed automation with Browser-Use and Playwright.
|
||||
"""
|
||||
print('🚀 Mixed Automation with Browser-Use and Actor API')
|
||||
|
||||
browser = Browser(keep_alive=True)
|
||||
await browser.start()
|
||||
|
||||
page = await browser.get_current_page() or await browser.new_page()
|
||||
|
||||
# Go to apple wikipedia page
|
||||
await page.goto('https://www.google.com/travel/flights')
|
||||
|
||||
await asyncio.sleep(1)
|
||||
|
||||
round_trip_button = await page.must_get_element_by_prompt('round trip button', llm)
|
||||
await round_trip_button.click()
|
||||
|
||||
one_way_button = await page.must_get_element_by_prompt('one way button', llm)
|
||||
await one_way_button.click()
|
||||
|
||||
await asyncio.sleep(1)
|
||||
|
||||
agent = Agent(task='Find the cheapest flight from London to Paris on 2025-10-15', llm=llm, browser_session=browser)
|
||||
await agent.run()
|
||||
|
||||
input('Press Enter to continue...')
|
||||
|
||||
await browser.stop()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,54 @@
|
||||
import asyncio
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use import Browser, ChatOpenAI
|
||||
|
||||
TASK = """
|
||||
On the current wikipedia page, find the latest huge edit and tell me what is was about.
|
||||
"""
|
||||
|
||||
|
||||
class LatestEditFinder(BaseModel):
|
||||
"""Find the latest huge edit on the current wikipedia page."""
|
||||
|
||||
latest_edit: str
|
||||
edit_time: str
|
||||
edit_author: str
|
||||
edit_summary: str
|
||||
edit_url: str
|
||||
|
||||
|
||||
llm = ChatOpenAI('gpt-4.1-mini')
|
||||
|
||||
|
||||
async def main():
|
||||
"""
|
||||
Main function demonstrating mixed automation with Browser-Use and Playwright.
|
||||
"""
|
||||
print('🚀 Mixed Automation with Browser-Use and Actor API')
|
||||
|
||||
browser = Browser(keep_alive=True)
|
||||
await browser.start()
|
||||
|
||||
page = await browser.get_current_page() or await browser.new_page()
|
||||
|
||||
# Go to apple wikipedia page
|
||||
await page.goto('https://browser-use.github.io/stress-tests/challenges/angularjs-form.html')
|
||||
|
||||
await asyncio.sleep(1)
|
||||
|
||||
element = await page.get_element_by_prompt('zip code input', llm)
|
||||
|
||||
print('Element found', element)
|
||||
|
||||
if element:
|
||||
await element.click()
|
||||
else:
|
||||
print('No element found')
|
||||
|
||||
await browser.stop()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
+236
@@ -0,0 +1,236 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Playground script to test the browser-use actor API.
|
||||
|
||||
This script demonstrates:
|
||||
- Starting a browser session
|
||||
- Using the actor API to navigate and interact
|
||||
- Finding elements, clicking, scrolling, JavaScript evaluation
|
||||
- Testing most of the available methods
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
|
||||
from browser_use import Browser
|
||||
|
||||
# Configure logging to see what's happening
|
||||
logging.basicConfig(level=logging.INFO)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main playground function."""
|
||||
logger.info('🚀 Starting browser actor playground')
|
||||
|
||||
# Create browser session
|
||||
browser = Browser()
|
||||
|
||||
try:
|
||||
# Start the browser
|
||||
await browser.start()
|
||||
logger.info('✅ Browser session started')
|
||||
|
||||
# Navigate to Wikipedia using integrated methods
|
||||
logger.info('📖 Navigating to Wikipedia...')
|
||||
page = await browser.new_page('https://en.wikipedia.org')
|
||||
|
||||
# Get basic page info
|
||||
url = await page.get_url()
|
||||
title = await page.get_title()
|
||||
logger.info(f'📄 Page loaded: {title} ({url})')
|
||||
|
||||
# Take a screenshot
|
||||
logger.info('📸 Taking initial screenshot...')
|
||||
screenshot_b64 = await page.screenshot()
|
||||
logger.info(f'📸 Screenshot captured: {len(screenshot_b64)} bytes')
|
||||
|
||||
# Set viewport size
|
||||
logger.info('🖥️ Setting viewport to 1920x1080...')
|
||||
await page.set_viewport_size(1920, 1080)
|
||||
|
||||
# Execute some JavaScript to count links
|
||||
logger.info('🔍 Counting article links using JavaScript...')
|
||||
js_code = """() => {
|
||||
// Find all article links on the page
|
||||
const links = Array.from(document.querySelectorAll('a[href*="/wiki/"]:not([href*=":"])'))
|
||||
.filter(link => !link.href.includes('Main_Page') && !link.href.includes('Special:'));
|
||||
|
||||
return {
|
||||
total: links.length,
|
||||
sample: links.slice(0, 3).map(link => ({
|
||||
href: link.href,
|
||||
text: link.textContent.trim()
|
||||
}))
|
||||
};
|
||||
}"""
|
||||
|
||||
link_info = json.loads(await page.evaluate(js_code))
|
||||
logger.info(f'🔗 Found {link_info["total"]} article links')
|
||||
# Try to find and interact with links using CSS selector
|
||||
try:
|
||||
# Find article links on the page
|
||||
links = await page.get_elements_by_css_selector('a[href*="/wiki/"]:not([href*=":"])')
|
||||
|
||||
if links:
|
||||
logger.info(f'📋 Found {len(links)} wiki links via CSS selector')
|
||||
|
||||
# Pick the first link
|
||||
link_element = links[0]
|
||||
|
||||
# Get link info using available methods
|
||||
basic_info = await link_element.get_basic_info()
|
||||
link_href = await link_element.get_attribute('href')
|
||||
|
||||
logger.info(f'🎯 Selected element: <{basic_info["nodeName"]}>')
|
||||
logger.info(f'🔗 Link href: {link_href}')
|
||||
|
||||
if basic_info['boundingBox']:
|
||||
bbox = basic_info['boundingBox']
|
||||
logger.info(f'📏 Position: ({bbox["x"]}, {bbox["y"]}) Size: {bbox["width"]}x{bbox["height"]}')
|
||||
|
||||
# Test element interactions with robust implementations
|
||||
logger.info('👆 Hovering over the element...')
|
||||
await link_element.hover()
|
||||
await asyncio.sleep(1)
|
||||
|
||||
logger.info('🔍 Focusing the element...')
|
||||
await link_element.focus()
|
||||
await asyncio.sleep(0.5)
|
||||
|
||||
# Click the link using robust click method
|
||||
logger.info('🖱️ Clicking the link with robust fallbacks...')
|
||||
await link_element.click()
|
||||
|
||||
# Wait for navigation
|
||||
await asyncio.sleep(3)
|
||||
|
||||
# Get new page info
|
||||
new_url = await page.get_url()
|
||||
new_title = await page.get_title()
|
||||
logger.info(f'📄 Navigated to: {new_title}')
|
||||
logger.info(f'🌐 New URL: {new_url}')
|
||||
else:
|
||||
logger.warning('❌ No links found to interact with')
|
||||
|
||||
except Exception as e:
|
||||
logger.warning(f'⚠️ Link interaction failed: {e}')
|
||||
|
||||
# Scroll down the page
|
||||
logger.info('📜 Scrolling down the page...')
|
||||
mouse = await page.mouse
|
||||
await mouse.scroll(x=0, y=100, delta_y=500)
|
||||
await asyncio.sleep(1)
|
||||
|
||||
# Test mouse operations
|
||||
logger.info('🖱️ Testing mouse operations...')
|
||||
await mouse.move(x=100, y=200)
|
||||
await mouse.click(x=150, y=250)
|
||||
|
||||
# Execute more JavaScript examples
|
||||
logger.info('🧪 Testing JavaScript evaluation...')
|
||||
|
||||
# Simple expressions
|
||||
page_height = await page.evaluate('() => document.body.scrollHeight')
|
||||
current_scroll = await page.evaluate('() => window.pageYOffset')
|
||||
logger.info(f'📏 Page height: {page_height}px, current scroll: {current_scroll}px')
|
||||
|
||||
# JavaScript with arguments
|
||||
result = await page.evaluate('(x) => x * 2', 21)
|
||||
logger.info(f'🧮 JavaScript with args: 21 * 2 = {result}')
|
||||
|
||||
# More complex JavaScript
|
||||
page_stats = json.loads(
|
||||
await page.evaluate("""() => {
|
||||
return {
|
||||
url: window.location.href,
|
||||
title: document.title,
|
||||
links: document.querySelectorAll('a').length,
|
||||
images: document.querySelectorAll('img').length,
|
||||
scrollTop: window.pageYOffset,
|
||||
viewportHeight: window.innerHeight
|
||||
};
|
||||
}""")
|
||||
)
|
||||
logger.info(f'📊 Page stats: {page_stats}')
|
||||
|
||||
# Get page title using different methods
|
||||
title_via_js = await page.evaluate('() => document.title')
|
||||
title_via_api = await page.get_title()
|
||||
logger.info(f'📝 Title via JS: "{title_via_js}"')
|
||||
logger.info(f'📝 Title via API: "{title_via_api}"')
|
||||
|
||||
# Take a final screenshot
|
||||
logger.info('📸 Taking final screenshot...')
|
||||
final_screenshot = await page.screenshot()
|
||||
logger.info(f'📸 Final screenshot: {len(final_screenshot)} bytes')
|
||||
|
||||
# Test browser navigation with error handling
|
||||
logger.info('⬅️ Testing browser back navigation...')
|
||||
try:
|
||||
await page.go_back()
|
||||
await asyncio.sleep(2)
|
||||
|
||||
back_url = await page.get_url()
|
||||
back_title = await page.get_title()
|
||||
logger.info(f'📄 After going back: {back_title}')
|
||||
logger.info(f'🌐 Back URL: {back_url}')
|
||||
except RuntimeError as e:
|
||||
logger.info(f'ℹ️ Navigation back failed as expected: {e}')
|
||||
|
||||
# Test creating new page
|
||||
logger.info('🆕 Creating new blank page...')
|
||||
new_page = await browser.new_page()
|
||||
new_page_url = await new_page.get_url()
|
||||
logger.info(f'🆕 New page created with URL: {new_page_url}')
|
||||
|
||||
# Get all pages
|
||||
all_pages = await browser.get_pages()
|
||||
logger.info(f'📑 Total pages: {len(all_pages)}')
|
||||
|
||||
# Test form interaction if we can find a form
|
||||
try:
|
||||
# Look for search input on the page
|
||||
search_inputs = await page.get_elements_by_css_selector('input[type="search"], input[name*="search"]')
|
||||
|
||||
if search_inputs:
|
||||
search_input = search_inputs[0]
|
||||
logger.info('🔍 Found search input, testing form interaction...')
|
||||
|
||||
await search_input.focus()
|
||||
await search_input.fill('test search query')
|
||||
await page.press('Enter')
|
||||
|
||||
logger.info('✅ Form interaction test completed')
|
||||
else:
|
||||
logger.info('ℹ️ No search inputs found for form testing')
|
||||
|
||||
except Exception as e:
|
||||
logger.info(f'ℹ️ Form interaction test skipped: {e}')
|
||||
|
||||
# wait 2 seconds before closing the new page
|
||||
logger.info('🕒 Waiting 2 seconds before closing the new page...')
|
||||
await asyncio.sleep(2)
|
||||
logger.info('🗑️ Closing new page...')
|
||||
await browser.close_page(new_page)
|
||||
|
||||
logger.info('✅ Playground completed successfully!')
|
||||
|
||||
input('Press Enter to continue...')
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f'❌ Error in playground: {e}', exc_info=True)
|
||||
|
||||
finally:
|
||||
# Clean up
|
||||
logger.info('🧹 Cleaning up...')
|
||||
try:
|
||||
await browser.stop()
|
||||
logger.info('✅ Browser session stopped')
|
||||
except Exception as e:
|
||||
logger.error(f'❌ Error stopping browser: {e}')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main())
|
||||
@@ -0,0 +1,282 @@
|
||||
import base64
|
||||
import os
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import anyio
|
||||
from bubus import BaseEvent
|
||||
from pydantic import Field, field_validator
|
||||
from uuid_extensions import uuid7str
|
||||
|
||||
MAX_STRING_LENGTH = 100000 # 100K chars ~ 25k tokens should be enough
|
||||
MAX_URL_LENGTH = 100000
|
||||
MAX_TASK_LENGTH = 100000
|
||||
MAX_COMMENT_LENGTH = 2000
|
||||
MAX_FILE_CONTENT_SIZE = 50 * 1024 * 1024 # 50MB
|
||||
|
||||
|
||||
class UpdateAgentTaskEvent(BaseEvent):
|
||||
# Required fields for identification
|
||||
id: str # The task ID to update
|
||||
user_id: str = Field(max_length=255) # For authorization
|
||||
device_id: str | None = Field(None, max_length=255) # Device ID for auth lookup
|
||||
|
||||
# Optional fields that can be updated
|
||||
stopped: bool | None = None
|
||||
paused: bool | None = None
|
||||
done_output: str | None = Field(None, max_length=MAX_STRING_LENGTH)
|
||||
finished_at: datetime | None = None
|
||||
agent_state: dict | None = None
|
||||
user_feedback_type: str | None = Field(None, max_length=10) # UserFeedbackType enum value as string
|
||||
user_comment: str | None = Field(None, max_length=MAX_COMMENT_LENGTH)
|
||||
gif_url: str | None = Field(None, max_length=MAX_URL_LENGTH)
|
||||
|
||||
@classmethod
|
||||
def from_agent(cls, agent) -> 'UpdateAgentTaskEvent':
|
||||
"""Create an UpdateAgentTaskEvent from an Agent instance"""
|
||||
if not hasattr(agent, '_task_start_time'):
|
||||
raise ValueError('Agent must have _task_start_time attribute')
|
||||
|
||||
done_output = agent.history.final_result() if agent.history else None
|
||||
return cls(
|
||||
id=str(agent.task_id),
|
||||
user_id='', # To be filled by cloud handler
|
||||
device_id=agent.cloud_sync.auth_client.device_id
|
||||
if hasattr(agent, 'cloud_sync') and agent.cloud_sync and agent.cloud_sync.auth_client
|
||||
else None,
|
||||
stopped=agent.state.stopped if hasattr(agent.state, 'stopped') else False,
|
||||
paused=agent.state.paused if hasattr(agent.state, 'paused') else False,
|
||||
done_output=done_output,
|
||||
finished_at=datetime.now(timezone.utc) if agent.history and agent.history.is_done() else None,
|
||||
agent_state=agent.state.model_dump() if hasattr(agent.state, 'model_dump') else {},
|
||||
user_feedback_type=None,
|
||||
user_comment=None,
|
||||
gif_url=None,
|
||||
# user_feedback_type and user_comment would be set by the API/frontend
|
||||
# gif_url would be set after GIF generation if needed
|
||||
)
|
||||
|
||||
|
||||
class CreateAgentOutputFileEvent(BaseEvent):
|
||||
# Model fields
|
||||
id: str = Field(default_factory=uuid7str)
|
||||
user_id: str = Field(max_length=255)
|
||||
device_id: str | None = Field(None, max_length=255) # Device ID for auth lookup
|
||||
task_id: str
|
||||
file_name: str = Field(max_length=255)
|
||||
file_content: str | None = None # Base64 encoded file content
|
||||
content_type: str | None = Field(None, max_length=100) # MIME type for file uploads
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
|
||||
@field_validator('file_content')
|
||||
@classmethod
|
||||
def validate_file_size(cls, v: str | None) -> str | None:
|
||||
"""Validate base64 file content size."""
|
||||
if v is None:
|
||||
return v
|
||||
# Remove data URL prefix if present
|
||||
if ',' in v:
|
||||
v = v.split(',')[1]
|
||||
# Estimate decoded size (base64 is ~33% larger)
|
||||
estimated_size = len(v) * 3 / 4
|
||||
if estimated_size > MAX_FILE_CONTENT_SIZE:
|
||||
raise ValueError(f'File content exceeds maximum size of {MAX_FILE_CONTENT_SIZE / 1024 / 1024}MB')
|
||||
return v
|
||||
|
||||
@classmethod
|
||||
async def from_agent_and_file(cls, agent, output_path: str) -> 'CreateAgentOutputFileEvent':
|
||||
"""Create a CreateAgentOutputFileEvent from a file path"""
|
||||
|
||||
gif_path = Path(output_path)
|
||||
if not gif_path.exists():
|
||||
raise FileNotFoundError(f'File not found: {output_path}')
|
||||
|
||||
gif_size = os.path.getsize(gif_path)
|
||||
|
||||
# Read GIF content for base64 encoding if needed
|
||||
gif_content = None
|
||||
if gif_size < 50 * 1024 * 1024: # Only read if < 50MB
|
||||
async with await anyio.open_file(gif_path, 'rb') as f:
|
||||
gif_bytes = await f.read()
|
||||
gif_content = base64.b64encode(gif_bytes).decode('utf-8')
|
||||
|
||||
return cls(
|
||||
user_id='', # To be filled by cloud handler
|
||||
device_id=agent.cloud_sync.auth_client.device_id
|
||||
if hasattr(agent, 'cloud_sync') and agent.cloud_sync and agent.cloud_sync.auth_client
|
||||
else None,
|
||||
task_id=str(agent.task_id),
|
||||
file_name=gif_path.name,
|
||||
file_content=gif_content, # Base64 encoded
|
||||
content_type='image/gif',
|
||||
)
|
||||
|
||||
|
||||
class CreateAgentStepEvent(BaseEvent):
|
||||
# Model fields
|
||||
id: str = Field(default_factory=uuid7str)
|
||||
user_id: str = Field(max_length=255) # Added for authorization checks
|
||||
device_id: str | None = Field(None, max_length=255) # Device ID for auth lookup
|
||||
created_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
agent_task_id: str
|
||||
step: int
|
||||
evaluation_previous_goal: str = Field(max_length=MAX_STRING_LENGTH)
|
||||
memory: str = Field(max_length=MAX_STRING_LENGTH)
|
||||
next_goal: str = Field(max_length=MAX_STRING_LENGTH)
|
||||
actions: list[dict]
|
||||
screenshot_url: str | None = Field(None, max_length=MAX_FILE_CONTENT_SIZE) # ~50MB for base64 images
|
||||
url: str = Field(default='', max_length=MAX_URL_LENGTH)
|
||||
|
||||
@field_validator('screenshot_url')
|
||||
@classmethod
|
||||
def validate_screenshot_size(cls, v: str | None) -> str | None:
|
||||
"""Validate screenshot URL or base64 content size."""
|
||||
if v is None or not v.startswith('data:'):
|
||||
return v
|
||||
# It's base64 data, check size
|
||||
if ',' in v:
|
||||
base64_part = v.split(',')[1]
|
||||
estimated_size = len(base64_part) * 3 / 4
|
||||
if estimated_size > MAX_FILE_CONTENT_SIZE:
|
||||
raise ValueError(f'Screenshot content exceeds maximum size of {MAX_FILE_CONTENT_SIZE / 1024 / 1024}MB')
|
||||
return v
|
||||
|
||||
@classmethod
|
||||
def from_agent_step(
|
||||
cls, agent, model_output, result: list, actions_data: list[dict], browser_state_summary
|
||||
) -> 'CreateAgentStepEvent':
|
||||
"""Create a CreateAgentStepEvent from agent step data"""
|
||||
# Get first action details if available
|
||||
first_action = model_output.action[0] if model_output.action else None
|
||||
|
||||
# Extract current state from model output
|
||||
current_state = model_output.current_state if hasattr(model_output, 'current_state') else None
|
||||
|
||||
# Capture screenshot as base64 data URL if available
|
||||
screenshot_url = None
|
||||
if browser_state_summary.screenshot:
|
||||
screenshot_url = f'data:image/png;base64,{browser_state_summary.screenshot}'
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.debug(f'📸 Including screenshot in CreateAgentStepEvent, length: {len(browser_state_summary.screenshot)}')
|
||||
else:
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
logger.debug('📸 No screenshot in browser_state_summary for CreateAgentStepEvent')
|
||||
|
||||
return cls(
|
||||
user_id='', # To be filled by cloud handler
|
||||
device_id=agent.cloud_sync.auth_client.device_id
|
||||
if hasattr(agent, 'cloud_sync') and agent.cloud_sync and agent.cloud_sync.auth_client
|
||||
else None,
|
||||
agent_task_id=str(agent.task_id),
|
||||
step=agent.state.n_steps,
|
||||
evaluation_previous_goal=current_state.evaluation_previous_goal if current_state else '',
|
||||
memory=current_state.memory if current_state else '',
|
||||
next_goal=current_state.next_goal if current_state else '',
|
||||
actions=actions_data, # List of action dicts
|
||||
url=browser_state_summary.url,
|
||||
screenshot_url=screenshot_url,
|
||||
)
|
||||
|
||||
|
||||
class CreateAgentTaskEvent(BaseEvent):
|
||||
# Model fields
|
||||
id: str = Field(default_factory=uuid7str)
|
||||
user_id: str = Field(max_length=255) # Added for authorization checks
|
||||
device_id: str | None = Field(None, max_length=255) # Device ID for auth lookup
|
||||
agent_session_id: str
|
||||
llm_model: str = Field(max_length=100) # LLMModel enum value as string
|
||||
stopped: bool = False
|
||||
paused: bool = False
|
||||
task: str = Field(max_length=MAX_TASK_LENGTH)
|
||||
done_output: str | None = Field(None, max_length=MAX_STRING_LENGTH)
|
||||
scheduled_task_id: str | None = None
|
||||
started_at: datetime = Field(default_factory=lambda: datetime.now(timezone.utc))
|
||||
finished_at: datetime | None = None
|
||||
agent_state: dict = Field(default_factory=dict)
|
||||
user_feedback_type: str | None = Field(None, max_length=10) # UserFeedbackType enum value as string
|
||||
user_comment: str | None = Field(None, max_length=MAX_COMMENT_LENGTH)
|
||||
gif_url: str | None = Field(None, max_length=MAX_URL_LENGTH)
|
||||
|
||||
@classmethod
|
||||
def from_agent(cls, agent) -> 'CreateAgentTaskEvent':
|
||||
"""Create a CreateAgentTaskEvent from an Agent instance"""
|
||||
return cls(
|
||||
id=str(agent.task_id),
|
||||
user_id='', # To be filled by cloud handler
|
||||
device_id=agent.cloud_sync.auth_client.device_id
|
||||
if hasattr(agent, 'cloud_sync') and agent.cloud_sync and agent.cloud_sync.auth_client
|
||||
else None,
|
||||
agent_session_id=str(agent.session_id),
|
||||
task=agent.task,
|
||||
llm_model=agent.llm.model_name,
|
||||
agent_state=agent.state.model_dump() if hasattr(agent.state, 'model_dump') else {},
|
||||
stopped=False,
|
||||
paused=False,
|
||||
done_output=None,
|
||||
started_at=datetime.fromtimestamp(agent._task_start_time, tz=timezone.utc),
|
||||
finished_at=None,
|
||||
user_feedback_type=None,
|
||||
user_comment=None,
|
||||
gif_url=None,
|
||||
)
|
||||
|
||||
|
||||
class CreateAgentSessionEvent(BaseEvent):
|
||||
# Model fields
|
||||
id: str = Field(default_factory=uuid7str)
|
||||
user_id: str = Field(max_length=255)
|
||||
device_id: str | None = Field(None, max_length=255) # Device ID for auth lookup
|
||||
browser_session_id: str = Field(max_length=255)
|
||||
browser_session_live_url: str = Field(max_length=MAX_URL_LENGTH)
|
||||
browser_session_cdp_url: str = Field(max_length=MAX_URL_LENGTH)
|
||||
browser_session_stopped: bool = False
|
||||
browser_session_stopped_at: datetime | None = None
|
||||
is_source_api: bool | None = None
|
||||
browser_state: dict = Field(default_factory=dict)
|
||||
browser_session_data: dict | None = None
|
||||
|
||||
@classmethod
|
||||
def from_agent(cls, agent) -> 'CreateAgentSessionEvent':
|
||||
"""Create a CreateAgentSessionEvent from an Agent instance"""
|
||||
return cls(
|
||||
id=str(agent.session_id),
|
||||
user_id='', # To be filled by cloud handler
|
||||
device_id=agent.cloud_sync.auth_client.device_id
|
||||
if hasattr(agent, 'cloud_sync') and agent.cloud_sync and agent.cloud_sync.auth_client
|
||||
else None,
|
||||
browser_session_id=agent.browser_session.id,
|
||||
browser_session_live_url='', # To be filled by cloud handler
|
||||
browser_session_cdp_url='', # To be filled by cloud handler
|
||||
browser_state={
|
||||
'viewport': agent.browser_profile.viewport if agent.browser_profile else {'width': 1280, 'height': 720},
|
||||
'user_agent': agent.browser_profile.user_agent if agent.browser_profile else None,
|
||||
'headless': agent.browser_profile.headless if agent.browser_profile else True,
|
||||
'initial_url': None, # Will be updated during execution
|
||||
'final_url': None, # Will be updated during execution
|
||||
'total_pages_visited': 0, # Will be updated during execution
|
||||
'session_duration_seconds': 0, # Will be updated during execution
|
||||
},
|
||||
browser_session_data={
|
||||
'cookies': [],
|
||||
'secrets': {},
|
||||
# TODO: send secrets safely so tasks can be replayed on cloud seamlessly
|
||||
# 'secrets': dict(agent.sensitive_data) if agent.sensitive_data else {},
|
||||
'allowed_domains': agent.browser_profile.allowed_domains if agent.browser_profile else [],
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class UpdateAgentSessionEvent(BaseEvent):
|
||||
"""Event to update an existing agent session"""
|
||||
|
||||
# Model fields
|
||||
id: str # Session ID to update
|
||||
user_id: str = Field(max_length=255)
|
||||
device_id: str | None = Field(None, max_length=255)
|
||||
browser_session_stopped: bool | None = None
|
||||
browser_session_stopped_at: datetime | None = None
|
||||
end_reason: str | None = Field(None, max_length=100) # Why the session ended
|
||||
@@ -0,0 +1,424 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
import platform
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from browser_use.agent.views import AgentHistoryList
|
||||
from browser_use.browser.views import PLACEHOLDER_4PX_SCREENSHOT
|
||||
from browser_use.config import CONFIG
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from PIL import Image, ImageFont
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def decode_unicode_escapes_to_utf8(text: str) -> str:
|
||||
"""Handle decoding any unicode escape sequences embedded in a string (needed to render non-ASCII languages like chinese or arabic in the GIF overlay text)"""
|
||||
|
||||
if r'\u' not in text:
|
||||
# doesn't have any escape sequences that need to be decoded
|
||||
return text
|
||||
|
||||
try:
|
||||
# Try to decode Unicode escape sequences
|
||||
return text.encode('latin1').decode('unicode_escape')
|
||||
except (UnicodeEncodeError, UnicodeDecodeError):
|
||||
# logger.debug(f"Failed to decode unicode escape sequences while generating gif text: {text}")
|
||||
return text
|
||||
|
||||
|
||||
def create_history_gif(
|
||||
task: str,
|
||||
history: AgentHistoryList,
|
||||
#
|
||||
output_path: str = 'agent_history.gif',
|
||||
duration: int = 3000,
|
||||
show_goals: bool = True,
|
||||
show_task: bool = True,
|
||||
show_logo: bool = False,
|
||||
font_size: int = 40,
|
||||
title_font_size: int = 56,
|
||||
goal_font_size: int = 44,
|
||||
margin: int = 40,
|
||||
line_spacing: float = 1.5,
|
||||
) -> None:
|
||||
"""Create a GIF from the agent's history with overlaid task and goal text."""
|
||||
if not history.history:
|
||||
logger.warning('No history to create GIF from')
|
||||
return
|
||||
|
||||
from PIL import Image, ImageFont
|
||||
|
||||
images = []
|
||||
|
||||
# if history is empty, we can't create a gif
|
||||
if not history.history:
|
||||
logger.warning('No history to create GIF from')
|
||||
return
|
||||
|
||||
# Get all screenshots from history (including None placeholders)
|
||||
screenshots = history.screenshots(return_none_if_not_screenshot=True)
|
||||
|
||||
if not screenshots:
|
||||
logger.warning('No screenshots found in history')
|
||||
return
|
||||
|
||||
# Find the first non-placeholder screenshot
|
||||
# A screenshot is considered a placeholder if:
|
||||
# 1. It's the exact 4px placeholder for about:blank pages, OR
|
||||
# 2. It comes from a new tab page (chrome://newtab/, about:blank, etc.)
|
||||
first_real_screenshot = None
|
||||
for screenshot in screenshots:
|
||||
if screenshot and screenshot != PLACEHOLDER_4PX_SCREENSHOT:
|
||||
first_real_screenshot = screenshot
|
||||
break
|
||||
|
||||
if not first_real_screenshot:
|
||||
logger.warning('No valid screenshots found (all are placeholders or from new tab pages)')
|
||||
return
|
||||
|
||||
# Try to load nicer fonts
|
||||
try:
|
||||
# Try different font options in order of preference
|
||||
# ArialUni is a font that comes with Office and can render most non-alphabet characters
|
||||
font_options = [
|
||||
'PingFang',
|
||||
'STHeiti Medium',
|
||||
'Microsoft YaHei', # 微软雅黑
|
||||
'SimHei', # 黑体
|
||||
'SimSun', # 宋体
|
||||
'Noto Sans CJK SC', # 思源黑体
|
||||
'WenQuanYi Micro Hei', # 文泉驿微米黑
|
||||
'Helvetica',
|
||||
'Arial',
|
||||
'DejaVuSans',
|
||||
'Verdana',
|
||||
]
|
||||
font_loaded = False
|
||||
|
||||
for font_name in font_options:
|
||||
try:
|
||||
if platform.system() == 'Windows':
|
||||
# Need to specify the abs font path on Windows
|
||||
font_name = os.path.join(CONFIG.WIN_FONT_DIR, font_name + '.ttf')
|
||||
regular_font = ImageFont.truetype(font_name, font_size)
|
||||
title_font = ImageFont.truetype(font_name, title_font_size)
|
||||
goal_font = ImageFont.truetype(font_name, goal_font_size)
|
||||
font_loaded = True
|
||||
break
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
if not font_loaded:
|
||||
raise OSError('No preferred fonts found')
|
||||
|
||||
except OSError:
|
||||
regular_font = ImageFont.load_default()
|
||||
title_font = ImageFont.load_default()
|
||||
|
||||
goal_font = regular_font
|
||||
|
||||
# Load logo if requested
|
||||
logo = None
|
||||
if show_logo:
|
||||
try:
|
||||
logo = Image.open('./static/browser-use.png')
|
||||
# Resize logo to be small (e.g., 40px height)
|
||||
logo_height = 150
|
||||
aspect_ratio = logo.width / logo.height
|
||||
logo_width = int(logo_height * aspect_ratio)
|
||||
logo = logo.resize((logo_width, logo_height), Image.Resampling.LANCZOS)
|
||||
except Exception as e:
|
||||
logger.warning(f'Could not load logo: {e}')
|
||||
|
||||
# Create task frame if requested
|
||||
if show_task and task:
|
||||
# Find the first non-placeholder screenshot for the task frame
|
||||
first_real_screenshot = None
|
||||
for item in history.history:
|
||||
screenshot_b64 = item.state.get_screenshot()
|
||||
if screenshot_b64 and screenshot_b64 != PLACEHOLDER_4PX_SCREENSHOT:
|
||||
first_real_screenshot = screenshot_b64
|
||||
break
|
||||
|
||||
if first_real_screenshot:
|
||||
task_frame = _create_task_frame(
|
||||
task,
|
||||
first_real_screenshot,
|
||||
title_font, # type: ignore
|
||||
regular_font, # type: ignore
|
||||
logo,
|
||||
line_spacing,
|
||||
)
|
||||
images.append(task_frame)
|
||||
else:
|
||||
logger.warning('No real screenshots found for task frame, skipping task frame')
|
||||
|
||||
# Process each history item with its corresponding screenshot
|
||||
for i, (item, screenshot) in enumerate(zip(history.history, screenshots), 1):
|
||||
if not screenshot:
|
||||
continue
|
||||
|
||||
# Skip placeholder screenshots from about:blank pages
|
||||
# These are 4x4 white PNGs encoded as a specific base64 string
|
||||
if screenshot == PLACEHOLDER_4PX_SCREENSHOT:
|
||||
logger.debug(f'Skipping placeholder screenshot from about:blank page at step {i}')
|
||||
continue
|
||||
|
||||
# Skip screenshots from new tab pages
|
||||
from browser_use.utils import is_new_tab_page
|
||||
|
||||
if is_new_tab_page(item.state.url):
|
||||
logger.debug(f'Skipping screenshot from new tab page ({item.state.url}) at step {i}')
|
||||
continue
|
||||
|
||||
# Convert base64 screenshot to PIL Image
|
||||
img_data = base64.b64decode(screenshot)
|
||||
image = Image.open(io.BytesIO(img_data))
|
||||
|
||||
if show_goals and item.model_output:
|
||||
image = _add_overlay_to_image(
|
||||
image=image,
|
||||
step_number=i,
|
||||
goal_text=item.model_output.current_state.next_goal,
|
||||
regular_font=regular_font, # type: ignore
|
||||
title_font=title_font, # type: ignore
|
||||
margin=margin,
|
||||
logo=logo,
|
||||
)
|
||||
|
||||
images.append(image)
|
||||
|
||||
if images:
|
||||
# Save the GIF
|
||||
images[0].save(
|
||||
output_path,
|
||||
save_all=True,
|
||||
append_images=images[1:],
|
||||
duration=duration,
|
||||
loop=0,
|
||||
optimize=False,
|
||||
)
|
||||
logger.info(f'Created GIF at {output_path}')
|
||||
else:
|
||||
logger.warning('No images found in history to create GIF')
|
||||
|
||||
|
||||
def _create_task_frame(
|
||||
task: str,
|
||||
first_screenshot: str,
|
||||
title_font: ImageFont.FreeTypeFont,
|
||||
regular_font: ImageFont.FreeTypeFont,
|
||||
logo: Image.Image | None = None,
|
||||
line_spacing: float = 1.5,
|
||||
) -> Image.Image:
|
||||
"""Create initial frame showing the task."""
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
img_data = base64.b64decode(first_screenshot)
|
||||
template = Image.open(io.BytesIO(img_data))
|
||||
image = Image.new('RGB', template.size, (0, 0, 0))
|
||||
draw = ImageDraw.Draw(image)
|
||||
|
||||
# Calculate vertical center of image
|
||||
center_y = image.height // 2
|
||||
|
||||
# Draw task text with dynamic font size based on task length
|
||||
margin = 140 # Increased margin
|
||||
max_width = image.width - (2 * margin)
|
||||
|
||||
# Dynamic font size calculation based on task length
|
||||
# Start with base font size (regular + 16)
|
||||
base_font_size = regular_font.size + 16
|
||||
min_font_size = max(regular_font.size - 10, 16) # Don't go below 16pt
|
||||
max_font_size = base_font_size # Cap at the base font size
|
||||
|
||||
# Calculate dynamic font size based on text length and complexity
|
||||
# Longer texts get progressively smaller fonts
|
||||
text_length = len(task)
|
||||
if text_length > 200:
|
||||
# For very long text, reduce font size logarithmically
|
||||
font_size = max(base_font_size - int(10 * (text_length / 200)), min_font_size)
|
||||
else:
|
||||
font_size = base_font_size
|
||||
|
||||
# Try to create a larger font, but fall back to regular font if it fails
|
||||
try:
|
||||
larger_font = ImageFont.truetype(regular_font.path, font_size) # type: ignore
|
||||
except (OSError, AttributeError):
|
||||
# Fall back to regular font if .path is not available or font loading fails
|
||||
larger_font = regular_font
|
||||
|
||||
# Generate wrapped text with the calculated font size
|
||||
wrapped_text = _wrap_text(task, larger_font, max_width)
|
||||
|
||||
# Calculate line height with spacing
|
||||
line_height = larger_font.size * line_spacing
|
||||
|
||||
# Split text into lines and draw with custom spacing
|
||||
lines = wrapped_text.split('\n')
|
||||
total_height = line_height * len(lines)
|
||||
|
||||
# Start position for first line
|
||||
text_y = center_y - (total_height / 2) + 50 # Shifted down slightly
|
||||
|
||||
for line in lines:
|
||||
# Get line width for centering
|
||||
line_bbox = draw.textbbox((0, 0), line, font=larger_font)
|
||||
text_x = (image.width - (line_bbox[2] - line_bbox[0])) // 2
|
||||
|
||||
draw.text(
|
||||
(text_x, text_y),
|
||||
line,
|
||||
font=larger_font,
|
||||
fill=(255, 255, 255),
|
||||
)
|
||||
text_y += line_height
|
||||
|
||||
# Add logo if provided (top right corner)
|
||||
if logo:
|
||||
logo_margin = 20
|
||||
logo_x = image.width - logo.width - logo_margin
|
||||
image.paste(logo, (logo_x, logo_margin), logo if logo.mode == 'RGBA' else None)
|
||||
|
||||
return image
|
||||
|
||||
|
||||
def _add_overlay_to_image(
|
||||
image: Image.Image,
|
||||
step_number: int,
|
||||
goal_text: str,
|
||||
regular_font: ImageFont.FreeTypeFont,
|
||||
title_font: ImageFont.FreeTypeFont,
|
||||
margin: int,
|
||||
logo: Image.Image | None = None,
|
||||
display_step: bool = True,
|
||||
text_color: tuple[int, int, int, int] = (255, 255, 255, 255),
|
||||
text_box_color: tuple[int, int, int, int] = (0, 0, 0, 255),
|
||||
) -> Image.Image:
|
||||
"""Add step number and goal overlay to an image."""
|
||||
|
||||
from PIL import Image, ImageDraw
|
||||
|
||||
goal_text = decode_unicode_escapes_to_utf8(goal_text)
|
||||
image = image.convert('RGBA')
|
||||
txt_layer = Image.new('RGBA', image.size, (0, 0, 0, 0))
|
||||
draw = ImageDraw.Draw(txt_layer)
|
||||
if display_step:
|
||||
# Add step number (bottom left)
|
||||
step_text = str(step_number)
|
||||
step_bbox = draw.textbbox((0, 0), step_text, font=title_font)
|
||||
step_width = step_bbox[2] - step_bbox[0]
|
||||
step_height = step_bbox[3] - step_bbox[1]
|
||||
|
||||
# Position step number in bottom left
|
||||
x_step = margin + 10 # Slight additional offset from edge
|
||||
y_step = image.height - margin - step_height - 10 # Slight offset from bottom
|
||||
|
||||
# Draw rounded rectangle background for step number
|
||||
padding = 20 # Increased padding
|
||||
step_bg_bbox = (
|
||||
x_step - padding,
|
||||
y_step - padding,
|
||||
x_step + step_width + padding,
|
||||
y_step + step_height + padding,
|
||||
)
|
||||
draw.rounded_rectangle(
|
||||
step_bg_bbox,
|
||||
radius=15, # Add rounded corners
|
||||
fill=text_box_color,
|
||||
)
|
||||
|
||||
# Draw step number
|
||||
draw.text(
|
||||
(x_step, y_step),
|
||||
step_text,
|
||||
font=title_font,
|
||||
fill=text_color,
|
||||
)
|
||||
|
||||
# Draw goal text (centered, bottom)
|
||||
max_width = image.width - (4 * margin)
|
||||
wrapped_goal = _wrap_text(goal_text, title_font, max_width)
|
||||
goal_bbox = draw.multiline_textbbox((0, 0), wrapped_goal, font=title_font)
|
||||
goal_width = goal_bbox[2] - goal_bbox[0]
|
||||
goal_height = goal_bbox[3] - goal_bbox[1]
|
||||
|
||||
# Center goal text horizontally, place above step number
|
||||
x_goal = (image.width - goal_width) // 2
|
||||
y_goal = y_step - goal_height - padding * 4 # More space between step and goal
|
||||
|
||||
# Draw rounded rectangle background for goal
|
||||
padding_goal = 25 # Increased padding for goal
|
||||
goal_bg_bbox = (
|
||||
x_goal - padding_goal, # Remove extra space for logo
|
||||
y_goal - padding_goal,
|
||||
x_goal + goal_width + padding_goal,
|
||||
y_goal + goal_height + padding_goal,
|
||||
)
|
||||
draw.rounded_rectangle(
|
||||
goal_bg_bbox,
|
||||
radius=15, # Add rounded corners
|
||||
fill=text_box_color,
|
||||
)
|
||||
|
||||
# Draw goal text
|
||||
draw.multiline_text(
|
||||
(x_goal, y_goal),
|
||||
wrapped_goal,
|
||||
font=title_font,
|
||||
fill=text_color,
|
||||
align='center',
|
||||
)
|
||||
|
||||
# Add logo if provided (top right corner)
|
||||
if logo:
|
||||
logo_layer = Image.new('RGBA', image.size, (0, 0, 0, 0))
|
||||
logo_margin = 20
|
||||
logo_x = image.width - logo.width - logo_margin
|
||||
logo_layer.paste(logo, (logo_x, logo_margin), logo if logo.mode == 'RGBA' else None)
|
||||
txt_layer = Image.alpha_composite(logo_layer, txt_layer)
|
||||
|
||||
# Composite and convert
|
||||
result = Image.alpha_composite(image, txt_layer)
|
||||
return result.convert('RGB')
|
||||
|
||||
|
||||
def _wrap_text(text: str, font: ImageFont.FreeTypeFont, max_width: int) -> str:
|
||||
"""
|
||||
Wrap text to fit within a given width.
|
||||
|
||||
Args:
|
||||
text: Text to wrap
|
||||
font: Font to use for text
|
||||
max_width: Maximum width in pixels
|
||||
|
||||
Returns:
|
||||
Wrapped text with newlines
|
||||
"""
|
||||
text = decode_unicode_escapes_to_utf8(text)
|
||||
words = text.split()
|
||||
lines = []
|
||||
current_line = []
|
||||
|
||||
for word in words:
|
||||
current_line.append(word)
|
||||
line = ' '.join(current_line)
|
||||
bbox = font.getbbox(line)
|
||||
if bbox[2] > max_width:
|
||||
if len(current_line) == 1:
|
||||
lines.append(current_line.pop())
|
||||
else:
|
||||
current_line.pop()
|
||||
lines.append(' '.join(current_line))
|
||||
current_line = [word]
|
||||
|
||||
if current_line:
|
||||
lines.append(' '.join(current_line))
|
||||
|
||||
return '\n'.join(lines)
|
||||
@@ -0,0 +1,422 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Literal
|
||||
|
||||
from browser_use.agent.message_manager.views import (
|
||||
HistoryItem,
|
||||
)
|
||||
from browser_use.agent.prompts import AgentMessagePrompt
|
||||
from browser_use.agent.views import (
|
||||
ActionResult,
|
||||
AgentOutput,
|
||||
AgentStepInfo,
|
||||
MessageManagerState,
|
||||
)
|
||||
from browser_use.browser.views import BrowserStateSummary
|
||||
from browser_use.filesystem.file_system import FileSystem
|
||||
from browser_use.llm.messages import (
|
||||
BaseMessage,
|
||||
ContentPartImageParam,
|
||||
ContentPartTextParam,
|
||||
SystemMessage,
|
||||
)
|
||||
from browser_use.observability import observe_debug
|
||||
from browser_use.utils import match_url_with_domain_pattern, time_execution_sync
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# ========== Logging Helper Functions ==========
|
||||
# These functions are used ONLY for formatting debug log output.
|
||||
# They do NOT affect the actual message content sent to the LLM.
|
||||
# All logging functions start with _log_ for easy identification.
|
||||
|
||||
|
||||
def _log_get_message_emoji(message: BaseMessage) -> str:
|
||||
"""Get emoji for a message type - used only for logging display"""
|
||||
emoji_map = {
|
||||
'UserMessage': '💬',
|
||||
'SystemMessage': '🧠',
|
||||
'AssistantMessage': '🔨',
|
||||
}
|
||||
return emoji_map.get(message.__class__.__name__, '🎮')
|
||||
|
||||
|
||||
def _log_format_message_line(message: BaseMessage, content: str, is_last_message: bool, terminal_width: int) -> list[str]:
|
||||
"""Format a single message for logging display"""
|
||||
try:
|
||||
lines = []
|
||||
|
||||
# Get emoji and token info
|
||||
emoji = _log_get_message_emoji(message)
|
||||
# token_str = str(message.metadata.tokens).rjust(4)
|
||||
# TODO: fix the token count
|
||||
token_str = '??? (TODO)'
|
||||
prefix = f'{emoji}[{token_str}]: '
|
||||
|
||||
# Calculate available width (emoji=2 visual cols + [token]: =8 chars)
|
||||
content_width = terminal_width - 10
|
||||
|
||||
# Handle last message wrapping
|
||||
if is_last_message and len(content) > content_width:
|
||||
# Find a good break point
|
||||
break_point = content.rfind(' ', 0, content_width)
|
||||
if break_point > content_width * 0.7: # Keep at least 70% of line
|
||||
first_line = content[:break_point]
|
||||
rest = content[break_point + 1 :]
|
||||
else:
|
||||
# No good break point, just truncate
|
||||
first_line = content[:content_width]
|
||||
rest = content[content_width:]
|
||||
|
||||
lines.append(prefix + first_line)
|
||||
|
||||
# Second line with 10-space indent
|
||||
if rest:
|
||||
if len(rest) > terminal_width - 10:
|
||||
rest = rest[: terminal_width - 10]
|
||||
lines.append(' ' * 10 + rest)
|
||||
else:
|
||||
# Single line - truncate if needed
|
||||
if len(content) > content_width:
|
||||
content = content[:content_width]
|
||||
lines.append(prefix + content)
|
||||
|
||||
return lines
|
||||
except Exception as e:
|
||||
logger.warning(f'Failed to format message line for logging: {e}')
|
||||
# Return a simple fallback line
|
||||
return ['❓[ ?]: [Error formatting message]']
|
||||
|
||||
|
||||
# ========== End of Logging Helper Functions ==========
|
||||
|
||||
|
||||
class MessageManager:
|
||||
vision_detail_level: Literal['auto', 'low', 'high']
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
task: str,
|
||||
system_message: SystemMessage,
|
||||
file_system: FileSystem,
|
||||
state: MessageManagerState = MessageManagerState(),
|
||||
use_thinking: bool = True,
|
||||
include_attributes: list[str] | None = None,
|
||||
sensitive_data: dict[str, str | dict[str, str]] | None = None,
|
||||
max_history_items: int | None = None,
|
||||
vision_detail_level: Literal['auto', 'low', 'high'] = 'auto',
|
||||
include_tool_call_examples: bool = False,
|
||||
include_recent_events: bool = False,
|
||||
sample_images: list[ContentPartTextParam | ContentPartImageParam] | None = None,
|
||||
):
|
||||
self.task = task
|
||||
self.state = state
|
||||
self.system_prompt = system_message
|
||||
self.file_system = file_system
|
||||
self.sensitive_data_description = ''
|
||||
self.use_thinking = use_thinking
|
||||
self.max_history_items = max_history_items
|
||||
self.vision_detail_level = vision_detail_level
|
||||
self.include_tool_call_examples = include_tool_call_examples
|
||||
self.include_recent_events = include_recent_events
|
||||
self.sample_images = sample_images
|
||||
|
||||
assert max_history_items is None or max_history_items > 5, 'max_history_items must be None or greater than 5'
|
||||
|
||||
# Store settings as direct attributes instead of in a settings object
|
||||
self.include_attributes = include_attributes or []
|
||||
self.sensitive_data = sensitive_data
|
||||
self.last_input_messages = []
|
||||
# Only initialize messages if state is empty
|
||||
if len(self.state.history.get_messages()) == 0:
|
||||
self._set_message_with_type(self.system_prompt, 'system')
|
||||
|
||||
@property
|
||||
def agent_history_description(self) -> str:
|
||||
"""Build agent history description from list of items, respecting max_history_items limit"""
|
||||
if self.max_history_items is None:
|
||||
# Include all items
|
||||
return '\n'.join(item.to_string() for item in self.state.agent_history_items)
|
||||
|
||||
total_items = len(self.state.agent_history_items)
|
||||
|
||||
# If we have fewer items than the limit, just return all items
|
||||
if total_items <= self.max_history_items:
|
||||
return '\n'.join(item.to_string() for item in self.state.agent_history_items)
|
||||
|
||||
# We have more items than the limit, so we need to omit some
|
||||
omitted_count = total_items - self.max_history_items
|
||||
|
||||
# Show first item + omitted message + most recent (max_history_items - 1) items
|
||||
# The omitted message doesn't count against the limit, only real history items do
|
||||
recent_items_count = self.max_history_items - 1 # -1 for first item
|
||||
|
||||
items_to_include = [
|
||||
self.state.agent_history_items[0].to_string(), # Keep first item (initialization)
|
||||
f'<sys>[... {omitted_count} previous steps omitted...]</sys>',
|
||||
]
|
||||
# Add most recent items
|
||||
items_to_include.extend([item.to_string() for item in self.state.agent_history_items[-recent_items_count:]])
|
||||
|
||||
return '\n'.join(items_to_include)
|
||||
|
||||
def add_new_task(self, new_task: str) -> None:
|
||||
new_task = '<follow_up_user_request> ' + new_task.strip() + ' </follow_up_user_request>'
|
||||
if '<initial_user_request>' not in self.task:
|
||||
self.task = '<initial_user_request>' + self.task + '</initial_user_request>'
|
||||
self.task += '\n' + new_task
|
||||
task_update_item = HistoryItem(system_message=new_task)
|
||||
self.state.agent_history_items.append(task_update_item)
|
||||
|
||||
def _update_agent_history_description(
|
||||
self,
|
||||
model_output: AgentOutput | None = None,
|
||||
result: list[ActionResult] | None = None,
|
||||
step_info: AgentStepInfo | None = None,
|
||||
) -> None:
|
||||
"""Update the agent history description"""
|
||||
|
||||
if result is None:
|
||||
result = []
|
||||
step_number = step_info.step_number if step_info else None
|
||||
|
||||
self.state.read_state_description = ''
|
||||
|
||||
action_results = ''
|
||||
result_len = len(result)
|
||||
read_state_idx = 0
|
||||
for idx, action_result in enumerate(result):
|
||||
if action_result.include_extracted_content_only_once and action_result.extracted_content:
|
||||
self.state.read_state_description += (
|
||||
f'<read_state_{read_state_idx}>\n{action_result.extracted_content}\n</read_state_{read_state_idx}>\n'
|
||||
)
|
||||
read_state_idx += 1
|
||||
logger.debug(f'Added extracted_content to read_state_description: {action_result.extracted_content}')
|
||||
|
||||
if action_result.long_term_memory:
|
||||
action_results += f'{action_result.long_term_memory}\n'
|
||||
logger.debug(f'Added long_term_memory to action_results: {action_result.long_term_memory}')
|
||||
elif action_result.extracted_content and not action_result.include_extracted_content_only_once:
|
||||
action_results += f'{action_result.extracted_content}\n'
|
||||
logger.debug(f'Added extracted_content to action_results: {action_result.extracted_content}')
|
||||
|
||||
if action_result.error:
|
||||
if len(action_result.error) > 200:
|
||||
error_text = action_result.error[:100] + '......' + action_result.error[-100:]
|
||||
else:
|
||||
error_text = action_result.error
|
||||
action_results += f'{error_text}\n'
|
||||
logger.debug(f'Added error to action_results: {error_text}')
|
||||
|
||||
self.state.read_state_description = self.state.read_state_description.strip('\n')
|
||||
|
||||
if action_results:
|
||||
action_results = f'Result:\n{action_results}'
|
||||
action_results = action_results.strip('\n') if action_results else None
|
||||
|
||||
# Build the history item
|
||||
if model_output is None:
|
||||
# Add history item for initial actions (step 0) or errors (step > 0)
|
||||
if step_number is not None:
|
||||
if step_number == 0 and action_results:
|
||||
# Step 0 with initial action results
|
||||
history_item = HistoryItem(step_number=step_number, action_results=action_results)
|
||||
self.state.agent_history_items.append(history_item)
|
||||
elif step_number > 0:
|
||||
# Error case for steps > 0
|
||||
history_item = HistoryItem(step_number=step_number, error='Agent failed to output in the right format.')
|
||||
self.state.agent_history_items.append(history_item)
|
||||
else:
|
||||
history_item = HistoryItem(
|
||||
step_number=step_number,
|
||||
evaluation_previous_goal=model_output.current_state.evaluation_previous_goal,
|
||||
memory=model_output.current_state.memory,
|
||||
next_goal=model_output.current_state.next_goal,
|
||||
action_results=action_results,
|
||||
)
|
||||
self.state.agent_history_items.append(history_item)
|
||||
|
||||
def _get_sensitive_data_description(self, current_page_url) -> str:
|
||||
sensitive_data = self.sensitive_data
|
||||
if not sensitive_data:
|
||||
return ''
|
||||
|
||||
# Collect placeholders for sensitive data
|
||||
placeholders: set[str] = set()
|
||||
|
||||
for key, value in sensitive_data.items():
|
||||
if isinstance(value, dict):
|
||||
# New format: {domain: {key: value}}
|
||||
if current_page_url and match_url_with_domain_pattern(current_page_url, key, True):
|
||||
placeholders.update(value.keys())
|
||||
else:
|
||||
# Old format: {key: value}
|
||||
placeholders.add(key)
|
||||
|
||||
if placeholders:
|
||||
placeholder_list = sorted(list(placeholders))
|
||||
info = f'Here are placeholders for sensitive data:\n{placeholder_list}\n'
|
||||
info += 'To use them, write <secret>the placeholder name</secret>'
|
||||
return info
|
||||
|
||||
return ''
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='create_state_messages')
|
||||
@time_execution_sync('--create_state_messages')
|
||||
def create_state_messages(
|
||||
self,
|
||||
browser_state_summary: BrowserStateSummary,
|
||||
model_output: AgentOutput | None = None,
|
||||
result: list[ActionResult] | None = None,
|
||||
step_info: AgentStepInfo | None = None,
|
||||
use_vision=True,
|
||||
page_filtered_actions: str | None = None,
|
||||
sensitive_data=None,
|
||||
available_file_paths: list[str] | None = None, # Always pass current available_file_paths
|
||||
) -> None:
|
||||
"""Create single state message with all content"""
|
||||
|
||||
# Clear contextual messages from previous steps to prevent accumulation
|
||||
self.state.history.context_messages.clear()
|
||||
|
||||
# First, update the agent history items with the latest step results
|
||||
self._update_agent_history_description(model_output, result, step_info)
|
||||
|
||||
# Use the passed sensitive_data parameter, falling back to instance variable
|
||||
effective_sensitive_data = sensitive_data if sensitive_data is not None else self.sensitive_data
|
||||
if effective_sensitive_data is not None:
|
||||
# Update instance variable to keep it in sync
|
||||
self.sensitive_data = effective_sensitive_data
|
||||
self.sensitive_data_description = self._get_sensitive_data_description(browser_state_summary.url)
|
||||
|
||||
# Use only the current screenshot
|
||||
screenshots = []
|
||||
if browser_state_summary.screenshot:
|
||||
screenshots.append(browser_state_summary.screenshot)
|
||||
|
||||
# Create single state message with all content
|
||||
assert browser_state_summary
|
||||
state_message = AgentMessagePrompt(
|
||||
browser_state_summary=browser_state_summary,
|
||||
file_system=self.file_system,
|
||||
agent_history_description=self.agent_history_description,
|
||||
read_state_description=self.state.read_state_description,
|
||||
task=self.task,
|
||||
include_attributes=self.include_attributes,
|
||||
step_info=step_info,
|
||||
page_filtered_actions=page_filtered_actions,
|
||||
sensitive_data=self.sensitive_data_description,
|
||||
available_file_paths=available_file_paths,
|
||||
screenshots=screenshots,
|
||||
vision_detail_level=self.vision_detail_level,
|
||||
include_recent_events=self.include_recent_events,
|
||||
sample_images=self.sample_images,
|
||||
).get_user_message(use_vision)
|
||||
|
||||
# Set the state message with caching enabled
|
||||
self._set_message_with_type(state_message, 'state')
|
||||
|
||||
def _log_history_lines(self) -> str:
|
||||
"""Generate a formatted log string of message history for debugging / printing to terminal"""
|
||||
# TODO: fix logging
|
||||
|
||||
# try:
|
||||
# total_input_tokens = 0
|
||||
# message_lines = []
|
||||
# terminal_width = shutil.get_terminal_size((80, 20)).columns
|
||||
|
||||
# for i, m in enumerate(self.state.history.messages):
|
||||
# try:
|
||||
# total_input_tokens += m.metadata.tokens
|
||||
# is_last_message = i == len(self.state.history.messages) - 1
|
||||
|
||||
# # Extract content for logging
|
||||
# content = _log_extract_message_content(m.message, is_last_message, m.metadata)
|
||||
|
||||
# # Format the message line(s)
|
||||
# lines = _log_format_message_line(m, content, is_last_message, terminal_width)
|
||||
# message_lines.extend(lines)
|
||||
# except Exception as e:
|
||||
# logger.warning(f'Failed to format message {i} for logging: {e}')
|
||||
# # Add a fallback line for this message
|
||||
# message_lines.append('❓[ ?]: [Error formatting this message]')
|
||||
|
||||
# # Build final log message
|
||||
# return (
|
||||
# f'📜 LLM Message history ({len(self.state.history.messages)} messages, {total_input_tokens} tokens):\n'
|
||||
# + '\n'.join(message_lines)
|
||||
# )
|
||||
# except Exception as e:
|
||||
# logger.warning(f'Failed to generate history log: {e}')
|
||||
# # Return a minimal fallback message
|
||||
# return f'📜 LLM Message history (error generating log: {e})'
|
||||
|
||||
return ''
|
||||
|
||||
@time_execution_sync('--get_messages')
|
||||
def get_messages(self) -> list[BaseMessage]:
|
||||
"""Get current message list, potentially trimmed to max tokens"""
|
||||
|
||||
# Log message history for debugging
|
||||
logger.debug(self._log_history_lines())
|
||||
self.last_input_messages = self.state.history.get_messages()
|
||||
return self.last_input_messages
|
||||
|
||||
def _set_message_with_type(self, message: BaseMessage, message_type: Literal['system', 'state']) -> None:
|
||||
"""Replace a specific state message slot with a new message"""
|
||||
# Don't filter system and state messages - they should contain placeholder tags or normal conversation
|
||||
if message_type == 'system':
|
||||
self.state.history.system_message = message
|
||||
elif message_type == 'state':
|
||||
self.state.history.state_message = message
|
||||
else:
|
||||
raise ValueError(f'Invalid state message type: {message_type}')
|
||||
|
||||
def _add_context_message(self, message: BaseMessage) -> None:
|
||||
"""Add a contextual message specific to this step (e.g., validation errors, retry instructions, timeout warnings)"""
|
||||
# Don't filter context messages - they should contain normal conversation or error messages
|
||||
self.state.history.context_messages.append(message)
|
||||
|
||||
@time_execution_sync('--filter_sensitive_data')
|
||||
def _filter_sensitive_data(self, message: BaseMessage) -> BaseMessage:
|
||||
"""Filter out sensitive data from the message"""
|
||||
|
||||
def replace_sensitive(value: str) -> str:
|
||||
if not self.sensitive_data:
|
||||
return value
|
||||
|
||||
# Collect all sensitive values, immediately converting old format to new format
|
||||
sensitive_values: dict[str, str] = {}
|
||||
|
||||
# Process all sensitive data entries
|
||||
for key_or_domain, content in self.sensitive_data.items():
|
||||
if isinstance(content, dict):
|
||||
# Already in new format: {domain: {key: value}}
|
||||
for key, val in content.items():
|
||||
if val: # Skip empty values
|
||||
sensitive_values[key] = val
|
||||
elif content: # Old format: {key: value} - convert to new format internally
|
||||
# We treat this as if it was {'http*://*': {key_or_domain: content}}
|
||||
sensitive_values[key_or_domain] = content
|
||||
|
||||
# If there are no valid sensitive data entries, just return the original value
|
||||
if not sensitive_values:
|
||||
logger.warning('No valid entries found in sensitive_data dictionary')
|
||||
return value
|
||||
|
||||
# Replace all valid sensitive data values with their placeholder tags
|
||||
for key, val in sensitive_values.items():
|
||||
value = value.replace(val, f'<secret>{key}</secret>')
|
||||
|
||||
return value
|
||||
|
||||
if isinstance(message.content, str):
|
||||
message.content = replace_sensitive(message.content)
|
||||
elif isinstance(message.content, list):
|
||||
for i, item in enumerate(message.content):
|
||||
if isinstance(item, ContentPartTextParam):
|
||||
item.text = replace_sensitive(item.text)
|
||||
message.content[i] = item
|
||||
return message
|
||||
@@ -0,0 +1,52 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import anyio
|
||||
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def save_conversation(
|
||||
input_messages: list[BaseMessage],
|
||||
response: Any,
|
||||
target: str | Path,
|
||||
encoding: str | None = None,
|
||||
) -> None:
|
||||
"""Save conversation history to file asynchronously."""
|
||||
target_path = Path(target)
|
||||
# create folders if not exists
|
||||
if target_path.parent:
|
||||
await anyio.Path(target_path.parent).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
await anyio.Path(target_path).write_text(
|
||||
await _format_conversation(input_messages, response),
|
||||
encoding=encoding or 'utf-8',
|
||||
)
|
||||
|
||||
|
||||
async def _format_conversation(messages: list[BaseMessage], response: Any) -> str:
|
||||
"""Format the conversation including messages and response."""
|
||||
lines = []
|
||||
|
||||
# Format messages
|
||||
for message in messages:
|
||||
lines.append(f' {message.role} ')
|
||||
|
||||
lines.append(message.text)
|
||||
lines.append('') # Empty line after each message
|
||||
|
||||
# Format response
|
||||
lines.append(' RESPONSE')
|
||||
lines.append(json.dumps(json.loads(response.model_dump_json(exclude_unset=True)), indent=2))
|
||||
|
||||
return '\n'.join(lines)
|
||||
|
||||
|
||||
# Note: _write_messages_to_file and _write_response_to_file have been merged into _format_conversation
|
||||
# This is more efficient for async operations and reduces file I/O
|
||||
@@ -0,0 +1,98 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
BaseMessage,
|
||||
)
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
class HistoryItem(BaseModel):
|
||||
"""Represents a single agent history item with its data and string representation"""
|
||||
|
||||
step_number: int | None = None
|
||||
evaluation_previous_goal: str | None = None
|
||||
memory: str | None = None
|
||||
next_goal: str | None = None
|
||||
action_results: str | None = None
|
||||
error: str | None = None
|
||||
system_message: str | None = None
|
||||
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True)
|
||||
|
||||
def model_post_init(self, __context) -> None:
|
||||
"""Validate that error and system_message are not both provided"""
|
||||
if self.error is not None and self.system_message is not None:
|
||||
raise ValueError('Cannot have both error and system_message at the same time')
|
||||
|
||||
def to_string(self) -> str:
|
||||
"""Get string representation of the history item"""
|
||||
step_str = 'step' if self.step_number is not None else 'step_unknown'
|
||||
|
||||
if self.error:
|
||||
return f"""<{step_str}>
|
||||
{self.error}
|
||||
</{step_str}>"""
|
||||
elif self.system_message:
|
||||
return self.system_message
|
||||
else:
|
||||
content_parts = []
|
||||
|
||||
# Only include evaluation_previous_goal if it's not None/empty
|
||||
if self.evaluation_previous_goal:
|
||||
content_parts.append(f'{self.evaluation_previous_goal}')
|
||||
|
||||
# Always include memory
|
||||
if self.memory:
|
||||
content_parts.append(f'{self.memory}')
|
||||
|
||||
# Only include next_goal if it's not None/empty
|
||||
if self.next_goal:
|
||||
content_parts.append(f'{self.next_goal}')
|
||||
|
||||
if self.action_results:
|
||||
content_parts.append(self.action_results)
|
||||
|
||||
content = '\n'.join(content_parts)
|
||||
|
||||
return f"""<{step_str}>
|
||||
{content}
|
||||
</{step_str}>"""
|
||||
|
||||
|
||||
class MessageHistory(BaseModel):
|
||||
"""History of messages"""
|
||||
|
||||
system_message: BaseMessage | None = None
|
||||
state_message: BaseMessage | None = None
|
||||
context_messages: list[BaseMessage] = Field(default_factory=list)
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True)
|
||||
|
||||
def get_messages(self) -> list[BaseMessage]:
|
||||
"""Get all messages in the correct order: system -> state -> contextual"""
|
||||
messages = []
|
||||
if self.system_message:
|
||||
messages.append(self.system_message)
|
||||
if self.state_message:
|
||||
messages.append(self.state_message)
|
||||
messages.extend(self.context_messages)
|
||||
|
||||
return messages
|
||||
|
||||
|
||||
class MessageManagerState(BaseModel):
|
||||
"""Holds the state for MessageManager"""
|
||||
|
||||
history: MessageHistory = Field(default_factory=MessageHistory)
|
||||
tool_id: int = 1
|
||||
agent_history_items: list[HistoryItem] = Field(
|
||||
default_factory=lambda: [HistoryItem(step_number=0, system_message='Agent initialized')]
|
||||
)
|
||||
read_state_description: str = ''
|
||||
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True)
|
||||
@@ -0,0 +1,378 @@
|
||||
import importlib.resources
|
||||
from datetime import datetime
|
||||
from typing import TYPE_CHECKING, Literal, Optional
|
||||
|
||||
from browser_use.dom.views import NodeType, SimplifiedNode
|
||||
from browser_use.llm.messages import ContentPartImageParam, ContentPartTextParam, ImageURL, SystemMessage, UserMessage
|
||||
from browser_use.observability import observe_debug
|
||||
from browser_use.utils import is_new_tab_page
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from browser_use.agent.views import AgentStepInfo
|
||||
from browser_use.browser.views import BrowserStateSummary
|
||||
from browser_use.filesystem.file_system import FileSystem
|
||||
|
||||
|
||||
class SystemPrompt:
|
||||
def __init__(
|
||||
self,
|
||||
action_description: str,
|
||||
max_actions_per_step: int = 10,
|
||||
override_system_message: str | None = None,
|
||||
extend_system_message: str | None = None,
|
||||
use_thinking: bool = True,
|
||||
flash_mode: bool = False,
|
||||
):
|
||||
self.default_action_description = action_description
|
||||
self.max_actions_per_step = max_actions_per_step
|
||||
self.use_thinking = use_thinking
|
||||
self.flash_mode = flash_mode
|
||||
prompt = ''
|
||||
if override_system_message:
|
||||
prompt = override_system_message
|
||||
else:
|
||||
self._load_prompt_template()
|
||||
prompt = self.prompt_template.format(max_actions=self.max_actions_per_step)
|
||||
|
||||
if extend_system_message:
|
||||
prompt += f'\n{extend_system_message}'
|
||||
|
||||
self.system_message = SystemMessage(content=prompt, cache=True)
|
||||
|
||||
def _load_prompt_template(self) -> None:
|
||||
"""Load the prompt template from the markdown file."""
|
||||
try:
|
||||
# Choose the appropriate template based on flash_mode and use_thinking settings
|
||||
if self.flash_mode:
|
||||
template_filename = 'system_prompt_flash.md'
|
||||
elif self.use_thinking:
|
||||
template_filename = 'system_prompt.md'
|
||||
else:
|
||||
template_filename = 'system_prompt_no_thinking.md'
|
||||
|
||||
# This works both in development and when installed as a package
|
||||
with importlib.resources.files('browser_use.agent').joinpath(template_filename).open('r', encoding='utf-8') as f:
|
||||
self.prompt_template = f.read()
|
||||
except Exception as e:
|
||||
raise RuntimeError(f'Failed to load system prompt template: {e}')
|
||||
|
||||
def get_system_message(self) -> SystemMessage:
|
||||
"""
|
||||
Get the system prompt for the agent.
|
||||
|
||||
Returns:
|
||||
SystemMessage: Formatted system prompt
|
||||
"""
|
||||
return self.system_message
|
||||
|
||||
|
||||
# Functions:
|
||||
# {self.default_action_description}
|
||||
|
||||
# Example:
|
||||
# {self.example_response()}
|
||||
# Your AVAILABLE ACTIONS:
|
||||
# {self.default_action_description}
|
||||
|
||||
|
||||
class AgentMessagePrompt:
|
||||
vision_detail_level: Literal['auto', 'low', 'high']
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
browser_state_summary: 'BrowserStateSummary',
|
||||
file_system: 'FileSystem',
|
||||
agent_history_description: str | None = None,
|
||||
read_state_description: str | None = None,
|
||||
task: str | None = None,
|
||||
include_attributes: list[str] | None = None,
|
||||
step_info: Optional['AgentStepInfo'] = None,
|
||||
page_filtered_actions: str | None = None,
|
||||
max_clickable_elements_length: int = 40000,
|
||||
sensitive_data: str | None = None,
|
||||
available_file_paths: list[str] | None = None,
|
||||
screenshots: list[str] | None = None,
|
||||
vision_detail_level: Literal['auto', 'low', 'high'] = 'auto',
|
||||
include_recent_events: bool = False,
|
||||
sample_images: list[ContentPartTextParam | ContentPartImageParam] | None = None,
|
||||
):
|
||||
self.browser_state: 'BrowserStateSummary' = browser_state_summary
|
||||
self.file_system: 'FileSystem | None' = file_system
|
||||
self.agent_history_description: str | None = agent_history_description
|
||||
self.read_state_description: str | None = read_state_description
|
||||
self.task: str | None = task
|
||||
self.include_attributes = include_attributes
|
||||
self.step_info = step_info
|
||||
self.page_filtered_actions: str | None = page_filtered_actions
|
||||
self.max_clickable_elements_length: int = max_clickable_elements_length
|
||||
self.sensitive_data: str | None = sensitive_data
|
||||
self.available_file_paths: list[str] | None = available_file_paths
|
||||
self.screenshots = screenshots or []
|
||||
self.vision_detail_level = vision_detail_level
|
||||
self.include_recent_events = include_recent_events
|
||||
self.sample_images = sample_images or []
|
||||
assert self.browser_state
|
||||
|
||||
def _extract_page_statistics(self) -> dict[str, int]:
|
||||
"""Extract high-level page statistics from DOM tree for LLM context"""
|
||||
stats = {
|
||||
'links': 0,
|
||||
'iframes': 0,
|
||||
'shadow_open': 0,
|
||||
'shadow_closed': 0,
|
||||
'scroll_containers': 0,
|
||||
'images': 0,
|
||||
'interactive_elements': 0,
|
||||
'total_elements': 0,
|
||||
}
|
||||
|
||||
if not self.browser_state.dom_state or not self.browser_state.dom_state._root:
|
||||
return stats
|
||||
|
||||
def traverse_node(node: SimplifiedNode) -> None:
|
||||
"""Recursively traverse simplified DOM tree to count elements"""
|
||||
if not node or not node.original_node:
|
||||
return
|
||||
|
||||
original = node.original_node
|
||||
stats['total_elements'] += 1
|
||||
|
||||
# Count by node type and tag
|
||||
if original.node_type == NodeType.ELEMENT_NODE:
|
||||
tag = original.tag_name.lower() if original.tag_name else ''
|
||||
|
||||
if tag == 'a':
|
||||
stats['links'] += 1
|
||||
elif tag in ('iframe', 'frame'):
|
||||
stats['iframes'] += 1
|
||||
elif tag == 'img':
|
||||
stats['images'] += 1
|
||||
|
||||
# Check if scrollable
|
||||
if original.is_actually_scrollable:
|
||||
stats['scroll_containers'] += 1
|
||||
|
||||
# Check if interactive
|
||||
if node.interactive_index is not None:
|
||||
stats['interactive_elements'] += 1
|
||||
|
||||
# Check if this element hosts shadow DOM
|
||||
if node.is_shadow_host:
|
||||
# Check if any shadow children are closed
|
||||
has_closed_shadow = any(
|
||||
child.original_node.node_type == NodeType.DOCUMENT_FRAGMENT_NODE
|
||||
and child.original_node.shadow_root_type
|
||||
and child.original_node.shadow_root_type.lower() == 'closed'
|
||||
for child in node.children
|
||||
)
|
||||
if has_closed_shadow:
|
||||
stats['shadow_closed'] += 1
|
||||
else:
|
||||
stats['shadow_open'] += 1
|
||||
|
||||
elif original.node_type == NodeType.DOCUMENT_FRAGMENT_NODE:
|
||||
# Shadow DOM fragment - these are the actual shadow roots
|
||||
# But don't double-count since we count them at the host level above
|
||||
pass
|
||||
|
||||
# Traverse children
|
||||
for child in node.children:
|
||||
traverse_node(child)
|
||||
|
||||
traverse_node(self.browser_state.dom_state._root)
|
||||
return stats
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='_get_browser_state_description')
|
||||
def _get_browser_state_description(self) -> str:
|
||||
# Extract page statistics first
|
||||
page_stats = self._extract_page_statistics()
|
||||
|
||||
# Format statistics for LLM
|
||||
stats_text = '<page_stats>'
|
||||
if page_stats['total_elements'] < 10:
|
||||
stats_text += 'Page appears empty (SPA not loaded?) - '
|
||||
stats_text += f'{page_stats["links"]} links, {page_stats["interactive_elements"]} interactive, '
|
||||
stats_text += f'{page_stats["iframes"]} iframes, {page_stats["scroll_containers"]} scroll containers'
|
||||
if page_stats['shadow_open'] > 0 or page_stats['shadow_closed'] > 0:
|
||||
stats_text += f', {page_stats["shadow_open"]} shadow(open), {page_stats["shadow_closed"]} shadow(closed)'
|
||||
if page_stats['images'] > 0:
|
||||
stats_text += f', {page_stats["images"]} images'
|
||||
stats_text += f', {page_stats["total_elements"]} total elements'
|
||||
stats_text += '</page_stats>\n\n'
|
||||
|
||||
elements_text = self.browser_state.dom_state.llm_representation(include_attributes=self.include_attributes)
|
||||
|
||||
if len(elements_text) > self.max_clickable_elements_length:
|
||||
elements_text = elements_text[: self.max_clickable_elements_length]
|
||||
truncated_text = f' (truncated to {self.max_clickable_elements_length} characters)'
|
||||
else:
|
||||
truncated_text = ''
|
||||
|
||||
has_content_above = False
|
||||
has_content_below = False
|
||||
# Enhanced page information for the model
|
||||
page_info_text = ''
|
||||
if self.browser_state.page_info:
|
||||
pi = self.browser_state.page_info
|
||||
# Compute page statistics dynamically
|
||||
pages_above = pi.pixels_above / pi.viewport_height if pi.viewport_height > 0 else 0
|
||||
pages_below = pi.pixels_below / pi.viewport_height if pi.viewport_height > 0 else 0
|
||||
has_content_above = pages_above > 0
|
||||
has_content_below = pages_below > 0
|
||||
total_pages = pi.page_height / pi.viewport_height if pi.viewport_height > 0 else 0
|
||||
current_page_position = pi.scroll_y / max(pi.page_height - pi.viewport_height, 1)
|
||||
page_info_text = '<page_info>'
|
||||
page_info_text += f'{pages_above:.1f} pages above, '
|
||||
page_info_text += f'{pages_below:.1f} pages below, '
|
||||
page_info_text += f'{total_pages:.1f} total pages'
|
||||
page_info_text += '</page_info>\n'
|
||||
# , at {current_page_position:.0%} of page
|
||||
if elements_text != '':
|
||||
if has_content_above:
|
||||
if self.browser_state.page_info:
|
||||
pi = self.browser_state.page_info
|
||||
pages_above = pi.pixels_above / pi.viewport_height if pi.viewport_height > 0 else 0
|
||||
elements_text = f'... {pages_above:.1f} pages above - scroll to see more or extract structured data if you are looking for specific information ...\n{elements_text}'
|
||||
else:
|
||||
elements_text = f'[Start of page]\n{elements_text}'
|
||||
if has_content_below:
|
||||
if self.browser_state.page_info:
|
||||
pi = self.browser_state.page_info
|
||||
pages_below = pi.pixels_below / pi.viewport_height if pi.viewport_height > 0 else 0
|
||||
elements_text = f'{elements_text}\n... {pages_below:.1f} pages below - scroll to see more or extract structured data if you are looking for specific information ...'
|
||||
else:
|
||||
elements_text = f'{elements_text}\n[End of page]'
|
||||
else:
|
||||
elements_text = 'empty page'
|
||||
|
||||
tabs_text = ''
|
||||
current_tab_candidates = []
|
||||
|
||||
# Find tabs that match both URL and title to identify current tab more reliably
|
||||
for tab in self.browser_state.tabs:
|
||||
if tab.url == self.browser_state.url and tab.title == self.browser_state.title:
|
||||
current_tab_candidates.append(tab.target_id)
|
||||
|
||||
# If we have exactly one match, mark it as current
|
||||
# Otherwise, don't mark any tab as current to avoid confusion
|
||||
current_target_id = current_tab_candidates[0] if len(current_tab_candidates) == 1 else None
|
||||
|
||||
for tab in self.browser_state.tabs:
|
||||
tabs_text += f'Tab {tab.target_id[-4:]}: {tab.url} - {tab.title[:30]}\n'
|
||||
|
||||
current_tab_text = f'Current tab: {current_target_id[-4:]}' if current_target_id is not None else ''
|
||||
|
||||
# Check if current page is a PDF viewer and add appropriate message
|
||||
pdf_message = ''
|
||||
if self.browser_state.is_pdf_viewer:
|
||||
pdf_message = 'PDF viewer cannot be rendered. In this page, DO NOT use the extract_structured_data action as PDF content cannot be rendered. Use the read_file action on the downloaded PDF in available_file_paths to read the full content.\n\n'
|
||||
|
||||
# Add recent events if available and requested
|
||||
recent_events_text = ''
|
||||
if self.include_recent_events and self.browser_state.recent_events:
|
||||
recent_events_text = f'Recent browser events: {self.browser_state.recent_events}\n'
|
||||
|
||||
browser_state = f"""{stats_text}{current_tab_text}
|
||||
Available tabs:
|
||||
{tabs_text}
|
||||
{page_info_text}
|
||||
{recent_events_text}{pdf_message}Elements you can interact with inside the viewport{truncated_text}:
|
||||
{elements_text}
|
||||
"""
|
||||
return browser_state
|
||||
|
||||
def _get_agent_state_description(self) -> str:
|
||||
if self.step_info:
|
||||
step_info_description = f'Step {self.step_info.step_number + 1}. Maximum steps: {self.step_info.max_steps}\n'
|
||||
else:
|
||||
step_info_description = ''
|
||||
|
||||
time_str = datetime.now().strftime('%Y-%m-%d')
|
||||
step_info_description += f'Current date: {time_str}'
|
||||
|
||||
_todo_contents = self.file_system.get_todo_contents() if self.file_system else ''
|
||||
if not len(_todo_contents):
|
||||
_todo_contents = '[Current todo.md is empty, fill it with your plan when applicable]'
|
||||
|
||||
agent_state = f"""
|
||||
<user_request>
|
||||
{self.task}
|
||||
</user_request>
|
||||
<file_system>
|
||||
{self.file_system.describe() if self.file_system else 'No file system available'}
|
||||
</file_system>
|
||||
<todo_contents>
|
||||
{_todo_contents}
|
||||
</todo_contents>
|
||||
"""
|
||||
if self.sensitive_data:
|
||||
agent_state += f'<sensitive_data>\n{self.sensitive_data}\n</sensitive_data>\n'
|
||||
|
||||
agent_state += f'<step_info>\n{step_info_description}\n</step_info>\n'
|
||||
if self.available_file_paths:
|
||||
available_file_paths_text = '\n'.join(self.available_file_paths)
|
||||
agent_state += f'<available_file_paths>\n{available_file_paths_text}\nUse absolute full paths when referencing these files.\n</available_file_paths>\n'
|
||||
return agent_state
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='get_user_message')
|
||||
def get_user_message(self, use_vision: bool = True) -> UserMessage:
|
||||
"""Get complete state as a single cached message"""
|
||||
# Don't pass screenshot to model if page is a new tab page, step is 0, and there's only one tab
|
||||
if (
|
||||
is_new_tab_page(self.browser_state.url)
|
||||
and self.step_info is not None
|
||||
and self.step_info.step_number == 0
|
||||
and len(self.browser_state.tabs) == 1
|
||||
):
|
||||
use_vision = False
|
||||
|
||||
# Build complete state description
|
||||
state_description = (
|
||||
'<agent_history>\n'
|
||||
+ (self.agent_history_description.strip('\n') if self.agent_history_description else '')
|
||||
+ '\n</agent_history>\n\n'
|
||||
)
|
||||
state_description += '<agent_state>\n' + self._get_agent_state_description().strip('\n') + '\n</agent_state>\n'
|
||||
state_description += '<browser_state>\n' + self._get_browser_state_description().strip('\n') + '\n</browser_state>\n'
|
||||
# Only add read_state if it has content
|
||||
read_state_description = self.read_state_description.strip('\n').strip() if self.read_state_description else ''
|
||||
if read_state_description:
|
||||
state_description += '<read_state>\n' + read_state_description + '\n</read_state>\n'
|
||||
|
||||
if self.page_filtered_actions:
|
||||
state_description += '<page_specific_actions>\n'
|
||||
state_description += self.page_filtered_actions + '\n'
|
||||
state_description += '</page_specific_actions>\n'
|
||||
|
||||
if use_vision is True and self.screenshots:
|
||||
# Start with text description
|
||||
content_parts: list[ContentPartTextParam | ContentPartImageParam] = [ContentPartTextParam(text=state_description)]
|
||||
|
||||
# Add sample images
|
||||
content_parts.extend(self.sample_images)
|
||||
|
||||
# Add screenshots with labels
|
||||
for i, screenshot in enumerate(self.screenshots):
|
||||
if i == len(self.screenshots) - 1:
|
||||
label = 'Current screenshot:'
|
||||
else:
|
||||
# Use simple, accurate labeling since we don't have actual step timing info
|
||||
label = 'Previous screenshot:'
|
||||
|
||||
# Add label as text content
|
||||
content_parts.append(ContentPartTextParam(text=label))
|
||||
|
||||
# Add the screenshot
|
||||
content_parts.append(
|
||||
ContentPartImageParam(
|
||||
image_url=ImageURL(
|
||||
url=f'data:image/png;base64,{screenshot}',
|
||||
media_type='image/png',
|
||||
detail=self.vision_detail_level,
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
return UserMessage(content=content_parts, cache=True)
|
||||
|
||||
return UserMessage(content=state_description, cache=True)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,216 @@
|
||||
You are an AI agent designed to operate in an iterative loop to automate browser tasks. Your ultimate goal is accomplishing the task provided in <user_request>.
|
||||
|
||||
<intro>
|
||||
You excel at following tasks:
|
||||
1. Navigating complex websites and extracting precise information
|
||||
2. Automating form submissions and interactive web actions
|
||||
3. Gathering and saving information
|
||||
4. Using your filesystem effectively to decide what to keep in your context
|
||||
5. Operate effectively in an agent loop
|
||||
6. Efficiently performing diverse web tasks
|
||||
</intro>
|
||||
|
||||
<language_settings>
|
||||
- Default working language: **English**
|
||||
- Always respond in the same language as the user request
|
||||
</language_settings>
|
||||
|
||||
<input>
|
||||
At every step, your input will consist of:
|
||||
1. <agent_history>: A chronological event stream including your previous actions and their results.
|
||||
2. <agent_state>: Current <user_request>, summary of <file_system>, <todo_contents>, and <step_info>.
|
||||
3. <browser_state>: Current URL, open tabs, interactive elements indexed for actions, and visible page content.
|
||||
4. <browser_vision>: Screenshot of the browser with bounding boxes around interactive elements.
|
||||
5. <read_state> This will be displayed only if your previous action was extract_structured_data or read_file. This data is only shown in the current step.
|
||||
</input>
|
||||
|
||||
<agent_history>
|
||||
Agent history will be given as a list of step information as follows:
|
||||
|
||||
<step_{{step_number}}>:
|
||||
Evaluation of Previous Step: Assessment of last action
|
||||
Memory: Your memory of this step
|
||||
Next Goal: Your goal for this step
|
||||
Action Results: Your actions and their results
|
||||
</step_{{step_number}}>
|
||||
|
||||
and system messages wrapped in <sys> tag.
|
||||
</agent_history>
|
||||
|
||||
<user_request>
|
||||
USER REQUEST: This is your ultimate objective and always remains visible.
|
||||
- This has the highest priority. Make the user happy.
|
||||
- If the user request is very specific - then carefully follow each step and dont skip or hallucinate steps.
|
||||
- If the task is open ended you can plan yourself how to get it done.
|
||||
</user_request>
|
||||
|
||||
<browser_state>
|
||||
1. Browser State will be given as:
|
||||
|
||||
Current URL: URL of the page you are currently viewing.
|
||||
Open Tabs: Open tabs with their indexes.
|
||||
Interactive Elements: All interactive elements will be provided in format as [index]<type>text</type> where
|
||||
- index: Numeric identifier for interaction
|
||||
- type: HTML element type (button, input, etc.)
|
||||
- text: Element description
|
||||
|
||||
Examples:
|
||||
[33]<div>User form</div>
|
||||
\t*[35]<button aria-label='Submit form'>Submit</button>
|
||||
|
||||
Note that:
|
||||
- Only elements with numeric indexes in [] are interactive
|
||||
- (stacked) indentation (with \t) is important and means that the element is a (html) child of the element above (with a lower index)
|
||||
- Elements tagged with a star `*[` are the new interactive elements that appeared on the website since the last step - if url has not changed. Your previous actions caused that change. Think if you need to interact with them, e.g. after input_text you might need to select the right option from the list.
|
||||
- Pure text elements without [] are not interactive.
|
||||
</browser_state>
|
||||
|
||||
<browser_vision>
|
||||
You will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
|
||||
If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
|
||||
</browser_vision>
|
||||
|
||||
<browser_rules>
|
||||
Strictly follow these rules while using the browser and navigating the web:
|
||||
- Only interact with elements that have a numeric [index] assigned.
|
||||
- Only use indexes that are explicitly provided.
|
||||
- If research is needed, open a **new tab** instead of reusing the current one.
|
||||
- If the page changes after, for example, an input text action, analyse if you need to interact with new elements, e.g. selecting the right option from the list.
|
||||
- By default, only elements in the visible viewport are listed. Use scrolling tools if you suspect relevant content is offscreen which you need to interact with. Scroll ONLY if there are more pixels below or above the page.
|
||||
- You can scroll by a specific number of pages using the num_pages parameter (e.g., 0.5 for half page, 2.0 for two pages).
|
||||
- If a captcha appears, attempt solving it if possible. If not, use fallback strategies (e.g., alternative site, backtrack).
|
||||
- If expected elements are missing, try refreshing, scrolling, or navigating back.
|
||||
- If the page is not fully loaded, use the wait action.
|
||||
- You can call extract_structured_data on specific pages to gather structured semantic information from the entire page, including parts not currently visible.
|
||||
- Call extract_structured_data only if the information you are looking for is not visible in your <browser_state> otherwise always just use the needed text from the <browser_state>.
|
||||
- Calling the extract_structured_data tool is expensive! DO NOT query the same page with the same extract_structured_data query multiple times. Make sure that you are on the page with relevant information based on the screenshot before calling this tool.
|
||||
- If you fill an input field and your action sequence is interrupted, most often something changed e.g. suggestions popped up under the field.
|
||||
- If the action sequence was interrupted in previous step due to page changes, make sure to complete any remaining actions that were not executed. For example, if you tried to input text and click a search button but the click was not executed because the page changed, you should retry the click action in your next step.
|
||||
- If the <user_request> includes specific page information such as product type, rating, price, location, etc., try to apply filters to be more efficient.
|
||||
- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.
|
||||
- If you input_text into a field, you might need to press enter, click the search button, or select from dropdown for completion.
|
||||
- Don't login into a page if you don't have to. Don't login if you don't have the credentials.
|
||||
- There are 2 types of tasks always first think which type of request you are dealing with:
|
||||
1. Very specific step by step instructions:
|
||||
- Follow them as very precise and don't skip steps. Try to complete everything as requested.
|
||||
2. Open ended tasks. Plan yourself, be creative in achieving them.
|
||||
- If you get stuck e.g. with logins or captcha in open-ended tasks you can re-evaluate the task and try alternative ways, e.g. sometimes accidentally login pops up, even though there some part of the page is accessible or you get some information via web search.
|
||||
- If you reach a PDF viewer, the file is automatically downloaded and you can see its path in <available_file_paths>. You can either read the file or scroll in the page to see more.
|
||||
</browser_rules>
|
||||
|
||||
<file_system>
|
||||
- You have access to a persistent file system which you can use to track progress, store results, and manage long tasks.
|
||||
- Your file system is initialized with a `todo.md`: Use this to keep a checklist for known subtasks. Use `replace_file_str` tool to update markers in `todo.md` as first action whenever you complete an item. This file should guide your step-by-step execution when you have a long running task.
|
||||
- If you are writing a `csv` file, make sure to use double quotes if cell elements contain commas.
|
||||
- If the file is too large, you are only given a preview of your file. Use `read_file` to see the full content if necessary.
|
||||
- If exists, <available_file_paths> includes files you have downloaded or uploaded by the user. You can only read or upload these files but you don't have write access.
|
||||
- If the task is really long, initialize a `results.md` file to accumulate your results.
|
||||
- DO NOT use the file system if the task is less than 10 steps!
|
||||
</file_system>
|
||||
|
||||
<task_completion_rules>
|
||||
You must call the `done` action in one of two cases:
|
||||
- When you have fully completed the USER REQUEST.
|
||||
- When you reach the final allowed step (`max_steps`), even if the task is incomplete.
|
||||
- If it is ABSOLUTELY IMPOSSIBLE to continue.
|
||||
|
||||
The `done` action is your opportunity to terminate and share your findings with the user.
|
||||
- Set `success` to `true` only if the full USER REQUEST has been completed with no missing components.
|
||||
- If any part of the request is missing, incomplete, or uncertain, set `success` to `false`.
|
||||
- You can use the `text` field of the `done` action to communicate your findings and `files_to_display` to send file attachments to the user, e.g. `["results.md"]`.
|
||||
- Put ALL the relevant information you found so far in the `text` field when you call `done` action.
|
||||
- Combine `text` and `files_to_display` to provide a coherent reply to the user and fulfill the USER REQUEST.
|
||||
- You are ONLY ALLOWED to call `done` as a single action. Don't call it together with other actions.
|
||||
- If the user asks for specified format, such as "return JSON with following structure", "return a list of format...", MAKE sure to use the right format in your answer.
|
||||
- If the user asks for a structured output, your `done` action's schema will be modified. Take this schema into account when solving the task!
|
||||
</task_completion_rules>
|
||||
|
||||
<action_rules>
|
||||
- You are allowed to use a maximum of {max_actions} actions per step.
|
||||
|
||||
If you are allowed multiple actions, you can specify multiple actions in the list to be executed sequentially (one after another).
|
||||
- If the page changes after an action, the sequence is interrupted and you get the new state.
|
||||
</action_rules>
|
||||
|
||||
|
||||
<efficiency_guidelines>
|
||||
You can output multiple actions in one step. Try to be efficient where it makes sense. Do not predict actions which do not make sense for the current page.
|
||||
|
||||
**Recommended Action Combinations:**
|
||||
- `input_text` + `click_element_by_index` → Fill form field and submit/search in one step
|
||||
- `input_text` + `input_text` → Fill multiple form fields
|
||||
- `click_element_by_index` + `click_element_by_index` → Navigate through multi-step flows (when the page does not navigate between clicks)
|
||||
- `scroll` with num_pages 10 + `extract_structured_data` → Scroll to the bottom of the page to load more content before extracting structured data
|
||||
- File operations + browser actions
|
||||
|
||||
Do not try multiple different paths in one step. Always have one clear goal per step.
|
||||
Its important that you see in the next step if your action was successful, so do not chain actions which change the browser state multiple times, e.g.
|
||||
- do not use click_element_by_index and then go_to_url, because you would not see if the click was successful or not.
|
||||
- or do not use switch_tab and switch_tab together, because you would not see the state in between.
|
||||
- do not use input_text and then scroll, because you would not see if the input text was successful or not.
|
||||
</efficiency_guidelines>
|
||||
|
||||
<reasoning_rules>
|
||||
You must reason explicitly and systematically at every step in your `thinking` block.
|
||||
|
||||
Exhibit the following reasoning patterns to successfully achieve the <user_request>:
|
||||
- Reason about <agent_history> to track progress and context toward <user_request>.
|
||||
- Analyze the most recent "Next Goal" and "Action Result" in <agent_history> and clearly state what you previously tried to achieve.
|
||||
- Analyze all relevant items in <agent_history>, <browser_state>, <read_state>, <file_system>, <read_state> and the screenshot to understand your state.
|
||||
- Explicitly judge success/failure/uncertainty of the last action. Never assume an action succeeded just because it appears to be executed in your last step in <agent_history>. For example, you might have "Action 1/1: Input '2025-05-05' into element 3." in your history even though inputting text failed. Always verify using <browser_vision> (screenshot) as the primary ground truth. If a screenshot is unavailable, fall back to <browser_state>. If the expected change is missing, mark the last action as failed (or uncertain) and plan a recovery.
|
||||
- If todo.md is empty and the task is multi-step, generate a stepwise plan in todo.md using file tools.
|
||||
- Analyze `todo.md` to guide and track your progress.
|
||||
- If any todo.md items are finished, mark them as complete in the file.
|
||||
- Analyze whether you are stuck, e.g. when you repeat the same actions multiple times without any progress. Then consider alternative approaches e.g. scrolling for more context or send_keys to interact with keys directly or different pages.
|
||||
- Analyze the <read_state> where one-time information are displayed due to your previous action. Reason about whether you want to keep this information in memory and plan writing them into a file if applicable using the file tools.
|
||||
- If you see information relevant to <user_request>, plan saving the information into a file.
|
||||
- Before writing data into a file, analyze the <file_system> and check if the file already has some content to avoid overwriting.
|
||||
- Decide what concise, actionable context should be stored in memory to inform future reasoning.
|
||||
- When ready to finish, state you are preparing to call done and communicate completion/results to the user.
|
||||
- Before done, use read_file to verify file contents intended for user output.
|
||||
- Always reason about the <user_request>. Make sure to carefully analyze the specific steps and information required. E.g. specific filters, specific form fields, specific information to search. Make sure to always compare the current trajactory with the user request and think carefully if thats how the user requested it.
|
||||
</reasoning_rules>
|
||||
|
||||
<examples>
|
||||
Here are examples of good output patterns. Use them as reference but never copy them directly.
|
||||
|
||||
<todo_examples>
|
||||
"write_file": {{
|
||||
"file_name": "todo.md",
|
||||
"content": "# ArXiv CS.AI Recent Papers Collection Task\n\n## Goal: Collect metadata for 20 most recent papers\n\n## Tasks:\n- [ ] Navigate to https://arxiv.org/list/cs.AI/recent\n- [ ] Initialize papers.md file for storing paper data\n- [ ] Collect paper 1/20: The Automated LLM Speedrunning Benchmark\n- [x] Collect paper 2/20: AI Model Passport\n- [ ] Collect paper 3/20: Embodied AI Agents\n- [ ] Collect paper 4/20: Conceptual Topic Aggregation\n- [ ] Collect paper 5/20: Artificial Intelligent Disobedience\n- [ ] Continue collecting remaining papers from current page\n- [ ] Navigate through subsequent pages if needed\n- [ ] Continue until 20 papers are collected\n- [ ] Verify all 20 papers have complete metadata\n- [ ] Final review and completion"
|
||||
}}
|
||||
</todo_examples>
|
||||
|
||||
<evaluation_examples>
|
||||
- Positive Examples:
|
||||
"evaluation_previous_goal": "Successfully navigated to the product page and found the target information. Verdict: Success"
|
||||
"evaluation_previous_goal": "Clicked the login button and user authentication form appeared. Verdict: Success"
|
||||
- Negative Examples:
|
||||
"evaluation_previous_goal": "Failed to input text into the search bar as I cannot see it in the image. Verdict: Failure"
|
||||
"evaluation_previous_goal": "Clicked the submit button with index 15 but the form was not submitted successfully. Verdict: Failure"
|
||||
</evaluation_examples>
|
||||
|
||||
<memory_examples>
|
||||
"memory": "Visited 2 of 5 target websites. Collected pricing data from Amazon ($39.99) and eBay ($42.00). Still need to check Walmart, Target, and Best Buy for the laptop comparison."
|
||||
"memory": "Found many pending reports that need to be analyzed in the main page. Successfully processed the first 2 reports on quarterly sales data and moving on to inventory analysis and customer feedback reports."
|
||||
</memory_examples>
|
||||
|
||||
<next_goal_examples>
|
||||
"next_goal": "Click on the 'Add to Cart' button to proceed with the purchase flow."
|
||||
"next_goal": "Extract details from the first item on the page."
|
||||
</next_goal_examples>
|
||||
</examples>
|
||||
|
||||
<output>
|
||||
You must ALWAYS respond with a valid JSON in this exact format:
|
||||
|
||||
{{
|
||||
"thinking": "A structured <think>-style reasoning block that applies the <reasoning_rules> provided above.",
|
||||
"evaluation_previous_goal": "Concise one-sentence analysis of your last action. Clearly state success, failure, or uncertain.",
|
||||
"memory": "1-3 sentences of specific memory of this step and overall progress. You should put here everything that will help you track progress in future steps. Like counting pages visited, items found, etc.",
|
||||
"next_goal": "State the next immediate goal and action to achieve it, in one clear sentence."
|
||||
"action":[{{"go_to_url": {{ "url": "url_value"}}}}, // ... more actions in sequence]
|
||||
}}
|
||||
|
||||
Action list should NEVER be empty.
|
||||
</output>
|
||||
@@ -0,0 +1,177 @@
|
||||
You are an AI agent designed to operate in an iterative loop to automate browser tasks. Your ultimate goal is accomplishing the task provided in <user_request>.
|
||||
|
||||
<intro>
|
||||
You excel at following tasks:
|
||||
1. Navigating complex websites and extracting precise information
|
||||
2. Automating form submissions and interactive web actions
|
||||
3. Gathering and saving information
|
||||
4. Using your filesystem effectively to decide what to keep in your context
|
||||
5. Operate effectively in an agent loop
|
||||
6. Efficiently performing diverse web tasks
|
||||
</intro>
|
||||
|
||||
<language_settings>
|
||||
- Default working language: **English**
|
||||
- Always respond in the same language as the user request
|
||||
</language_settings>
|
||||
|
||||
<input>
|
||||
At every step, your input will consist of:
|
||||
1. <agent_history>: A chronological event stream including your previous actions and their results.
|
||||
2. <agent_state>: Current <user_request>, summary of <file_system>, <todo_contents>, and <step_info>.
|
||||
3. <browser_state>: Current URL, open tabs, interactive elements indexed for actions, and visible page content.
|
||||
4. <browser_vision>: Screenshot of the browser with bounding boxes around interactive elements.
|
||||
5. <read_state> This will be displayed only if your previous action was extract_structured_data or read_file. This data is only shown in the current step.
|
||||
</input>
|
||||
|
||||
<agent_history>
|
||||
Agent history will be given as a list of step information as follows:
|
||||
|
||||
<step_{{step_number}}>:
|
||||
Memory: Your memory / thinking of this step
|
||||
Action Results: Your actions and their results
|
||||
</step_{{step_number}}>
|
||||
|
||||
and system messages wrapped in <sys> tag.
|
||||
</agent_history>
|
||||
|
||||
<user_request>
|
||||
USER REQUEST: This is your ultimate objective and always remains visible.
|
||||
- This has the highest priority. Make the user happy.
|
||||
- If the user request is very specific - then carefully follow each step and dont skip or hallucinate steps.
|
||||
- If the task is open ended you can plan yourself how to get it done.
|
||||
</user_request>
|
||||
|
||||
<browser_state>
|
||||
1. Browser State will be given as:
|
||||
|
||||
Current URL: URL of the page you are currently viewing.
|
||||
Open Tabs: Open tabs with their indexes.
|
||||
Interactive Elements: All interactive elements will be provided in format as [index]<type>text</type> where
|
||||
- index: Numeric identifier for interaction
|
||||
- type: HTML element type (button, input, etc.)
|
||||
- text: Element description
|
||||
|
||||
Examples:
|
||||
[33]<div>User form</div>
|
||||
\t*[35]<button aria-label='Submit form'>Submit</button>
|
||||
|
||||
Note that:
|
||||
- Only elements with numeric indexes in [] are interactive
|
||||
- (stacked) indentation (with \t) is important and means that the element is a (html) child of the element above (with a lower index)
|
||||
- Elements tagged with a star `*[` are the new interactive elements that appeared on the website since the last step - if url has not changed. Your previous actions caused that change. Think if you need to interact with them, e.g. after input_text you might need to select the right option from the list.
|
||||
- Pure text elements without [] are not interactive.
|
||||
</browser_state>
|
||||
|
||||
<browser_vision>
|
||||
You will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
|
||||
If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
|
||||
</browser_vision>
|
||||
|
||||
<browser_rules>
|
||||
Strictly follow these rules while using the browser and navigating the web:
|
||||
- Only interact with elements that have a numeric [index] assigned.
|
||||
- Only use indexes that are explicitly provided.
|
||||
- If research is needed, open a **new tab** instead of reusing the current one.
|
||||
- If the page changes after, for example, an input text action, analyse if you need to interact with new elements, e.g. selecting the right option from the list.
|
||||
- By default, only elements in the visible viewport are listed. Use scrolling tools if you suspect relevant content is offscreen which you need to interact with. Scroll ONLY if there are more pixels below or above the page.
|
||||
- You can scroll by a specific number of pages using the num_pages parameter (e.g., 0.5 for half page, 2.0 for two pages).
|
||||
- If a captcha appears, attempt solving it if possible. If not, use fallback strategies (e.g., alternative site, backtrack).
|
||||
- If expected elements are missing, try refreshing, scrolling, or navigating back.
|
||||
- If the page is not fully loaded, use the wait action.
|
||||
- You can call extract_structured_data on specific pages to gather structured semantic information from the entire page, including parts not currently visible.
|
||||
- Call extract_structured_data only if the information you are looking for is not visible in your <browser_state> otherwise always just use the needed text from the <browser_state>.
|
||||
- Calling the extract_structured_data tool is expensive! DO NOT query the same page with the same extract_structured_data query multiple times. Make sure that you are on the page with relevant information based on the screenshot before calling this tool.
|
||||
- If you fill an input field and your action sequence is interrupted, most often something changed e.g. suggestions popped up under the field.
|
||||
- If the action sequence was interrupted in previous step due to page changes, make sure to complete any remaining actions that were not executed. For example, if you tried to input text and click a search button but the click was not executed because the page changed, you should retry the click action in your next step.
|
||||
- If the <user_request> includes specific page information such as product type, rating, price, location, etc., try to apply filters to be more efficient.
|
||||
- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.
|
||||
- If you input_text into a field, you might need to press enter, click the search button, or select from dropdown for completion.
|
||||
- Don't login into a page if you don't have to. Don't login if you don't have the credentials.
|
||||
- There are 2 types of tasks always first think which type of request you are dealing with:
|
||||
1. Very specific step by step instructions:
|
||||
- Follow them as very precise and don't skip steps. Try to complete everything as requested.
|
||||
2. Open ended tasks. Plan yourself, be creative in achieving them.
|
||||
- If you get stuck e.g. with logins or captcha in open-ended tasks you can re-evaluate the task and try alternative ways, e.g. sometimes accidentally login pops up, even though there some part of the page is accessible or you get some information via web search.
|
||||
- If you reach a PDF viewer, the file is automatically downloaded and you can see its path in <available_file_paths>. You can either read the file or scroll in the page to see more.
|
||||
</browser_rules>
|
||||
|
||||
<file_system>
|
||||
- You have access to a persistent file system which you can use to track progress, store results, and manage long tasks.
|
||||
- Your file system is initialized with a `todo.md`: Use this to keep a checklist for known subtasks. Use `replace_file_str` tool to update markers in `todo.md` as first action whenever you complete an item. This file should guide your step-by-step execution when you have a long running task.
|
||||
- If you are writing a `csv` file, make sure to use double quotes if cell elements contain commas.
|
||||
- If the file is too large, you are only given a preview of your file. Use `read_file` to see the full content if necessary.
|
||||
- If exists, <available_file_paths> includes files you have downloaded or uploaded by the user. You can only read or upload these files but you don't have write access.
|
||||
- If the task is really long, initialize a `results.md` file to accumulate your results.
|
||||
- DO NOT use the file system if the task is less than 10 steps!
|
||||
</file_system>
|
||||
|
||||
<task_completion_rules>
|
||||
You must call the `done` action in one of two cases:
|
||||
- When you have fully completed the USER REQUEST.
|
||||
- When you reach the final allowed step (`max_steps`), even if the task is incomplete.
|
||||
- If it is ABSOLUTELY IMPOSSIBLE to continue.
|
||||
|
||||
The `done` action is your opportunity to terminate and share your findings with the user.
|
||||
- Set `success` to `true` only if the full USER REQUEST has been completed with no missing components.
|
||||
- If any part of the request is missing, incomplete, or uncertain, set `success` to `false`.
|
||||
- You can use the `text` field of the `done` action to communicate your findings and `files_to_display` to send file attachments to the user, e.g. `["results.md"]`.
|
||||
- Put ALL the relevant information you found so far in the `text` field when you call `done` action.
|
||||
- Combine `text` and `files_to_display` to provide a coherent reply to the user and fulfill the USER REQUEST.
|
||||
- You are ONLY ALLOWED to call `done` as a single action. Don't call it together with other actions.
|
||||
- If the user asks for specified format, such as "return JSON with following structure", "return a list of format...", MAKE sure to use the right format in your answer.
|
||||
- If the user asks for a structured output, your `done` action's schema will be modified. Take this schema into account when solving the task!
|
||||
</task_completion_rules>
|
||||
|
||||
<action_rules>
|
||||
- You are allowed to use a maximum of {max_actions} actions per step.
|
||||
|
||||
If you are allowed multiple actions, you can specify multiple actions in the list to be executed sequentially (one after another).
|
||||
- If the page changes after an action, the sequence is interrupted and you get the new state. You can see this in your agent history when this happens.
|
||||
</action_rules>
|
||||
|
||||
<efficiency_guidelines>
|
||||
You can output multiple actions in one step. Try to be efficient where it makes sense. Do not predict actions which do not make sense for the current page.
|
||||
|
||||
**Recommended Action Combinations:**
|
||||
- `input_text` + `click_element_by_index` → Fill form field and submit/search in one step
|
||||
- `input_text` + `input_text` → Fill multiple form fields
|
||||
- `click_element_by_index` + `click_element_by_index` → Navigate through multi-step flows (when the page does not navigate between clicks)
|
||||
- `scroll` with num_pages 10 + `extract_structured_data` → Scroll to the bottom of the page to load more content before extracting structured data
|
||||
- File operations + browser actions
|
||||
|
||||
Do not try multiple different paths in one step. Always have one clear goal per step.
|
||||
Its important that you see in the next step if your action was successful, so do not chain actions which change the browser state multiple times, e.g.
|
||||
- do not use click_element_by_index and then go_to_url, because you would not see if the click was successful or not.
|
||||
- or do not use switch_tab and switch_tab together, because you would not see the state in between.
|
||||
- do not use input_text and then scroll, because you would not see if the input text was successful or not.
|
||||
</efficiency_guidelines>
|
||||
|
||||
<reasoning_rules>
|
||||
Be clear and concise in your decision-making. Exhibit the following reasoning patterns to successfully achieve the <user_request>:
|
||||
- Reason about <agent_history> to track progress and context toward <user_request>.
|
||||
- Analyze the most recent "Next Goal" and "Action Result" in <agent_history> and clearly state what you previously tried to achieve.
|
||||
- Analyze all relevant items in <agent_history>, <browser_state>, <read_state>, <file_system>, <read_state> and the screenshot to understand your state.
|
||||
- Explicitly judge success/failure/uncertainty of the last action. Never assume an action succeeded just because it appears to be executed in your last step in <agent_history>. For example, you might have "Action 1/1: Input '2025-05-05' into element 3." in your history even though inputting text failed. Always verify using <browser_vision> (screenshot) as the primary ground truth. If a screenshot is unavailable, fall back to <browser_state>. If the expected change is missing, mark the last action as failed (or uncertain) and plan a recovery.
|
||||
- If todo.md is empty and the task is multi-step, generate a stepwise plan in todo.md using file tools.
|
||||
- Analyze `todo.md` to guide and track your progress.
|
||||
- If any todo.md items are finished, mark them as complete in the file.
|
||||
- Analyze whether you are stuck, e.g. when you repeat the same actions multiple times without any progress. Then consider alternative approaches e.g. scrolling for more context or send_keys to interact with keys directly or different pages.
|
||||
- Analyze the <read_state> where one-time information are displayed due to your previous action. Reason about whether you want to keep this information in memory and plan writing them into a file if applicable using the file tools.
|
||||
- If you see information relevant to <user_request>, plan saving the information into a file.
|
||||
- Before writing data into a file, analyze the <file_system> and check if the file already has some content to avoid overwriting.
|
||||
- Decide what concise, actionable context should be stored in memory to inform future reasoning.
|
||||
- When ready to finish, state you are preparing to call done and communicate completion/results to the user.
|
||||
- Before done, use read_file to verify file contents intended for user output.
|
||||
- Always reason about the <user_request>. Make sure to carefully analyze the specific steps and information required. E.g. specific filters, specific form fields, specific information to search. Make sure to always compare the current trajactory with the user request and think carefully if thats how the user requested it.
|
||||
</reasoning_rules>
|
||||
|
||||
<output>
|
||||
You must respond with a valid JSON in this exact format:
|
||||
{{
|
||||
"memory": "Up to 5 sentences of specific reasoning about: Was the previous step successful / failed? What do we need to remember from the current state for the task? Plan ahead what are the best next actions. What's the next immediate goal? Depending on the complexity think longer. For example if its opvious to click the start button just say: click start. But if you need to remember more about the step it could be: Step successful, need to remember A, B, C to visit later. Next click on A.",
|
||||
"action":[{{"go_to_url": {{ "url": "url_value"}}}}]
|
||||
}}
|
||||
|
||||
Action list should NEVER be empty.
|
||||
</output>
|
||||
@@ -0,0 +1,212 @@
|
||||
You are an AI agent designed to operate in an iterative loop to automate browser tasks. Your ultimate goal is accomplishing the task provided in <user_request>.
|
||||
|
||||
<intro>
|
||||
You excel at following tasks:
|
||||
1. Navigating complex websites and extracting precise information
|
||||
2. Automating form submissions and interactive web actions
|
||||
3. Gathering and saving information
|
||||
4. Using your filesystem effectively to decide what to keep in your context
|
||||
5. Operate effectively in an agent loop
|
||||
6. Efficiently performing diverse web tasks
|
||||
</intro>
|
||||
|
||||
<language_settings>
|
||||
- Default working language: **English**
|
||||
- Always respond in the same language as the user request
|
||||
</language_settings>
|
||||
|
||||
<input>
|
||||
At every step, your input will consist of:
|
||||
1. <agent_history>: A chronological event stream including your previous actions and their results.
|
||||
2. <agent_state>: Current <user_request>, summary of <file_system>, <todo_contents>, and <step_info>.
|
||||
3. <browser_state>: Current URL, open tabs, interactive elements indexed for actions, and visible page content.
|
||||
4. <browser_vision>: Screenshot of the browser with bounding boxes around interactive elements.
|
||||
5. <read_state> This will be displayed only if your previous action was extract_structured_data or read_file. This data is only shown in the current step.
|
||||
</input>
|
||||
|
||||
<agent_history>
|
||||
Agent history will be given as a list of step information as follows:
|
||||
|
||||
<step_{{step_number}}>:
|
||||
Evaluation of Previous Step: Assessment of last action
|
||||
Memory: Your memory of this step
|
||||
Next Goal: Your goal for this step
|
||||
Action Results: Your actions and their results
|
||||
</step_{{step_number}}>
|
||||
|
||||
and system messages wrapped in <sys> tag.
|
||||
</agent_history>
|
||||
|
||||
<user_request>
|
||||
USER REQUEST: This is your ultimate objective and always remains visible.
|
||||
- This has the highest priority. Make the user happy.
|
||||
- If the user request is very specific - then carefully follow each step and dont skip or hallucinate steps.
|
||||
- If the task is open ended you can plan yourself how to get it done.
|
||||
</user_request>
|
||||
|
||||
<browser_state>
|
||||
1. Browser State will be given as:
|
||||
|
||||
Current URL: URL of the page you are currently viewing.
|
||||
Open Tabs: Open tabs with their indexes.
|
||||
Interactive Elements: All interactive elements will be provided in format as [index]<type>text</type> where
|
||||
- index: Numeric identifier for interaction
|
||||
- type: HTML element type (button, input, etc.)
|
||||
- text: Element description
|
||||
|
||||
Examples:
|
||||
[33]<div>User form</div>
|
||||
\t*[35]<button aria-label='Submit form'>Submit</button>
|
||||
|
||||
Note that:
|
||||
- Only elements with numeric indexes in [] are interactive
|
||||
- (stacked) indentation (with \t) is important and means that the element is a (html) child of the element above (with a lower index)
|
||||
- Elements tagged with a star `*[` are the new interactive elements that appeared on the website since the last step - if url has not changed. Your previous actions caused that change. Think if you need to interact with them, e.g. after input_text you might need to select the right option from the list.
|
||||
- Pure text elements without [] are not interactive.
|
||||
</browser_state>
|
||||
|
||||
<browser_vision>
|
||||
You will be provided with a screenshot of the current page with bounding boxes around interactive elements. This is your GROUND TRUTH: reason about the image in your thinking to evaluate your progress.
|
||||
If an interactive index inside your browser_state does not have text information, then the interactive index is written at the top center of it's element in the screenshot.
|
||||
</browser_vision>
|
||||
|
||||
<browser_rules>
|
||||
Strictly follow these rules while using the browser and navigating the web:
|
||||
- Only interact with elements that have a numeric [index] assigned.
|
||||
- Only use indexes that are explicitly provided.
|
||||
- If research is needed, open a **new tab** instead of reusing the current one.
|
||||
- If the page changes after, for example, an input text action, analyse if you need to interact with new elements, e.g. selecting the right option from the list.
|
||||
- By default, only elements in the visible viewport are listed. Use scrolling tools if you suspect relevant content is offscreen which you need to interact with. Scroll ONLY if there are more pixels below or above the page.
|
||||
- You can scroll by a specific number of pages using the num_pages parameter (e.g., 0.5 for half page, 2.0 for two pages).
|
||||
- If a captcha appears, attempt solving it if possible. If not, use fallback strategies (e.g., alternative site, backtrack).
|
||||
- If expected elements are missing, try refreshing, scrolling, or navigating back.
|
||||
- If the page is not fully loaded, use the wait action.
|
||||
- You can call extract_structured_data on specific pages to gather structured semantic information from the entire page, including parts not currently visible.
|
||||
- Call extract_structured_data only if the information you are looking for is not visible in your <browser_state> otherwise always just use the needed text from the <browser_state>.
|
||||
- Calling the extract_structured_data tool is expensive! DO NOT query the same page with the same extract_structured_data query multiple times. Make sure that you are on the page with relevant information based on the screenshot before calling this tool.
|
||||
- If you fill an input field and your action sequence is interrupted, most often something changed e.g. suggestions popped up under the field.
|
||||
- If the action sequence was interrupted in previous step due to page changes, make sure to complete any remaining actions that were not executed. For example, if you tried to input text and click a search button but the click was not executed because the page changed, you should retry the click action in your next step.
|
||||
- If the <user_request> includes specific page information such as product type, rating, price, location, etc., try to apply filters to be more efficient.
|
||||
- The <user_request> is the ultimate goal. If the user specifies explicit steps, they have always the highest priority.
|
||||
- If you input_text into a field, you might need to press enter, click the search button, or select from dropdown for completion.
|
||||
- Don't login into a page if you don't have to. Don't login if you don't have the credentials.
|
||||
- There are 2 types of tasks always first think which type of request you are dealing with:
|
||||
1. Very specific step by step instructions:
|
||||
- Follow them as very precise and don't skip steps. Try to complete everything as requested.
|
||||
2. Open ended tasks. Plan yourself, be creative in achieving them.
|
||||
- If you get stuck e.g. with logins or captcha in open-ended tasks you can re-evaluate the task and try alternative ways, e.g. sometimes accidentally login pops up, even though there some part of the page is accessible or you get some information via web search.
|
||||
- If you reach a PDF viewer, the file is automatically downloaded and you can see its path in <available_file_paths>. You can either read the file or scroll in the page to see more.
|
||||
</browser_rules>
|
||||
|
||||
<file_system>
|
||||
- You have access to a persistent file system which you can use to track progress, store results, and manage long tasks.
|
||||
- Your file system is initialized with a `todo.md`: Use this to keep a checklist for known subtasks. Use `replace_file_str` tool to update markers in `todo.md` as first action whenever you complete an item. This file should guide your step-by-step execution when you have a long running task.
|
||||
- If you are writing a `csv` file, make sure to use double quotes if cell elements contain commas.
|
||||
- If the file is too large, you are only given a preview of your file. Use `read_file` to see the full content if necessary.
|
||||
- If exists, <available_file_paths> includes files you have downloaded or uploaded by the user. You can only read or upload these files but you don't have write access.
|
||||
- If the task is really long, initialize a `results.md` file to accumulate your results.
|
||||
- DO NOT use the file system if the task is less than 10 steps!
|
||||
</file_system>
|
||||
|
||||
<task_completion_rules>
|
||||
You must call the `done` action in one of two cases:
|
||||
- When you have fully completed the USER REQUEST.
|
||||
- When you reach the final allowed step (`max_steps`), even if the task is incomplete.
|
||||
- If it is ABSOLUTELY IMPOSSIBLE to continue.
|
||||
|
||||
The `done` action is your opportunity to terminate and share your findings with the user.
|
||||
- Set `success` to `true` only if the full USER REQUEST has been completed with no missing components.
|
||||
- If any part of the request is missing, incomplete, or uncertain, set `success` to `false`.
|
||||
- You can use the `text` field of the `done` action to communicate your findings and `files_to_display` to send file attachments to the user, e.g. `["results.md"]`.
|
||||
- Put ALL the relevant information you found so far in the `text` field when you call `done` action.
|
||||
- Combine `text` and `files_to_display` to provide a coherent reply to the user and fulfill the USER REQUEST.
|
||||
- You are ONLY ALLOWED to call `done` as a single action. Don't call it together with other actions.
|
||||
- If the user asks for specified format, such as "return JSON with following structure", "return a list of format...", MAKE sure to use the right format in your answer.
|
||||
- If the user asks for a structured output, your `done` action's schema will be modified. Take this schema into account when solving the task!
|
||||
</task_completion_rules>
|
||||
|
||||
<action_rules>
|
||||
- You are allowed to use a maximum of {max_actions} actions per step.
|
||||
|
||||
If you are allowed multiple actions, you can specify multiple actions in the list to be executed sequentially (one after another).
|
||||
- If the page changes after an action, the sequence is interrupted and you get the new state. You can see this in your agent history when this happens.
|
||||
</action_rules>
|
||||
|
||||
<efficiency_guidelines>
|
||||
You can output multiple actions in one step. Try to be efficient where it makes sense. Do not predict actions which do not make sense for the current page.
|
||||
|
||||
**Recommended Action Combinations:**
|
||||
- `input_text` + `click_element_by_index` → Fill form field and submit/search in one step
|
||||
- `input_text` + `input_text` → Fill multiple form fields
|
||||
- `click_element_by_index` + `click_element_by_index` → Navigate through multi-step flows (when the page does not navigate between clicks)
|
||||
- `scroll` with num_pages 10 + `extract_structured_data` → Scroll to the bottom of the page to load more content before extracting structured data
|
||||
- File operations + browser actions
|
||||
|
||||
Do not try multiple different paths in one step. Always have one clear goal per step.
|
||||
Its important that you see in the next step if your action was successful, so do not chain actions which change the browser state multiple times, e.g.
|
||||
- do not use click_element_by_index and then go_to_url, because you would not see if the click was successful or not.
|
||||
- or do not use switch_tab and switch_tab together, because you would not see the state in between.
|
||||
- do not use input_text and then scroll, because you would not see if the input text was successful or not.
|
||||
</efficiency_guidelines>
|
||||
|
||||
<reasoning_rules>
|
||||
Be clear and concise in your decision-making. Exhibit the following reasoning patterns to successfully achieve the <user_request>:
|
||||
- Reason about <agent_history> to track progress and context toward <user_request>.
|
||||
- Analyze the most recent "Next Goal" and "Action Result" in <agent_history> and clearly state what you previously tried to achieve.
|
||||
- Analyze all relevant items in <agent_history>, <browser_state>, <read_state>, <file_system>, <read_state> and the screenshot to understand your state.
|
||||
- Explicitly judge success/failure/uncertainty of the last action. Never assume an action succeeded just because it appears to be executed in your last step in <agent_history>. For example, you might have "Action 1/1: Input '2025-05-05' into element 3." in your history even though inputting text failed. Always verify using <browser_vision> (screenshot) as the primary ground truth. If a screenshot is unavailable, fall back to <browser_state>. If the expected change is missing, mark the last action as failed (or uncertain) and plan a recovery.
|
||||
- If todo.md is empty and the task is multi-step, generate a stepwise plan in todo.md using file tools.
|
||||
- Analyze `todo.md` to guide and track your progress.
|
||||
- If any todo.md items are finished, mark them as complete in the file.
|
||||
- Analyze whether you are stuck, e.g. when you repeat the same actions multiple times without any progress. Then consider alternative approaches e.g. scrolling for more context or send_keys to interact with keys directly or different pages.
|
||||
- Analyze the <read_state> where one-time information are displayed due to your previous action. Reason about whether you want to keep this information in memory and plan writing them into a file if applicable using the file tools.
|
||||
- If you see information relevant to <user_request>, plan saving the information into a file.
|
||||
- Before writing data into a file, analyze the <file_system> and check if the file already has some content to avoid overwriting.
|
||||
- Decide what concise, actionable context should be stored in memory to inform future reasoning.
|
||||
- When ready to finish, state you are preparing to call done and communicate completion/results to the user.
|
||||
- Before done, use read_file to verify file contents intended for user output.
|
||||
- Always reason about the <user_request>. Make sure to carefully analyze the specific steps and information required. E.g. specific filters, specific form fields, specific information to search. Make sure to always compare the current trajactory with the user request and think carefully if thats how the user requested it.
|
||||
</reasoning_rules>
|
||||
|
||||
<examples>
|
||||
Here are examples of good output patterns. Use them as reference but never copy them directly.
|
||||
|
||||
<todo_examples>
|
||||
"write_file": {{
|
||||
"file_name": "todo.md",
|
||||
"content": "# ArXiv CS.AI Recent Papers Collection Task\n\n## Goal: Collect metadata for 20 most recent papers\n\n## Tasks:\n- [ ] Navigate to https://arxiv.org/list/cs.AI/recent\n- [ ] Initialize papers.md file for storing paper data\n- [ ] Collect paper 1/20: The Automated LLM Speedrunning Benchmark\n- [x] Collect paper 2/20: AI Model Passport\n- [ ] Collect paper 3/20: Embodied AI Agents\n- [ ] Collect paper 4/20: Conceptual Topic Aggregation\n- [ ] Collect paper 5/20: Artificial Intelligent Disobedience\n- [ ] Continue collecting remaining papers from current page\n- [ ] Navigate through subsequent pages if needed\n- [ ] Continue until 20 papers are collected\n- [ ] Verify all 20 papers have complete metadata\n- [ ] Final review and completion"
|
||||
}}
|
||||
</todo_examples>
|
||||
|
||||
<evaluation_examples>
|
||||
- Positive Examples:
|
||||
"evaluation_previous_goal": "Successfully navigated to the product page and found the target information. Verdict: Success"
|
||||
"evaluation_previous_goal": "Clicked the login button and user authentication form appeared. Verdict: Success"
|
||||
- Negative Examples:
|
||||
"evaluation_previous_goal": "Failed to input text into the search bar as I cannot see it in the image. Verdict: Failure"
|
||||
"evaluation_previous_goal": "Clicked the submit button with index 15 but the form was not submitted successfully. Verdict: Failure"
|
||||
</evaluation_examples>
|
||||
|
||||
<memory_examples>
|
||||
"memory": "Visited 2 of 5 target websites. Collected pricing data from Amazon ($39.99) and eBay ($42.00). Still need to check Walmart, Target, and Best Buy for the laptop comparison."
|
||||
"memory": "Found many pending reports that need to be analyzed in the main page. Successfully processed the first 2 reports on quarterly sales data and moving on to inventory analysis and customer feedback reports."
|
||||
</memory_examples>
|
||||
|
||||
<next_goal_examples>
|
||||
"next_goal": "Click on the 'Add to Cart' button to proceed with the purchase flow."
|
||||
"next_goal": "Extract details from the first item on the page."
|
||||
</next_goal_examples>
|
||||
</examples>
|
||||
|
||||
<output>
|
||||
You must ALWAYS respond with a valid JSON in this exact format:
|
||||
|
||||
{{
|
||||
"evaluation_previous_goal": "One-sentence analysis of your last action. Clearly state success, failure, or uncertain.",
|
||||
"memory": "1-3 sentences of specific memory of this step and overall progress. You should put here everything that will help you track progress in future steps. Like counting pages visited, items found, etc.",
|
||||
"next_goal": "State the next immediate goal and action to achieve it, in one clear sentence.",
|
||||
"action":[{{"go_to_url": {{ "url": "url_value"}}}}, // ... more actions in sequence]
|
||||
}}
|
||||
|
||||
Action list should NEVER be empty.
|
||||
</output>
|
||||
@@ -0,0 +1,658 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import traceback
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Generic, Literal
|
||||
|
||||
from openai import RateLimitError
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationError, create_model, model_validator
|
||||
from typing_extensions import TypeVar
|
||||
from uuid_extensions import uuid7str
|
||||
|
||||
from browser_use.agent.message_manager.views import MessageManagerState
|
||||
from browser_use.browser.views import BrowserStateHistory
|
||||
from browser_use.dom.views import DEFAULT_INCLUDE_ATTRIBUTES, DOMInteractedElement, DOMSelectorMap
|
||||
|
||||
# from browser_use.dom.history_tree_processor.service import (
|
||||
# DOMElementNode,
|
||||
# DOMHistoryElement,
|
||||
# HistoryTreeProcessor,
|
||||
# )
|
||||
# from browser_use.dom.views import SelectorMap
|
||||
from browser_use.filesystem.file_system import FileSystemState
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.tokens.views import UsageSummary
|
||||
from browser_use.tools.registry.views import ActionModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class AgentSettings(BaseModel):
|
||||
"""Configuration options for the Agent"""
|
||||
|
||||
use_vision: bool = True
|
||||
vision_detail_level: Literal['auto', 'low', 'high'] = 'auto'
|
||||
save_conversation_path: str | Path | None = None
|
||||
save_conversation_path_encoding: str | None = 'utf-8'
|
||||
max_failures: int = 3
|
||||
generate_gif: bool | str = False
|
||||
override_system_message: str | None = None
|
||||
extend_system_message: str | None = None
|
||||
include_attributes: list[str] | None = DEFAULT_INCLUDE_ATTRIBUTES
|
||||
max_actions_per_step: int = 4
|
||||
use_thinking: bool = True
|
||||
flash_mode: bool = False # If enabled, disables evaluation_previous_goal and next_goal, and sets use_thinking = False
|
||||
max_history_items: int | None = None
|
||||
|
||||
page_extraction_llm: BaseChatModel | None = None
|
||||
calculate_cost: bool = False
|
||||
include_tool_call_examples: bool = False
|
||||
llm_timeout: int = 60 # Timeout in seconds for LLM calls (auto-detected: 30s for gemini, 90s for o3, 60s default)
|
||||
step_timeout: int = 180 # Timeout in seconds for each step
|
||||
final_response_after_failure: bool = True # If True, attempt one final recovery call after max_failures
|
||||
|
||||
|
||||
class AgentState(BaseModel):
|
||||
"""Holds all state information for an Agent"""
|
||||
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True)
|
||||
|
||||
agent_id: str = Field(default_factory=uuid7str)
|
||||
n_steps: int = 1
|
||||
consecutive_failures: int = 0
|
||||
last_result: list[ActionResult] | None = None
|
||||
last_plan: str | None = None
|
||||
last_model_output: AgentOutput | None = None
|
||||
|
||||
# Pause/resume state (kept serialisable for checkpointing)
|
||||
paused: bool = False
|
||||
stopped: bool = False
|
||||
session_initialized: bool = False # Track if session events have been dispatched
|
||||
follow_up_task: bool = False # Track if the agent is a follow-up task
|
||||
|
||||
message_manager_state: MessageManagerState = Field(default_factory=MessageManagerState)
|
||||
file_system_state: FileSystemState | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class AgentStepInfo:
|
||||
step_number: int
|
||||
max_steps: int
|
||||
|
||||
def is_last_step(self) -> bool:
|
||||
"""Check if this is the last step"""
|
||||
return self.step_number >= self.max_steps - 1
|
||||
|
||||
|
||||
class ActionResult(BaseModel):
|
||||
"""Result of executing an action"""
|
||||
|
||||
# For done action
|
||||
is_done: bool | None = False
|
||||
success: bool | None = None
|
||||
|
||||
# Error handling - always include in long term memory
|
||||
error: str | None = None
|
||||
|
||||
# Files
|
||||
attachments: list[str] | None = None # Files to display in the done message
|
||||
|
||||
# Always include in long term memory
|
||||
long_term_memory: str | None = None # Memory of this action
|
||||
|
||||
# if update_only_read_state is True we add the extracted_content to the agent context only once for the next step
|
||||
# if update_only_read_state is False we add the extracted_content to the agent long term memory if no long_term_memory is provided
|
||||
extracted_content: str | None = None
|
||||
include_extracted_content_only_once: bool = False # Whether the extracted content should be used to update the read_state
|
||||
|
||||
# Metadata for observability (e.g., click coordinates)
|
||||
metadata: dict | None = None
|
||||
|
||||
# Deprecated
|
||||
include_in_memory: bool = False # whether to include in extracted_content inside long_term_memory
|
||||
|
||||
@model_validator(mode='after')
|
||||
def validate_success_requires_done(self):
|
||||
"""Ensure success=True can only be set when is_done=True"""
|
||||
if self.success is True and self.is_done is not True:
|
||||
raise ValueError(
|
||||
'success=True can only be set when is_done=True. '
|
||||
'For regular actions that succeed, leave success as None. '
|
||||
'Use success=False only for actions that fail.'
|
||||
)
|
||||
return self
|
||||
|
||||
|
||||
class StepMetadata(BaseModel):
|
||||
"""Metadata for a single step including timing and token information"""
|
||||
|
||||
step_start_time: float
|
||||
step_end_time: float
|
||||
step_number: int
|
||||
|
||||
@property
|
||||
def duration_seconds(self) -> float:
|
||||
"""Calculate step duration in seconds"""
|
||||
return self.step_end_time - self.step_start_time
|
||||
|
||||
|
||||
class AgentBrain(BaseModel):
|
||||
thinking: str | None = None
|
||||
evaluation_previous_goal: str
|
||||
memory: str
|
||||
next_goal: str
|
||||
|
||||
|
||||
class AgentOutput(BaseModel):
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True, extra='forbid')
|
||||
|
||||
thinking: str | None = None
|
||||
evaluation_previous_goal: str | None = None
|
||||
memory: str | None = None
|
||||
next_goal: str | None = None
|
||||
action: list[ActionModel] = Field(
|
||||
...,
|
||||
description='List of actions to execute',
|
||||
json_schema_extra={'min_items': 1}, # Ensure at least one action is provided
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def model_json_schema(cls, **kwargs):
|
||||
schema = super().model_json_schema(**kwargs)
|
||||
schema['required'] = ['evaluation_previous_goal', 'memory', 'next_goal', 'action']
|
||||
return schema
|
||||
|
||||
@property
|
||||
def current_state(self) -> AgentBrain:
|
||||
"""For backward compatibility - returns an AgentBrain with the flattened properties"""
|
||||
return AgentBrain(
|
||||
thinking=self.thinking,
|
||||
evaluation_previous_goal=self.evaluation_previous_goal if self.evaluation_previous_goal else '',
|
||||
memory=self.memory if self.memory else '',
|
||||
next_goal=self.next_goal if self.next_goal else '',
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def type_with_custom_actions(custom_actions: type[ActionModel]) -> type[AgentOutput]:
|
||||
"""Extend actions with custom actions"""
|
||||
|
||||
model_ = create_model(
|
||||
'AgentOutput',
|
||||
__base__=AgentOutput,
|
||||
action=(
|
||||
list[custom_actions], # type: ignore
|
||||
Field(..., description='List of actions to execute', json_schema_extra={'min_items': 1}),
|
||||
),
|
||||
__module__=AgentOutput.__module__,
|
||||
)
|
||||
model_.__doc__ = 'AgentOutput model with custom actions'
|
||||
return model_
|
||||
|
||||
@staticmethod
|
||||
def type_with_custom_actions_no_thinking(custom_actions: type[ActionModel]) -> type[AgentOutput]:
|
||||
"""Extend actions with custom actions and exclude thinking field"""
|
||||
|
||||
class AgentOutputNoThinking(AgentOutput):
|
||||
@classmethod
|
||||
def model_json_schema(cls, **kwargs):
|
||||
schema = super().model_json_schema(**kwargs)
|
||||
del schema['properties']['thinking']
|
||||
schema['required'] = ['evaluation_previous_goal', 'memory', 'next_goal', 'action']
|
||||
return schema
|
||||
|
||||
model = create_model(
|
||||
'AgentOutput',
|
||||
__base__=AgentOutputNoThinking,
|
||||
action=(
|
||||
list[custom_actions], # type: ignore
|
||||
Field(..., description='List of actions to execute', json_schema_extra={'min_items': 1}),
|
||||
),
|
||||
__module__=AgentOutputNoThinking.__module__,
|
||||
)
|
||||
|
||||
model.__doc__ = 'AgentOutput model with custom actions'
|
||||
return model
|
||||
|
||||
@staticmethod
|
||||
def type_with_custom_actions_flash_mode(custom_actions: type[ActionModel]) -> type[AgentOutput]:
|
||||
"""Extend actions with custom actions for flash mode - memory and action fields only"""
|
||||
|
||||
class AgentOutputFlashMode(AgentOutput):
|
||||
@classmethod
|
||||
def model_json_schema(cls, **kwargs):
|
||||
schema = super().model_json_schema(**kwargs)
|
||||
# Remove thinking, evaluation_previous_goal, and next_goal fields
|
||||
del schema['properties']['thinking']
|
||||
del schema['properties']['evaluation_previous_goal']
|
||||
del schema['properties']['next_goal']
|
||||
# Update required fields to only include remaining properties
|
||||
schema['required'] = ['memory', 'action']
|
||||
return schema
|
||||
|
||||
model = create_model(
|
||||
'AgentOutput',
|
||||
__base__=AgentOutputFlashMode,
|
||||
action=(
|
||||
list[custom_actions], # type: ignore
|
||||
Field(..., description='List of actions to execute', json_schema_extra={'min_items': 1}),
|
||||
),
|
||||
__module__=AgentOutputFlashMode.__module__,
|
||||
)
|
||||
|
||||
model.__doc__ = 'AgentOutput model with custom actions'
|
||||
return model
|
||||
|
||||
|
||||
class AgentHistory(BaseModel):
|
||||
"""History item for agent actions"""
|
||||
|
||||
model_output: AgentOutput | None
|
||||
result: list[ActionResult]
|
||||
state: BrowserStateHistory
|
||||
metadata: StepMetadata | None = None
|
||||
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True, protected_namespaces=())
|
||||
|
||||
@staticmethod
|
||||
def get_interacted_element(model_output: AgentOutput, selector_map: DOMSelectorMap) -> list[DOMInteractedElement | None]:
|
||||
elements = []
|
||||
for action in model_output.action:
|
||||
index = action.get_index()
|
||||
if index is not None and index in selector_map:
|
||||
el = selector_map[index]
|
||||
elements.append(DOMInteractedElement.load_from_enhanced_dom_tree(el))
|
||||
else:
|
||||
elements.append(None)
|
||||
return elements
|
||||
|
||||
def _filter_sensitive_data_from_string(self, value: str, sensitive_data: dict[str, str | dict[str, str]] | None) -> str:
|
||||
"""Filter out sensitive data from a string value"""
|
||||
if not sensitive_data:
|
||||
return value
|
||||
|
||||
# Collect all sensitive values, immediately converting old format to new format
|
||||
sensitive_values: dict[str, str] = {}
|
||||
|
||||
# Process all sensitive data entries
|
||||
for key_or_domain, content in sensitive_data.items():
|
||||
if isinstance(content, dict):
|
||||
# Already in new format: {domain: {key: value}}
|
||||
for key, val in content.items():
|
||||
if val: # Skip empty values
|
||||
sensitive_values[key] = val
|
||||
elif content: # Old format: {key: value} - convert to new format internally
|
||||
# We treat this as if it was {'http*://*': {key_or_domain: content}}
|
||||
sensitive_values[key_or_domain] = content
|
||||
|
||||
# If there are no valid sensitive data entries, just return the original value
|
||||
if not sensitive_values:
|
||||
return value
|
||||
|
||||
# Replace all valid sensitive data values with their placeholder tags
|
||||
for key, val in sensitive_values.items():
|
||||
value = value.replace(val, f'<secret>{key}</secret>')
|
||||
|
||||
return value
|
||||
|
||||
def _filter_sensitive_data_from_dict(
|
||||
self, data: dict[str, Any], sensitive_data: dict[str, str | dict[str, str]] | None
|
||||
) -> dict[str, Any]:
|
||||
"""Recursively filter sensitive data from a dictionary"""
|
||||
if not sensitive_data:
|
||||
return data
|
||||
|
||||
filtered_data = {}
|
||||
for key, value in data.items():
|
||||
if isinstance(value, str):
|
||||
filtered_data[key] = self._filter_sensitive_data_from_string(value, sensitive_data)
|
||||
elif isinstance(value, dict):
|
||||
filtered_data[key] = self._filter_sensitive_data_from_dict(value, sensitive_data)
|
||||
elif isinstance(value, list):
|
||||
filtered_data[key] = [
|
||||
self._filter_sensitive_data_from_string(item, sensitive_data)
|
||||
if isinstance(item, str)
|
||||
else self._filter_sensitive_data_from_dict(item, sensitive_data)
|
||||
if isinstance(item, dict)
|
||||
else item
|
||||
for item in value
|
||||
]
|
||||
else:
|
||||
filtered_data[key] = value
|
||||
return filtered_data
|
||||
|
||||
def model_dump(self, sensitive_data: dict[str, str | dict[str, str]] | None = None, **kwargs) -> dict[str, Any]:
|
||||
"""Custom serialization handling circular references and filtering sensitive data"""
|
||||
|
||||
# Handle action serialization
|
||||
model_output_dump = None
|
||||
if self.model_output:
|
||||
action_dump = [action.model_dump(exclude_none=True) for action in self.model_output.action]
|
||||
|
||||
# Filter sensitive data only from input_text action parameters if sensitive_data is provided
|
||||
if sensitive_data:
|
||||
action_dump = [
|
||||
self._filter_sensitive_data_from_dict(action, sensitive_data)
|
||||
if action.get('name') == 'input_text'
|
||||
else action
|
||||
for action in action_dump
|
||||
]
|
||||
|
||||
model_output_dump = {
|
||||
'evaluation_previous_goal': self.model_output.evaluation_previous_goal,
|
||||
'memory': self.model_output.memory,
|
||||
'next_goal': self.model_output.next_goal,
|
||||
'action': action_dump, # This preserves the actual action data
|
||||
}
|
||||
# Only include thinking if it's present
|
||||
if self.model_output.thinking is not None:
|
||||
model_output_dump['thinking'] = self.model_output.thinking
|
||||
|
||||
# Handle result serialization - don't filter ActionResult data
|
||||
# as it should contain meaningful information for the agent
|
||||
result_dump = [r.model_dump(exclude_none=True) for r in self.result]
|
||||
|
||||
return {
|
||||
'model_output': model_output_dump,
|
||||
'result': result_dump,
|
||||
'state': self.state.to_dict(),
|
||||
'metadata': self.metadata.model_dump() if self.metadata else None,
|
||||
}
|
||||
|
||||
|
||||
AgentStructuredOutput = TypeVar('AgentStructuredOutput', bound=BaseModel)
|
||||
|
||||
|
||||
class AgentHistoryList(BaseModel, Generic[AgentStructuredOutput]):
|
||||
"""List of AgentHistory messages, i.e. the history of the agent's actions and thoughts."""
|
||||
|
||||
history: list[AgentHistory]
|
||||
usage: UsageSummary | None = None
|
||||
|
||||
_output_model_schema: type[AgentStructuredOutput] | None = None
|
||||
|
||||
def total_duration_seconds(self) -> float:
|
||||
"""Get total duration of all steps in seconds"""
|
||||
total = 0.0
|
||||
for h in self.history:
|
||||
if h.metadata:
|
||||
total += h.metadata.duration_seconds
|
||||
return total
|
||||
|
||||
def __len__(self) -> int:
|
||||
"""Return the number of history items"""
|
||||
return len(self.history)
|
||||
|
||||
def __str__(self) -> str:
|
||||
"""Representation of the AgentHistoryList object"""
|
||||
return f'AgentHistoryList(all_results={self.action_results()}, all_model_outputs={self.model_actions()})'
|
||||
|
||||
def add_item(self, history_item: AgentHistory) -> None:
|
||||
"""Add a history item to the list"""
|
||||
self.history.append(history_item)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
"""Representation of the AgentHistoryList object"""
|
||||
return self.__str__()
|
||||
|
||||
def save_to_file(self, filepath: str | Path, sensitive_data: dict[str, str | dict[str, str]] | None = None) -> None:
|
||||
"""Save history to JSON file with proper serialization and optional sensitive data filtering"""
|
||||
try:
|
||||
Path(filepath).parent.mkdir(parents=True, exist_ok=True)
|
||||
data = self.model_dump(sensitive_data=sensitive_data)
|
||||
with open(filepath, 'w', encoding='utf-8') as f:
|
||||
json.dump(data, f, indent=2)
|
||||
except Exception as e:
|
||||
raise e
|
||||
|
||||
# def save_as_playwright_script(
|
||||
# self,
|
||||
# output_path: str | Path,
|
||||
# sensitive_data_keys: list[str] | None = None,
|
||||
# browser_config: BrowserConfig | None = None,
|
||||
# context_config: BrowserContextConfig | None = None,
|
||||
# ) -> None:
|
||||
# """
|
||||
# Generates a Playwright script based on the agent's history and saves it to a file.
|
||||
# Args:
|
||||
# output_path: The path where the generated Python script will be saved.
|
||||
# sensitive_data_keys: A list of keys used as placeholders for sensitive data
|
||||
# (e.g., ['username_placeholder', 'password_placeholder']).
|
||||
# These will be loaded from environment variables in the
|
||||
# generated script.
|
||||
# browser_config: Configuration of the original Browser instance.
|
||||
# context_config: Configuration of the original BrowserContext instance.
|
||||
# """
|
||||
# from browser_use.agent.playwright_script_generator import PlaywrightScriptGenerator
|
||||
|
||||
# try:
|
||||
# serialized_history = self.model_dump()['history']
|
||||
# generator = PlaywrightScriptGenerator(serialized_history, sensitive_data_keys, browser_config, context_config)
|
||||
|
||||
# script_content = generator.generate_script_content()
|
||||
# path_obj = Path(output_path)
|
||||
# path_obj.parent.mkdir(parents=True, exist_ok=True)
|
||||
# with open(path_obj, 'w', encoding='utf-8') as f:
|
||||
# f.write(script_content)
|
||||
# except Exception as e:
|
||||
# raise e
|
||||
|
||||
def model_dump(self, **kwargs) -> dict[str, Any]:
|
||||
"""Custom serialization that properly uses AgentHistory's model_dump"""
|
||||
return {
|
||||
'history': [h.model_dump(**kwargs) for h in self.history],
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def load_from_file(cls, filepath: str | Path, output_model: type[AgentOutput]) -> AgentHistoryList:
|
||||
"""Load history from JSON file"""
|
||||
with open(filepath, encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
# loop through history and validate output_model actions to enrich with custom actions
|
||||
for h in data['history']:
|
||||
if h['model_output']:
|
||||
if isinstance(h['model_output'], dict):
|
||||
h['model_output'] = output_model.model_validate(h['model_output'])
|
||||
else:
|
||||
h['model_output'] = None
|
||||
if 'interacted_element' not in h['state']:
|
||||
h['state']['interacted_element'] = None
|
||||
history = cls.model_validate(data)
|
||||
return history
|
||||
|
||||
def last_action(self) -> None | dict:
|
||||
"""Last action in history"""
|
||||
if self.history and self.history[-1].model_output:
|
||||
return self.history[-1].model_output.action[-1].model_dump(exclude_none=True)
|
||||
return None
|
||||
|
||||
def errors(self) -> list[str | None]:
|
||||
"""Get all errors from history, with None for steps without errors"""
|
||||
errors = []
|
||||
for h in self.history:
|
||||
step_errors = [r.error for r in h.result if r.error]
|
||||
|
||||
# each step can have only one error
|
||||
errors.append(step_errors[0] if step_errors else None)
|
||||
return errors
|
||||
|
||||
def final_result(self) -> None | str:
|
||||
"""Final result from history"""
|
||||
if self.history and self.history[-1].result[-1].extracted_content:
|
||||
return self.history[-1].result[-1].extracted_content
|
||||
return None
|
||||
|
||||
def is_done(self) -> bool:
|
||||
"""Check if the agent is done"""
|
||||
if self.history and len(self.history[-1].result) > 0:
|
||||
last_result = self.history[-1].result[-1]
|
||||
return last_result.is_done is True
|
||||
return False
|
||||
|
||||
def is_successful(self) -> bool | None:
|
||||
"""Check if the agent completed successfully - the agent decides in the last step if it was successful or not. None if not done yet."""
|
||||
if self.history and len(self.history[-1].result) > 0:
|
||||
last_result = self.history[-1].result[-1]
|
||||
if last_result.is_done is True:
|
||||
return last_result.success
|
||||
return None
|
||||
|
||||
def has_errors(self) -> bool:
|
||||
"""Check if the agent has any non-None errors"""
|
||||
return any(error is not None for error in self.errors())
|
||||
|
||||
def urls(self) -> list[str | None]:
|
||||
"""Get all unique URLs from history"""
|
||||
return [h.state.url if h.state.url is not None else None for h in self.history]
|
||||
|
||||
def screenshot_paths(self, n_last: int | None = None, return_none_if_not_screenshot: bool = True) -> list[str | None]:
|
||||
"""Get all screenshot paths from history"""
|
||||
if n_last == 0:
|
||||
return []
|
||||
if n_last is None:
|
||||
if return_none_if_not_screenshot:
|
||||
return [h.state.screenshot_path if h.state.screenshot_path is not None else None for h in self.history]
|
||||
else:
|
||||
return [h.state.screenshot_path for h in self.history if h.state.screenshot_path is not None]
|
||||
else:
|
||||
if return_none_if_not_screenshot:
|
||||
return [h.state.screenshot_path if h.state.screenshot_path is not None else None for h in self.history[-n_last:]]
|
||||
else:
|
||||
return [h.state.screenshot_path for h in self.history[-n_last:] if h.state.screenshot_path is not None]
|
||||
|
||||
def screenshots(self, n_last: int | None = None, return_none_if_not_screenshot: bool = True) -> list[str | None]:
|
||||
"""Get all screenshots from history as base64 strings"""
|
||||
if n_last == 0:
|
||||
return []
|
||||
|
||||
history_items = self.history if n_last is None else self.history[-n_last:]
|
||||
screenshots = []
|
||||
|
||||
for item in history_items:
|
||||
screenshot_b64 = item.state.get_screenshot()
|
||||
if screenshot_b64:
|
||||
screenshots.append(screenshot_b64)
|
||||
else:
|
||||
if return_none_if_not_screenshot:
|
||||
screenshots.append(None)
|
||||
# If return_none_if_not_screenshot is False, we skip None values
|
||||
|
||||
return screenshots
|
||||
|
||||
def action_names(self) -> list[str]:
|
||||
"""Get all action names from history"""
|
||||
action_names = []
|
||||
for action in self.model_actions():
|
||||
actions = list(action.keys())
|
||||
if actions:
|
||||
action_names.append(actions[0])
|
||||
return action_names
|
||||
|
||||
def model_thoughts(self) -> list[AgentBrain]:
|
||||
"""Get all thoughts from history"""
|
||||
return [h.model_output.current_state for h in self.history if h.model_output]
|
||||
|
||||
def model_outputs(self) -> list[AgentOutput]:
|
||||
"""Get all model outputs from history"""
|
||||
return [h.model_output for h in self.history if h.model_output]
|
||||
|
||||
# get all actions with params
|
||||
def model_actions(self) -> list[dict]:
|
||||
"""Get all actions from history"""
|
||||
outputs = []
|
||||
|
||||
for h in self.history:
|
||||
if h.model_output:
|
||||
# Guard against None interacted_element before zipping
|
||||
interacted_elements = h.state.interacted_element or [None] * len(h.model_output.action)
|
||||
for action, interacted_element in zip(h.model_output.action, interacted_elements):
|
||||
output = action.model_dump(exclude_none=True)
|
||||
output['interacted_element'] = interacted_element
|
||||
outputs.append(output)
|
||||
return outputs
|
||||
|
||||
def action_history(self) -> list[list[dict]]:
|
||||
"""Get truncated action history with only essential fields"""
|
||||
step_outputs = []
|
||||
|
||||
for h in self.history:
|
||||
step_actions = []
|
||||
if h.model_output:
|
||||
# Guard against None interacted_element before zipping
|
||||
interacted_elements = h.state.interacted_element or [None] * len(h.model_output.action)
|
||||
# Zip actions with interacted elements and results
|
||||
for action, interacted_element, result in zip(h.model_output.action, interacted_elements, h.result):
|
||||
action_output = action.model_dump(exclude_none=True)
|
||||
action_output['interacted_element'] = interacted_element
|
||||
# Only keep long_term_memory from result
|
||||
action_output['result'] = result.long_term_memory if result and result.long_term_memory else None
|
||||
step_actions.append(action_output)
|
||||
step_outputs.append(step_actions)
|
||||
|
||||
return step_outputs
|
||||
|
||||
def action_results(self) -> list[ActionResult]:
|
||||
"""Get all results from history"""
|
||||
results = []
|
||||
for h in self.history:
|
||||
results.extend([r for r in h.result if r])
|
||||
return results
|
||||
|
||||
def extracted_content(self) -> list[str]:
|
||||
"""Get all extracted content from history"""
|
||||
content = []
|
||||
for h in self.history:
|
||||
content.extend([r.extracted_content for r in h.result if r.extracted_content])
|
||||
return content
|
||||
|
||||
def model_actions_filtered(self, include: list[str] | None = None) -> list[dict]:
|
||||
"""Get all model actions from history as JSON"""
|
||||
if include is None:
|
||||
include = []
|
||||
outputs = self.model_actions()
|
||||
result = []
|
||||
for o in outputs:
|
||||
for i in include:
|
||||
if i == list(o.keys())[0]:
|
||||
result.append(o)
|
||||
return result
|
||||
|
||||
def number_of_steps(self) -> int:
|
||||
"""Get the number of steps in the history"""
|
||||
return len(self.history)
|
||||
|
||||
@property
|
||||
def structured_output(self) -> AgentStructuredOutput | None:
|
||||
"""Get the structured output from the history
|
||||
|
||||
Returns:
|
||||
The structured output if both final_result and _output_model_schema are available,
|
||||
otherwise None
|
||||
"""
|
||||
final_result = self.final_result()
|
||||
if final_result is not None and self._output_model_schema is not None:
|
||||
return self._output_model_schema.model_validate_json(final_result)
|
||||
|
||||
return None
|
||||
|
||||
|
||||
class AgentError:
|
||||
"""Container for agent error handling"""
|
||||
|
||||
VALIDATION_ERROR = 'Invalid model output format. Please follow the correct schema.'
|
||||
RATE_LIMIT_ERROR = 'Rate limit reached. Waiting before retry.'
|
||||
NO_VALID_ACTION = 'No valid action found'
|
||||
|
||||
@staticmethod
|
||||
def format_error(error: Exception, include_trace: bool = False) -> str:
|
||||
"""Format error message based on error type and optionally include trace"""
|
||||
message = ''
|
||||
if isinstance(error, ValidationError):
|
||||
return f'{AgentError.VALIDATION_ERROR}\nDetails: {str(error)}'
|
||||
if isinstance(error, RateLimitError):
|
||||
return AgentError.RATE_LIMIT_ERROR
|
||||
if include_trace:
|
||||
return f'{str(error)}\nStacktrace:\n{traceback.format_exc()}'
|
||||
return f'{str(error)}'
|
||||
@@ -0,0 +1,41 @@
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
# Type stubs for lazy imports
|
||||
if TYPE_CHECKING:
|
||||
from .profile import BrowserProfile, ProxySettings
|
||||
from .session import BrowserSession
|
||||
|
||||
|
||||
# Lazy imports mapping for heavy browser components
|
||||
_LAZY_IMPORTS = {
|
||||
'ProxySettings': ('.profile', 'ProxySettings'),
|
||||
'BrowserProfile': ('.profile', 'BrowserProfile'),
|
||||
'BrowserSession': ('.session', 'BrowserSession'),
|
||||
}
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
"""Lazy import mechanism for heavy browser components."""
|
||||
if name in _LAZY_IMPORTS:
|
||||
module_path, attr_name = _LAZY_IMPORTS[name]
|
||||
try:
|
||||
from importlib import import_module
|
||||
|
||||
# Use relative import for current package
|
||||
full_module_path = f'browser_use.browser{module_path}'
|
||||
module = import_module(full_module_path)
|
||||
attr = getattr(module, attr_name)
|
||||
# Cache the imported attribute in the module's globals
|
||||
globals()[name] = attr
|
||||
return attr
|
||||
except ImportError as e:
|
||||
raise ImportError(f'Failed to import {name} from {full_module_path}: {e}') from e
|
||||
|
||||
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
|
||||
|
||||
|
||||
__all__ = [
|
||||
'BrowserSession',
|
||||
'BrowserProfile',
|
||||
'ProxySettings',
|
||||
]
|
||||
@@ -0,0 +1,287 @@
|
||||
"""Cloud browser service integration for browser-use.
|
||||
|
||||
This module provides integration with the browser-use cloud browser service.
|
||||
When cloud_browser=True, it automatically creates a cloud browser instance
|
||||
and returns the CDP URL for connection.
|
||||
"""
|
||||
|
||||
import logging
|
||||
import os
|
||||
|
||||
import httpx
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from browser_use.sync.auth import CloudAuthConfig
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class CloudBrowserResponse(BaseModel):
|
||||
"""Response from cloud browser API."""
|
||||
|
||||
id: str
|
||||
status: str
|
||||
liveUrl: str = Field(alias='liveUrl')
|
||||
cdpUrl: str = Field(alias='cdpUrl')
|
||||
timeoutAt: str = Field(alias='timeoutAt')
|
||||
startedAt: str = Field(alias='startedAt')
|
||||
finishedAt: str | None = Field(alias='finishedAt', default=None)
|
||||
|
||||
|
||||
class CloudBrowserError(Exception):
|
||||
"""Exception raised when cloud browser operations fail."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class CloudBrowserAuthError(CloudBrowserError):
|
||||
"""Exception raised when cloud browser authentication fails."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class CloudBrowserClient:
|
||||
"""Client for browser-use cloud browser service."""
|
||||
|
||||
def __init__(self, api_base_url: str = 'https://api.browser-use.com'):
|
||||
self.api_base_url = api_base_url
|
||||
self.client = httpx.AsyncClient(timeout=30.0)
|
||||
self.current_session_id: str | None = None
|
||||
|
||||
async def create_browser(self) -> CloudBrowserResponse:
|
||||
"""Create a new cloud browser instance.
|
||||
|
||||
Returns:
|
||||
CloudBrowserResponse: Contains CDP URL and other browser info
|
||||
|
||||
Raises:
|
||||
CloudBrowserAuthError: If authentication fails
|
||||
CloudBrowserError: If browser creation fails
|
||||
"""
|
||||
url = f'{self.api_base_url}/api/v2/browsers'
|
||||
|
||||
# Try to get API key from environment variable first, then auth config
|
||||
api_token = os.getenv('BROWSER_USE_API_KEY')
|
||||
|
||||
if not api_token:
|
||||
# Fallback to auth config file
|
||||
try:
|
||||
auth_config = CloudAuthConfig.load_from_file()
|
||||
api_token = auth_config.api_token
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not api_token:
|
||||
raise CloudBrowserAuthError(
|
||||
'No authentication token found. Please set BROWSER_USE_API_KEY environment variable to authenticate with the cloud service. You can also create an API key at https://cloud.browser-use.com'
|
||||
)
|
||||
|
||||
headers = {'X-Browser-Use-API-Key': api_token, 'Content-Type': 'application/json'}
|
||||
|
||||
# Empty request body as per API specification
|
||||
request_body = {}
|
||||
|
||||
try:
|
||||
logger.info('🌤️ Creating cloud browser instance...')
|
||||
|
||||
response = await self.client.post(url, headers=headers, json=request_body)
|
||||
|
||||
if response.status_code == 401:
|
||||
raise CloudBrowserAuthError(
|
||||
'Authentication failed. Please make sure you have set BROWSER_USE_API_KEY environment variable to authenticate with the cloud service. You can also create an API key at https://cloud.browser-use.com'
|
||||
)
|
||||
elif response.status_code == 403:
|
||||
raise CloudBrowserAuthError('Access forbidden. Please check your browser-use cloud subscription status.')
|
||||
elif not response.is_success:
|
||||
error_msg = f'Failed to create cloud browser: HTTP {response.status_code}'
|
||||
try:
|
||||
error_data = response.json()
|
||||
if 'detail' in error_data:
|
||||
error_msg += f' - {error_data["detail"]}'
|
||||
except Exception:
|
||||
pass
|
||||
raise CloudBrowserError(error_msg)
|
||||
|
||||
browser_data = response.json()
|
||||
browser_response = CloudBrowserResponse(**browser_data)
|
||||
|
||||
# Store session ID for cleanup
|
||||
self.current_session_id = browser_response.id
|
||||
|
||||
logger.info(f'🌤️ Cloud browser created successfully: {browser_response.id}')
|
||||
logger.debug(f'🌤️ CDP URL: {browser_response.cdpUrl}')
|
||||
# Cyan color for live URL
|
||||
logger.info(f'\033[36m🔗 Live URL: {browser_response.liveUrl}\033[0m')
|
||||
|
||||
return browser_response
|
||||
|
||||
except httpx.TimeoutException:
|
||||
raise CloudBrowserError('Timeout while creating cloud browser. Please try again.')
|
||||
except httpx.ConnectError:
|
||||
raise CloudBrowserError('Failed to connect to cloud browser service. Please check your internet connection.')
|
||||
except Exception as e:
|
||||
if isinstance(e, (CloudBrowserError, CloudBrowserAuthError)):
|
||||
raise
|
||||
raise CloudBrowserError(f'Unexpected error creating cloud browser: {e}')
|
||||
|
||||
async def stop_browser(self, session_id: str | None = None) -> CloudBrowserResponse:
|
||||
"""Stop a cloud browser session.
|
||||
|
||||
Args:
|
||||
session_id: Session ID to stop. If None, uses current session.
|
||||
|
||||
Returns:
|
||||
CloudBrowserResponse: Updated browser info with stopped status
|
||||
|
||||
Raises:
|
||||
CloudBrowserAuthError: If authentication fails
|
||||
CloudBrowserError: If stopping fails
|
||||
"""
|
||||
if session_id is None:
|
||||
session_id = self.current_session_id
|
||||
|
||||
if not session_id:
|
||||
raise CloudBrowserError('No session ID provided and no current session available')
|
||||
|
||||
url = f'{self.api_base_url}/api/v2/browsers/{session_id}'
|
||||
|
||||
# Try to get API key from environment variable first, then auth config
|
||||
api_token = os.getenv('BROWSER_USE_API_KEY')
|
||||
|
||||
if not api_token:
|
||||
# Fallback to auth config file
|
||||
try:
|
||||
auth_config = CloudAuthConfig.load_from_file()
|
||||
api_token = auth_config.api_token
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
if not api_token:
|
||||
raise CloudBrowserAuthError(
|
||||
'No authentication token found. Please set BROWSER_USE_API_KEY environment variable to authenticate with the cloud service. You can also create an API key at https://cloud.browser-use.com'
|
||||
)
|
||||
|
||||
headers = {'X-Browser-Use-API-Key': api_token, 'Content-Type': 'application/json'}
|
||||
|
||||
request_body = {'action': 'stop'}
|
||||
|
||||
try:
|
||||
logger.info(f'🌤️ Stopping cloud browser session: {session_id}')
|
||||
|
||||
response = await self.client.patch(url, headers=headers, json=request_body)
|
||||
|
||||
if response.status_code == 401:
|
||||
raise CloudBrowserAuthError(
|
||||
'Authentication failed. Please make sure you have set the BROWSER_USE_API_KEY environment variable to authenticate with the cloud service.'
|
||||
)
|
||||
elif response.status_code == 404:
|
||||
# Session already stopped or doesn't exist - treating as error and clearing session
|
||||
logger.debug(f'🌤️ Cloud browser session {session_id} not found (already stopped)')
|
||||
# Clear current session if it was this one
|
||||
if session_id == self.current_session_id:
|
||||
self.current_session_id = None
|
||||
raise CloudBrowserError(f'Cloud browser session {session_id} not found')
|
||||
elif not response.is_success:
|
||||
error_msg = f'Failed to stop cloud browser: HTTP {response.status_code}'
|
||||
try:
|
||||
error_data = response.json()
|
||||
if 'detail' in error_data:
|
||||
error_msg += f' - {error_data["detail"]}'
|
||||
except Exception:
|
||||
pass
|
||||
raise CloudBrowserError(error_msg)
|
||||
|
||||
browser_data = response.json()
|
||||
browser_response = CloudBrowserResponse(**browser_data)
|
||||
|
||||
# Clear current session if it was this one
|
||||
if session_id == self.current_session_id:
|
||||
self.current_session_id = None
|
||||
|
||||
logger.info(f'🌤️ Cloud browser session stopped: {browser_response.id}')
|
||||
logger.debug(f'🌤️ Status: {browser_response.status}')
|
||||
|
||||
return browser_response
|
||||
|
||||
except httpx.TimeoutException:
|
||||
raise CloudBrowserError('Timeout while stopping cloud browser. Please try again.')
|
||||
except httpx.ConnectError:
|
||||
raise CloudBrowserError('Failed to connect to cloud browser service. Please check your internet connection.')
|
||||
except Exception as e:
|
||||
if isinstance(e, (CloudBrowserError, CloudBrowserAuthError)):
|
||||
raise
|
||||
raise CloudBrowserError(f'Unexpected error stopping cloud browser: {e}')
|
||||
|
||||
async def close(self):
|
||||
"""Close the HTTP client and cleanup any active sessions."""
|
||||
# Try to stop current session if active
|
||||
if self.current_session_id:
|
||||
try:
|
||||
await self.stop_browser()
|
||||
except Exception as e:
|
||||
logger.debug(f'Failed to stop cloud browser session during cleanup: {e}')
|
||||
|
||||
await self.client.aclose()
|
||||
|
||||
|
||||
# Global client instance
|
||||
_cloud_client: CloudBrowserClient | None = None
|
||||
|
||||
|
||||
async def get_cloud_browser_cdp_url() -> str:
|
||||
"""Get a CDP URL for a new cloud browser instance.
|
||||
|
||||
Returns:
|
||||
str: CDP URL for connecting to the cloud browser
|
||||
|
||||
Raises:
|
||||
CloudBrowserAuthError: If authentication fails
|
||||
CloudBrowserError: If browser creation fails
|
||||
"""
|
||||
global _cloud_client
|
||||
|
||||
if _cloud_client is None:
|
||||
_cloud_client = CloudBrowserClient()
|
||||
|
||||
try:
|
||||
browser_response = await _cloud_client.create_browser()
|
||||
return browser_response.cdpUrl
|
||||
except Exception:
|
||||
# Clean up client on error
|
||||
if _cloud_client:
|
||||
await _cloud_client.close()
|
||||
_cloud_client = None
|
||||
raise
|
||||
|
||||
|
||||
async def stop_cloud_browser_session(session_id: str | None = None) -> CloudBrowserResponse:
|
||||
"""Stop a cloud browser session.
|
||||
|
||||
Args:
|
||||
session_id: Session ID to stop. If None, uses current session from global client.
|
||||
|
||||
Returns:
|
||||
CloudBrowserResponse: Updated browser info with stopped status
|
||||
|
||||
Raises:
|
||||
CloudBrowserAuthError: If authentication fails
|
||||
CloudBrowserError: If stopping fails
|
||||
"""
|
||||
global _cloud_client
|
||||
|
||||
if _cloud_client is None:
|
||||
_cloud_client = CloudBrowserClient()
|
||||
|
||||
try:
|
||||
return await _cloud_client.stop_browser(session_id)
|
||||
except Exception:
|
||||
# Don't clean up client on stop errors - session might still be valid
|
||||
raise
|
||||
|
||||
|
||||
async def cleanup_cloud_client():
|
||||
"""Clean up the global cloud client."""
|
||||
global _cloud_client
|
||||
if _cloud_client:
|
||||
await _cloud_client.close()
|
||||
_cloud_client = None
|
||||
@@ -0,0 +1,584 @@
|
||||
"""Event definitions for browser communication."""
|
||||
|
||||
import inspect
|
||||
import os
|
||||
from typing import Any, Literal
|
||||
|
||||
from bubus import BaseEvent
|
||||
from bubus.models import T_EventResultType
|
||||
from cdp_use.cdp.target import TargetID
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
|
||||
from browser_use.browser.views import BrowserStateSummary
|
||||
from browser_use.dom.views import EnhancedDOMTreeNode
|
||||
|
||||
|
||||
def _get_timeout(env_var: str, default: float) -> float | None:
|
||||
"""
|
||||
Safely parse environment variable timeout values with robust error handling.
|
||||
|
||||
Args:
|
||||
env_var: Environment variable name (e.g. 'TIMEOUT_NavigateToUrlEvent')
|
||||
default: Default timeout value as float (e.g. 15.0)
|
||||
|
||||
Returns:
|
||||
Parsed float value or the default if parsing fails
|
||||
|
||||
Raises:
|
||||
ValueError: Only if both env_var and default are invalid (should not happen with valid defaults)
|
||||
"""
|
||||
# Try environment variable first
|
||||
env_value = os.getenv(env_var)
|
||||
if env_value:
|
||||
try:
|
||||
parsed = float(env_value)
|
||||
if parsed < 0:
|
||||
print(f'Warning: {env_var}={env_value} is negative, using default {default}')
|
||||
return default
|
||||
return parsed
|
||||
except (ValueError, TypeError):
|
||||
print(f'Warning: {env_var}={env_value} is not a valid number, using default {default}')
|
||||
|
||||
# Fall back to default
|
||||
return default
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Agent/Tools -> BrowserSession Events (High-level browser actions)
|
||||
# ============================================================================
|
||||
|
||||
|
||||
class ElementSelectedEvent(BaseEvent[T_EventResultType]):
|
||||
"""An element was selected."""
|
||||
|
||||
node: EnhancedDOMTreeNode
|
||||
|
||||
@field_validator('node', mode='before')
|
||||
@classmethod
|
||||
def serialize_node(cls, data: EnhancedDOMTreeNode | None) -> EnhancedDOMTreeNode | None:
|
||||
if data is None:
|
||||
return None
|
||||
return EnhancedDOMTreeNode(
|
||||
element_index=data.element_index,
|
||||
node_id=data.node_id,
|
||||
backend_node_id=data.backend_node_id,
|
||||
session_id=data.session_id,
|
||||
frame_id=data.frame_id,
|
||||
target_id=data.target_id,
|
||||
node_type=data.node_type,
|
||||
node_name=data.node_name,
|
||||
node_value=data.node_value,
|
||||
attributes=data.attributes,
|
||||
is_scrollable=data.is_scrollable,
|
||||
is_visible=data.is_visible,
|
||||
absolute_position=data.absolute_position,
|
||||
# override the circular reference fields in EnhancedDOMTreeNode as they cant be serialized and aren't needed by event handlers
|
||||
# only used internally by the DOM service during DOM tree building process, not intended public API use
|
||||
content_document=None,
|
||||
shadow_root_type=None,
|
||||
shadow_roots=[],
|
||||
parent_node=None,
|
||||
children_nodes=[],
|
||||
ax_node=None,
|
||||
snapshot_node=None,
|
||||
)
|
||||
|
||||
|
||||
# TODO: add page handle to events
|
||||
# class PageHandle(share a base with browser.session.CDPSession?):
|
||||
# url: str
|
||||
# target_id: TargetID
|
||||
# @classmethod
|
||||
# def from_target_id(cls, target_id: TargetID) -> Self:
|
||||
# return cls(target_id=target_id)
|
||||
# @classmethod
|
||||
# def from_target_id(cls, target_id: TargetID) -> Self:
|
||||
# return cls(target_id=target_id)
|
||||
# @classmethod
|
||||
# def from_url(cls, url: str) -> Self:
|
||||
# @property
|
||||
# def root_frame_id(self) -> str:
|
||||
# return self.target_id
|
||||
# @property
|
||||
# def session_id(self) -> str:
|
||||
# return browser_session.get_or_create_cdp_session(self.target_id).session_id
|
||||
|
||||
# class PageSelectedEvent(BaseEvent[T_EventResultType]):
|
||||
# """An event like SwitchToTabEvent(page=PageHandle) or CloseTabEvent(page=PageHandle)"""
|
||||
# page: PageHandle
|
||||
|
||||
|
||||
class NavigateToUrlEvent(BaseEvent[None]):
|
||||
"""Navigate to a specific URL."""
|
||||
|
||||
url: str
|
||||
wait_until: Literal['load', 'domcontentloaded', 'networkidle', 'commit'] = 'load'
|
||||
timeout_ms: int | None = None
|
||||
new_tab: bool = Field(
|
||||
default=False, description='Set True to leave the current tab alone and open a new tab in the foreground for the new URL'
|
||||
)
|
||||
# existing_tab: PageHandle | None = None # TODO
|
||||
|
||||
# time limits enforced by bubus, not exposed to LLM:
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_NavigateToUrlEvent', 15.0) # seconds
|
||||
|
||||
|
||||
class ClickElementEvent(ElementSelectedEvent[dict[str, Any] | None]):
|
||||
"""Click an element."""
|
||||
|
||||
node: 'EnhancedDOMTreeNode'
|
||||
button: Literal['left', 'right', 'middle'] = 'left'
|
||||
while_holding_ctrl: bool = Field(
|
||||
default=False,
|
||||
description='Set True to open any link clicked in a new tab in the background, can use switch_tab(tab_id=None) after to focus it',
|
||||
)
|
||||
# click_count: int = 1 # TODO
|
||||
# expect_download: bool = False # moved to downloads_watchdog.py
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_ClickElementEvent', 15.0) # seconds
|
||||
|
||||
|
||||
class TypeTextEvent(ElementSelectedEvent[dict | None]):
|
||||
"""Type text into an element."""
|
||||
|
||||
node: 'EnhancedDOMTreeNode'
|
||||
text: str
|
||||
clear_existing: bool = True
|
||||
is_sensitive: bool = False # Flag to indicate if text contains sensitive data
|
||||
sensitive_key_name: str | None = None # Name of the sensitive key being typed (e.g., 'username', 'password')
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_TypeTextEvent', 15.0) # seconds
|
||||
|
||||
|
||||
class ScrollEvent(ElementSelectedEvent[None]):
|
||||
"""Scroll the page or element."""
|
||||
|
||||
direction: Literal['up', 'down', 'left', 'right']
|
||||
amount: int # pixels
|
||||
node: 'EnhancedDOMTreeNode | None' = None # None means scroll page
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_ScrollEvent', 8.0) # seconds
|
||||
|
||||
|
||||
class SwitchTabEvent(BaseEvent[TargetID]):
|
||||
"""Switch to a different tab."""
|
||||
|
||||
target_id: TargetID | None = Field(default=None, description='None means switch to the most recently opened tab')
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_SwitchTabEvent', 10.0) # seconds
|
||||
|
||||
|
||||
class CloseTabEvent(BaseEvent[None]):
|
||||
"""Close a tab."""
|
||||
|
||||
target_id: TargetID
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_CloseTabEvent', 10.0) # seconds
|
||||
|
||||
|
||||
class ScreenshotEvent(BaseEvent[str]):
|
||||
"""Request to take a screenshot."""
|
||||
|
||||
full_page: bool = False
|
||||
clip: dict[str, float] | None = None # {x, y, width, height}
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_ScreenshotEvent', 8.0) # seconds
|
||||
|
||||
|
||||
class BrowserStateRequestEvent(BaseEvent[BrowserStateSummary]):
|
||||
"""Request current browser state."""
|
||||
|
||||
include_dom: bool = True
|
||||
include_screenshot: bool = True
|
||||
cache_clickable_elements_hashes: bool = True
|
||||
include_recent_events: bool = False
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserStateRequestEvent', 30.0) # seconds
|
||||
|
||||
|
||||
# class WaitForConditionEvent(BaseEvent):
|
||||
# """Wait for a condition."""
|
||||
|
||||
# condition: Literal['navigation', 'selector', 'timeout', 'load_state']
|
||||
# timeout: float = 30000
|
||||
# selector: str | None = None
|
||||
# state: Literal['attached', 'detached', 'visible', 'hidden'] | None = None
|
||||
|
||||
|
||||
class GoBackEvent(BaseEvent[None]):
|
||||
"""Navigate back in browser history."""
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_GoBackEvent', 15.0) # seconds
|
||||
|
||||
|
||||
class GoForwardEvent(BaseEvent[None]):
|
||||
"""Navigate forward in browser history."""
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_GoForwardEvent', 15.0) # seconds
|
||||
|
||||
|
||||
class RefreshEvent(BaseEvent[None]):
|
||||
"""Refresh/reload the current page."""
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_RefreshEvent', 15.0) # seconds
|
||||
|
||||
|
||||
class WaitEvent(BaseEvent[None]):
|
||||
"""Wait for a specified number of seconds."""
|
||||
|
||||
seconds: float = 3.0
|
||||
max_seconds: float = 10.0 # Safety cap
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_WaitEvent', 60.0) # seconds
|
||||
|
||||
|
||||
class SendKeysEvent(BaseEvent[None]):
|
||||
"""Send keyboard keys/shortcuts."""
|
||||
|
||||
keys: str # e.g., "ctrl+a", "cmd+c", "Enter"
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_SendKeysEvent', 15.0) # seconds
|
||||
|
||||
|
||||
class UploadFileEvent(ElementSelectedEvent[None]):
|
||||
"""Upload a file to an element."""
|
||||
|
||||
node: 'EnhancedDOMTreeNode'
|
||||
file_path: str
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_UploadFileEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class GetDropdownOptionsEvent(ElementSelectedEvent[dict[str, str]]):
|
||||
"""Get all options from any dropdown (native <select>, ARIA menus, or custom dropdowns).
|
||||
|
||||
Returns a dict containing dropdown type, options list, and element metadata."""
|
||||
|
||||
node: 'EnhancedDOMTreeNode'
|
||||
|
||||
event_timeout: float | None = _get_timeout(
|
||||
'TIMEOUT_GetDropdownOptionsEvent',
|
||||
15.0,
|
||||
) # some dropdowns lazy-load the list of options on first interaction, so we need to wait for them to load (e.g. table filter lists can have thousands of options)
|
||||
|
||||
|
||||
class SelectDropdownOptionEvent(ElementSelectedEvent[dict[str, str]]):
|
||||
"""Select a dropdown option by exact text from any dropdown type.
|
||||
|
||||
Returns a dict containing success status and selection details."""
|
||||
|
||||
node: 'EnhancedDOMTreeNode'
|
||||
text: str # The option text to select
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_SelectDropdownOptionEvent', 8.0) # seconds
|
||||
|
||||
|
||||
class ScrollToTextEvent(BaseEvent[None]):
|
||||
"""Scroll to specific text on the page. Raises exception if text not found."""
|
||||
|
||||
text: str
|
||||
direction: Literal['up', 'down'] = 'down'
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_ScrollToTextEvent', 15.0) # seconds
|
||||
|
||||
|
||||
# ============================================================================
|
||||
|
||||
|
||||
class BrowserStartEvent(BaseEvent):
|
||||
"""Start/connect to browser."""
|
||||
|
||||
cdp_url: str | None = None
|
||||
launch_options: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserStartEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class BrowserStopEvent(BaseEvent):
|
||||
"""Stop/disconnect from browser."""
|
||||
|
||||
force: bool = False
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserStopEvent', 45.0) # seconds
|
||||
|
||||
|
||||
class BrowserLaunchResult(BaseModel):
|
||||
"""Result of launching a browser."""
|
||||
|
||||
# TODO: add browser executable_path, pid, version, latency, user_data_dir, X11 $DISPLAY, host IP address, etc.
|
||||
cdp_url: str
|
||||
|
||||
|
||||
class BrowserLaunchEvent(BaseEvent[BrowserLaunchResult]):
|
||||
"""Launch a local browser process."""
|
||||
|
||||
# TODO: add executable_path, proxy settings, preferences, extra launch args, etc.
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserLaunchEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class BrowserKillEvent(BaseEvent):
|
||||
"""Kill local browser subprocess."""
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserKillEvent', 30.0) # seconds
|
||||
|
||||
|
||||
# TODO: replace all Runtime.evaluate() calls with this event
|
||||
# class ExecuteJavaScriptEvent(BaseEvent):
|
||||
# """Execute JavaScript in page context."""
|
||||
|
||||
# target_id: TargetID
|
||||
# expression: str
|
||||
# await_promise: bool = True
|
||||
|
||||
# event_timeout: float | None = 60.0 # seconds
|
||||
|
||||
# TODO: add this and use the old BrowserProfile.viewport options to set it
|
||||
# class SetViewportEvent(BaseEvent):
|
||||
# """Set the viewport size."""
|
||||
|
||||
# width: int
|
||||
# height: int
|
||||
# device_scale_factor: float = 1.0
|
||||
|
||||
# event_timeout: float | None = 15.0 # seconds
|
||||
|
||||
|
||||
# Moved to storage state
|
||||
# class SetCookiesEvent(BaseEvent):
|
||||
# """Set browser cookies."""
|
||||
|
||||
# cookies: list[dict[str, Any]]
|
||||
|
||||
# event_timeout: float | None = (
|
||||
# 30.0 # only long to support the edge case of restoring a big localStorage / on many origins (has to O(n) visit each origin to restore)
|
||||
# )
|
||||
|
||||
|
||||
# class GetCookiesEvent(BaseEvent):
|
||||
# """Get browser cookies."""
|
||||
|
||||
# urls: list[str] | None = None
|
||||
|
||||
# event_timeout: float | None = 30.0 # seconds
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# DOM-related Events
|
||||
# ============================================================================
|
||||
|
||||
|
||||
class BrowserConnectedEvent(BaseEvent):
|
||||
"""Browser has started/connected."""
|
||||
|
||||
cdp_url: str
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserConnectedEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class BrowserStoppedEvent(BaseEvent):
|
||||
"""Browser has stopped/disconnected."""
|
||||
|
||||
reason: str | None = None
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserStoppedEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class TabCreatedEvent(BaseEvent):
|
||||
"""A new tab was created."""
|
||||
|
||||
target_id: TargetID
|
||||
url: str
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_TabCreatedEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class TabClosedEvent(BaseEvent):
|
||||
"""A tab was closed."""
|
||||
|
||||
target_id: TargetID
|
||||
|
||||
# TODO:
|
||||
# new_focus_target_id: int | None = None
|
||||
# new_focus_url: str | None = None
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_TabClosedEvent', 10.0) # seconds
|
||||
|
||||
|
||||
# TODO: emit this when DOM changes significantly, inner frame navigates, form submits, history.pushState(), etc.
|
||||
# class TabUpdatedEvent(BaseEvent):
|
||||
# """Tab information updated (URL changed, etc.)."""
|
||||
|
||||
# target_id: TargetID
|
||||
# url: str
|
||||
|
||||
|
||||
class AgentFocusChangedEvent(BaseEvent):
|
||||
"""Agent focus changed to a different tab."""
|
||||
|
||||
target_id: TargetID
|
||||
url: str
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_AgentFocusChangedEvent', 10.0) # seconds
|
||||
|
||||
|
||||
class TargetCrashedEvent(BaseEvent):
|
||||
"""A target has crashed."""
|
||||
|
||||
target_id: TargetID
|
||||
error: str
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_TargetCrashedEvent', 10.0) # seconds
|
||||
|
||||
|
||||
class NavigationStartedEvent(BaseEvent):
|
||||
"""Navigation started."""
|
||||
|
||||
target_id: TargetID
|
||||
url: str
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_NavigationStartedEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class NavigationCompleteEvent(BaseEvent):
|
||||
"""Navigation completed."""
|
||||
|
||||
target_id: TargetID
|
||||
url: str
|
||||
status: int | None = None
|
||||
error_message: str | None = None # Error/timeout message if navigation had issues
|
||||
loading_status: str | None = None # Detailed loading status (e.g., network timeout info)
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_NavigationCompleteEvent', 30.0) # seconds
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Error Events
|
||||
# ============================================================================
|
||||
|
||||
|
||||
class BrowserErrorEvent(BaseEvent):
|
||||
"""An error occurred in the browser layer."""
|
||||
|
||||
error_type: str
|
||||
message: str
|
||||
details: dict[str, Any] = Field(default_factory=dict)
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_BrowserErrorEvent', 30.0) # seconds
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Storage State Events
|
||||
# ============================================================================
|
||||
|
||||
|
||||
class SaveStorageStateEvent(BaseEvent):
|
||||
"""Request to save browser storage state."""
|
||||
|
||||
path: str | None = None # Optional path, uses profile default if not provided
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_SaveStorageStateEvent', 45.0) # seconds
|
||||
|
||||
|
||||
class StorageStateSavedEvent(BaseEvent):
|
||||
"""Notification that storage state was saved."""
|
||||
|
||||
path: str
|
||||
cookies_count: int
|
||||
origins_count: int
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_StorageStateSavedEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class LoadStorageStateEvent(BaseEvent):
|
||||
"""Request to load browser storage state."""
|
||||
|
||||
path: str | None = None # Optional path, uses profile default if not provided
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_LoadStorageStateEvent', 45.0) # seconds
|
||||
|
||||
|
||||
# TODO: refactor this to:
|
||||
# - on_BrowserConnectedEvent() -> dispatch(LoadStorageStateEvent()) -> _copy_storage_state_from_json_to_browser(json_file, new_cdp_session) + return storage_state from handler
|
||||
# - on_BrowserStopEvent() -> dispatch(SaveStorageStateEvent()) -> _copy_storage_state_from_browser_to_json(new_cdp_session, json_file)
|
||||
# and get rid of StorageStateSavedEvent and StorageStateLoadedEvent, have the original events + provide handler return values for any results
|
||||
class StorageStateLoadedEvent(BaseEvent):
|
||||
"""Notification that storage state was loaded."""
|
||||
|
||||
path: str
|
||||
cookies_count: int
|
||||
origins_count: int
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_StorageStateLoadedEvent', 30.0) # seconds
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# File Download Events
|
||||
# ============================================================================
|
||||
|
||||
|
||||
class FileDownloadedEvent(BaseEvent):
|
||||
"""A file has been downloaded."""
|
||||
|
||||
url: str
|
||||
path: str
|
||||
file_name: str
|
||||
file_size: int
|
||||
file_type: str | None = None # e.g., 'pdf', 'zip', 'docx', etc.
|
||||
mime_type: str | None = None # e.g., 'application/pdf'
|
||||
from_cache: bool = False
|
||||
auto_download: bool = False # Whether this was an automatic download (e.g., PDF auto-download)
|
||||
|
||||
event_timeout: float | None = _get_timeout('TIMEOUT_FileDownloadedEvent', 30.0) # seconds
|
||||
|
||||
|
||||
class AboutBlankDVDScreensaverShownEvent(BaseEvent):
|
||||
"""AboutBlankWatchdog has shown DVD screensaver animation on an about:blank tab."""
|
||||
|
||||
target_id: TargetID
|
||||
error: str | None = None
|
||||
|
||||
|
||||
class DialogOpenedEvent(BaseEvent):
|
||||
"""Event dispatched when a JavaScript dialog is opened and handled."""
|
||||
|
||||
dialog_type: str # 'alert', 'confirm', 'prompt', or 'beforeunload'
|
||||
message: str
|
||||
url: str
|
||||
frame_id: str | None = None # Can be None when frameId is not provided by CDP
|
||||
# target_id: TargetID # TODO: add this to avoid needing target_id_from_frame() later
|
||||
|
||||
|
||||
# Note: Model rebuilding for forward references is handled in the importing modules
|
||||
# Events with 'EnhancedDOMTreeNode' forward references (ClickElementEvent, TypeTextEvent,
|
||||
# ScrollEvent, UploadFileEvent) need model_rebuild() called after imports are complete
|
||||
|
||||
|
||||
def _check_event_names_dont_overlap():
|
||||
"""
|
||||
check that event names defined in this file are valid and non-overlapping
|
||||
(naiively n^2 so it's pretty slow but ok for now, optimize when >20 events)
|
||||
"""
|
||||
event_names = {
|
||||
name.split('[')[0]
|
||||
for name in globals().keys()
|
||||
if not name.startswith('_')
|
||||
and inspect.isclass(globals()[name])
|
||||
and issubclass(globals()[name], BaseEvent)
|
||||
and name != 'BaseEvent'
|
||||
}
|
||||
for name_a in event_names:
|
||||
assert name_a.endswith('Event'), f'Event with name {name_a} does not end with "Event"'
|
||||
for name_b in event_names:
|
||||
if name_a != name_b: # Skip self-comparison
|
||||
assert name_a not in name_b, (
|
||||
f'Event with name {name_a} is a substring of {name_b}, all events must be completely unique to avoid find-and-replace accidents'
|
||||
)
|
||||
|
||||
|
||||
# overlapping event names are a nightmare to trace and rename later, dont do it!
|
||||
# e.g. prevent ClickEvent and FailedClickEvent are terrible names because one is a substring of the other,
|
||||
# must be ClickEvent and ClickFailedEvent to preserve the usefulnes of codebase grep/sed/awk as refactoring tools.
|
||||
# at import time, we do a quick check that all event names defined above are valid and non-overlapping.
|
||||
# this is hand written in blood by a human! not LLM slop. feel free to optimize but do not remove it without a good reason.
|
||||
_check_event_names_dont_overlap()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,548 @@
|
||||
"""Python-based highlighting system for drawing bounding boxes on screenshots.
|
||||
|
||||
This module replaces JavaScript-based highlighting with fast Python image processing
|
||||
to draw bounding boxes around interactive elements directly on screenshots.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import io
|
||||
import logging
|
||||
import os
|
||||
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
from browser_use.dom.views import DOMSelectorMap, EnhancedDOMTreeNode
|
||||
from browser_use.observability import observe_debug
|
||||
from browser_use.utils import time_execution_async
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Font cache to prevent repeated font loading and reduce memory usage
|
||||
_FONT_CACHE: dict[tuple[str, int], ImageFont.FreeTypeFont | None] = {}
|
||||
|
||||
# Cross-platform font paths
|
||||
_FONT_PATHS = [
|
||||
'/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf', # Linux (Debian/Ubuntu)
|
||||
'/usr/share/fonts/TTF/DejaVuSans-Bold.ttf', # Linux (Arch/Fedora)
|
||||
'/System/Library/Fonts/Arial.ttf', # macOS
|
||||
'C:\\Windows\\Fonts\\arial.ttf', # Windows
|
||||
'arial.ttf', # Windows (system path)
|
||||
'Arial Bold.ttf', # macOS alternative
|
||||
'/usr/share/fonts/truetype/liberation/LiberationSans-Bold.ttf', # Linux alternative
|
||||
]
|
||||
|
||||
|
||||
def get_cross_platform_font(font_size: int) -> ImageFont.FreeTypeFont | None:
|
||||
"""Get a cross-platform compatible font with caching to prevent memory leaks.
|
||||
|
||||
Args:
|
||||
font_size: Size of the font to load
|
||||
|
||||
Returns:
|
||||
ImageFont object or None if no system fonts are available
|
||||
"""
|
||||
# Use cache key based on font size
|
||||
cache_key = ('system_font', font_size)
|
||||
|
||||
# Return cached font if available
|
||||
if cache_key in _FONT_CACHE:
|
||||
return _FONT_CACHE[cache_key]
|
||||
|
||||
# Try to load a system font
|
||||
font = None
|
||||
for font_path in _FONT_PATHS:
|
||||
try:
|
||||
font = ImageFont.truetype(font_path, font_size)
|
||||
break
|
||||
except OSError:
|
||||
continue
|
||||
|
||||
# Cache the result (even if None) to avoid repeated attempts
|
||||
_FONT_CACHE[cache_key] = font
|
||||
return font
|
||||
|
||||
|
||||
def cleanup_font_cache() -> None:
|
||||
"""Clean up the font cache to prevent memory leaks in long-running applications."""
|
||||
global _FONT_CACHE
|
||||
_FONT_CACHE.clear()
|
||||
|
||||
|
||||
# Color scheme for different element types
|
||||
ELEMENT_COLORS = {
|
||||
'button': '#FF6B6B', # Red for buttons
|
||||
'input': '#4ECDC4', # Teal for inputs
|
||||
'select': '#45B7D1', # Blue for dropdowns
|
||||
'a': '#96CEB4', # Green for links
|
||||
'textarea': '#FF8C42', # Orange for text areas (was yellow, now more visible)
|
||||
'default': '#DDA0DD', # Light purple for other interactive elements
|
||||
}
|
||||
|
||||
# Element type mappings
|
||||
ELEMENT_TYPE_MAP = {
|
||||
'button': 'button',
|
||||
'input': 'input',
|
||||
'select': 'select',
|
||||
'a': 'a',
|
||||
'textarea': 'textarea',
|
||||
}
|
||||
|
||||
|
||||
def get_element_color(tag_name: str, element_type: str | None = None) -> str:
|
||||
"""Get color for element based on tag name and type."""
|
||||
# Check input type first
|
||||
if tag_name == 'input' and element_type:
|
||||
if element_type in ['button', 'submit']:
|
||||
return ELEMENT_COLORS['button']
|
||||
|
||||
# Use tag-based color
|
||||
return ELEMENT_COLORS.get(tag_name.lower(), ELEMENT_COLORS['default'])
|
||||
|
||||
|
||||
def should_show_index_overlay(element_index: int | None) -> bool:
|
||||
"""Determine if index overlay should be shown."""
|
||||
return element_index is not None
|
||||
|
||||
|
||||
def draw_enhanced_bounding_box_with_text(
|
||||
draw, # ImageDraw.Draw - avoiding type annotation due to PIL typing issues
|
||||
bbox: tuple[int, int, int, int],
|
||||
color: str,
|
||||
text: str | None = None,
|
||||
font: ImageFont.FreeTypeFont | None = None,
|
||||
element_type: str = 'div',
|
||||
image_size: tuple[int, int] = (2000, 1500),
|
||||
device_pixel_ratio: float = 1.0,
|
||||
) -> None:
|
||||
"""Draw an enhanced bounding box with much bigger index containers and dashed borders."""
|
||||
x1, y1, x2, y2 = bbox
|
||||
|
||||
# Draw dashed bounding box with pattern: 1 line, 2 spaces, 1 line, 2 spaces...
|
||||
dash_length = 4
|
||||
gap_length = 8
|
||||
line_width = 2
|
||||
|
||||
# Helper function to draw dashed line
|
||||
def draw_dashed_line(start_x, start_y, end_x, end_y):
|
||||
if start_x == end_x: # Vertical line
|
||||
y = start_y
|
||||
while y < end_y:
|
||||
dash_end = min(y + dash_length, end_y)
|
||||
draw.line([(start_x, y), (start_x, dash_end)], fill=color, width=line_width)
|
||||
y += dash_length + gap_length
|
||||
else: # Horizontal line
|
||||
x = start_x
|
||||
while x < end_x:
|
||||
dash_end = min(x + dash_length, end_x)
|
||||
draw.line([(x, start_y), (dash_end, start_y)], fill=color, width=line_width)
|
||||
x += dash_length + gap_length
|
||||
|
||||
# Draw dashed rectangle
|
||||
draw_dashed_line(x1, y1, x2, y1) # Top
|
||||
draw_dashed_line(x2, y1, x2, y2) # Right
|
||||
draw_dashed_line(x2, y2, x1, y2) # Bottom
|
||||
draw_dashed_line(x1, y2, x1, y1) # Left
|
||||
|
||||
# Draw much bigger index overlay if we have index text
|
||||
if text:
|
||||
try:
|
||||
# Scale font size for appropriate sizing across different resolutions
|
||||
img_width, img_height = image_size
|
||||
|
||||
css_width = img_width # / device_pixel_ratio
|
||||
# Much smaller scaling - 1% of CSS viewport width, max 16px to prevent huge highlights
|
||||
base_font_size = max(10, min(20, int(css_width * 0.01)))
|
||||
# Use shared font loading function with caching
|
||||
big_font = get_cross_platform_font(base_font_size)
|
||||
if big_font is None:
|
||||
big_font = font # Fallback to original font if no system fonts found
|
||||
|
||||
# Get text size with bigger font
|
||||
if big_font:
|
||||
bbox_text = draw.textbbox((0, 0), text, font=big_font)
|
||||
text_width = bbox_text[2] - bbox_text[0]
|
||||
text_height = bbox_text[3] - bbox_text[1]
|
||||
else:
|
||||
# Fallback for default font
|
||||
bbox_text = draw.textbbox((0, 0), text)
|
||||
text_width = bbox_text[2] - bbox_text[0]
|
||||
text_height = bbox_text[3] - bbox_text[1]
|
||||
|
||||
# Scale padding appropriately for different resolutions
|
||||
padding = max(4, min(10, int(css_width * 0.005))) # 0.3% of CSS width, max 4px
|
||||
element_width = x2 - x1
|
||||
element_height = y2 - y1
|
||||
|
||||
# Container dimensions
|
||||
container_width = text_width + padding * 2
|
||||
container_height = text_height + padding * 2
|
||||
|
||||
# Position in top center - for small elements, place further up to avoid blocking content
|
||||
# Center horizontally within the element
|
||||
bg_x1 = x1 + (element_width - container_width) // 2
|
||||
|
||||
# Simple rule: if element is small, place index further up to avoid blocking icons
|
||||
if element_width < 60 or element_height < 30:
|
||||
# Small element: place well above to avoid blocking content
|
||||
bg_y1 = max(0, y1 - container_height - 5)
|
||||
else:
|
||||
# Regular element: place inside with small offset
|
||||
bg_y1 = y1 + 2
|
||||
|
||||
bg_x2 = bg_x1 + container_width
|
||||
bg_y2 = bg_y1 + container_height
|
||||
|
||||
# Center the number within the index box with proper baseline handling
|
||||
text_x = bg_x1 + (container_width - text_width) // 2
|
||||
# Add extra vertical space to prevent clipping
|
||||
text_y = bg_y1 + (container_height - text_height) // 2 - bbox_text[1] # Subtract top offset
|
||||
|
||||
# Ensure container stays within image bounds
|
||||
img_width, img_height = image_size
|
||||
if bg_x1 < 0:
|
||||
offset = -bg_x1
|
||||
bg_x1 += offset
|
||||
bg_x2 += offset
|
||||
text_x += offset
|
||||
if bg_y1 < 0:
|
||||
offset = -bg_y1
|
||||
bg_y1 += offset
|
||||
bg_y2 += offset
|
||||
text_y += offset
|
||||
if bg_x2 > img_width:
|
||||
offset = bg_x2 - img_width
|
||||
bg_x1 -= offset
|
||||
bg_x2 -= offset
|
||||
text_x -= offset
|
||||
if bg_y2 > img_height:
|
||||
offset = bg_y2 - img_height
|
||||
bg_y1 -= offset
|
||||
bg_y2 -= offset
|
||||
text_y -= offset
|
||||
|
||||
# Draw bigger background rectangle with thicker border
|
||||
draw.rectangle([bg_x1, bg_y1, bg_x2, bg_y2], fill=color, outline='white', width=2)
|
||||
|
||||
# Draw white text centered in the index box
|
||||
draw.text((text_x, text_y), text, fill='white', font=big_font or font)
|
||||
|
||||
except Exception as e:
|
||||
logger.debug(f'Failed to draw enhanced text overlay: {e}')
|
||||
|
||||
|
||||
def draw_bounding_box_with_text(
|
||||
draw, # ImageDraw.Draw - avoiding type annotation due to PIL typing issues
|
||||
bbox: tuple[int, int, int, int],
|
||||
color: str,
|
||||
text: str | None = None,
|
||||
font: ImageFont.FreeTypeFont | None = None,
|
||||
) -> None:
|
||||
"""Draw a bounding box with optional text overlay."""
|
||||
x1, y1, x2, y2 = bbox
|
||||
|
||||
# Draw dashed bounding box
|
||||
dash_length = 2
|
||||
gap_length = 6
|
||||
|
||||
# Top edge
|
||||
x = x1
|
||||
while x < x2:
|
||||
end_x = min(x + dash_length, x2)
|
||||
draw.line([(x, y1), (end_x, y1)], fill=color, width=2)
|
||||
draw.line([(x, y1 + 1), (end_x, y1 + 1)], fill=color, width=2)
|
||||
x += dash_length + gap_length
|
||||
|
||||
# Bottom edge
|
||||
x = x1
|
||||
while x < x2:
|
||||
end_x = min(x + dash_length, x2)
|
||||
draw.line([(x, y2), (end_x, y2)], fill=color, width=2)
|
||||
draw.line([(x, y2 - 1), (end_x, y2 - 1)], fill=color, width=2)
|
||||
x += dash_length + gap_length
|
||||
|
||||
# Left edge
|
||||
y = y1
|
||||
while y < y2:
|
||||
end_y = min(y + dash_length, y2)
|
||||
draw.line([(x1, y), (x1, end_y)], fill=color, width=2)
|
||||
draw.line([(x1 + 1, y), (x1 + 1, end_y)], fill=color, width=2)
|
||||
y += dash_length + gap_length
|
||||
|
||||
# Right edge
|
||||
y = y1
|
||||
while y < y2:
|
||||
end_y = min(y + dash_length, y2)
|
||||
draw.line([(x2, y), (x2, end_y)], fill=color, width=2)
|
||||
draw.line([(x2 - 1, y), (x2 - 1, end_y)], fill=color, width=2)
|
||||
y += dash_length + gap_length
|
||||
|
||||
# Draw index overlay if we have index text
|
||||
if text:
|
||||
try:
|
||||
# Get text size
|
||||
if font:
|
||||
bbox_text = draw.textbbox((0, 0), text, font=font)
|
||||
text_width = bbox_text[2] - bbox_text[0]
|
||||
text_height = bbox_text[3] - bbox_text[1]
|
||||
else:
|
||||
# Fallback for default font
|
||||
bbox_text = draw.textbbox((0, 0), text)
|
||||
text_width = bbox_text[2] - bbox_text[0]
|
||||
text_height = bbox_text[3] - bbox_text[1]
|
||||
|
||||
# Smart positioning based on element size
|
||||
padding = 5
|
||||
element_width = x2 - x1
|
||||
element_height = y2 - y1
|
||||
element_area = element_width * element_height
|
||||
index_box_area = (text_width + padding * 2) * (text_height + padding * 2)
|
||||
|
||||
# Calculate size ratio to determine positioning strategy
|
||||
size_ratio = element_area / max(index_box_area, 1)
|
||||
|
||||
if size_ratio < 4:
|
||||
# Very small elements: place outside in bottom-right corner
|
||||
text_x = x2 + padding
|
||||
text_y = y2 - text_height
|
||||
# Ensure it doesn't go off screen
|
||||
text_x = min(text_x, 1200 - text_width - padding)
|
||||
text_y = max(text_y, 0)
|
||||
elif size_ratio < 16:
|
||||
# Medium elements: place in bottom-right corner inside
|
||||
text_x = x2 - text_width - padding
|
||||
text_y = y2 - text_height - padding
|
||||
else:
|
||||
# Large elements: place in center
|
||||
text_x = x1 + (element_width - text_width) // 2
|
||||
text_y = y1 + (element_height - text_height) // 2
|
||||
|
||||
# Ensure text stays within bounds
|
||||
text_x = max(0, min(text_x, 1200 - text_width))
|
||||
text_y = max(0, min(text_y, 800 - text_height))
|
||||
|
||||
# Draw background rectangle for maximum contrast
|
||||
bg_x1 = text_x - padding
|
||||
bg_y1 = text_y - padding
|
||||
bg_x2 = text_x + text_width + padding
|
||||
bg_y2 = text_y + text_height + padding
|
||||
|
||||
# Use white background with thick black border for maximum visibility
|
||||
draw.rectangle([bg_x1, bg_y1, bg_x2, bg_y2], fill='white', outline='black', width=2)
|
||||
|
||||
# Draw bold dark text on light background for best contrast
|
||||
draw.text((text_x, text_y), text, fill='black', font=font)
|
||||
|
||||
except Exception as e:
|
||||
logger.debug(f'Failed to draw text overlay: {e}')
|
||||
|
||||
|
||||
def process_element_highlight(
|
||||
element_id: int,
|
||||
element: EnhancedDOMTreeNode,
|
||||
draw,
|
||||
device_pixel_ratio: float,
|
||||
font,
|
||||
filter_highlight_ids: bool,
|
||||
image_size: tuple[int, int],
|
||||
) -> None:
|
||||
"""Process a single element for highlighting."""
|
||||
try:
|
||||
# Use absolute_position coordinates directly
|
||||
if not element.absolute_position:
|
||||
return
|
||||
|
||||
bounds = element.absolute_position
|
||||
|
||||
# Scale coordinates from CSS pixels to device pixels for screenshot
|
||||
# The screenshot is captured at device pixel resolution, but coordinates are in CSS pixels
|
||||
x1 = int(bounds.x * device_pixel_ratio)
|
||||
y1 = int(bounds.y * device_pixel_ratio)
|
||||
x2 = int((bounds.x + bounds.width) * device_pixel_ratio)
|
||||
y2 = int((bounds.y + bounds.height) * device_pixel_ratio)
|
||||
|
||||
# Ensure coordinates are within image bounds
|
||||
img_width, img_height = image_size
|
||||
x1 = max(0, min(x1, img_width))
|
||||
y1 = max(0, min(y1, img_height))
|
||||
x2 = max(x1, min(x2, img_width))
|
||||
y2 = max(y1, min(y2, img_height))
|
||||
|
||||
# Skip if bounding box is too small or invalid
|
||||
if x2 - x1 < 2 or y2 - y1 < 2:
|
||||
return
|
||||
|
||||
# Get element color based on type
|
||||
tag_name = element.tag_name if hasattr(element, 'tag_name') else 'div'
|
||||
element_type = None
|
||||
if hasattr(element, 'attributes') and element.attributes:
|
||||
element_type = element.attributes.get('type')
|
||||
|
||||
color = get_element_color(tag_name, element_type)
|
||||
|
||||
# Get element index for overlay and apply filtering
|
||||
element_index = getattr(element, 'element_index', None)
|
||||
index_text = None
|
||||
|
||||
if element_index is not None:
|
||||
if filter_highlight_ids:
|
||||
# Use the meaningful text that matches what the LLM sees
|
||||
meaningful_text = element.get_meaningful_text_for_llm()
|
||||
# Show ID only if meaningful text is less than 5 characters
|
||||
if len(meaningful_text) < 3:
|
||||
index_text = str(element_index)
|
||||
else:
|
||||
# Always show ID when filter is disabled
|
||||
index_text = str(element_index)
|
||||
|
||||
# Draw enhanced bounding box with bigger index
|
||||
draw_enhanced_bounding_box_with_text(
|
||||
draw, (x1, y1, x2, y2), color, index_text, font, tag_name, image_size, device_pixel_ratio
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.debug(f'Failed to draw highlight for element {element_id}: {e}')
|
||||
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='create_highlighted_screenshot')
|
||||
@time_execution_async('create_highlighted_screenshot')
|
||||
async def create_highlighted_screenshot(
|
||||
screenshot_b64: str,
|
||||
selector_map: DOMSelectorMap,
|
||||
device_pixel_ratio: float = 1.0,
|
||||
viewport_offset_x: int = 0,
|
||||
viewport_offset_y: int = 0,
|
||||
filter_highlight_ids: bool = True,
|
||||
) -> str:
|
||||
"""Create a highlighted screenshot with bounding boxes around interactive elements.
|
||||
|
||||
Args:
|
||||
screenshot_b64: Base64 encoded screenshot
|
||||
selector_map: Map of interactive elements with their positions
|
||||
device_pixel_ratio: Device pixel ratio for scaling coordinates
|
||||
viewport_offset_x: X offset for viewport positioning
|
||||
viewport_offset_y: Y offset for viewport positioning
|
||||
|
||||
Returns:
|
||||
Base64 encoded highlighted screenshot
|
||||
"""
|
||||
try:
|
||||
# Decode screenshot
|
||||
screenshot_data = base64.b64decode(screenshot_b64)
|
||||
image = Image.open(io.BytesIO(screenshot_data)).convert('RGBA')
|
||||
|
||||
# Create drawing context
|
||||
draw = ImageDraw.Draw(image)
|
||||
|
||||
# Load font using shared function with caching
|
||||
font = get_cross_platform_font(12)
|
||||
# If no system fonts found, font remains None and will use default font
|
||||
|
||||
# Process elements sequentially to avoid ImageDraw thread safety issues
|
||||
# PIL ImageDraw is not thread-safe, so we process elements one by one
|
||||
for element_id, element in selector_map.items():
|
||||
process_element_highlight(element_id, element, draw, device_pixel_ratio, font, filter_highlight_ids, image.size)
|
||||
|
||||
# Convert back to base64
|
||||
output_buffer = io.BytesIO()
|
||||
try:
|
||||
image.save(output_buffer, format='PNG')
|
||||
output_buffer.seek(0)
|
||||
highlighted_b64 = base64.b64encode(output_buffer.getvalue()).decode('utf-8')
|
||||
|
||||
logger.debug(f'Successfully created highlighted screenshot with {len(selector_map)} elements')
|
||||
return highlighted_b64
|
||||
finally:
|
||||
# Explicit cleanup to prevent memory leaks
|
||||
output_buffer.close()
|
||||
if 'image' in locals():
|
||||
image.close()
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to create highlighted screenshot: {e}')
|
||||
# Clean up on error as well
|
||||
if 'image' in locals():
|
||||
image.close()
|
||||
# Return original screenshot on error
|
||||
return screenshot_b64
|
||||
|
||||
|
||||
async def get_viewport_info_from_cdp(cdp_session) -> tuple[float, int, int]:
|
||||
"""Get viewport information from CDP session.
|
||||
|
||||
Returns:
|
||||
Tuple of (device_pixel_ratio, scroll_x, scroll_y)
|
||||
"""
|
||||
try:
|
||||
# Get layout metrics which includes viewport info and device pixel ratio
|
||||
metrics = await cdp_session.cdp_client.send.Page.getLayoutMetrics(session_id=cdp_session.session_id)
|
||||
|
||||
# Extract viewport information
|
||||
visual_viewport = metrics.get('visualViewport', {})
|
||||
css_visual_viewport = metrics.get('cssVisualViewport', {})
|
||||
css_layout_viewport = metrics.get('cssLayoutViewport', {})
|
||||
|
||||
# Calculate device pixel ratio
|
||||
css_width = css_visual_viewport.get('clientWidth', css_layout_viewport.get('clientWidth', 1280.0))
|
||||
device_width = visual_viewport.get('clientWidth', css_width)
|
||||
device_pixel_ratio = device_width / css_width if css_width > 0 else 1.0
|
||||
|
||||
# Get scroll position in CSS pixels
|
||||
scroll_x = int(css_visual_viewport.get('pageX', 0))
|
||||
scroll_y = int(css_visual_viewport.get('pageY', 0))
|
||||
|
||||
return float(device_pixel_ratio), scroll_x, scroll_y
|
||||
|
||||
except Exception as e:
|
||||
logger.debug(f'Failed to get viewport info from CDP: {e}')
|
||||
return 1.0, 0, 0
|
||||
|
||||
|
||||
@time_execution_async('create_highlighted_screenshot_async')
|
||||
async def create_highlighted_screenshot_async(
|
||||
screenshot_b64: str, selector_map: DOMSelectorMap, cdp_session=None, filter_highlight_ids: bool = True
|
||||
) -> str:
|
||||
"""Async wrapper for creating highlighted screenshots.
|
||||
|
||||
Args:
|
||||
screenshot_b64: Base64 encoded screenshot
|
||||
selector_map: Map of interactive elements
|
||||
cdp_session: CDP session for getting viewport info
|
||||
filter_highlight_ids: Whether to filter element IDs based on meaningful text
|
||||
|
||||
Returns:
|
||||
Base64 encoded highlighted screenshot
|
||||
"""
|
||||
# Get viewport information if CDP session is available
|
||||
device_pixel_ratio = 1.0
|
||||
viewport_offset_x = 0
|
||||
viewport_offset_y = 0
|
||||
|
||||
if cdp_session:
|
||||
try:
|
||||
device_pixel_ratio, viewport_offset_x, viewport_offset_y = await get_viewport_info_from_cdp(cdp_session)
|
||||
except Exception as e:
|
||||
logger.debug(f'Failed to get viewport info from CDP: {e}')
|
||||
|
||||
# Create highlighted screenshot with async processing
|
||||
final_screenshot = await create_highlighted_screenshot(
|
||||
screenshot_b64, selector_map, device_pixel_ratio, viewport_offset_x, viewport_offset_y, filter_highlight_ids
|
||||
)
|
||||
|
||||
filename = os.getenv('BROWSER_USE_SCREENSHOT_FILE')
|
||||
if filename:
|
||||
|
||||
def _write_screenshot():
|
||||
try:
|
||||
with open(filename, 'wb') as f:
|
||||
f.write(base64.b64decode(final_screenshot))
|
||||
logger.debug('Saved screenshot to ' + str(filename))
|
||||
except Exception as e:
|
||||
logger.warning(f'Failed to save screenshot to {filename}: {e}')
|
||||
|
||||
await asyncio.to_thread(_write_screenshot)
|
||||
return final_screenshot
|
||||
|
||||
|
||||
# Export the cleanup function for external use in long-running applications
|
||||
__all__ = ['create_highlighted_screenshot', 'create_highlighted_screenshot_async', 'cleanup_font_cache']
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,162 @@
|
||||
"""Video Recording Service for Browser Use Sessions."""
|
||||
|
||||
import base64
|
||||
import logging
|
||||
import math
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from browser_use.browser.profile import ViewportSize
|
||||
|
||||
try:
|
||||
import imageio.v2 as iio # type: ignore[import-not-found]
|
||||
import imageio_ffmpeg # type: ignore[import-not-found]
|
||||
import numpy as np # type: ignore[import-not-found]
|
||||
from imageio.core.format import Format # type: ignore[import-not-found]
|
||||
|
||||
IMAGEIO_AVAILABLE = True
|
||||
except ImportError:
|
||||
IMAGEIO_AVAILABLE = False
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _get_padded_size(size: ViewportSize, macro_block_size: int = 16) -> ViewportSize:
|
||||
"""Calculates the dimensions padded to the nearest multiple of macro_block_size."""
|
||||
width = int(math.ceil(size['width'] / macro_block_size)) * macro_block_size
|
||||
height = int(math.ceil(size['height'] / macro_block_size)) * macro_block_size
|
||||
return ViewportSize(width=width, height=height)
|
||||
|
||||
|
||||
class VideoRecorderService:
|
||||
"""
|
||||
Handles the video encoding process for a browser session using imageio.
|
||||
|
||||
This service captures individual frames from the CDP screencast, decodes them,
|
||||
and appends them to a video file using a pip-installable ffmpeg backend.
|
||||
It automatically resizes frames to match the target video dimensions.
|
||||
"""
|
||||
|
||||
def __init__(self, output_path: Path, size: ViewportSize, framerate: int):
|
||||
"""
|
||||
Initializes the video recorder.
|
||||
|
||||
Args:
|
||||
output_path: The full path where the video will be saved.
|
||||
size: A ViewportSize object specifying the width and height of the video.
|
||||
framerate: The desired framerate for the output video.
|
||||
"""
|
||||
self.output_path = output_path
|
||||
self.size = size
|
||||
self.framerate = framerate
|
||||
self._writer: Optional['Format.Writer'] = None
|
||||
self._is_active = False
|
||||
self.padded_size = _get_padded_size(self.size)
|
||||
|
||||
def start(self) -> None:
|
||||
"""
|
||||
Prepares and starts the video writer.
|
||||
|
||||
If the required optional dependencies are not installed, this method will
|
||||
log an error and do nothing.
|
||||
"""
|
||||
if not IMAGEIO_AVAILABLE:
|
||||
logger.error(
|
||||
'MP4 recording requires optional dependencies. Please install them with: pip install "browser-use[video]"'
|
||||
)
|
||||
return
|
||||
|
||||
try:
|
||||
self.output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
# The macro_block_size is set to None because we handle padding ourselves
|
||||
self._writer = iio.get_writer(
|
||||
str(self.output_path),
|
||||
fps=self.framerate,
|
||||
codec='libx264',
|
||||
quality=8, # A good balance of quality and file size (1-10 scale)
|
||||
pixelformat='yuv420p', # Ensures compatibility with most players
|
||||
macro_block_size=None,
|
||||
)
|
||||
self._is_active = True
|
||||
logger.debug(f'Video recorder started. Output will be saved to {self.output_path}')
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to initialize video writer: {e}')
|
||||
self._is_active = False
|
||||
|
||||
def add_frame(self, frame_data_b64: str) -> None:
|
||||
"""
|
||||
Decodes a base64-encoded PNG frame, resizes it, pads it to be codec-compatible,
|
||||
and appends it to the video.
|
||||
|
||||
Args:
|
||||
frame_data_b64: A base64-encoded string of the PNG frame data.
|
||||
"""
|
||||
if not self._is_active or not self._writer:
|
||||
return
|
||||
|
||||
try:
|
||||
frame_bytes = base64.b64decode(frame_data_b64)
|
||||
|
||||
# Build a filter chain for ffmpeg:
|
||||
# 1. scale: Resizes the frame to the user-specified dimensions.
|
||||
# 2. pad: Adds black bars to meet codec's macro-block requirements,
|
||||
# centering the original content.
|
||||
vf_chain = (
|
||||
f'scale={self.size["width"]}:{self.size["height"]},'
|
||||
f'pad={self.padded_size["width"]}:{self.padded_size["height"]}:(ow-iw)/2:(oh-ih)/2:color=black'
|
||||
)
|
||||
|
||||
output_pix_fmt = 'rgb24'
|
||||
command = [
|
||||
imageio_ffmpeg.get_ffmpeg_exe(),
|
||||
'-f',
|
||||
'image2pipe', # Input format from a pipe
|
||||
'-c:v',
|
||||
'png', # Specify input codec is PNG
|
||||
'-i',
|
||||
'-', # Input from stdin
|
||||
'-vf',
|
||||
vf_chain, # Video filter for resizing and padding
|
||||
'-f',
|
||||
'rawvideo', # Output format is raw video
|
||||
'-pix_fmt',
|
||||
output_pix_fmt, # Output pixel format
|
||||
'-', # Output to stdout
|
||||
]
|
||||
|
||||
# Execute ffmpeg as a subprocess
|
||||
proc = subprocess.Popen(command, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
|
||||
out, err = proc.communicate(input=frame_bytes)
|
||||
|
||||
if proc.returncode != 0:
|
||||
err_msg = err.decode(errors='ignore').strip()
|
||||
if 'deprecated pixel format used' not in err_msg.lower():
|
||||
raise OSError(f'ffmpeg error during resizing/padding: {err_msg}')
|
||||
else:
|
||||
logger.debug(f'ffmpeg warning during resizing/padding: {err_msg}')
|
||||
|
||||
# Convert the raw output bytes to a numpy array with the padded dimensions
|
||||
img_array = np.frombuffer(out, dtype=np.uint8).reshape((self.padded_size['height'], self.padded_size['width'], 3))
|
||||
|
||||
self._writer.append_data(img_array)
|
||||
except Exception as e:
|
||||
logger.warning(f'Could not process and add video frame: {e}')
|
||||
|
||||
def stop_and_save(self) -> None:
|
||||
"""
|
||||
Finalizes the video file by closing the writer.
|
||||
|
||||
This method should be called when the recording session is complete.
|
||||
"""
|
||||
if not self._is_active or not self._writer:
|
||||
return
|
||||
|
||||
try:
|
||||
self._writer.close()
|
||||
logger.info(f'📹 Video recording saved successfully to: {self.output_path}')
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to finalize and save video: {e}')
|
||||
finally:
|
||||
self._is_active = False
|
||||
self._writer = None
|
||||
@@ -0,0 +1,176 @@
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any
|
||||
|
||||
from bubus import BaseEvent
|
||||
from cdp_use.cdp.target import TargetID
|
||||
from pydantic import AliasChoices, BaseModel, ConfigDict, Field, field_serializer
|
||||
|
||||
from browser_use.dom.views import DOMInteractedElement, SerializedDOMState
|
||||
|
||||
# Known placeholder image data for about:blank pages - a 4x4 white PNG
|
||||
PLACEHOLDER_4PX_SCREENSHOT = (
|
||||
'iVBORw0KGgoAAAANSUhEUgAAAAQAAAAECAIAAAAmkwkpAAAAFElEQVR4nGP8//8/AwwwMSAB3BwAlm4DBfIlvvkAAAAASUVORK5CYII='
|
||||
)
|
||||
|
||||
|
||||
# Pydantic
|
||||
class TabInfo(BaseModel):
|
||||
"""Represents information about a browser tab"""
|
||||
|
||||
model_config = ConfigDict(
|
||||
extra='forbid',
|
||||
validate_by_name=True,
|
||||
validate_by_alias=True,
|
||||
populate_by_name=True,
|
||||
)
|
||||
|
||||
# Original fields
|
||||
url: str
|
||||
title: str
|
||||
target_id: TargetID = Field(serialization_alias='tab_id', validation_alias=AliasChoices('tab_id', 'target_id'))
|
||||
parent_target_id: TargetID | None = Field(
|
||||
default=None, serialization_alias='parent_tab_id', validation_alias=AliasChoices('parent_tab_id', 'parent_target_id')
|
||||
) # parent page that contains this popup or cross-origin iframe
|
||||
|
||||
@field_serializer('target_id')
|
||||
def serialize_target_id(self, target_id: TargetID, _info: Any) -> str:
|
||||
return target_id[-4:]
|
||||
|
||||
@field_serializer('parent_target_id')
|
||||
def serialize_parent_target_id(self, parent_target_id: TargetID | None, _info: Any) -> str | None:
|
||||
return parent_target_id[-4:] if parent_target_id else None
|
||||
|
||||
|
||||
class PageInfo(BaseModel):
|
||||
"""Comprehensive page size and scroll information"""
|
||||
|
||||
# Current viewport dimensions
|
||||
viewport_width: int
|
||||
viewport_height: int
|
||||
|
||||
# Total page dimensions
|
||||
page_width: int
|
||||
page_height: int
|
||||
|
||||
# Current scroll position
|
||||
scroll_x: int
|
||||
scroll_y: int
|
||||
|
||||
# Calculated scroll information
|
||||
pixels_above: int
|
||||
pixels_below: int
|
||||
pixels_left: int
|
||||
pixels_right: int
|
||||
|
||||
# Page statistics are now computed dynamically instead of stored
|
||||
|
||||
|
||||
@dataclass
|
||||
class BrowserStateSummary:
|
||||
"""The summary of the browser's current state designed for an LLM to process"""
|
||||
|
||||
# provided by SerializedDOMState:
|
||||
dom_state: SerializedDOMState
|
||||
|
||||
url: str
|
||||
title: str
|
||||
tabs: list[TabInfo]
|
||||
screenshot: str | None = field(default=None, repr=False)
|
||||
page_info: PageInfo | None = None # Enhanced page information
|
||||
|
||||
# Keep legacy fields for backward compatibility
|
||||
pixels_above: int = 0
|
||||
pixels_below: int = 0
|
||||
browser_errors: list[str] = field(default_factory=list)
|
||||
is_pdf_viewer: bool = False # Whether the current page is a PDF viewer
|
||||
recent_events: str | None = None # Text summary of recent browser events
|
||||
|
||||
|
||||
@dataclass
|
||||
class BrowserStateHistory:
|
||||
"""The summary of the browser's state at a past point in time to usse in LLM message history"""
|
||||
|
||||
url: str
|
||||
title: str
|
||||
tabs: list[TabInfo]
|
||||
interacted_element: list[DOMInteractedElement | None] | list[None]
|
||||
screenshot_path: str | None = None
|
||||
|
||||
def get_screenshot(self) -> str | None:
|
||||
"""Load screenshot from disk and return as base64 string"""
|
||||
if not self.screenshot_path:
|
||||
return None
|
||||
|
||||
import base64
|
||||
from pathlib import Path
|
||||
|
||||
path_obj = Path(self.screenshot_path)
|
||||
if not path_obj.exists():
|
||||
return None
|
||||
|
||||
try:
|
||||
with open(path_obj, 'rb') as f:
|
||||
screenshot_data = f.read()
|
||||
return base64.b64encode(screenshot_data).decode('utf-8')
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
data = {}
|
||||
data['tabs'] = [tab.model_dump() for tab in self.tabs]
|
||||
data['screenshot_path'] = self.screenshot_path
|
||||
data['interacted_element'] = [el.to_dict() if el else None for el in self.interacted_element]
|
||||
data['url'] = self.url
|
||||
data['title'] = self.title
|
||||
return data
|
||||
|
||||
|
||||
class BrowserError(Exception):
|
||||
"""Browser error with structured memory for LLM context management.
|
||||
|
||||
This exception class provides separate memory contexts for browser actions:
|
||||
- short_term_memory: Immediate context shown once to the LLM for the next action
|
||||
- long_term_memory: Persistent error information stored across steps
|
||||
"""
|
||||
|
||||
message: str
|
||||
short_term_memory: str | None = None
|
||||
long_term_memory: str | None = None
|
||||
details: dict[str, Any] | None = None
|
||||
while_handling_event: BaseEvent[Any] | None = None
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
short_term_memory: str | None = None,
|
||||
long_term_memory: str | None = None,
|
||||
details: dict[str, Any] | None = None,
|
||||
event: BaseEvent[Any] | None = None,
|
||||
):
|
||||
"""Initialize a BrowserError with structured memory contexts.
|
||||
|
||||
Args:
|
||||
message: Technical error message for logging and debugging
|
||||
short_term_memory: Context shown once to LLM (e.g., available actions, options)
|
||||
long_term_memory: Persistent error info stored in agent memory
|
||||
details: Additional metadata for debugging
|
||||
event: The browser event that triggered this error
|
||||
"""
|
||||
self.message = message
|
||||
self.short_term_memory = short_term_memory
|
||||
self.long_term_memory = long_term_memory
|
||||
self.details = details
|
||||
self.while_handling_event = event
|
||||
super().__init__(message)
|
||||
|
||||
def __str__(self) -> str:
|
||||
if self.details:
|
||||
return f'{self.message} ({self.details}) during: {self.while_handling_event}'
|
||||
elif self.while_handling_event:
|
||||
return f'{self.message} (while handling: {self.while_handling_event})'
|
||||
else:
|
||||
return self.message
|
||||
|
||||
|
||||
class URLNotAllowedError(BrowserError):
|
||||
"""Error raised when a URL is not allowed"""
|
||||
@@ -0,0 +1,268 @@
|
||||
"""Base watchdog class for browser monitoring components."""
|
||||
|
||||
import inspect
|
||||
import time
|
||||
from collections.abc import Iterable
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from bubus import BaseEvent, EventBus
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
|
||||
from browser_use.browser.session import BrowserSession
|
||||
|
||||
|
||||
class BaseWatchdog(BaseModel):
|
||||
"""Base class for all browser watchdogs.
|
||||
|
||||
Watchdogs monitor browser state and emit events based on changes.
|
||||
They automatically register event handlers based on method names.
|
||||
|
||||
Handler methods should be named: on_EventTypeName(self, event: EventTypeName)
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(
|
||||
arbitrary_types_allowed=True, # allow non-serializable objects like EventBus/BrowserSession in fields
|
||||
extra='forbid', # dont allow implicit class/instance state, everything must be a properly typed Field or PrivateAttr
|
||||
validate_assignment=False, # avoid re-triggering __init__ / validators on values on every assignment
|
||||
revalidate_instances='never', # avoid re-triggering __init__ / validators and erasing private attrs
|
||||
)
|
||||
|
||||
# Class variables to statically define the list of events relevant to each watchdog
|
||||
# (not enforced, just to make it easier to understand the code and debug watchdogs at runtime)
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent[Any]]]] = [] # Events this watchdog listens to
|
||||
EMITS: ClassVar[list[type[BaseEvent[Any]]]] = [] # Events this watchdog emits
|
||||
|
||||
# Core dependencies
|
||||
event_bus: EventBus = Field()
|
||||
browser_session: BrowserSession = Field()
|
||||
|
||||
# Shared state that other watchdogs might need to access should not be defined on BrowserSession, not here!
|
||||
# Shared helper methods needed by other watchdogs should be defined on BrowserSession, not here!
|
||||
# Alternatively, expose some events on the watchdog to allow access to state/helpers via event_bus system.
|
||||
|
||||
# Private state internal to the watchdog can be defined like this on BaseWatchdog subclasses:
|
||||
# _screenshot_cache: dict[str, bytes] = PrivateAttr(default_factory=dict)
|
||||
# _browser_crash_watcher_task: asyncio.Task | None = PrivateAttr(default=None)
|
||||
# _cdp_download_tasks: WeakSet[asyncio.Task] = PrivateAttr(default_factory=WeakSet)
|
||||
# ...
|
||||
|
||||
@property
|
||||
def logger(self):
|
||||
"""Get the logger from the browser session."""
|
||||
return self.browser_session.logger
|
||||
|
||||
@staticmethod
|
||||
def attach_handler_to_session(browser_session: 'BrowserSession', event_class: type[BaseEvent[Any]], handler) -> None:
|
||||
"""Attach a single event handler to a browser session.
|
||||
|
||||
Args:
|
||||
browser_session: The browser session to attach to
|
||||
event_class: The event class to listen for
|
||||
handler: The handler method (must start with 'on_' and end with event type)
|
||||
"""
|
||||
event_bus = browser_session.event_bus
|
||||
|
||||
# Validate handler naming convention
|
||||
assert hasattr(handler, '__name__'), 'Handler must have a __name__ attribute'
|
||||
assert handler.__name__.startswith('on_'), f'Handler {handler.__name__} must start with "on_"'
|
||||
assert handler.__name__.endswith(event_class.__name__), (
|
||||
f'Handler {handler.__name__} must end with event type {event_class.__name__}'
|
||||
)
|
||||
|
||||
# Get the watchdog instance if this is a bound method
|
||||
watchdog_instance = getattr(handler, '__self__', None)
|
||||
watchdog_class_name = watchdog_instance.__class__.__name__ if watchdog_instance else 'Unknown'
|
||||
|
||||
# Color codes for logging
|
||||
red = '\033[91m'
|
||||
green = '\033[92m'
|
||||
yellow = '\033[93m'
|
||||
magenta = '\033[95m'
|
||||
cyan = '\033[96m'
|
||||
reset = '\033[0m'
|
||||
|
||||
# Create a wrapper function with unique name to avoid duplicate handler warnings
|
||||
# Capture handler by value to avoid closure issues
|
||||
def make_unique_handler(actual_handler):
|
||||
async def unique_handler(event):
|
||||
# just for debug logging, not used for anything else
|
||||
parent_event = event_bus.event_history.get(event.event_parent_id) if event.event_parent_id else None
|
||||
grandparent_event = (
|
||||
event_bus.event_history.get(parent_event.event_parent_id)
|
||||
if parent_event and parent_event.event_parent_id
|
||||
else None
|
||||
)
|
||||
parent = (
|
||||
f'{yellow}↲ triggered by {cyan}on_{parent_event.event_type}#{parent_event.event_id[-4:]}{reset}'
|
||||
if parent_event
|
||||
else f'{magenta}👈 by Agent{reset}'
|
||||
)
|
||||
grandparent = (
|
||||
(
|
||||
f'{yellow}↲ under {cyan}{grandparent_event.event_type}#{grandparent_event.event_id[-4:]}{reset}'
|
||||
if grandparent_event
|
||||
else f'{magenta}👈 by Agent{reset}'
|
||||
)
|
||||
if parent_event
|
||||
else ''
|
||||
)
|
||||
event_str = f'#{event.event_id[-4:]}'
|
||||
time_start = time.time()
|
||||
watchdog_and_handler_str = f'[{watchdog_class_name}.{actual_handler.__name__}({event_str})]'.ljust(54)
|
||||
browser_session.logger.debug(
|
||||
f'{cyan}🚌 {watchdog_and_handler_str} ⏳ Starting... {reset} {parent} {grandparent}'
|
||||
)
|
||||
|
||||
try:
|
||||
# **EXECUTE THE EVENT HANDLER FUNCTION**
|
||||
result = await actual_handler(event)
|
||||
|
||||
if isinstance(result, Exception):
|
||||
raise result
|
||||
|
||||
# just for debug logging, not used for anything else
|
||||
time_end = time.time()
|
||||
time_elapsed = time_end - time_start
|
||||
result_summary = '' if result is None else f' ➡️ {magenta}<{type(result).__name__}>{reset}'
|
||||
parents_summary = f' {parent}'.replace('↲ triggered by ', f'⤴ {green}returned to {cyan}').replace(
|
||||
'👈 by Agent', f'👉 {green}returned to {magenta}Agent{reset}'
|
||||
)
|
||||
browser_session.logger.debug(
|
||||
f'{green}🚌 {watchdog_and_handler_str} ✅ Succeeded ({time_elapsed:.2f}s){reset}{result_summary}{parents_summary}'
|
||||
)
|
||||
return result
|
||||
except Exception as e:
|
||||
time_end = time.time()
|
||||
time_elapsed = time_end - time_start
|
||||
original_error = e
|
||||
browser_session.logger.error(
|
||||
f'{red}🚌 {watchdog_and_handler_str} ❌ Failed ({time_elapsed:.2f}s): {type(e).__name__}: {e}{reset}'
|
||||
)
|
||||
|
||||
# attempt to repair potentially crashed CDP session
|
||||
try:
|
||||
if browser_session.agent_focus and browser_session.agent_focus.target_id:
|
||||
# Common issue with CDP, some calls need the target to be active/foreground to succeed:
|
||||
# screenshot, scroll, Page.handleJavaScriptDialog, and some others
|
||||
browser_session.logger.debug(
|
||||
f'{yellow}🚌 {watchdog_and_handler_str} ⚠️ Re-foregrounding target to try and recover crashed CDP session\n\t{browser_session.agent_focus}{reset}'
|
||||
)
|
||||
del browser_session._cdp_session_pool[browser_session.agent_focus.target_id]
|
||||
browser_session.agent_focus = await browser_session.get_or_create_cdp_session(
|
||||
target_id=browser_session.agent_focus.target_id, new_socket=True
|
||||
)
|
||||
await browser_session.agent_focus.cdp_client.send.Target.activateTarget(
|
||||
params={'targetId': browser_session.agent_focus.target_id}
|
||||
)
|
||||
else:
|
||||
await browser_session.get_or_create_cdp_session(target_id=None, new_socket=True, focus=True)
|
||||
except Exception as sub_error:
|
||||
if 'ConnectionClosedError' in str(type(sub_error)) or 'ConnectionError' in str(type(sub_error)):
|
||||
browser_session.logger.error(
|
||||
f'{red}🚌 {watchdog_and_handler_str} ❌ Browser closed or CDP Connection disconnected by remote. {red}{type(sub_error).__name__}: {sub_error}{reset}\n'
|
||||
)
|
||||
raise
|
||||
else:
|
||||
browser_session.logger.error(
|
||||
f'{red}🚌 {watchdog_and_handler_str} ❌ CDP connected but failed to re-create CDP session after error "{type(original_error).__name__}: {original_error}" in {cyan}{actual_handler.__name__}({event.event_type}#{event.event_id[-4:]}){reset}: due to {red}{type(sub_error).__name__}: {sub_error}{reset}\n'
|
||||
)
|
||||
|
||||
raise
|
||||
|
||||
return unique_handler
|
||||
|
||||
unique_handler = make_unique_handler(handler)
|
||||
unique_handler.__name__ = f'{watchdog_class_name}.{handler.__name__}'
|
||||
|
||||
# Check if this handler is already registered - throw error if duplicate
|
||||
existing_handlers = event_bus.handlers.get(event_class.__name__, [])
|
||||
handler_names = [getattr(h, '__name__', str(h)) for h in existing_handlers]
|
||||
|
||||
if unique_handler.__name__ in handler_names:
|
||||
raise RuntimeError(
|
||||
f'[{watchdog_class_name}] Duplicate handler registration attempted! '
|
||||
f'Handler {unique_handler.__name__} is already registered for {event_class.__name__}. '
|
||||
f'This likely means attach_to_session() was called multiple times.'
|
||||
)
|
||||
|
||||
event_bus.on(event_class, unique_handler)
|
||||
|
||||
def attach_to_session(self) -> None:
|
||||
"""Attach watchdog to its browser session and start monitoring.
|
||||
|
||||
This method handles event listener registration. The watchdog is already
|
||||
bound to a browser session via self.browser_session from initialization.
|
||||
"""
|
||||
# Register event handlers automatically based on method names
|
||||
assert self.browser_session is not None, 'Root CDP client not initialized - browser may not be connected yet'
|
||||
|
||||
from browser_use.browser import events
|
||||
|
||||
event_classes = {}
|
||||
for name in dir(events):
|
||||
obj = getattr(events, name)
|
||||
if inspect.isclass(obj) and issubclass(obj, BaseEvent) and obj is not BaseEvent:
|
||||
event_classes[name] = obj
|
||||
|
||||
# Find all handler methods (on_EventName)
|
||||
registered_events = set()
|
||||
for method_name in dir(self):
|
||||
if method_name.startswith('on_') and callable(getattr(self, method_name)):
|
||||
# Extract event name from method name (on_EventName -> EventName)
|
||||
event_name = method_name[3:] # Remove 'on_' prefix
|
||||
|
||||
if event_name in event_classes:
|
||||
event_class = event_classes[event_name]
|
||||
|
||||
# ASSERTION: If LISTENS_TO is defined, enforce it
|
||||
if self.LISTENS_TO:
|
||||
assert event_class in self.LISTENS_TO, (
|
||||
f'[{self.__class__.__name__}] Handler {method_name} listens to {event_name} '
|
||||
f'but {event_name} is not declared in LISTENS_TO: {[e.__name__ for e in self.LISTENS_TO]}'
|
||||
)
|
||||
|
||||
handler = getattr(self, method_name)
|
||||
|
||||
# Use the static helper to attach the handler
|
||||
self.attach_handler_to_session(self.browser_session, event_class, handler)
|
||||
registered_events.add(event_class)
|
||||
|
||||
# ASSERTION: If LISTENS_TO is defined, ensure all declared events have handlers
|
||||
if self.LISTENS_TO:
|
||||
missing_handlers = set(self.LISTENS_TO) - registered_events
|
||||
if missing_handlers:
|
||||
missing_names = [e.__name__ for e in missing_handlers]
|
||||
self.logger.warning(
|
||||
f'[{self.__class__.__name__}] LISTENS_TO declares {missing_names} '
|
||||
f'but no handlers found (missing on_{"_, on_".join(missing_names)} methods)'
|
||||
)
|
||||
|
||||
def __del__(self) -> None:
|
||||
"""Clean up any running tasks during garbage collection."""
|
||||
|
||||
# A BIT OF MAGIC: Cancel any private attributes that look like asyncio tasks
|
||||
try:
|
||||
for attr_name in dir(self):
|
||||
# e.g. _browser_crash_watcher_task = asyncio.Task
|
||||
if attr_name.startswith('_') and attr_name.endswith('_task'):
|
||||
try:
|
||||
task = getattr(self, attr_name)
|
||||
if hasattr(task, 'cancel') and callable(task.cancel) and not task.done():
|
||||
task.cancel()
|
||||
# self.logger.debug(f'[{self.__class__.__name__}] Cancelled {attr_name} during cleanup')
|
||||
except Exception:
|
||||
pass # Ignore errors during cleanup
|
||||
|
||||
# e.g. _cdp_download_tasks = WeakSet[asyncio.Task] or list[asyncio.Task]
|
||||
if attr_name.startswith('_') and attr_name.endswith('_tasks') and isinstance(getattr(self, attr_name), Iterable):
|
||||
for task in getattr(self, attr_name):
|
||||
try:
|
||||
if hasattr(task, 'cancel') and callable(task.cancel) and not task.done():
|
||||
task.cancel()
|
||||
# self.logger.debug(f'[{self.__class__.__name__}] Cancelled {attr_name} during cleanup')
|
||||
except Exception:
|
||||
pass # Ignore errors during cleanup
|
||||
except Exception as e:
|
||||
from browser_use.utils import logger
|
||||
|
||||
logger.error(f'⚠️ Error during BrowserSession {self.__class__.__name__} gargabe collection __del__(): {type(e)}: {e}')
|
||||
+253
@@ -0,0 +1,253 @@
|
||||
"""About:blank watchdog for managing about:blank tabs with DVD screensaver."""
|
||||
|
||||
from typing import TYPE_CHECKING, ClassVar
|
||||
|
||||
from bubus import BaseEvent
|
||||
from cdp_use.cdp.target import TargetID
|
||||
from pydantic import PrivateAttr
|
||||
|
||||
from browser_use.browser.events import (
|
||||
AboutBlankDVDScreensaverShownEvent,
|
||||
BrowserStopEvent,
|
||||
BrowserStoppedEvent,
|
||||
CloseTabEvent,
|
||||
NavigateToUrlEvent,
|
||||
TabClosedEvent,
|
||||
TabCreatedEvent,
|
||||
)
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
class AboutBlankWatchdog(BaseWatchdog):
|
||||
"""Ensures there's always exactly one about:blank tab with DVD screensaver."""
|
||||
|
||||
# Event contracts
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [
|
||||
BrowserStopEvent,
|
||||
BrowserStoppedEvent,
|
||||
TabCreatedEvent,
|
||||
TabClosedEvent,
|
||||
]
|
||||
EMITS: ClassVar[list[type[BaseEvent]]] = [
|
||||
NavigateToUrlEvent,
|
||||
CloseTabEvent,
|
||||
AboutBlankDVDScreensaverShownEvent,
|
||||
]
|
||||
|
||||
_stopping: bool = PrivateAttr(default=False)
|
||||
|
||||
async def on_BrowserStopEvent(self, event: BrowserStopEvent) -> None:
|
||||
"""Handle browser stop request - stop creating new tabs."""
|
||||
# logger.info('[AboutBlankWatchdog] Browser stop requested, stopping tab creation')
|
||||
self._stopping = True
|
||||
|
||||
async def on_BrowserStoppedEvent(self, event: BrowserStoppedEvent) -> None:
|
||||
"""Handle browser stopped event."""
|
||||
# logger.info('[AboutBlankWatchdog] Browser stopped')
|
||||
self._stopping = True
|
||||
|
||||
async def on_TabCreatedEvent(self, event: TabCreatedEvent) -> None:
|
||||
"""Check tabs when a new tab is created."""
|
||||
# logger.debug(f'[AboutBlankWatchdog] ➕ New tab created: {event.url}')
|
||||
|
||||
# If an about:blank tab was created, show DVD screensaver on all about:blank tabs
|
||||
if event.url == 'about:blank':
|
||||
await self._show_dvd_screensaver_on_about_blank_tabs()
|
||||
|
||||
async def on_TabClosedEvent(self, event: TabClosedEvent) -> None:
|
||||
"""Check tabs when a tab is closed and proactively create about:blank if needed."""
|
||||
# logger.debug('[AboutBlankWatchdog] Tab closing, checking if we need to create about:blank tab')
|
||||
|
||||
# Don't create new tabs if browser is shutting down
|
||||
if self._stopping:
|
||||
# logger.debug('[AboutBlankWatchdog] Browser is stopping, not creating new tabs')
|
||||
return
|
||||
|
||||
# Check if we're about to close the last tab (event happens BEFORE tab closes)
|
||||
# Use _cdp_get_all_pages for quick check without fetching titles
|
||||
page_targets = await self.browser_session._cdp_get_all_pages()
|
||||
if len(page_targets) <= 1:
|
||||
self.logger.debug(
|
||||
'[AboutBlankWatchdog] Last tab closing, creating new about:blank tab to avoid closing entire browser'
|
||||
)
|
||||
# Create the animation tab since no tabs should remain
|
||||
navigate_event = self.event_bus.dispatch(NavigateToUrlEvent(url='about:blank', new_tab=True))
|
||||
await navigate_event
|
||||
# Show DVD screensaver on the new tab
|
||||
await self._show_dvd_screensaver_on_about_blank_tabs()
|
||||
else:
|
||||
# Multiple tabs exist, check after close
|
||||
await self._check_and_ensure_about_blank_tab()
|
||||
|
||||
async def attach_to_target(self, target_id: TargetID) -> None:
|
||||
"""AboutBlankWatchdog doesn't monitor individual targets."""
|
||||
pass
|
||||
|
||||
async def _check_and_ensure_about_blank_tab(self) -> None:
|
||||
"""Check current tabs and ensure exactly one about:blank tab with animation exists."""
|
||||
try:
|
||||
# For quick checks, just get page targets without titles to reduce noise
|
||||
page_targets = await self.browser_session._cdp_get_all_pages()
|
||||
|
||||
# If no tabs exist at all, create one to keep browser alive
|
||||
if len(page_targets) == 0:
|
||||
# Only create a new tab if there are no tabs at all
|
||||
self.logger.debug('[AboutBlankWatchdog] No tabs exist, creating new about:blank DVD screensaver tab')
|
||||
navigate_event = self.event_bus.dispatch(NavigateToUrlEvent(url='about:blank', new_tab=True))
|
||||
await navigate_event
|
||||
# Show DVD screensaver on the new tab
|
||||
await self._show_dvd_screensaver_on_about_blank_tabs()
|
||||
# Otherwise there are tabs, don't create new ones to avoid interfering
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'[AboutBlankWatchdog] Error ensuring about:blank tab: {e}')
|
||||
|
||||
async def _show_dvd_screensaver_on_about_blank_tabs(self) -> None:
|
||||
"""Show DVD screensaver on all about:blank pages only."""
|
||||
try:
|
||||
# Get just the page targets without expensive title fetching
|
||||
page_targets = await self.browser_session._cdp_get_all_pages()
|
||||
browser_session_label = str(self.browser_session.id)[-4:]
|
||||
|
||||
for page_target in page_targets:
|
||||
target_id = page_target['targetId']
|
||||
url = page_target['url']
|
||||
|
||||
# Only target about:blank pages specifically
|
||||
if url == 'about:blank':
|
||||
await self._show_dvd_screensaver_loading_animation_cdp(target_id, browser_session_label)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'[AboutBlankWatchdog] Error showing DVD screensaver: {e}')
|
||||
|
||||
async def _show_dvd_screensaver_loading_animation_cdp(self, target_id: TargetID, browser_session_label: str) -> None:
|
||||
"""
|
||||
Injects a DVD screensaver-style bouncing logo loading animation overlay into the target using CDP.
|
||||
This is used to visually indicate that the browser is setting up or waiting.
|
||||
"""
|
||||
try:
|
||||
# Create temporary session for this target without switching focus
|
||||
temp_session = await self.browser_session.get_or_create_cdp_session(target_id, focus=False)
|
||||
|
||||
# Inject the DVD screensaver script (from main branch with idempotency added)
|
||||
script = f"""
|
||||
(function(browser_session_label) {{
|
||||
// Idempotency check
|
||||
if (window.__dvdAnimationRunning) {{
|
||||
return; // Already running, don't add another
|
||||
}}
|
||||
window.__dvdAnimationRunning = true;
|
||||
|
||||
// Ensure document.body exists before proceeding
|
||||
if (!document.body) {{
|
||||
// Try again after DOM is ready
|
||||
window.__dvdAnimationRunning = false; // Reset flag to retry
|
||||
if (document.readyState === 'loading') {{
|
||||
document.addEventListener('DOMContentLoaded', () => arguments.callee(browser_session_label));
|
||||
}}
|
||||
return;
|
||||
}}
|
||||
|
||||
const animated_title = `Starting agent ${{browser_session_label}}...`;
|
||||
if (document.title === animated_title) {{
|
||||
return; // already run on this tab, dont run again
|
||||
}}
|
||||
document.title = animated_title;
|
||||
|
||||
// Create the main overlay
|
||||
const loadingOverlay = document.createElement('div');
|
||||
loadingOverlay.id = 'pretty-loading-animation';
|
||||
loadingOverlay.style.position = 'fixed';
|
||||
loadingOverlay.style.top = '0';
|
||||
loadingOverlay.style.left = '0';
|
||||
loadingOverlay.style.width = '100vw';
|
||||
loadingOverlay.style.height = '100vh';
|
||||
loadingOverlay.style.background = '#000';
|
||||
loadingOverlay.style.zIndex = '99999';
|
||||
loadingOverlay.style.overflow = 'hidden';
|
||||
|
||||
// Create the image element
|
||||
const img = document.createElement('img');
|
||||
img.src = 'https://cf.browser-use.com/logo.svg';
|
||||
img.alt = 'Browser-Use';
|
||||
img.style.width = '200px';
|
||||
img.style.height = 'auto';
|
||||
img.style.position = 'absolute';
|
||||
img.style.left = '0px';
|
||||
img.style.top = '0px';
|
||||
img.style.zIndex = '2';
|
||||
img.style.opacity = '0.8';
|
||||
|
||||
loadingOverlay.appendChild(img);
|
||||
document.body.appendChild(loadingOverlay);
|
||||
|
||||
// DVD screensaver bounce logic
|
||||
let x = Math.random() * (window.innerWidth - 300);
|
||||
let y = Math.random() * (window.innerHeight - 300);
|
||||
let dx = 1.2 + Math.random() * 0.4; // px per frame
|
||||
let dy = 1.2 + Math.random() * 0.4;
|
||||
// Randomize direction
|
||||
if (Math.random() > 0.5) dx = -dx;
|
||||
if (Math.random() > 0.5) dy = -dy;
|
||||
|
||||
function animate() {{
|
||||
const imgWidth = img.offsetWidth || 300;
|
||||
const imgHeight = img.offsetHeight || 300;
|
||||
x += dx;
|
||||
y += dy;
|
||||
|
||||
if (x <= 0) {{
|
||||
x = 0;
|
||||
dx = Math.abs(dx);
|
||||
}} else if (x + imgWidth >= window.innerWidth) {{
|
||||
x = window.innerWidth - imgWidth;
|
||||
dx = -Math.abs(dx);
|
||||
}}
|
||||
if (y <= 0) {{
|
||||
y = 0;
|
||||
dy = Math.abs(dy);
|
||||
}} else if (y + imgHeight >= window.innerHeight) {{
|
||||
y = window.innerHeight - imgHeight;
|
||||
dy = -Math.abs(dy);
|
||||
}}
|
||||
|
||||
img.style.left = `${{x}}px`;
|
||||
img.style.top = `${{y}}px`;
|
||||
|
||||
requestAnimationFrame(animate);
|
||||
}}
|
||||
animate();
|
||||
|
||||
// Responsive: update bounds on resize
|
||||
window.addEventListener('resize', () => {{
|
||||
x = Math.min(x, window.innerWidth - img.offsetWidth);
|
||||
y = Math.min(y, window.innerHeight - img.offsetHeight);
|
||||
}});
|
||||
|
||||
// Add a little CSS for smoothness
|
||||
const style = document.createElement('style');
|
||||
style.textContent = `
|
||||
#pretty-loading-animation {{
|
||||
/*backdrop-filter: blur(2px) brightness(0.9);*/
|
||||
}}
|
||||
#pretty-loading-animation img {{
|
||||
user-select: none;
|
||||
pointer-events: none;
|
||||
}}
|
||||
`;
|
||||
document.head.appendChild(style);
|
||||
}})('{browser_session_label}');
|
||||
"""
|
||||
|
||||
await temp_session.cdp_client.send.Runtime.evaluate(params={'expression': script}, session_id=temp_session.session_id)
|
||||
|
||||
# No need to detach - session is cached
|
||||
|
||||
# Dispatch event
|
||||
self.event_bus.dispatch(AboutBlankDVDScreensaverShownEvent(target_id=target_id))
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'[AboutBlankWatchdog] Error injecting DVD screensaver: {e}')
|
||||
@@ -0,0 +1,362 @@
|
||||
"""Browser watchdog for monitoring crashes and network timeouts using CDP."""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from typing import TYPE_CHECKING, ClassVar
|
||||
|
||||
import psutil
|
||||
from bubus import BaseEvent
|
||||
from cdp_use.cdp.target import SessionID, TargetID
|
||||
from cdp_use.cdp.target.events import TargetCrashedEvent
|
||||
from pydantic import Field, PrivateAttr
|
||||
|
||||
from browser_use.browser.events import (
|
||||
BrowserConnectedEvent,
|
||||
BrowserErrorEvent,
|
||||
BrowserStoppedEvent,
|
||||
TabCreatedEvent,
|
||||
)
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
class NetworkRequestTracker:
|
||||
"""Tracks ongoing network requests."""
|
||||
|
||||
def __init__(self, request_id: str, start_time: float, url: str, method: str, resource_type: str | None = None):
|
||||
self.request_id = request_id
|
||||
self.start_time = start_time
|
||||
self.url = url
|
||||
self.method = method
|
||||
self.resource_type = resource_type
|
||||
|
||||
|
||||
class CrashWatchdog(BaseWatchdog):
|
||||
"""Monitors browser health for crashes and network timeouts using CDP."""
|
||||
|
||||
# Event contracts
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [
|
||||
BrowserConnectedEvent,
|
||||
BrowserStoppedEvent,
|
||||
TabCreatedEvent,
|
||||
]
|
||||
EMITS: ClassVar[list[type[BaseEvent]]] = [BrowserErrorEvent]
|
||||
|
||||
# Configuration
|
||||
network_timeout_seconds: float = Field(default=10.0)
|
||||
check_interval_seconds: float = Field(default=5.0) # Reduced frequency to reduce noise
|
||||
|
||||
# Private state
|
||||
_active_requests: dict[str, NetworkRequestTracker] = PrivateAttr(default_factory=dict)
|
||||
_monitoring_task: asyncio.Task | None = PrivateAttr(default=None)
|
||||
_last_responsive_checks: dict[str, float] = PrivateAttr(default_factory=dict) # target_url -> timestamp
|
||||
_cdp_event_tasks: set[asyncio.Task] = PrivateAttr(default_factory=set) # Track CDP event handler tasks
|
||||
_sessions_with_listeners: set[str] = PrivateAttr(default_factory=set) # Track sessions that already have event listeners
|
||||
|
||||
async def on_BrowserConnectedEvent(self, event: BrowserConnectedEvent) -> None:
|
||||
"""Start monitoring when browser is connected."""
|
||||
# logger.debug('[CrashWatchdog] Browser connected event received, beginning monitoring')
|
||||
|
||||
asyncio.create_task(self._start_monitoring())
|
||||
# logger.debug(f'[CrashWatchdog] Monitoring task started: {self._monitoring_task and not self._monitoring_task.done()}')
|
||||
|
||||
async def on_BrowserStoppedEvent(self, event: BrowserStoppedEvent) -> None:
|
||||
"""Stop monitoring when browser stops."""
|
||||
# logger.debug('[CrashWatchdog] Browser stopped, ending monitoring')
|
||||
await self._stop_monitoring()
|
||||
|
||||
async def on_TabCreatedEvent(self, event: TabCreatedEvent) -> None:
|
||||
"""Attach to new tab."""
|
||||
assert self.browser_session.agent_focus is not None, 'No current target ID'
|
||||
await self.attach_to_target(self.browser_session.agent_focus.target_id)
|
||||
|
||||
async def attach_to_target(self, target_id: TargetID) -> None:
|
||||
"""Set up crash monitoring for a specific target using CDP."""
|
||||
try:
|
||||
# Create temporary session for monitoring without switching focus
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session(target_id, focus=False)
|
||||
|
||||
# Check if we already have listeners for this session
|
||||
if cdp_session.session_id in self._sessions_with_listeners:
|
||||
self.logger.debug(f'[CrashWatchdog] Event listeners already exist for session: {cdp_session.session_id}')
|
||||
return
|
||||
|
||||
# Set up network event handlers
|
||||
# def on_request_will_be_sent(event):
|
||||
# # Create and track the task
|
||||
# task = asyncio.create_task(self._on_request_cdp(event))
|
||||
# self._cdp_event_tasks.add(task)
|
||||
# # Remove from set when done
|
||||
# task.add_done_callback(lambda t: self._cdp_event_tasks.discard(t))
|
||||
|
||||
# def on_response_received(event):
|
||||
# self._on_response_cdp(event)
|
||||
|
||||
# def on_loading_failed(event):
|
||||
# self._on_request_failed_cdp(event)
|
||||
|
||||
# def on_loading_finished(event):
|
||||
# self._on_request_finished_cdp(event)
|
||||
|
||||
# Register event handlers
|
||||
# TEMPORARILY DISABLED: Network events causing too much logging
|
||||
# cdp_client.on('Network.requestWillBeSent', on_request_will_be_sent, session_id=session_id)
|
||||
# cdp_client.on('Network.responseReceived', on_response_received, session_id=session_id)
|
||||
# cdp_client.on('Network.loadingFailed', on_loading_failed, session_id=session_id)
|
||||
# cdp_client.on('Network.loadingFinished', on_loading_finished, session_id=session_id)
|
||||
|
||||
def on_target_crashed(event: TargetCrashedEvent, session_id: SessionID | None = None):
|
||||
# Create and track the task
|
||||
task = asyncio.create_task(self._on_target_crash_cdp(target_id))
|
||||
self._cdp_event_tasks.add(task)
|
||||
# Remove from set when done
|
||||
task.add_done_callback(lambda t: self._cdp_event_tasks.discard(t))
|
||||
|
||||
cdp_session.cdp_client.register.Target.targetCrashed(on_target_crashed)
|
||||
|
||||
# Track that we've added listeners to this session
|
||||
self._sessions_with_listeners.add(cdp_session.session_id)
|
||||
|
||||
# Get target info for logging
|
||||
targets = await cdp_session.cdp_client.send.Target.getTargets()
|
||||
target_info = next((t for t in targets['targetInfos'] if t['targetId'] == target_id), None)
|
||||
if target_info:
|
||||
self.logger.debug(f'[CrashWatchdog] Added target to monitoring: {target_info.get("url", "unknown")}')
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f'[CrashWatchdog] Failed to attach to target {target_id}: {e}')
|
||||
|
||||
async def _on_request_cdp(self, event: dict) -> None:
|
||||
"""Track new network request from CDP event."""
|
||||
request_id = event.get('requestId', '')
|
||||
request = event.get('request', {})
|
||||
|
||||
self._active_requests[request_id] = NetworkRequestTracker(
|
||||
request_id=request_id,
|
||||
start_time=time.time(),
|
||||
url=request.get('url', ''),
|
||||
method=request.get('method', ''),
|
||||
resource_type=event.get('type'),
|
||||
)
|
||||
# logger.debug(f'[CrashWatchdog] Tracking request: {request.get("method", "")} {request.get("url", "")[:50]}...')
|
||||
|
||||
def _on_response_cdp(self, event: dict) -> None:
|
||||
"""Remove request from tracking on response."""
|
||||
request_id = event.get('requestId', '')
|
||||
if request_id in self._active_requests:
|
||||
elapsed = time.time() - self._active_requests[request_id].start_time
|
||||
response = event.get('response', {})
|
||||
self.logger.debug(f'[CrashWatchdog] Request completed in {elapsed:.2f}s: {response.get("url", "")[:50]}...')
|
||||
# Don't remove yet - wait for loadingFinished
|
||||
|
||||
def _on_request_failed_cdp(self, event: dict) -> None:
|
||||
"""Remove request from tracking on failure."""
|
||||
request_id = event.get('requestId', '')
|
||||
if request_id in self._active_requests:
|
||||
elapsed = time.time() - self._active_requests[request_id].start_time
|
||||
self.logger.debug(
|
||||
f'[CrashWatchdog] Request failed after {elapsed:.2f}s: {self._active_requests[request_id].url[:50]}...'
|
||||
)
|
||||
del self._active_requests[request_id]
|
||||
|
||||
def _on_request_finished_cdp(self, event: dict) -> None:
|
||||
"""Remove request from tracking when loading is finished."""
|
||||
request_id = event.get('requestId', '')
|
||||
self._active_requests.pop(request_id, None)
|
||||
|
||||
async def _on_target_crash_cdp(self, target_id: TargetID) -> None:
|
||||
"""Handle target crash detected via CDP."""
|
||||
# Remove crashed session from pool
|
||||
if session := self.browser_session._cdp_session_pool.pop(target_id, None):
|
||||
await session.disconnect()
|
||||
self.logger.debug(f'[CrashWatchdog] Removed crashed session from pool: {target_id}')
|
||||
|
||||
# Get target info
|
||||
cdp_client = self.browser_session.cdp_client
|
||||
targets = await cdp_client.send.Target.getTargets()
|
||||
target_info = next((t for t in targets['targetInfos'] if t['targetId'] == target_id), None)
|
||||
if (
|
||||
target_info
|
||||
and self.browser_session.agent_focus
|
||||
and target_info['targetId'] == self.browser_session.agent_focus.target_id
|
||||
):
|
||||
self.browser_session.agent_focus.target_id = None # type: ignore
|
||||
self.browser_session.agent_focus.session_id = None # type: ignore
|
||||
self.logger.error(
|
||||
f'[CrashWatchdog] 💥 Target crashed, navigating Agent to a new tab: {target_info.get("url", "unknown")}'
|
||||
)
|
||||
|
||||
# Also emit generic browser error
|
||||
self.event_bus.dispatch(
|
||||
BrowserErrorEvent(
|
||||
error_type='TargetCrash',
|
||||
message=f'Target crashed: {target_id}',
|
||||
details={
|
||||
# 'url': target_url, # TODO: add url to details
|
||||
'target_id': target_id,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
async def _start_monitoring(self) -> None:
|
||||
"""Start the monitoring loop."""
|
||||
assert self.browser_session.cdp_client is not None, 'Root CDP client not initialized - browser may not be connected yet'
|
||||
|
||||
if self._monitoring_task and not self._monitoring_task.done():
|
||||
# logger.info('[CrashWatchdog] Monitoring already running')
|
||||
return
|
||||
|
||||
self._monitoring_task = asyncio.create_task(self._monitoring_loop())
|
||||
# logger.debug('[CrashWatchdog] Monitoring loop created and started')
|
||||
|
||||
async def _stop_monitoring(self) -> None:
|
||||
"""Stop the monitoring loop."""
|
||||
if self._monitoring_task and not self._monitoring_task.done():
|
||||
self._monitoring_task.cancel()
|
||||
try:
|
||||
await self._monitoring_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
self.logger.debug('[CrashWatchdog] Monitoring loop stopped')
|
||||
|
||||
# Cancel all CDP event handler tasks
|
||||
for task in list(self._cdp_event_tasks):
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
# Wait for all tasks to complete cancellation
|
||||
if self._cdp_event_tasks:
|
||||
await asyncio.gather(*self._cdp_event_tasks, return_exceptions=True)
|
||||
self._cdp_event_tasks.clear()
|
||||
|
||||
# Clear tracking (CDP sessions are cached and managed by BrowserSession)
|
||||
self._active_requests.clear()
|
||||
self._sessions_with_listeners.clear()
|
||||
|
||||
async def _monitoring_loop(self) -> None:
|
||||
"""Main monitoring loop."""
|
||||
await asyncio.sleep(10) # give browser time to start up and load the first page after first LLM call
|
||||
while True:
|
||||
try:
|
||||
await self._check_network_timeouts()
|
||||
await self._check_browser_health()
|
||||
await asyncio.sleep(self.check_interval_seconds)
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
self.logger.error(f'[CrashWatchdog] Error in monitoring loop: {e}')
|
||||
|
||||
async def _check_network_timeouts(self) -> None:
|
||||
"""Check for network requests exceeding timeout."""
|
||||
current_time = time.time()
|
||||
timed_out_requests = []
|
||||
|
||||
# Debug logging
|
||||
if self._active_requests:
|
||||
self.logger.debug(
|
||||
f'[CrashWatchdog] Checking {len(self._active_requests)} active requests for timeouts (threshold: {self.network_timeout_seconds}s)'
|
||||
)
|
||||
|
||||
for request_id, tracker in self._active_requests.items():
|
||||
elapsed = current_time - tracker.start_time
|
||||
self.logger.debug(
|
||||
f'[CrashWatchdog] Request {tracker.url[:30]}... elapsed: {elapsed:.1f}s, timeout: {self.network_timeout_seconds}s'
|
||||
)
|
||||
if elapsed >= self.network_timeout_seconds:
|
||||
timed_out_requests.append((request_id, tracker))
|
||||
|
||||
# Emit events for timed out requests
|
||||
for request_id, tracker in timed_out_requests:
|
||||
self.logger.warning(
|
||||
f'[CrashWatchdog] Network request timeout after {self.network_timeout_seconds}s: '
|
||||
f'{tracker.method} {tracker.url[:100]}...'
|
||||
)
|
||||
|
||||
self.event_bus.dispatch(
|
||||
BrowserErrorEvent(
|
||||
error_type='NetworkTimeout',
|
||||
message=f'Network request timed out after {self.network_timeout_seconds}s',
|
||||
details={
|
||||
'url': tracker.url,
|
||||
'method': tracker.method,
|
||||
'resource_type': tracker.resource_type,
|
||||
'elapsed_seconds': current_time - tracker.start_time,
|
||||
},
|
||||
)
|
||||
)
|
||||
|
||||
# Remove from tracking
|
||||
del self._active_requests[request_id]
|
||||
|
||||
async def _check_browser_health(self) -> None:
|
||||
"""Check if browser and targets are still responsive."""
|
||||
|
||||
try:
|
||||
try:
|
||||
self.logger.debug(f'[CrashWatchdog] Checking browser health for target {self.browser_session.agent_focus}')
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session()
|
||||
except Exception as e:
|
||||
self.logger.debug(
|
||||
f'[CrashWatchdog] Checking browser health for target {self.browser_session.agent_focus} error: {type(e).__name__}: {e}'
|
||||
)
|
||||
self.agent_focus = cdp_session = await self.browser_session.get_or_create_cdp_session(
|
||||
target_id=self.agent_focus.target_id, new_socket=True, focus=True
|
||||
)
|
||||
|
||||
for target in (await self.browser_session.cdp_client.send.Target.getTargets()).get('targetInfos', []):
|
||||
if target.get('type') == 'page':
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session(target_id=target.get('targetId'))
|
||||
if self._is_new_tab_page(target.get('url')) and target.get('url') != 'about:blank':
|
||||
self.logger.debug(
|
||||
f'[CrashWatchdog] Redirecting chrome://new-tab-page/ to about:blank {target.get("url")}'
|
||||
)
|
||||
await cdp_session.cdp_client.send.Page.navigate(
|
||||
params={'url': 'about:blank'}, session_id=cdp_session.session_id
|
||||
)
|
||||
|
||||
# Quick ping to check if session is alive
|
||||
self.logger.debug(f'[CrashWatchdog] Attempting to run simple JS test expression in session {cdp_session} 1+1')
|
||||
await asyncio.wait_for(
|
||||
cdp_session.cdp_client.send.Runtime.evaluate(params={'expression': '1+1'}, session_id=cdp_session.session_id),
|
||||
timeout=1.0,
|
||||
)
|
||||
self.logger.debug(f'[CrashWatchdog] Browser health check passed for target {self.browser_session.agent_focus}')
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f'[CrashWatchdog] ❌ Crashed session detected for target {self.browser_session.agent_focus} error: {type(e).__name__}: {e}'
|
||||
)
|
||||
# Remove crashed session from pool
|
||||
if self.browser_session.agent_focus and (target_id := self.browser_session.agent_focus.target_id):
|
||||
if session := self.browser_session._cdp_session_pool.pop(target_id, None):
|
||||
await session.disconnect()
|
||||
self.logger.debug(f'[CrashWatchdog] Removed crashed session from pool: {target_id}')
|
||||
self.browser_session.agent_focus.target_id = None # type: ignore
|
||||
|
||||
# Check browser process if we have PID
|
||||
if self.browser_session._local_browser_watchdog and (proc := self.browser_session._local_browser_watchdog._subprocess):
|
||||
try:
|
||||
if proc.status() in (psutil.STATUS_ZOMBIE, psutil.STATUS_DEAD):
|
||||
self.logger.error(f'[CrashWatchdog] Browser process {proc.pid} has crashed')
|
||||
# Clear all sessions from pool when browser crashes
|
||||
for session in self.browser_session._cdp_session_pool.values():
|
||||
await session.disconnect()
|
||||
self.browser_session._cdp_session_pool.clear()
|
||||
self.logger.debug('[CrashWatchdog] Cleared all sessions from pool due to browser crash')
|
||||
|
||||
self.event_bus.dispatch(
|
||||
BrowserErrorEvent(
|
||||
error_type='BrowserProcessCrashed',
|
||||
message=f'Browser process {proc.pid} has crashed',
|
||||
details={'pid': proc.pid, 'status': proc.status()},
|
||||
)
|
||||
)
|
||||
await self._stop_monitoring()
|
||||
return
|
||||
except Exception:
|
||||
pass # psutil not available or process doesn't exist
|
||||
|
||||
@staticmethod
|
||||
def _is_new_tab_page(url: str) -> bool:
|
||||
"""Check if URL is a new tab page."""
|
||||
return url in ['about:blank', 'chrome://new-tab-page/', 'chrome://newtab/']
|
||||
+2344
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,582 @@
|
||||
"""DOM watchdog for browser DOM tree management using CDP."""
|
||||
|
||||
import asyncio
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from browser_use.browser.events import (
|
||||
BrowserErrorEvent,
|
||||
BrowserStateRequestEvent,
|
||||
ScreenshotEvent,
|
||||
TabCreatedEvent,
|
||||
)
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
from browser_use.dom.service import DomService
|
||||
from browser_use.dom.views import (
|
||||
EnhancedDOMTreeNode,
|
||||
SerializedDOMState,
|
||||
)
|
||||
from browser_use.observability import observe_debug
|
||||
from browser_use.utils import time_execution_async
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from browser_use.browser.views import BrowserStateSummary, PageInfo
|
||||
|
||||
|
||||
class DOMWatchdog(BaseWatchdog):
|
||||
"""Handles DOM tree building, serialization, and element access via CDP.
|
||||
|
||||
This watchdog acts as a bridge between the event-driven browser session
|
||||
and the DomService implementation, maintaining cached state and providing
|
||||
helper methods for other watchdogs.
|
||||
"""
|
||||
|
||||
LISTENS_TO = [TabCreatedEvent, BrowserStateRequestEvent]
|
||||
EMITS = [BrowserErrorEvent]
|
||||
|
||||
# Public properties for other watchdogs
|
||||
selector_map: dict[int, EnhancedDOMTreeNode] | None = None
|
||||
current_dom_state: SerializedDOMState | None = None
|
||||
enhanced_dom_tree: EnhancedDOMTreeNode | None = None
|
||||
|
||||
# Internal DOM service
|
||||
_dom_service: DomService | None = None
|
||||
|
||||
async def on_TabCreatedEvent(self, event: TabCreatedEvent) -> None:
|
||||
# self.logger.debug('Setting up init scripts in browser')
|
||||
return None
|
||||
|
||||
def _get_recent_events_str(self, limit: int = 10) -> str | None:
|
||||
"""Get the most recent events from the event bus as JSON.
|
||||
|
||||
Args:
|
||||
limit: Maximum number of recent events to include
|
||||
|
||||
Returns:
|
||||
JSON string of recent events or None if not available
|
||||
"""
|
||||
import json
|
||||
|
||||
try:
|
||||
# Get all events from history, sorted by creation time (most recent first)
|
||||
all_events = sorted(
|
||||
self.browser_session.event_bus.event_history.values(), key=lambda e: e.event_created_at.timestamp(), reverse=True
|
||||
)
|
||||
|
||||
# Take the most recent events and create JSON-serializable data
|
||||
recent_events_data = []
|
||||
for event in all_events[:limit]:
|
||||
event_data = {
|
||||
'event_type': event.event_type,
|
||||
'timestamp': event.event_created_at.isoformat(),
|
||||
}
|
||||
# Add specific fields for certain event types
|
||||
if hasattr(event, 'url'):
|
||||
event_data['url'] = getattr(event, 'url')
|
||||
if hasattr(event, 'error_message'):
|
||||
event_data['error_message'] = getattr(event, 'error_message')
|
||||
if hasattr(event, 'target_id'):
|
||||
event_data['target_id'] = getattr(event, 'target_id')
|
||||
recent_events_data.append(event_data)
|
||||
|
||||
return json.dumps(recent_events_data) # Return empty array if no events
|
||||
except Exception as e:
|
||||
self.logger.debug(f'Failed to get recent events: {e}')
|
||||
|
||||
return json.dumps([]) # Return empty JSON array on error
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='browser_state_request_event')
|
||||
async def on_BrowserStateRequestEvent(self, event: BrowserStateRequestEvent) -> 'BrowserStateSummary':
|
||||
"""Handle browser state request by coordinating DOM building and screenshot capture.
|
||||
|
||||
This is the main entry point for getting the complete browser state.
|
||||
|
||||
Args:
|
||||
event: The browser state request event with options
|
||||
|
||||
Returns:
|
||||
Complete BrowserStateSummary with DOM, screenshot, and target info
|
||||
"""
|
||||
from browser_use.browser.views import BrowserStateSummary, PageInfo
|
||||
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: STARTING browser state request')
|
||||
page_url = await self.browser_session.get_current_page_url()
|
||||
self.logger.debug(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Got page URL: {page_url}')
|
||||
if self.browser_session.agent_focus:
|
||||
self.logger.debug(
|
||||
f'Current page URL: {page_url}, target_id: {self.browser_session.agent_focus.target_id}, session_id: {self.browser_session.agent_focus.session_id}'
|
||||
)
|
||||
else:
|
||||
self.logger.debug(f'Current page URL: {page_url}, no cdp_session attached')
|
||||
|
||||
# check if we should skip DOM tree build for pointless pages
|
||||
not_a_meaningful_website = page_url.lower().split(':', 1)[0] not in ('http', 'https')
|
||||
|
||||
# Wait for page stability using browser profile settings (main branch pattern)
|
||||
if not not_a_meaningful_website:
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: ⏳ Waiting for page stability...')
|
||||
try:
|
||||
await self._wait_for_stable_network()
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: ✅ Page stability complete')
|
||||
except Exception as e:
|
||||
self.logger.warning(
|
||||
f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Network waiting failed: {e}, continuing anyway...'
|
||||
)
|
||||
|
||||
# Get tabs info once at the beginning for all paths
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: Getting tabs info...')
|
||||
tabs_info = await self.browser_session.get_tabs()
|
||||
self.logger.debug(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Got {len(tabs_info)} tabs')
|
||||
self.logger.debug(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Tabs info: {tabs_info}')
|
||||
|
||||
# Get viewport / scroll position info, remember changing scroll position should invalidate selector_map cache because it only includes visible elements
|
||||
# cdp_session = await self.browser_session.get_or_create_cdp_session(focus=True)
|
||||
# scroll_info = await cdp_session.cdp_client.send.Runtime.evaluate(
|
||||
# params={'expression': 'JSON.stringify({y: document.body.scrollTop, x: document.body.scrollLeft, width: document.documentElement.clientWidth, height: document.documentElement.clientHeight})'},
|
||||
# session_id=cdp_session.session_id,
|
||||
# )
|
||||
# self.logger.debug(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Got scroll info: {scroll_info["result"]}')
|
||||
|
||||
try:
|
||||
# Fast path for empty pages
|
||||
if not_a_meaningful_website:
|
||||
self.logger.debug(f'⚡ Skipping BuildDOMTree for empty target: {page_url}')
|
||||
self.logger.debug(f'📸 Not taking screenshot for empty page: {page_url} (non-http/https URL)')
|
||||
|
||||
# Create minimal DOM state
|
||||
content = SerializedDOMState(_root=None, selector_map={})
|
||||
|
||||
# Skip screenshot for empty pages
|
||||
screenshot_b64 = None
|
||||
|
||||
# Try to get page info from CDP, fall back to defaults if unavailable
|
||||
try:
|
||||
page_info = await self._get_page_info()
|
||||
except Exception as e:
|
||||
self.logger.debug(f'Failed to get page info from CDP for empty page: {e}, using fallback')
|
||||
# Use default viewport dimensions
|
||||
viewport = self.browser_session.browser_profile.viewport or {'width': 1280, 'height': 720}
|
||||
page_info = PageInfo(
|
||||
viewport_width=viewport['width'],
|
||||
viewport_height=viewport['height'],
|
||||
page_width=viewport['width'],
|
||||
page_height=viewport['height'],
|
||||
scroll_x=0,
|
||||
scroll_y=0,
|
||||
pixels_above=0,
|
||||
pixels_below=0,
|
||||
pixels_left=0,
|
||||
pixels_right=0,
|
||||
)
|
||||
|
||||
return BrowserStateSummary(
|
||||
dom_state=content,
|
||||
url=page_url,
|
||||
title='Empty Tab',
|
||||
tabs=tabs_info,
|
||||
screenshot=screenshot_b64,
|
||||
page_info=page_info,
|
||||
pixels_above=0,
|
||||
pixels_below=0,
|
||||
browser_errors=[],
|
||||
is_pdf_viewer=False,
|
||||
recent_events=self._get_recent_events_str() if event.include_recent_events else None,
|
||||
)
|
||||
|
||||
# Execute DOM building and screenshot capture in parallel
|
||||
dom_task = None
|
||||
screenshot_task = None
|
||||
|
||||
# Start DOM building task if requested
|
||||
if event.include_dom:
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: 🌳 Starting DOM tree build task...')
|
||||
|
||||
previous_state = (
|
||||
self.browser_session._cached_browser_state_summary.dom_state
|
||||
if self.browser_session._cached_browser_state_summary
|
||||
else None
|
||||
)
|
||||
|
||||
dom_task = asyncio.create_task(self._build_dom_tree_without_highlights(previous_state))
|
||||
|
||||
# Start clean screenshot task if requested (without JS highlights)
|
||||
if event.include_screenshot:
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: 📸 Starting clean screenshot task...')
|
||||
screenshot_task = asyncio.create_task(self._capture_clean_screenshot())
|
||||
|
||||
# Wait for both tasks to complete
|
||||
content = None
|
||||
screenshot_b64 = None
|
||||
|
||||
if dom_task:
|
||||
try:
|
||||
content = await dom_task
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: ✅ DOM tree build completed')
|
||||
except Exception as e:
|
||||
self.logger.warning(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: DOM build failed: {e}, using minimal state')
|
||||
content = SerializedDOMState(_root=None, selector_map={})
|
||||
else:
|
||||
content = SerializedDOMState(_root=None, selector_map={})
|
||||
|
||||
if screenshot_task:
|
||||
try:
|
||||
screenshot_b64 = await screenshot_task
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: ✅ Clean screenshot captured')
|
||||
except Exception as e:
|
||||
self.logger.warning(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Clean screenshot failed: {e}')
|
||||
screenshot_b64 = None
|
||||
|
||||
# Apply Python-based highlighting if both DOM and screenshot are available
|
||||
if screenshot_b64 and content and content.selector_map and self.browser_session.browser_profile.highlight_elements:
|
||||
try:
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: 🎨 Applying Python-based highlighting...')
|
||||
from browser_use.browser.python_highlights import create_highlighted_screenshot_async
|
||||
|
||||
# Get CDP session for viewport info
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session()
|
||||
start = time.time()
|
||||
screenshot_b64 = await create_highlighted_screenshot_async(
|
||||
screenshot_b64,
|
||||
content.selector_map,
|
||||
cdp_session,
|
||||
self.browser_session.browser_profile.filter_highlight_ids,
|
||||
)
|
||||
self.logger.debug(
|
||||
f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: ✅ Applied highlights to {len(content.selector_map)} elements in {time.time() - start:.2f}s'
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.warning(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Python highlighting failed: {e}')
|
||||
|
||||
# Ensure we have valid content
|
||||
if not content:
|
||||
content = SerializedDOMState(_root=None, selector_map={})
|
||||
|
||||
# Tabs info already fetched at the beginning
|
||||
|
||||
# Get target title safely
|
||||
try:
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: Getting page title...')
|
||||
title = await asyncio.wait_for(self.browser_session.get_current_page_title(), timeout=1.0)
|
||||
self.logger.debug(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Got title: {title}')
|
||||
except Exception as e:
|
||||
self.logger.debug(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Failed to get title: {e}')
|
||||
title = 'Page'
|
||||
|
||||
# Get comprehensive page info from CDP with timeout
|
||||
try:
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: Getting page info from CDP...')
|
||||
page_info = await asyncio.wait_for(self._get_page_info(), timeout=1.0)
|
||||
self.logger.debug(f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Got page info from CDP: {page_info}')
|
||||
except Exception as e:
|
||||
self.logger.debug(
|
||||
f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: Failed to get page info from CDP: {e}, using fallback'
|
||||
)
|
||||
# Fallback to default viewport dimensions
|
||||
viewport = self.browser_session.browser_profile.viewport or {'width': 1280, 'height': 720}
|
||||
page_info = PageInfo(
|
||||
viewport_width=viewport['width'],
|
||||
viewport_height=viewport['height'],
|
||||
page_width=viewport['width'],
|
||||
page_height=viewport['height'],
|
||||
scroll_x=0,
|
||||
scroll_y=0,
|
||||
pixels_above=0,
|
||||
pixels_below=0,
|
||||
pixels_left=0,
|
||||
pixels_right=0,
|
||||
)
|
||||
|
||||
# Check for PDF viewer
|
||||
is_pdf_viewer = page_url.endswith('.pdf') or '/pdf/' in page_url
|
||||
|
||||
# Build and cache the browser state summary
|
||||
if screenshot_b64:
|
||||
self.logger.debug(
|
||||
f'🔍 DOMWatchdog.on_BrowserStateRequestEvent: 📸 Creating BrowserStateSummary with screenshot, length: {len(screenshot_b64)}'
|
||||
)
|
||||
else:
|
||||
self.logger.debug(
|
||||
'🔍 DOMWatchdog.on_BrowserStateRequestEvent: 📸 Creating BrowserStateSummary WITHOUT screenshot'
|
||||
)
|
||||
|
||||
browser_state = BrowserStateSummary(
|
||||
dom_state=content,
|
||||
url=page_url,
|
||||
title=title,
|
||||
tabs=tabs_info,
|
||||
screenshot=screenshot_b64,
|
||||
page_info=page_info,
|
||||
pixels_above=0,
|
||||
pixels_below=0,
|
||||
browser_errors=[],
|
||||
is_pdf_viewer=is_pdf_viewer,
|
||||
recent_events=self._get_recent_events_str() if event.include_recent_events else None,
|
||||
)
|
||||
|
||||
# Cache the state
|
||||
self.browser_session._cached_browser_state_summary = browser_state
|
||||
|
||||
self.logger.debug('🔍 DOMWatchdog.on_BrowserStateRequestEvent: ✅ COMPLETED - Returning browser state')
|
||||
return browser_state
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'Failed to get browser state: {e}')
|
||||
|
||||
# Return minimal recovery state
|
||||
return BrowserStateSummary(
|
||||
dom_state=SerializedDOMState(_root=None, selector_map={}),
|
||||
url=page_url if 'page_url' in locals() else '',
|
||||
title='Error',
|
||||
tabs=[],
|
||||
screenshot=None,
|
||||
page_info=PageInfo(
|
||||
viewport_width=1280,
|
||||
viewport_height=720,
|
||||
page_width=1280,
|
||||
page_height=720,
|
||||
scroll_x=0,
|
||||
scroll_y=0,
|
||||
pixels_above=0,
|
||||
pixels_below=0,
|
||||
pixels_left=0,
|
||||
pixels_right=0,
|
||||
),
|
||||
pixels_above=0,
|
||||
pixels_below=0,
|
||||
browser_errors=[str(e)],
|
||||
is_pdf_viewer=False,
|
||||
recent_events=None,
|
||||
)
|
||||
|
||||
@time_execution_async('build_dom_tree_without_highlights')
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='build_dom_tree_without_highlights')
|
||||
async def _build_dom_tree_without_highlights(self, previous_state: SerializedDOMState | None = None) -> SerializedDOMState:
|
||||
"""Build DOM tree without injecting JavaScript highlights (for parallel execution)."""
|
||||
try:
|
||||
self.logger.debug('🔍 DOMWatchdog._build_dom_tree_without_highlights: STARTING DOM tree build')
|
||||
|
||||
# Create or reuse DOM service
|
||||
if self._dom_service is None:
|
||||
self._dom_service = DomService(
|
||||
browser_session=self.browser_session,
|
||||
logger=self.logger,
|
||||
cross_origin_iframes=self.browser_session.browser_profile.cross_origin_iframes,
|
||||
paint_order_filtering=self.browser_session.browser_profile.paint_order_filtering,
|
||||
max_iframes=self.browser_session.browser_profile.max_iframes,
|
||||
max_iframe_depth=self.browser_session.browser_profile.max_iframe_depth,
|
||||
)
|
||||
|
||||
# Get serialized DOM tree using the service
|
||||
self.logger.debug('🔍 DOMWatchdog._build_dom_tree_without_highlights: Calling DomService.get_serialized_dom_tree...')
|
||||
start = time.time()
|
||||
self.current_dom_state, self.enhanced_dom_tree, timing_info = await self._dom_service.get_serialized_dom_tree(
|
||||
previous_cached_state=previous_state,
|
||||
)
|
||||
end = time.time()
|
||||
self.logger.debug(
|
||||
'🔍 DOMWatchdog._build_dom_tree_without_highlights: ✅ DomService.get_serialized_dom_tree completed'
|
||||
)
|
||||
|
||||
self.logger.debug(f'Time taken to get DOM tree: {end - start} seconds')
|
||||
self.logger.debug(f'Timing breakdown: {timing_info}')
|
||||
|
||||
# Update selector map for other watchdogs
|
||||
self.logger.debug('🔍 DOMWatchdog._build_dom_tree_without_highlights: Updating selector maps...')
|
||||
self.selector_map = self.current_dom_state.selector_map
|
||||
# Update BrowserSession's cached selector map
|
||||
if self.browser_session:
|
||||
self.browser_session.update_cached_selector_map(self.selector_map)
|
||||
self.logger.debug(
|
||||
f'🔍 DOMWatchdog._build_dom_tree_without_highlights: ✅ Selector maps updated, {len(self.selector_map)} elements'
|
||||
)
|
||||
|
||||
# Skip JavaScript highlighting injection - Python highlighting will be applied later
|
||||
self.logger.debug('🔍 DOMWatchdog._build_dom_tree_without_highlights: ✅ COMPLETED DOM tree build (no JS highlights)')
|
||||
return self.current_dom_state
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'Failed to build DOM tree without highlights: {e}')
|
||||
self.event_bus.dispatch(
|
||||
BrowserErrorEvent(
|
||||
error_type='DOMBuildFailed',
|
||||
message=str(e),
|
||||
)
|
||||
)
|
||||
raise
|
||||
|
||||
@time_execution_async('capture_clean_screenshot')
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='capture_clean_screenshot')
|
||||
async def _capture_clean_screenshot(self) -> str:
|
||||
"""Capture a clean screenshot without JavaScript highlights."""
|
||||
try:
|
||||
self.logger.debug('🔍 DOMWatchdog._capture_clean_screenshot: Capturing clean screenshot...')
|
||||
|
||||
# Ensure we have a focused CDP session
|
||||
assert self.browser_session.agent_focus is not None, 'No current target ID'
|
||||
await self.browser_session.get_or_create_cdp_session(target_id=self.browser_session.agent_focus.target_id, focus=True)
|
||||
|
||||
# Check if handler is registered
|
||||
handlers = self.event_bus.handlers.get('ScreenshotEvent', [])
|
||||
handler_names = [getattr(h, '__name__', str(h)) for h in handlers]
|
||||
self.logger.debug(f'📸 ScreenshotEvent handlers registered: {len(handlers)} - {handler_names}')
|
||||
|
||||
screenshot_event = self.event_bus.dispatch(ScreenshotEvent(full_page=False))
|
||||
self.logger.debug('📸 Dispatched ScreenshotEvent, waiting for event to complete...')
|
||||
|
||||
# Wait for the event itself to complete (this waits for all handlers)
|
||||
await screenshot_event
|
||||
|
||||
# Get the single handler result
|
||||
screenshot_b64 = await screenshot_event.event_result(raise_if_any=True, raise_if_none=True)
|
||||
if screenshot_b64 is None:
|
||||
raise RuntimeError('Screenshot handler returned None')
|
||||
self.logger.debug('🔍 DOMWatchdog._capture_clean_screenshot: ✅ Clean screenshot captured successfully')
|
||||
return str(screenshot_b64)
|
||||
|
||||
except TimeoutError:
|
||||
self.logger.warning('📸 Clean screenshot timed out after 6 seconds - no handler registered or slow page?')
|
||||
raise
|
||||
except Exception as e:
|
||||
self.logger.warning(f'📸 Clean screenshot failed: {type(e).__name__}: {e}')
|
||||
raise
|
||||
|
||||
async def _wait_for_stable_network(self):
|
||||
"""Wait for page stability - simplified for CDP-only branch."""
|
||||
start_time = time.time()
|
||||
|
||||
# Apply minimum wait time first (let page settle)
|
||||
min_wait = self.browser_session.browser_profile.minimum_wait_page_load_time
|
||||
if min_wait > 0:
|
||||
self.logger.debug(f'⏳ Minimum wait: {min_wait}s')
|
||||
await asyncio.sleep(min_wait)
|
||||
|
||||
# Apply network idle wait time (for dynamic content like iframes)
|
||||
network_idle_wait = self.browser_session.browser_profile.wait_for_network_idle_page_load_time
|
||||
if network_idle_wait > 0:
|
||||
self.logger.debug(f'⏳ Network idle wait: {network_idle_wait}s')
|
||||
await asyncio.sleep(network_idle_wait)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
self.logger.debug(f'✅ Page stability wait completed in {elapsed:.2f}s')
|
||||
|
||||
async def _get_page_info(self) -> 'PageInfo':
|
||||
"""Get comprehensive page information using a single CDP call.
|
||||
|
||||
TODO: should we make this an event as well?
|
||||
|
||||
Returns:
|
||||
PageInfo with all viewport, page dimensions, and scroll information
|
||||
"""
|
||||
|
||||
from browser_use.browser.views import PageInfo
|
||||
|
||||
# Get CDP session for the current target
|
||||
if not self.browser_session.agent_focus:
|
||||
raise RuntimeError('No active CDP session - browser may not be connected yet')
|
||||
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session(
|
||||
target_id=self.browser_session.agent_focus.target_id, focus=True
|
||||
)
|
||||
|
||||
# Get layout metrics which includes all the information we need
|
||||
metrics = await asyncio.wait_for(
|
||||
cdp_session.cdp_client.send.Page.getLayoutMetrics(session_id=cdp_session.session_id), timeout=10.0
|
||||
)
|
||||
|
||||
# Extract different viewport types
|
||||
layout_viewport = metrics.get('layoutViewport', {})
|
||||
visual_viewport = metrics.get('visualViewport', {})
|
||||
css_visual_viewport = metrics.get('cssVisualViewport', {})
|
||||
css_layout_viewport = metrics.get('cssLayoutViewport', {})
|
||||
content_size = metrics.get('contentSize', {})
|
||||
|
||||
# Calculate device pixel ratio to convert between device pixels and CSS pixels
|
||||
# This matches the approach in dom/service.py _get_viewport_ratio method
|
||||
css_width = css_visual_viewport.get('clientWidth', css_layout_viewport.get('clientWidth', 1280.0))
|
||||
device_width = visual_viewport.get('clientWidth', css_width)
|
||||
device_pixel_ratio = device_width / css_width if css_width > 0 else 1.0
|
||||
|
||||
# For viewport dimensions, use CSS pixels (what JavaScript sees)
|
||||
# Prioritize CSS layout viewport, then fall back to layout viewport
|
||||
viewport_width = int(css_layout_viewport.get('clientWidth') or layout_viewport.get('clientWidth', 1280))
|
||||
viewport_height = int(css_layout_viewport.get('clientHeight') or layout_viewport.get('clientHeight', 720))
|
||||
|
||||
# For total page dimensions, content size is typically in device pixels, so convert to CSS pixels
|
||||
# by dividing by device pixel ratio
|
||||
raw_page_width = content_size.get('width', viewport_width * device_pixel_ratio)
|
||||
raw_page_height = content_size.get('height', viewport_height * device_pixel_ratio)
|
||||
page_width = int(raw_page_width / device_pixel_ratio)
|
||||
page_height = int(raw_page_height / device_pixel_ratio)
|
||||
|
||||
# For scroll position, use CSS visual viewport if available, otherwise CSS layout viewport
|
||||
# These should already be in CSS pixels
|
||||
scroll_x = int(css_visual_viewport.get('pageX') or css_layout_viewport.get('pageX', 0))
|
||||
scroll_y = int(css_visual_viewport.get('pageY') or css_layout_viewport.get('pageY', 0))
|
||||
|
||||
# Calculate scroll information - pixels that are above/below/left/right of current viewport
|
||||
pixels_above = scroll_y
|
||||
pixels_below = max(0, page_height - viewport_height - scroll_y)
|
||||
pixels_left = scroll_x
|
||||
pixels_right = max(0, page_width - viewport_width - scroll_x)
|
||||
|
||||
page_info = PageInfo(
|
||||
viewport_width=viewport_width,
|
||||
viewport_height=viewport_height,
|
||||
page_width=page_width,
|
||||
page_height=page_height,
|
||||
scroll_x=scroll_x,
|
||||
scroll_y=scroll_y,
|
||||
pixels_above=pixels_above,
|
||||
pixels_below=pixels_below,
|
||||
pixels_left=pixels_left,
|
||||
pixels_right=pixels_right,
|
||||
)
|
||||
|
||||
return page_info
|
||||
|
||||
# ========== Public Helper Methods ==========
|
||||
|
||||
async def get_element_by_index(self, index: int) -> EnhancedDOMTreeNode | None:
|
||||
"""Get DOM element by index from cached selector map.
|
||||
|
||||
Builds DOM if not cached.
|
||||
|
||||
Returns:
|
||||
EnhancedDOMTreeNode or None if index not found
|
||||
"""
|
||||
if not self.selector_map:
|
||||
# Build DOM if not cached
|
||||
await self._build_dom_tree_without_highlights()
|
||||
|
||||
return self.selector_map.get(index) if self.selector_map else None
|
||||
|
||||
def clear_cache(self) -> None:
|
||||
"""Clear cached DOM state to force rebuild on next access."""
|
||||
self.selector_map = None
|
||||
self.current_dom_state = None
|
||||
self.enhanced_dom_tree = None
|
||||
# Keep the DOM service instance to reuse its CDP client connection
|
||||
|
||||
def is_file_input(self, element: EnhancedDOMTreeNode) -> bool:
|
||||
"""Check if element is a file input."""
|
||||
return element.node_name.upper() == 'INPUT' and element.attributes.get('type', '').lower() == 'file'
|
||||
|
||||
@staticmethod
|
||||
def is_element_visible_according_to_all_parents(node: EnhancedDOMTreeNode, html_frames: list[EnhancedDOMTreeNode]) -> bool:
|
||||
"""Check if the element is visible according to all its parent HTML frames.
|
||||
|
||||
Delegates to the DomService static method.
|
||||
"""
|
||||
return DomService.is_element_visible_according_to_all_parents(node, html_frames)
|
||||
|
||||
async def __aexit__(self, exc_type, exc_value, traceback):
|
||||
"""Clean up DOM service on exit."""
|
||||
if self._dom_service:
|
||||
await self._dom_service.__aexit__(exc_type, exc_value, traceback)
|
||||
self._dom_service = None
|
||||
|
||||
def __del__(self):
|
||||
"""Clean up DOM service on deletion."""
|
||||
super().__del__()
|
||||
# DOM service will clean up its own CDP client
|
||||
self._dom_service = None
|
||||
+933
@@ -0,0 +1,933 @@
|
||||
"""Downloads watchdog for monitoring and handling file downloads."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, ClassVar
|
||||
from urllib.parse import urlparse
|
||||
|
||||
import anyio
|
||||
from bubus import BaseEvent
|
||||
from cdp_use.cdp.browser import DownloadProgressEvent, DownloadWillBeginEvent
|
||||
from cdp_use.cdp.target import SessionID, TargetID
|
||||
from pydantic import PrivateAttr
|
||||
|
||||
from browser_use.browser.events import (
|
||||
BrowserLaunchEvent,
|
||||
BrowserStateRequestEvent,
|
||||
BrowserStoppedEvent,
|
||||
FileDownloadedEvent,
|
||||
NavigationCompleteEvent,
|
||||
TabClosedEvent,
|
||||
TabCreatedEvent,
|
||||
)
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
class DownloadsWatchdog(BaseWatchdog):
|
||||
"""Monitors downloads and handles file download events."""
|
||||
|
||||
# Events this watchdog listens to (for documentation)
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent[Any]]]] = [
|
||||
BrowserLaunchEvent,
|
||||
BrowserStateRequestEvent,
|
||||
BrowserStoppedEvent,
|
||||
TabCreatedEvent,
|
||||
TabClosedEvent,
|
||||
NavigationCompleteEvent,
|
||||
]
|
||||
|
||||
# Events this watchdog emits
|
||||
EMITS: ClassVar[list[type[BaseEvent[Any]]]] = [
|
||||
FileDownloadedEvent,
|
||||
]
|
||||
|
||||
# Private state
|
||||
_sessions_with_listeners: set[str] = PrivateAttr(default_factory=set) # Track sessions that already have download listeners
|
||||
_active_downloads: dict[str, Any] = PrivateAttr(default_factory=dict)
|
||||
_pdf_viewer_cache: dict[str, bool] = PrivateAttr(default_factory=dict) # Cache PDF viewer status by target URL
|
||||
_download_cdp_session_setup: bool = PrivateAttr(default=False) # Track if CDP session is set up
|
||||
_download_cdp_session: Any = PrivateAttr(default=None) # Store CDP session reference
|
||||
_cdp_event_tasks: set[asyncio.Task] = PrivateAttr(default_factory=set) # Track CDP event handler tasks
|
||||
_cdp_downloads_info: dict[str, dict[str, Any]] = PrivateAttr(default_factory=dict) # Map guid -> info
|
||||
_use_js_fetch_for_local: bool = PrivateAttr(default=False) # Guard JS fetch path for local regular downloads
|
||||
|
||||
async def on_BrowserLaunchEvent(self, event: BrowserLaunchEvent) -> None:
|
||||
self.logger.debug(f'[DownloadsWatchdog] Received BrowserLaunchEvent, EventBus ID: {id(self.event_bus)}')
|
||||
# Ensure downloads directory exists
|
||||
downloads_path = self.browser_session.browser_profile.downloads_path
|
||||
if downloads_path:
|
||||
expanded_path = Path(downloads_path).expanduser().resolve()
|
||||
expanded_path.mkdir(parents=True, exist_ok=True)
|
||||
self.logger.debug(f'[DownloadsWatchdog] Ensured downloads directory exists: {expanded_path}')
|
||||
|
||||
async def on_TabCreatedEvent(self, event: TabCreatedEvent) -> None:
|
||||
"""Monitor new tabs for downloads."""
|
||||
# logger.info(f'[DownloadsWatchdog] TabCreatedEvent received for tab {event.target_id[-4:]}: {event.url}')
|
||||
|
||||
# Assert downloads path is configured (should always be set by BrowserProfile default)
|
||||
assert self.browser_session.browser_profile.downloads_path is not None, 'Downloads path must be configured'
|
||||
|
||||
if event.target_id:
|
||||
# logger.info(f'[DownloadsWatchdog] Found target for tab {event.target_id}, calling attach_to_target')
|
||||
await self.attach_to_target(event.target_id)
|
||||
else:
|
||||
self.logger.warning(f'[DownloadsWatchdog] No target found for tab {event.target_id}')
|
||||
|
||||
async def on_TabClosedEvent(self, event: TabClosedEvent) -> None:
|
||||
"""Stop monitoring closed tabs."""
|
||||
pass # No cleanup needed, browser context handles target lifecycle
|
||||
|
||||
async def on_BrowserStateRequestEvent(self, event: BrowserStateRequestEvent) -> None:
|
||||
"""Handle browser state request events."""
|
||||
cdp_session = self.browser_session.agent_focus
|
||||
if not cdp_session:
|
||||
return
|
||||
|
||||
url = await self.browser_session.get_current_page_url()
|
||||
if not url:
|
||||
return
|
||||
|
||||
target_id = cdp_session.target_id
|
||||
self.event_bus.dispatch(
|
||||
NavigationCompleteEvent(
|
||||
event_type='NavigationCompleteEvent',
|
||||
url=url,
|
||||
target_id=target_id,
|
||||
event_parent_id=event.event_id,
|
||||
)
|
||||
)
|
||||
|
||||
async def on_BrowserStoppedEvent(self, event: BrowserStoppedEvent) -> None:
|
||||
"""Clean up when browser stops."""
|
||||
# Cancel all CDP event handler tasks
|
||||
for task in list(self._cdp_event_tasks):
|
||||
if not task.done():
|
||||
task.cancel()
|
||||
# Wait for all tasks to complete cancellation
|
||||
if self._cdp_event_tasks:
|
||||
await asyncio.gather(*self._cdp_event_tasks, return_exceptions=True)
|
||||
self._cdp_event_tasks.clear()
|
||||
|
||||
# Clean up CDP session
|
||||
# CDP sessions are now cached and managed by BrowserSession
|
||||
self._download_cdp_session = None
|
||||
self._download_cdp_session_setup = False
|
||||
|
||||
# Clear other state
|
||||
self._sessions_with_listeners.clear()
|
||||
self._active_downloads.clear()
|
||||
self._pdf_viewer_cache.clear()
|
||||
|
||||
async def on_NavigationCompleteEvent(self, event: NavigationCompleteEvent) -> None:
|
||||
"""Check for PDFs after navigation completes."""
|
||||
self.logger.debug(f'[DownloadsWatchdog] NavigationCompleteEvent received for {event.url}, tab #{event.target_id[-4:]}')
|
||||
|
||||
# Clear PDF cache for the navigated URL since content may have changed
|
||||
if event.url in self._pdf_viewer_cache:
|
||||
del self._pdf_viewer_cache[event.url]
|
||||
|
||||
# Check if auto-download is enabled
|
||||
auto_download_enabled = self._is_auto_download_enabled()
|
||||
if not auto_download_enabled:
|
||||
return
|
||||
|
||||
# Note: Using network-based PDF detection that doesn't require JavaScript
|
||||
|
||||
target_id = event.target_id
|
||||
self.logger.debug(f'[DownloadsWatchdog] Got target_id={target_id} for tab #{event.target_id[-4:]}')
|
||||
|
||||
is_pdf = await self.check_for_pdf_viewer(target_id)
|
||||
if is_pdf:
|
||||
self.logger.debug(f'[DownloadsWatchdog] 📄 PDF detected at {event.url}, triggering auto-download...')
|
||||
download_path = await self.trigger_pdf_download(target_id)
|
||||
if not download_path:
|
||||
self.logger.warning(f'[DownloadsWatchdog] ⚠️ PDF download failed for {event.url}')
|
||||
|
||||
def _is_auto_download_enabled(self) -> bool:
|
||||
"""Check if auto-download PDFs is enabled in browser profile."""
|
||||
return self.browser_session.browser_profile.auto_download_pdfs
|
||||
|
||||
async def attach_to_target(self, target_id: TargetID) -> None:
|
||||
"""Set up download monitoring for a specific target."""
|
||||
|
||||
# Define CDP event handlers outside of try to avoid indentation/scope issues
|
||||
async def download_will_begin_handler(event: DownloadWillBeginEvent, session_id: SessionID | None):
|
||||
self.logger.debug(f'[DownloadsWatchdog] Download will begin: {event}')
|
||||
# Cache info for later completion event handling (esp. remote browsers)
|
||||
guid = event.get('guid', '')
|
||||
try:
|
||||
suggested_filename = event.get('suggestedFilename')
|
||||
assert suggested_filename, 'CDP DownloadWillBegin missing suggestedFilename'
|
||||
self._cdp_downloads_info[guid] = {
|
||||
'url': event.get('url', ''),
|
||||
'suggested_filename': suggested_filename,
|
||||
'handled': False,
|
||||
}
|
||||
except (AssertionError, KeyError):
|
||||
pass
|
||||
# Create and track the task
|
||||
task = asyncio.create_task(self._handle_cdp_download(event, target_id, session_id))
|
||||
self._cdp_event_tasks.add(task)
|
||||
# Remove from set when done
|
||||
task.add_done_callback(lambda t: self._cdp_event_tasks.discard(t))
|
||||
|
||||
async def download_progress_handler(event: DownloadProgressEvent, session_id: SessionID | None):
|
||||
# Check if download is complete
|
||||
if event.get('state') == 'completed':
|
||||
file_path = event.get('filePath')
|
||||
guid = event.get('guid', '')
|
||||
if self.browser_session.is_local:
|
||||
if file_path:
|
||||
self.logger.debug(f'[DownloadsWatchdog] Download completed: {file_path}')
|
||||
# Track the download
|
||||
self._track_download(file_path)
|
||||
# Mark as handled to prevent fallback duplicate dispatch
|
||||
try:
|
||||
if guid in self._cdp_downloads_info:
|
||||
self._cdp_downloads_info[guid]['handled'] = True
|
||||
except (KeyError, AttributeError):
|
||||
pass
|
||||
else:
|
||||
# No local file path provided, local polling in _handle_cdp_download will handle it
|
||||
self.logger.debug(
|
||||
'[DownloadsWatchdog] No filePath in progress event (local); polling will handle detection'
|
||||
)
|
||||
else:
|
||||
# Remote browser: do not touch local filesystem. Fallback to downloadPath+suggestedFilename
|
||||
info = self._cdp_downloads_info.get(guid, {})
|
||||
try:
|
||||
suggested_filename = info.get('suggested_filename') or (Path(file_path).name if file_path else 'download')
|
||||
downloads_path = str(self.browser_session.browser_profile.downloads_path or '')
|
||||
effective_path = file_path or str(Path(downloads_path) / suggested_filename)
|
||||
file_name = Path(effective_path).name
|
||||
file_ext = Path(file_name).suffix.lower().lstrip('.')
|
||||
self.event_bus.dispatch(
|
||||
FileDownloadedEvent(
|
||||
url=info.get('url', ''),
|
||||
path=str(effective_path),
|
||||
file_name=file_name,
|
||||
file_size=0,
|
||||
file_type=file_ext if file_ext else None,
|
||||
)
|
||||
)
|
||||
self.logger.debug(f'[DownloadsWatchdog] ✅ (remote) Download completed: {effective_path}')
|
||||
finally:
|
||||
if guid in self._cdp_downloads_info:
|
||||
del self._cdp_downloads_info[guid]
|
||||
|
||||
try:
|
||||
downloads_path_raw = self.browser_session.browser_profile.downloads_path
|
||||
if not downloads_path_raw:
|
||||
# logger.info(f'[DownloadsWatchdog] No downloads path configured, skipping target: {target_id}')
|
||||
return # No downloads path configured
|
||||
|
||||
# Check if we already have a download listener on this session
|
||||
# to prevent duplicate listeners from being added
|
||||
# Note: Since download listeners are set up once per browser session, not per target,
|
||||
# we just track if we've set up the browser-level listener
|
||||
if self._download_cdp_session_setup:
|
||||
self.logger.debug('[DownloadsWatchdog] Download listener already set up for browser session')
|
||||
return
|
||||
|
||||
# logger.debug(f'[DownloadsWatchdog] Setting up CDP download listener for target: {target_id}')
|
||||
|
||||
# Use CDP session for download events but store reference in watchdog
|
||||
if not self._download_cdp_session_setup:
|
||||
# Set up CDP session for downloads (only once per browser session)
|
||||
cdp_client = self.browser_session.cdp_client
|
||||
|
||||
# Set download behavior to allow downloads and enable events
|
||||
downloads_path = self.browser_session.browser_profile.downloads_path
|
||||
if not downloads_path:
|
||||
self.logger.warning('[DownloadsWatchdog] No downloads path configured, skipping CDP download setup')
|
||||
return
|
||||
# Ensure path is properly expanded (~ -> absolute path)
|
||||
expanded_downloads_path = Path(downloads_path).expanduser().resolve()
|
||||
await cdp_client.send.Browser.setDownloadBehavior(
|
||||
params={
|
||||
'behavior': 'allow',
|
||||
'downloadPath': str(expanded_downloads_path), # Use expanded absolute path
|
||||
'eventsEnabled': True,
|
||||
}
|
||||
)
|
||||
|
||||
# Register the handlers with CDP
|
||||
cdp_client.register.Browser.downloadWillBegin(download_will_begin_handler) # type: ignore[arg-type]
|
||||
cdp_client.register.Browser.downloadProgress(download_progress_handler) # type: ignore[arg-type]
|
||||
|
||||
self._download_cdp_session_setup = True
|
||||
self.logger.debug('[DownloadsWatchdog] Set up CDP download listeners')
|
||||
|
||||
# No need to track individual targets since download listener is browser-level
|
||||
# logger.debug(f'[DownloadsWatchdog] Successfully set up CDP download listener for target: {target_id}')
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f'[DownloadsWatchdog] Failed to set up CDP download listener for target {target_id}: {e}')
|
||||
|
||||
def _track_download(self, file_path: str) -> None:
|
||||
"""Track a completed download and dispatch the appropriate event.
|
||||
|
||||
Args:
|
||||
file_path: The path to the downloaded file
|
||||
"""
|
||||
try:
|
||||
# Get file info
|
||||
path = Path(file_path)
|
||||
if path.exists():
|
||||
file_size = path.stat().st_size
|
||||
self.logger.debug(f'[DownloadsWatchdog] Tracked download: {path.name} ({file_size} bytes)')
|
||||
|
||||
# Dispatch download event
|
||||
from browser_use.browser.events import FileDownloadedEvent
|
||||
|
||||
self.event_bus.dispatch(
|
||||
FileDownloadedEvent(
|
||||
url=str(path), # Use the file path as URL for local files
|
||||
path=str(path),
|
||||
file_name=path.name,
|
||||
file_size=file_size,
|
||||
)
|
||||
)
|
||||
else:
|
||||
self.logger.warning(f'[DownloadsWatchdog] Downloaded file not found: {file_path}')
|
||||
except Exception as e:
|
||||
self.logger.error(f'[DownloadsWatchdog] Error tracking download: {e}')
|
||||
|
||||
async def _handle_cdp_download(
|
||||
self, event: DownloadWillBeginEvent, target_id: TargetID, session_id: SessionID | None
|
||||
) -> None:
|
||||
"""Handle a CDP Page.downloadWillBegin event."""
|
||||
downloads_dir = (
|
||||
Path(
|
||||
self.browser_session.browser_profile.downloads_path
|
||||
or f'{tempfile.gettempdir()}/browser_use_downloads.{str(self.browser_session.id)[-4:]}'
|
||||
)
|
||||
.expanduser()
|
||||
.resolve()
|
||||
) # Ensure path is properly expanded
|
||||
|
||||
# Initialize variables that may be used outside try blocks
|
||||
unique_filename = None
|
||||
file_size = 0
|
||||
expected_path = None
|
||||
download_result = None
|
||||
download_url = event.get('url', '')
|
||||
suggested_filename = event.get('suggestedFilename', 'download')
|
||||
guid = event.get('guid', '')
|
||||
|
||||
try:
|
||||
self.logger.debug(f'[DownloadsWatchdog] ⬇️ File download starting: {suggested_filename} from {download_url[:100]}...')
|
||||
self.logger.debug(f'[DownloadsWatchdog] Full CDP event: {event}')
|
||||
|
||||
# Since Browser.setDownloadBehavior is already configured, the browser will download the file
|
||||
# We just need to wait for it to appear in the downloads directory
|
||||
expected_path = downloads_dir / suggested_filename
|
||||
|
||||
# Debug: List current directory contents
|
||||
self.logger.debug(f'[DownloadsWatchdog] Downloads directory: {downloads_dir}')
|
||||
if downloads_dir.exists():
|
||||
files_before = list(downloads_dir.iterdir())
|
||||
self.logger.debug(f'[DownloadsWatchdog] Files before download: {[f.name for f in files_before]}')
|
||||
|
||||
# Try manual JavaScript fetch as a fallback for local browsers (disabled for regular local downloads)
|
||||
if self.browser_session.is_local and self._use_js_fetch_for_local:
|
||||
self.logger.debug(f'[DownloadsWatchdog] Attempting JS fetch fallback for {download_url}')
|
||||
|
||||
unique_filename = None
|
||||
file_size = None
|
||||
download_result = None
|
||||
try:
|
||||
# Escape the URL for JavaScript
|
||||
import json
|
||||
|
||||
escaped_url = json.dumps(download_url)
|
||||
|
||||
# Get the proper session for the frame that initiated the download
|
||||
cdp_session = await self.browser_session.cdp_client_for_frame(event.get('frameId'))
|
||||
assert cdp_session
|
||||
|
||||
result = await cdp_session.cdp_client.send.Runtime.evaluate(
|
||||
params={
|
||||
'expression': f"""
|
||||
(async () => {{
|
||||
try {{
|
||||
const response = await fetch({escaped_url});
|
||||
if (!response.ok) {{
|
||||
throw new Error(`HTTP error! status: ${{response.status}}`);
|
||||
}}
|
||||
const blob = await response.blob();
|
||||
const arrayBuffer = await blob.arrayBuffer();
|
||||
const uint8Array = new Uint8Array(arrayBuffer);
|
||||
return {{
|
||||
data: Array.from(uint8Array),
|
||||
size: uint8Array.length,
|
||||
contentType: response.headers.get('content-type') || 'application/octet-stream'
|
||||
}};
|
||||
}} catch (error) {{
|
||||
throw new Error(`Fetch failed: ${{error.message}}`);
|
||||
}}
|
||||
}})()
|
||||
""",
|
||||
'awaitPromise': True,
|
||||
'returnByValue': True,
|
||||
},
|
||||
session_id=cdp_session.session_id,
|
||||
)
|
||||
download_result = result.get('result', {}).get('value')
|
||||
|
||||
if download_result and download_result.get('data'):
|
||||
# Save the file
|
||||
file_data = bytes(download_result['data'])
|
||||
file_size = len(file_data)
|
||||
|
||||
# Ensure unique filename
|
||||
unique_filename = await self._get_unique_filename(str(downloads_dir), suggested_filename)
|
||||
final_path = downloads_dir / unique_filename
|
||||
|
||||
# Write the file
|
||||
import anyio
|
||||
|
||||
async with await anyio.open_file(final_path, 'wb') as f:
|
||||
await f.write(file_data)
|
||||
|
||||
self.logger.debug(f'[DownloadsWatchdog] ✅ Downloaded and saved file: {final_path} ({file_size} bytes)')
|
||||
expected_path = final_path
|
||||
# Emit download event immediately
|
||||
file_ext = expected_path.suffix.lower().lstrip('.')
|
||||
file_type = file_ext if file_ext else None
|
||||
self.event_bus.dispatch(
|
||||
FileDownloadedEvent(
|
||||
url=download_url,
|
||||
path=str(expected_path),
|
||||
file_name=unique_filename or expected_path.name,
|
||||
file_size=file_size or 0,
|
||||
file_type=file_type,
|
||||
mime_type=(download_result.get('contentType') if download_result else None),
|
||||
from_cache=False,
|
||||
auto_download=False,
|
||||
)
|
||||
)
|
||||
# Mark as handled to prevent duplicate dispatch from progress/polling paths
|
||||
try:
|
||||
if guid in self._cdp_downloads_info:
|
||||
self._cdp_downloads_info[guid]['handled'] = True
|
||||
except (KeyError, AttributeError):
|
||||
pass
|
||||
self.logger.debug(
|
||||
f'[DownloadsWatchdog] ✅ File download completed via CDP: {suggested_filename} ({file_size} bytes) saved to {expected_path}'
|
||||
)
|
||||
return
|
||||
else:
|
||||
self.logger.error('[DownloadsWatchdog] ❌ No data received from fetch')
|
||||
|
||||
except Exception as fetch_error:
|
||||
self.logger.error(f'[DownloadsWatchdog] ❌ Failed to download file via fetch: {fetch_error}')
|
||||
|
||||
# For remote browsers, don't poll local filesystem; downloadProgress handler will emit the event
|
||||
if not self.browser_session.is_local:
|
||||
return
|
||||
except Exception as e:
|
||||
self.logger.error(f'[DownloadsWatchdog] ❌ Error handling CDP download: {type(e).__name__} {e}')
|
||||
|
||||
# If we reach here, the fetch method failed, so wait for native download
|
||||
# Poll the downloads directory for new files
|
||||
self.logger.debug(f'[DownloadsWatchdog] Checking if browser auto-download saved the file for us: {suggested_filename}')
|
||||
|
||||
# Get initial list of files in downloads directory
|
||||
initial_files = set()
|
||||
if Path(downloads_dir).exists():
|
||||
for f in Path(downloads_dir).iterdir():
|
||||
if f.is_file() and not f.name.startswith('.'):
|
||||
initial_files.add(f.name)
|
||||
|
||||
# Poll for new files
|
||||
max_wait = 20 # seconds
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
|
||||
while asyncio.get_event_loop().time() - start_time < max_wait:
|
||||
await asyncio.sleep(5.0) # Check every 5 seconds
|
||||
|
||||
if Path(downloads_dir).exists():
|
||||
for file_path in Path(downloads_dir).iterdir():
|
||||
# Skip hidden files and files that were already there
|
||||
if file_path.is_file() and not file_path.name.startswith('.') and file_path.name not in initial_files:
|
||||
# Check if file has content (> 4 bytes)
|
||||
try:
|
||||
file_size = file_path.stat().st_size
|
||||
if file_size > 4:
|
||||
# Found a new download!
|
||||
self.logger.debug(
|
||||
f'[DownloadsWatchdog] ✅ Found downloaded file: {file_path} ({file_size} bytes)'
|
||||
)
|
||||
|
||||
# Determine file type from extension
|
||||
file_ext = file_path.suffix.lower().lstrip('.')
|
||||
file_type = file_ext if file_ext else None
|
||||
|
||||
# Dispatch download event
|
||||
# Skip if already handled by progress/JS fetch
|
||||
info = self._cdp_downloads_info.get(guid, {})
|
||||
if info.get('handled'):
|
||||
return
|
||||
self.event_bus.dispatch(
|
||||
FileDownloadedEvent(
|
||||
url=download_url,
|
||||
path=str(file_path),
|
||||
file_name=file_path.name,
|
||||
file_size=file_size,
|
||||
file_type=file_type,
|
||||
)
|
||||
)
|
||||
# Mark as handled after dispatch
|
||||
try:
|
||||
if guid in self._cdp_downloads_info:
|
||||
self._cdp_downloads_info[guid]['handled'] = True
|
||||
except (KeyError, AttributeError):
|
||||
pass
|
||||
return
|
||||
except Exception as e:
|
||||
self.logger.debug(f'[DownloadsWatchdog] Error checking file {file_path}: {e}')
|
||||
|
||||
self.logger.warning(f'[DownloadsWatchdog] Download did not complete within {max_wait} seconds')
|
||||
|
||||
async def _handle_download(self, download: Any) -> None:
|
||||
"""Handle a download event."""
|
||||
download_id = f'{id(download)}'
|
||||
self._active_downloads[download_id] = download
|
||||
self.logger.debug(f'[DownloadsWatchdog] ⬇️ Handling download: {download.suggested_filename} from {download.url[:100]}...')
|
||||
|
||||
# Debug: Check if download is already being handled elsewhere
|
||||
failure = (
|
||||
await download.failure()
|
||||
) # TODO: it always fails for some reason, figure out why connect_over_cdp makes accept_downloads not work
|
||||
self.logger.warning(f'[DownloadsWatchdog] ❌ Download state - canceled: {failure}, url: {download.url}')
|
||||
# logger.info(f'[DownloadsWatchdog] Active downloads count: {len(self._active_downloads)}')
|
||||
|
||||
try:
|
||||
current_step = 'getting_download_info'
|
||||
# Get download info immediately
|
||||
url = download.url
|
||||
suggested_filename = download.suggested_filename
|
||||
|
||||
current_step = 'determining_download_directory'
|
||||
# Determine download directory from browser profile
|
||||
downloads_dir = self.browser_session.browser_profile.downloads_path
|
||||
if not downloads_dir:
|
||||
downloads_dir = str(Path.home() / 'Downloads')
|
||||
else:
|
||||
downloads_dir = str(downloads_dir) # Ensure it's a string
|
||||
|
||||
# Check if Playwright already auto-downloaded the file (due to CDP setup)
|
||||
original_path = Path(downloads_dir) / suggested_filename
|
||||
if original_path.exists() and original_path.stat().st_size > 0:
|
||||
self.logger.debug(
|
||||
f'[DownloadsWatchdog] File already downloaded by Playwright: {original_path} ({original_path.stat().st_size} bytes)'
|
||||
)
|
||||
|
||||
# Use the existing file instead of creating a duplicate
|
||||
download_path = original_path
|
||||
file_size = original_path.stat().st_size
|
||||
unique_filename = suggested_filename
|
||||
else:
|
||||
current_step = 'generating_unique_filename'
|
||||
# Ensure unique filename
|
||||
unique_filename = await self._get_unique_filename(downloads_dir, suggested_filename)
|
||||
download_path = Path(downloads_dir) / unique_filename
|
||||
|
||||
self.logger.debug(f'[DownloadsWatchdog] Download started: {unique_filename} from {url[:100]}...')
|
||||
|
||||
current_step = 'calling_save_as'
|
||||
# Save the download using Playwright's save_as method
|
||||
self.logger.debug(f'[DownloadsWatchdog] Saving download to: {download_path}')
|
||||
self.logger.debug(f'[DownloadsWatchdog] Download path exists: {download_path.parent.exists()}')
|
||||
self.logger.debug(f'[DownloadsWatchdog] Download path writable: {os.access(download_path.parent, os.W_OK)}')
|
||||
|
||||
try:
|
||||
self.logger.debug('[DownloadsWatchdog] About to call download.save_as()...')
|
||||
await download.save_as(str(download_path))
|
||||
self.logger.debug(f'[DownloadsWatchdog] Successfully saved download to: {download_path}')
|
||||
current_step = 'save_as_completed'
|
||||
except Exception as save_error:
|
||||
self.logger.error(f'[DownloadsWatchdog] save_as() failed with error: {save_error}')
|
||||
raise save_error
|
||||
|
||||
# Get file info
|
||||
file_size = download_path.stat().st_size if download_path.exists() else 0
|
||||
|
||||
# Determine file type from extension
|
||||
file_ext = download_path.suffix.lower().lstrip('.')
|
||||
file_type = file_ext if file_ext else None
|
||||
|
||||
# Try to get MIME type from response headers if available
|
||||
mime_type = None
|
||||
# Note: Playwright doesn't expose response headers directly from Download object
|
||||
|
||||
# Check if this was a PDF auto-download
|
||||
auto_download = False
|
||||
if file_type == 'pdf':
|
||||
auto_download = self._is_auto_download_enabled()
|
||||
|
||||
# Emit download event
|
||||
self.event_bus.dispatch(
|
||||
FileDownloadedEvent(
|
||||
url=url,
|
||||
path=str(download_path),
|
||||
file_name=suggested_filename,
|
||||
file_size=file_size,
|
||||
file_type=file_type,
|
||||
mime_type=mime_type,
|
||||
from_cache=False,
|
||||
auto_download=auto_download,
|
||||
)
|
||||
)
|
||||
|
||||
self.logger.debug(
|
||||
f'[DownloadsWatchdog] ✅ Download completed: {suggested_filename} ({file_size} bytes) saved to {download_path}'
|
||||
)
|
||||
|
||||
# File is now tracked on filesystem, no need to track in memory
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(
|
||||
f'[DownloadsWatchdog] Error handling download at step "{locals().get("current_step", "unknown")}", error: {e}'
|
||||
)
|
||||
self.logger.error(
|
||||
f'[DownloadsWatchdog] Download state - URL: {download.url}, filename: {download.suggested_filename}'
|
||||
)
|
||||
finally:
|
||||
# Clean up tracking
|
||||
if download_id in self._active_downloads:
|
||||
del self._active_downloads[download_id]
|
||||
|
||||
async def check_for_pdf_viewer(self, target_id: TargetID) -> bool:
|
||||
"""Check if the current target is a PDF using network-based detection.
|
||||
|
||||
This method avoids JavaScript execution that can crash WebSocket connections.
|
||||
Returns True if a PDF is detected and should be downloaded.
|
||||
"""
|
||||
self.logger.debug(f'[DownloadsWatchdog] Checking if target {target_id} is PDF viewer...')
|
||||
|
||||
# Get target info to get URL
|
||||
cdp_client = self.browser_session.cdp_client
|
||||
targets = await cdp_client.send.Target.getTargets()
|
||||
target_info = next((t for t in targets['targetInfos'] if t['targetId'] == target_id), None)
|
||||
if not target_info:
|
||||
self.logger.warning(f'[DownloadsWatchdog] No target info found for {target_id}')
|
||||
return False
|
||||
|
||||
page_url = target_info.get('url', '')
|
||||
|
||||
# Check cache first
|
||||
if page_url in self._pdf_viewer_cache:
|
||||
cached_result = self._pdf_viewer_cache[page_url]
|
||||
self.logger.debug(f'[DownloadsWatchdog] Using cached PDF check result for {page_url}: {cached_result}')
|
||||
return cached_result
|
||||
|
||||
try:
|
||||
# Method 1: Check URL patterns (fastest, most reliable)
|
||||
url_is_pdf = self._check_url_for_pdf(page_url)
|
||||
if url_is_pdf:
|
||||
self.logger.debug(f'[DownloadsWatchdog] PDF detected via URL pattern: {page_url}')
|
||||
self._pdf_viewer_cache[page_url] = True
|
||||
return True
|
||||
|
||||
# Method 2: Check network response headers via CDP (safer than JavaScript)
|
||||
header_is_pdf = await self._check_network_headers_for_pdf(target_id)
|
||||
if header_is_pdf:
|
||||
self.logger.debug(f'[DownloadsWatchdog] PDF detected via network headers: {page_url}')
|
||||
self._pdf_viewer_cache[page_url] = True
|
||||
return True
|
||||
|
||||
# Method 3: Check Chrome's PDF viewer specific URLs
|
||||
chrome_pdf_viewer = self._is_chrome_pdf_viewer_url(page_url)
|
||||
if chrome_pdf_viewer:
|
||||
self.logger.debug(f'[DownloadsWatchdog] Chrome PDF viewer detected: {page_url}')
|
||||
self._pdf_viewer_cache[page_url] = True
|
||||
return True
|
||||
|
||||
# Not a PDF
|
||||
self._pdf_viewer_cache[page_url] = False
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f'[DownloadsWatchdog] ❌ Error checking for PDF viewer: {e}')
|
||||
self._pdf_viewer_cache[page_url] = False
|
||||
return False
|
||||
|
||||
def _check_url_for_pdf(self, url: str) -> bool:
|
||||
"""Check if URL indicates a PDF file."""
|
||||
if not url:
|
||||
return False
|
||||
|
||||
url_lower = url.lower()
|
||||
|
||||
# Direct PDF file extensions
|
||||
if url_lower.endswith('.pdf'):
|
||||
return True
|
||||
|
||||
# PDF in path
|
||||
if '.pdf' in url_lower:
|
||||
return True
|
||||
|
||||
# PDF MIME type in URL parameters
|
||||
if any(
|
||||
param in url_lower
|
||||
for param in [
|
||||
'content-type=application/pdf',
|
||||
'content-type=application%2fpdf',
|
||||
'mimetype=application/pdf',
|
||||
'type=application/pdf',
|
||||
]
|
||||
):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
def _is_chrome_pdf_viewer_url(self, url: str) -> bool:
|
||||
"""Check if this is Chrome's internal PDF viewer URL."""
|
||||
if not url:
|
||||
return False
|
||||
|
||||
url_lower = url.lower()
|
||||
|
||||
# Chrome PDF viewer uses chrome-extension:// URLs
|
||||
if 'chrome-extension://' in url_lower and 'pdf' in url_lower:
|
||||
return True
|
||||
|
||||
# Chrome PDF viewer internal URLs
|
||||
if url_lower.startswith('chrome://') and 'pdf' in url_lower:
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
async def _check_network_headers_for_pdf(self, target_id: TargetID) -> bool:
|
||||
"""Infer PDF via navigation history/URL; headers are not available post-navigation in this context."""
|
||||
try:
|
||||
import asyncio
|
||||
|
||||
# Get CDP session
|
||||
temp_session = await self.browser_session.get_or_create_cdp_session(target_id, focus=False)
|
||||
|
||||
# Get navigation history to find the main resource
|
||||
history = await asyncio.wait_for(
|
||||
temp_session.cdp_client.send.Page.getNavigationHistory(session_id=temp_session.session_id), timeout=3.0
|
||||
)
|
||||
|
||||
current_entry = history.get('entries', [])
|
||||
if current_entry:
|
||||
current_index = history.get('currentIndex', 0)
|
||||
if 0 <= current_index < len(current_entry):
|
||||
current_url = current_entry[current_index].get('url', '')
|
||||
|
||||
# Check if the URL itself suggests PDF
|
||||
if self._check_url_for_pdf(current_url):
|
||||
return True
|
||||
|
||||
# Note: CDP doesn't easily expose response headers for completed navigations
|
||||
# For more complex cases, we'd need to set up Network.responseReceived listeners
|
||||
# before navigation, but that's overkill for most PDF detection cases
|
||||
|
||||
return False
|
||||
|
||||
except Exception as e:
|
||||
self.logger.debug(f'[DownloadsWatchdog] Network headers check failed (non-critical): {e}')
|
||||
return False
|
||||
|
||||
async def trigger_pdf_download(self, target_id: TargetID) -> str | None:
|
||||
"""Trigger download of a PDF from Chrome's PDF viewer.
|
||||
|
||||
Returns the download path if successful, None otherwise.
|
||||
"""
|
||||
self.logger.debug(f'[DownloadsWatchdog] trigger_pdf_download called for target_id={target_id}')
|
||||
|
||||
if not self.browser_session.browser_profile.downloads_path:
|
||||
self.logger.warning('[DownloadsWatchdog] ❌ No downloads path configured, cannot save PDF download')
|
||||
return None
|
||||
|
||||
downloads_path = self.browser_session.browser_profile.downloads_path
|
||||
self.logger.debug(f'[DownloadsWatchdog] Downloads path: {downloads_path}')
|
||||
|
||||
try:
|
||||
# Create a temporary CDP session for this target without switching focus
|
||||
import asyncio
|
||||
|
||||
self.logger.debug(f'[DownloadsWatchdog] Creating CDP session for PDF download from target {target_id}')
|
||||
temp_session = await self.browser_session.get_or_create_cdp_session(target_id, focus=False)
|
||||
|
||||
# Try to get the PDF URL with timeout
|
||||
result = await asyncio.wait_for(
|
||||
temp_session.cdp_client.send.Runtime.evaluate(
|
||||
params={
|
||||
'expression': """
|
||||
(() => {
|
||||
// For Chrome's PDF viewer, the actual URL is in window.location.href
|
||||
// The embed element's src is often "about:blank"
|
||||
const embedElement = document.querySelector('embed[type="application/x-google-chrome-pdf"]') ||
|
||||
document.querySelector('embed[type="application/pdf"]');
|
||||
if (embedElement) {
|
||||
// Chrome PDF viewer detected - use the page URL
|
||||
return { url: window.location.href };
|
||||
}
|
||||
// Fallback to window.location.href anyway
|
||||
return { url: window.location.href };
|
||||
})()
|
||||
""",
|
||||
'returnByValue': True,
|
||||
},
|
||||
session_id=temp_session.session_id,
|
||||
),
|
||||
timeout=5.0, # 5 second timeout to prevent hanging
|
||||
)
|
||||
pdf_info = result.get('result', {}).get('value', {})
|
||||
|
||||
pdf_url = pdf_info.get('url', '')
|
||||
if not pdf_url:
|
||||
self.logger.warning(f'[DownloadsWatchdog] ❌ Could not determine PDF URL for download {pdf_info}')
|
||||
return None
|
||||
|
||||
# Generate filename from URL
|
||||
pdf_filename = os.path.basename(pdf_url.split('?')[0]) # Remove query params
|
||||
if not pdf_filename or not pdf_filename.endswith('.pdf'):
|
||||
parsed = urlparse(pdf_url)
|
||||
pdf_filename = os.path.basename(parsed.path) or 'document.pdf'
|
||||
if not pdf_filename.endswith('.pdf'):
|
||||
pdf_filename += '.pdf'
|
||||
|
||||
self.logger.debug(f'[DownloadsWatchdog] Generated filename: {pdf_filename}')
|
||||
|
||||
# Check if already downloaded by looking in the downloads directory
|
||||
downloads_dir = str(self.browser_session.browser_profile.downloads_path)
|
||||
if os.path.exists(downloads_dir):
|
||||
existing_files = os.listdir(downloads_dir)
|
||||
if pdf_filename in existing_files:
|
||||
self.logger.debug(f'[DownloadsWatchdog] PDF already downloaded: {pdf_filename}')
|
||||
return None
|
||||
|
||||
self.logger.debug(f'[DownloadsWatchdog] Starting PDF download from: {pdf_url[:100]}...')
|
||||
|
||||
# Download using JavaScript fetch to leverage browser cache
|
||||
try:
|
||||
# Properly escape the URL to prevent JavaScript injection
|
||||
escaped_pdf_url = json.dumps(pdf_url)
|
||||
|
||||
result = await asyncio.wait_for(
|
||||
temp_session.cdp_client.send.Runtime.evaluate(
|
||||
params={
|
||||
'expression': f"""
|
||||
(async () => {{
|
||||
try {{
|
||||
// Use fetch with cache: 'force-cache' to prioritize cached version
|
||||
const response = await fetch({escaped_pdf_url}, {{
|
||||
cache: 'force-cache'
|
||||
}});
|
||||
if (!response.ok) {{
|
||||
throw new Error(`HTTP error! status: ${{response.status}}`);
|
||||
}}
|
||||
const blob = await response.blob();
|
||||
const arrayBuffer = await blob.arrayBuffer();
|
||||
const uint8Array = new Uint8Array(arrayBuffer);
|
||||
|
||||
// Check if served from cache
|
||||
const fromCache = response.headers.has('age') ||
|
||||
!response.headers.has('date');
|
||||
|
||||
return {{
|
||||
data: Array.from(uint8Array),
|
||||
fromCache: fromCache,
|
||||
responseSize: uint8Array.length,
|
||||
transferSize: response.headers.get('content-length') || 'unknown'
|
||||
}};
|
||||
}} catch (error) {{
|
||||
throw new Error(`Fetch failed: ${{error.message}}`);
|
||||
}}
|
||||
}})()
|
||||
""",
|
||||
'awaitPromise': True,
|
||||
'returnByValue': True,
|
||||
},
|
||||
session_id=temp_session.session_id,
|
||||
),
|
||||
timeout=10.0, # 10 second timeout for download operation
|
||||
)
|
||||
download_result = result.get('result', {}).get('value', {})
|
||||
|
||||
if download_result and download_result.get('data') and len(download_result['data']) > 0:
|
||||
# Ensure unique filename
|
||||
downloads_dir = str(self.browser_session.browser_profile.downloads_path)
|
||||
# Ensure downloads directory exists
|
||||
os.makedirs(downloads_dir, exist_ok=True)
|
||||
unique_filename = await self._get_unique_filename(downloads_dir, pdf_filename)
|
||||
download_path = os.path.join(downloads_dir, unique_filename)
|
||||
|
||||
# Save the PDF asynchronously
|
||||
async with await anyio.open_file(download_path, 'wb') as f:
|
||||
await f.write(bytes(download_result['data']))
|
||||
|
||||
# Verify file was written successfully
|
||||
if os.path.exists(download_path):
|
||||
actual_size = os.path.getsize(download_path)
|
||||
self.logger.debug(
|
||||
f'[DownloadsWatchdog] PDF file written successfully: {download_path} ({actual_size} bytes)'
|
||||
)
|
||||
else:
|
||||
self.logger.error(f'[DownloadsWatchdog] ❌ Failed to write PDF file to: {download_path}')
|
||||
return None
|
||||
|
||||
# Log cache information
|
||||
cache_status = 'from cache' if download_result.get('fromCache') else 'from network'
|
||||
response_size = download_result.get('responseSize', 0)
|
||||
self.logger.debug(
|
||||
f'[DownloadsWatchdog] ✅ Auto-downloaded PDF ({cache_status}, {response_size:,} bytes): {download_path}'
|
||||
)
|
||||
|
||||
# Emit file downloaded event
|
||||
self.logger.debug(f'[DownloadsWatchdog] Dispatching FileDownloadedEvent for {unique_filename}')
|
||||
self.event_bus.dispatch(
|
||||
FileDownloadedEvent(
|
||||
url=pdf_url,
|
||||
path=download_path,
|
||||
file_name=unique_filename,
|
||||
file_size=response_size,
|
||||
file_type='pdf',
|
||||
mime_type='application/pdf',
|
||||
from_cache=download_result.get('fromCache', False),
|
||||
auto_download=True,
|
||||
)
|
||||
)
|
||||
|
||||
# No need to detach - session is cached
|
||||
return download_path
|
||||
else:
|
||||
self.logger.warning(f'[DownloadsWatchdog] No data received when downloading PDF from {pdf_url}')
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f'[DownloadsWatchdog] Failed to auto-download PDF from {pdf_url}: {type(e).__name__}: {e}')
|
||||
return None
|
||||
|
||||
except TimeoutError:
|
||||
self.logger.debug('[DownloadsWatchdog] PDF download operation timed out')
|
||||
return None
|
||||
except Exception as e:
|
||||
self.logger.error(f'[DownloadsWatchdog] Error in PDF download: {type(e).__name__}: {e}')
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
async def _get_unique_filename(directory: str, filename: str) -> str:
|
||||
"""Generate a unique filename for downloads by appending (1), (2), etc., if a file already exists."""
|
||||
base, ext = os.path.splitext(filename)
|
||||
counter = 1
|
||||
new_filename = filename
|
||||
while os.path.exists(os.path.join(directory, new_filename)):
|
||||
new_filename = f'{base} ({counter}){ext}'
|
||||
counter += 1
|
||||
return new_filename
|
||||
|
||||
|
||||
# Fix Pydantic circular dependency - this will be called from session.py after BrowserSession is defined
|
||||
+456
@@ -0,0 +1,456 @@
|
||||
"""Local browser watchdog for managing browser subprocess lifecycle."""
|
||||
|
||||
import asyncio
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, ClassVar
|
||||
|
||||
import psutil
|
||||
from bubus import BaseEvent
|
||||
from pydantic import PrivateAttr
|
||||
|
||||
from browser_use.browser.events import (
|
||||
BrowserKillEvent,
|
||||
BrowserLaunchEvent,
|
||||
BrowserLaunchResult,
|
||||
BrowserStopEvent,
|
||||
)
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
from browser_use.observability import observe_debug
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
class LocalBrowserWatchdog(BaseWatchdog):
|
||||
"""Manages local browser subprocess lifecycle."""
|
||||
|
||||
# Events this watchdog listens to
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent[Any]]]] = [
|
||||
BrowserLaunchEvent,
|
||||
BrowserKillEvent,
|
||||
BrowserStopEvent,
|
||||
]
|
||||
|
||||
# Events this watchdog emits
|
||||
EMITS: ClassVar[list[type[BaseEvent[Any]]]] = []
|
||||
|
||||
# Private state for subprocess management
|
||||
_subprocess: psutil.Process | None = PrivateAttr(default=None)
|
||||
_owns_browser_resources: bool = PrivateAttr(default=True)
|
||||
_temp_dirs_to_cleanup: list[Path] = PrivateAttr(default_factory=list)
|
||||
_original_user_data_dir: str | None = PrivateAttr(default=None)
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='browser_launch_event')
|
||||
async def on_BrowserLaunchEvent(self, event: BrowserLaunchEvent) -> BrowserLaunchResult:
|
||||
"""Launch a local browser process."""
|
||||
|
||||
try:
|
||||
self.logger.debug('[LocalBrowserWatchdog] Received BrowserLaunchEvent, launching local browser...')
|
||||
|
||||
# self.logger.debug('[LocalBrowserWatchdog] Calling _launch_browser...')
|
||||
process, cdp_url = await self._launch_browser()
|
||||
self._subprocess = process
|
||||
# self.logger.debug(f'[LocalBrowserWatchdog] _launch_browser returned: process={process}, cdp_url={cdp_url}')
|
||||
|
||||
return BrowserLaunchResult(cdp_url=cdp_url)
|
||||
except Exception as e:
|
||||
self.logger.error(f'[LocalBrowserWatchdog] Exception in on_BrowserLaunchEvent: {e}', exc_info=True)
|
||||
raise
|
||||
|
||||
async def on_BrowserKillEvent(self, event: BrowserKillEvent) -> None:
|
||||
"""Kill the local browser subprocess."""
|
||||
self.logger.debug('[LocalBrowserWatchdog] Killing local browser process')
|
||||
|
||||
if self._subprocess:
|
||||
await self._cleanup_process(self._subprocess)
|
||||
self._subprocess = None
|
||||
|
||||
# Clean up temp directories if any were created
|
||||
for temp_dir in self._temp_dirs_to_cleanup:
|
||||
self._cleanup_temp_dir(temp_dir)
|
||||
self._temp_dirs_to_cleanup.clear()
|
||||
|
||||
# Restore original user_data_dir if it was modified
|
||||
if self._original_user_data_dir is not None:
|
||||
self.browser_session.browser_profile.user_data_dir = self._original_user_data_dir
|
||||
self._original_user_data_dir = None
|
||||
|
||||
self.logger.debug('[LocalBrowserWatchdog] Browser cleanup completed')
|
||||
|
||||
async def on_BrowserStopEvent(self, event: BrowserStopEvent) -> None:
|
||||
"""Listen for BrowserStopEvent and dispatch BrowserKillEvent without awaiting it."""
|
||||
if self.browser_session.is_local and self._subprocess:
|
||||
self.logger.debug('[LocalBrowserWatchdog] BrowserStopEvent received, dispatching BrowserKillEvent')
|
||||
# Dispatch BrowserKillEvent without awaiting so it gets processed after all BrowserStopEvent handlers
|
||||
self.event_bus.dispatch(BrowserKillEvent())
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='launch_browser_process')
|
||||
async def _launch_browser(self, max_retries: int = 3) -> tuple[psutil.Process, str]:
|
||||
"""Launch browser process and return (process, cdp_url).
|
||||
|
||||
Handles launch errors by falling back to temporary directories if needed.
|
||||
|
||||
Returns:
|
||||
Tuple of (psutil.Process, cdp_url)
|
||||
"""
|
||||
# Keep track of original user_data_dir to restore if needed
|
||||
profile = self.browser_session.browser_profile
|
||||
self._original_user_data_dir = str(profile.user_data_dir) if profile.user_data_dir else None
|
||||
self._temp_dirs_to_cleanup = []
|
||||
|
||||
for attempt in range(max_retries):
|
||||
try:
|
||||
# Get launch args from profile
|
||||
launch_args = profile.get_args()
|
||||
|
||||
# Add debugging port
|
||||
debug_port = self._find_free_port()
|
||||
launch_args.extend(
|
||||
[
|
||||
f'--remote-debugging-port={debug_port}',
|
||||
]
|
||||
)
|
||||
assert '--user-data-dir' in str(launch_args), (
|
||||
'User data dir must be set somewhere in launch args to a non-default path, otherwise Chrome will not let us attach via CDP'
|
||||
)
|
||||
|
||||
# Get browser executable
|
||||
# Priority: custom executable > fallback paths > playwright subprocess
|
||||
if profile.executable_path:
|
||||
browser_path = profile.executable_path
|
||||
self.logger.debug(f'[LocalBrowserWatchdog] 📦 Using custom local browser executable_path= {browser_path}')
|
||||
else:
|
||||
# self.logger.debug('[LocalBrowserWatchdog] 🔍 Looking for local browser binary path...')
|
||||
# Try fallback paths first (system browsers preferred)
|
||||
browser_path = self._find_installed_browser_path()
|
||||
if not browser_path:
|
||||
self.logger.error(
|
||||
'[LocalBrowserWatchdog] ⚠️ No local browser binary found, installing browser using playwright subprocess...'
|
||||
)
|
||||
browser_path = await self._install_browser_with_playwright()
|
||||
|
||||
self.logger.debug(f'[LocalBrowserWatchdog] 📦 Found local browser installed at executable_path= {browser_path}')
|
||||
if not browser_path:
|
||||
raise RuntimeError('No local Chrome/Chromium install found, and failed to install with playwright')
|
||||
|
||||
# Launch browser subprocess directly
|
||||
self.logger.debug(f'[LocalBrowserWatchdog] 🚀 Launching browser subprocess with {len(launch_args)} args...')
|
||||
subprocess = await asyncio.create_subprocess_exec(
|
||||
browser_path,
|
||||
*launch_args,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
self.logger.debug(
|
||||
f'[LocalBrowserWatchdog] 🎭 Browser running with browser_pid= {subprocess.pid} 🔗 listening on CDP port :{debug_port}'
|
||||
)
|
||||
|
||||
# Convert to psutil.Process
|
||||
process = psutil.Process(subprocess.pid)
|
||||
|
||||
# Wait for CDP to be ready and get the URL
|
||||
cdp_url = await self._wait_for_cdp_url(debug_port)
|
||||
|
||||
# Success! Clean up any temp dirs we created but didn't use
|
||||
for tmp_dir in self._temp_dirs_to_cleanup:
|
||||
try:
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return process, cdp_url
|
||||
|
||||
except Exception as e:
|
||||
error_str = str(e).lower()
|
||||
|
||||
# Check if this is a user_data_dir related error
|
||||
if any(err in error_str for err in ['singletonlock', 'user data directory', 'cannot create', 'already in use']):
|
||||
self.logger.warning(f'Browser launch failed (attempt {attempt + 1}/{max_retries}): {e}')
|
||||
|
||||
if attempt < max_retries - 1:
|
||||
# Create a temporary directory for next attempt
|
||||
tmp_dir = Path(tempfile.mkdtemp(prefix='browseruse-tmp-'))
|
||||
self._temp_dirs_to_cleanup.append(tmp_dir)
|
||||
|
||||
# Update profile to use temp directory
|
||||
profile.user_data_dir = str(tmp_dir)
|
||||
self.logger.debug(f'Retrying with temporary user_data_dir: {tmp_dir}')
|
||||
|
||||
# Small delay before retry
|
||||
await asyncio.sleep(0.5)
|
||||
continue
|
||||
|
||||
# Not a recoverable error or last attempt failed
|
||||
# Restore original user_data_dir before raising
|
||||
if self._original_user_data_dir is not None:
|
||||
profile.user_data_dir = self._original_user_data_dir
|
||||
|
||||
# Clean up any temp dirs we created
|
||||
for tmp_dir in self._temp_dirs_to_cleanup:
|
||||
try:
|
||||
shutil.rmtree(tmp_dir, ignore_errors=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
raise
|
||||
|
||||
# Should not reach here, but just in case
|
||||
if self._original_user_data_dir is not None:
|
||||
profile.user_data_dir = self._original_user_data_dir
|
||||
raise RuntimeError(f'Failed to launch browser after {max_retries} attempts')
|
||||
|
||||
@staticmethod
|
||||
def _find_installed_browser_path() -> str | None:
|
||||
"""Try to find browser executable from common fallback locations.
|
||||
|
||||
Prioritizes:
|
||||
1. System Chrome Stable
|
||||
1. Playwright chromium
|
||||
2. Other system native browsers (Chromium -> Chrome Canary/Dev -> Brave)
|
||||
3. Playwright headless-shell fallback
|
||||
|
||||
Returns:
|
||||
Path to browser executable or None if not found
|
||||
"""
|
||||
import glob
|
||||
import platform
|
||||
from pathlib import Path
|
||||
|
||||
system = platform.system()
|
||||
patterns = []
|
||||
|
||||
# Get playwright browsers path from environment variable if set
|
||||
playwright_path = os.environ.get('PLAYWRIGHT_BROWSERS_PATH')
|
||||
|
||||
if system == 'Darwin': # macOS
|
||||
if not playwright_path:
|
||||
playwright_path = '~/Library/Caches/ms-playwright'
|
||||
patterns = [
|
||||
'/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
|
||||
f'{playwright_path}/chromium-*/chrome-mac/Chromium.app/Contents/MacOS/Chromium',
|
||||
'/Applications/Chromium.app/Contents/MacOS/Chromium',
|
||||
'/Applications/Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary',
|
||||
'/Applications/Brave Browser.app/Contents/MacOS/Brave Browser',
|
||||
f'{playwright_path}/chromium_headless_shell-*/chrome-mac/Chromium.app/Contents/MacOS/Chromium',
|
||||
]
|
||||
elif system == 'Linux':
|
||||
if not playwright_path:
|
||||
playwright_path = '~/.cache/ms-playwright'
|
||||
patterns = [
|
||||
'/usr/bin/google-chrome-stable',
|
||||
'/usr/bin/google-chrome',
|
||||
'/usr/local/bin/google-chrome',
|
||||
f'{playwright_path}/chromium-*/chrome-linux/chrome',
|
||||
'/usr/bin/chromium',
|
||||
'/usr/bin/chromium-browser',
|
||||
'/usr/local/bin/chromium',
|
||||
'/snap/bin/chromium',
|
||||
'/usr/bin/google-chrome-beta',
|
||||
'/usr/bin/google-chrome-dev',
|
||||
'/usr/bin/brave-browser',
|
||||
f'{playwright_path}/chromium_headless_shell-*/chrome-linux/chrome',
|
||||
]
|
||||
elif system == 'Windows':
|
||||
if not playwright_path:
|
||||
playwright_path = r'%LOCALAPPDATA%\ms-playwright'
|
||||
patterns = [
|
||||
r'C:\Program Files\Google\Chrome\Application\chrome.exe',
|
||||
r'C:\Program Files (x86)\Google\Chrome\Application\chrome.exe',
|
||||
r'%LOCALAPPDATA%\Google\Chrome\Application\chrome.exe',
|
||||
r'%PROGRAMFILES%\Google\Chrome\Application\chrome.exe',
|
||||
r'%PROGRAMFILES(X86)%\Google\Chrome\Application\chrome.exe',
|
||||
f'{playwright_path}\\chromium-*\\chrome-win\\chrome.exe',
|
||||
r'C:\Program Files\Chromium\Application\chrome.exe',
|
||||
r'C:\Program Files (x86)\Chromium\Application\chrome.exe',
|
||||
r'%LOCALAPPDATA%\Chromium\Application\chrome.exe',
|
||||
r'C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe',
|
||||
r'C:\Program Files (x86)\BraveSoftware\Brave-Browser\Application\brave.exe',
|
||||
r'C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe',
|
||||
r'C:\Program Files\Microsoft\Edge\Application\msedge.exe',
|
||||
r'%LOCALAPPDATA%\Microsoft\Edge\Application\msedge.exe',
|
||||
f'{playwright_path}\\chromium_headless_shell-*\\chrome-win\\chrome.exe',
|
||||
]
|
||||
|
||||
for pattern in patterns:
|
||||
# Expand user home directory
|
||||
expanded_pattern = Path(pattern).expanduser()
|
||||
|
||||
# Handle Windows environment variables
|
||||
if system == 'Windows':
|
||||
pattern_str = str(expanded_pattern)
|
||||
for env_var in ['%LOCALAPPDATA%', '%PROGRAMFILES%', '%PROGRAMFILES(X86)%']:
|
||||
if env_var in pattern_str:
|
||||
env_key = env_var.strip('%').replace('(X86)', ' (x86)')
|
||||
env_value = os.environ.get(env_key, '')
|
||||
if env_value:
|
||||
pattern_str = pattern_str.replace(env_var, env_value)
|
||||
expanded_pattern = Path(pattern_str)
|
||||
|
||||
# Convert to string for glob
|
||||
pattern_str = str(expanded_pattern)
|
||||
|
||||
# Check if pattern contains wildcards
|
||||
if '*' in pattern_str:
|
||||
# Use glob to expand the pattern
|
||||
matches = glob.glob(pattern_str)
|
||||
if matches:
|
||||
# Sort matches and take the last one (alphanumerically highest version)
|
||||
matches.sort()
|
||||
browser_path = matches[-1]
|
||||
if Path(browser_path).exists() and Path(browser_path).is_file():
|
||||
return browser_path
|
||||
else:
|
||||
# Direct path check
|
||||
if expanded_pattern.exists() and expanded_pattern.is_file():
|
||||
return str(expanded_pattern)
|
||||
|
||||
return None
|
||||
|
||||
async def _install_browser_with_playwright(self) -> str:
|
||||
"""Get browser executable path from playwright in a subprocess to avoid thread issues."""
|
||||
|
||||
# Run in subprocess with timeout
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
'uvx',
|
||||
'playwright',
|
||||
'install',
|
||||
'chrome',
|
||||
'--with-deps',
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
|
||||
try:
|
||||
stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=60.0)
|
||||
self.logger.debug(f'[LocalBrowserWatchdog] 📦 Playwright install output: {stdout}')
|
||||
browser_path = self._find_installed_browser_path()
|
||||
if browser_path:
|
||||
return browser_path
|
||||
self.logger.error(f'[LocalBrowserWatchdog] ❌ Playwright local browser installation error: \n{stdout}\n{stderr}')
|
||||
raise RuntimeError('No local browser path found after: uvx playwright install chrome --with-deps')
|
||||
except TimeoutError:
|
||||
# Kill the subprocess if it times out
|
||||
process.kill()
|
||||
await process.wait()
|
||||
raise RuntimeError('Timeout getting browser path from playwright')
|
||||
except Exception as e:
|
||||
# Make sure subprocess is terminated
|
||||
if process.returncode is None:
|
||||
process.kill()
|
||||
await process.wait()
|
||||
raise RuntimeError(f'Error getting browser path: {e}')
|
||||
|
||||
@staticmethod
|
||||
def _find_free_port() -> int:
|
||||
"""Find a free port for the debugging interface."""
|
||||
import socket
|
||||
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
||||
s.bind(('127.0.0.1', 0))
|
||||
s.listen(1)
|
||||
port = s.getsockname()[1]
|
||||
return port
|
||||
|
||||
@staticmethod
|
||||
async def _wait_for_cdp_url(port: int, timeout: float = 30) -> str:
|
||||
"""Wait for the browser to start and return the CDP URL."""
|
||||
import aiohttp
|
||||
|
||||
start_time = asyncio.get_event_loop().time()
|
||||
|
||||
while asyncio.get_event_loop().time() - start_time < timeout:
|
||||
try:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(f'http://localhost:{port}/json/version') as resp:
|
||||
if resp.status == 200:
|
||||
# Chrome is ready
|
||||
return f'http://localhost:{port}/'
|
||||
else:
|
||||
# Chrome is starting up and returning 502/500 errors
|
||||
await asyncio.sleep(0.1)
|
||||
except Exception:
|
||||
# Connection error - Chrome might not be ready yet
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
raise TimeoutError(f'Browser did not start within {timeout} seconds')
|
||||
|
||||
@staticmethod
|
||||
async def _cleanup_process(process: psutil.Process) -> None:
|
||||
"""Clean up browser process.
|
||||
|
||||
Args:
|
||||
process: psutil.Process to terminate
|
||||
"""
|
||||
if not process:
|
||||
return
|
||||
|
||||
try:
|
||||
# Try graceful shutdown first
|
||||
process.terminate()
|
||||
|
||||
# Use async wait instead of blocking wait
|
||||
for _ in range(50): # Wait up to 5 seconds (50 * 0.1)
|
||||
if not process.is_running():
|
||||
return
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
# If still running after 5 seconds, force kill
|
||||
if process.is_running():
|
||||
process.kill()
|
||||
# Give it a moment to die
|
||||
await asyncio.sleep(0.1)
|
||||
|
||||
except psutil.NoSuchProcess:
|
||||
# Process already gone
|
||||
pass
|
||||
except Exception:
|
||||
# Ignore any other errors during cleanup
|
||||
pass
|
||||
|
||||
def _cleanup_temp_dir(self, temp_dir: Path | str) -> None:
|
||||
"""Clean up temporary directory.
|
||||
|
||||
Args:
|
||||
temp_dir: Path to temporary directory to remove
|
||||
"""
|
||||
if not temp_dir:
|
||||
return
|
||||
|
||||
try:
|
||||
temp_path = Path(temp_dir)
|
||||
# Only remove if it's actually a temp directory we created
|
||||
if 'browseruse-tmp-' in str(temp_path):
|
||||
shutil.rmtree(temp_path, ignore_errors=True)
|
||||
except Exception as e:
|
||||
self.logger.debug(f'Failed to cleanup temp dir {temp_dir}: {e}')
|
||||
|
||||
@property
|
||||
def browser_pid(self) -> int | None:
|
||||
"""Get the browser process ID."""
|
||||
if self._subprocess:
|
||||
return self._subprocess.pid
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
async def get_browser_pid_via_cdp(browser) -> int | None:
|
||||
"""Get the browser process ID via CDP SystemInfo.getProcessInfo.
|
||||
|
||||
Args:
|
||||
browser: Playwright Browser instance
|
||||
|
||||
Returns:
|
||||
Process ID or None if failed
|
||||
"""
|
||||
try:
|
||||
cdp_session = await browser.new_browser_cdp_session()
|
||||
result = await cdp_session.send('SystemInfo.getProcessInfo')
|
||||
process_info = result.get('processInfo', {})
|
||||
pid = process_info.get('id')
|
||||
await cdp_session.detach()
|
||||
return pid
|
||||
except Exception:
|
||||
# If we can't get PID via CDP, it's not critical
|
||||
return None
|
||||
+43
@@ -0,0 +1,43 @@
|
||||
"""Permissions watchdog for granting browser permissions on connection."""
|
||||
|
||||
from typing import TYPE_CHECKING, ClassVar
|
||||
|
||||
from bubus import BaseEvent
|
||||
|
||||
from browser_use.browser.events import BrowserConnectedEvent
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
class PermissionsWatchdog(BaseWatchdog):
|
||||
"""Grants browser permissions when browser connects."""
|
||||
|
||||
# Event contracts
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [
|
||||
BrowserConnectedEvent,
|
||||
]
|
||||
EMITS: ClassVar[list[type[BaseEvent]]] = []
|
||||
|
||||
async def on_BrowserConnectedEvent(self, event: BrowserConnectedEvent) -> None:
|
||||
"""Grant permissions when browser connects."""
|
||||
permissions = self.browser_session.browser_profile.permissions
|
||||
|
||||
if not permissions:
|
||||
self.logger.debug('No permissions to grant')
|
||||
return
|
||||
|
||||
self.logger.debug(f'🔓 Granting browser permissions: {permissions}')
|
||||
|
||||
try:
|
||||
# Grant permissions using CDP Browser.grantPermissions
|
||||
# origin=None means grant to all origins
|
||||
# Browser domain commands don't use session_id
|
||||
await self.browser_session.cdp_client.send.Browser.grantPermissions(
|
||||
params={'permissions': permissions} # type: ignore
|
||||
)
|
||||
self.logger.debug(f'✅ Successfully granted permissions: {permissions}')
|
||||
except Exception as e:
|
||||
self.logger.error(f'❌ Failed to grant permissions: {str(e)}')
|
||||
# Don't raise - permissions are not critical to browser operation
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
"""Watchdog for handling JavaScript dialogs (alert, confirm, prompt) automatically."""
|
||||
|
||||
import asyncio
|
||||
from typing import ClassVar
|
||||
|
||||
from bubus import BaseEvent
|
||||
from pydantic import PrivateAttr
|
||||
|
||||
from browser_use.browser.events import TabCreatedEvent
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
|
||||
class PopupsWatchdog(BaseWatchdog):
|
||||
"""Handles JavaScript dialogs (alert, confirm, prompt) by automatically accepting them immediately."""
|
||||
|
||||
# Events this watchdog listens to and emits
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [TabCreatedEvent]
|
||||
EMITS: ClassVar[list[type[BaseEvent]]] = []
|
||||
|
||||
# Track which targets have dialog handlers registered
|
||||
_dialog_listeners_registered: set[str] = PrivateAttr(default_factory=set)
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
super().__init__(**kwargs)
|
||||
self.logger.debug(f'🚀 PopupsWatchdog initialized with browser_session={self.browser_session}, ID={id(self)}')
|
||||
|
||||
async def on_TabCreatedEvent(self, event: TabCreatedEvent) -> None:
|
||||
"""Set up JavaScript dialog handling when a new tab is created."""
|
||||
target_id = event.target_id
|
||||
self.logger.debug(f'🎯 PopupsWatchdog received TabCreatedEvent for target {target_id}')
|
||||
|
||||
# Skip if we've already registered for this target
|
||||
if target_id in self._dialog_listeners_registered:
|
||||
self.logger.debug(f'Already registered dialog handlers for target {target_id}')
|
||||
return
|
||||
|
||||
self.logger.debug(f'📌 Starting dialog handler setup for target {target_id}')
|
||||
try:
|
||||
# Get all CDP sessions for this target and any child frames
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session(
|
||||
target_id, focus=False
|
||||
) # don't auto-focus new tabs! sometimes we need to open tabs in background
|
||||
|
||||
# Also register for the root CDP client to catch dialogs from any frame
|
||||
if self.browser_session._cdp_client_root:
|
||||
self.logger.debug('📌 Also registering handler on root CDP client')
|
||||
|
||||
# Set up async handler for JavaScript dialogs - accept immediately without event dispatch
|
||||
async def handle_dialog(event_data, session_id: str | None = None):
|
||||
"""Handle JavaScript dialog events - accept immediately."""
|
||||
try:
|
||||
dialog_type = event_data.get('type', 'alert')
|
||||
message = event_data.get('message', '')
|
||||
|
||||
self.logger.info(f"🔔 JavaScript {dialog_type} dialog: '{message[:100]}' - attempting to accept...")
|
||||
|
||||
self.logger.debug('Trying all approaches to accept dialog...')
|
||||
|
||||
# Approach 1: Use the session that detected the dialog
|
||||
if self.browser_session._cdp_client_root and session_id:
|
||||
try:
|
||||
self.logger.debug(f'🔄 Approach 1: Using session {session_id}')
|
||||
await asyncio.wait_for(
|
||||
self.browser_session._cdp_client_root.send.Page.handleJavaScriptDialog(
|
||||
params={'accept': True},
|
||||
session_id=session_id,
|
||||
),
|
||||
timeout=0.25,
|
||||
)
|
||||
except (TimeoutError, Exception) as e:
|
||||
pass
|
||||
|
||||
# Approach 2: Try with current agent focus session
|
||||
if self.browser_session._cdp_client_root and self.browser_session.agent_focus:
|
||||
try:
|
||||
self.logger.debug(
|
||||
f'🔄 Approach 2: Using agent focus session {self.browser_session.agent_focus.session_id}'
|
||||
)
|
||||
await asyncio.wait_for(
|
||||
self.browser_session._cdp_client_root.send.Page.handleJavaScriptDialog(
|
||||
params={'accept': True},
|
||||
session_id=self.browser_session.agent_focus.session_id,
|
||||
),
|
||||
timeout=0.25,
|
||||
)
|
||||
except (TimeoutError, Exception) as e:
|
||||
pass
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'❌ Critical error in dialog handler: {type(e).__name__}: {e}')
|
||||
|
||||
# Register handler on the specific session
|
||||
cdp_session.cdp_client.register.Page.javascriptDialogOpening(handle_dialog) # type: ignore[arg-type]
|
||||
self.logger.debug(
|
||||
f'Successfully registered Page.javascriptDialogOpening handler for session {cdp_session.session_id}'
|
||||
)
|
||||
|
||||
# Also register on root CDP client to catch dialogs from any frame
|
||||
if hasattr(self.browser_session._cdp_client_root, 'register'):
|
||||
try:
|
||||
self.browser_session._cdp_client_root.register.Page.javascriptDialogOpening(handle_dialog) # type: ignore[arg-type]
|
||||
self.logger.debug('Successfully registered dialog handler on root CDP client for all frames')
|
||||
except Exception as root_error:
|
||||
self.logger.warning(f'Failed to register on root CDP client: {root_error}')
|
||||
|
||||
# Mark this target as having dialog handling set up
|
||||
self._dialog_listeners_registered.add(target_id)
|
||||
|
||||
self.logger.debug(f'Set up JavaScript dialog handling for tab {target_id}')
|
||||
|
||||
except Exception as e:
|
||||
self.logger.warning(f'Failed to set up popup handling for tab {target_id}: {e}')
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
"""Recording Watchdog for Browser Use Sessions."""
|
||||
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from typing import ClassVar
|
||||
|
||||
from bubus import BaseEvent
|
||||
from cdp_use.cdp.page.events import ScreencastFrameEvent
|
||||
from uuid_extensions import uuid7str
|
||||
|
||||
from browser_use.browser.events import BrowserConnectedEvent, BrowserStopEvent
|
||||
from browser_use.browser.profile import ViewportSize
|
||||
from browser_use.browser.video_recorder import VideoRecorderService
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
|
||||
class RecordingWatchdog(BaseWatchdog):
|
||||
"""
|
||||
Manages video recording of a browser session using CDP screencasting.
|
||||
"""
|
||||
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [BrowserConnectedEvent, BrowserStopEvent]
|
||||
EMITS: ClassVar[list[type[BaseEvent]]] = []
|
||||
|
||||
_recorder: VideoRecorderService | None = None
|
||||
|
||||
async def on_BrowserConnectedEvent(self, event: BrowserConnectedEvent) -> None:
|
||||
"""
|
||||
Starts video recording if it is configured in the browser profile.
|
||||
"""
|
||||
profile = self.browser_session.browser_profile
|
||||
if not profile.record_video_dir:
|
||||
return
|
||||
|
||||
# Dynamically determine video size
|
||||
size = profile.record_video_size
|
||||
if not size:
|
||||
self.logger.debug('record_video_size not specified, detecting viewport size...')
|
||||
size = await self._get_current_viewport_size()
|
||||
|
||||
if not size:
|
||||
self.logger.warning('Cannot start video recording: viewport size could not be determined.')
|
||||
return
|
||||
|
||||
video_format = getattr(profile, 'record_video_format', 'mp4').strip('.')
|
||||
output_path = Path(profile.record_video_dir) / f'{uuid7str()}.{video_format}'
|
||||
|
||||
self.logger.debug(f'Initializing video recorder for format: {video_format}')
|
||||
self._recorder = VideoRecorderService(output_path=output_path, size=size, framerate=profile.record_video_framerate)
|
||||
self._recorder.start()
|
||||
|
||||
if not self._recorder._is_active:
|
||||
self._recorder = None
|
||||
return
|
||||
|
||||
self.browser_session.cdp_client.register.Page.screencastFrame(self.on_screencastFrame)
|
||||
|
||||
try:
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session()
|
||||
await cdp_session.cdp_client.send.Page.startScreencast(
|
||||
params={
|
||||
'format': 'png',
|
||||
'quality': 90,
|
||||
'maxWidth': size['width'],
|
||||
'maxHeight': size['height'],
|
||||
'everyNthFrame': 1,
|
||||
},
|
||||
session_id=cdp_session.session_id,
|
||||
)
|
||||
self.logger.info(f'📹 Started video recording to {output_path}')
|
||||
except Exception as e:
|
||||
self.logger.error(f'Failed to start screencast via CDP: {e}')
|
||||
if self._recorder:
|
||||
self._recorder.stop_and_save()
|
||||
self._recorder = None
|
||||
|
||||
async def _get_current_viewport_size(self) -> ViewportSize | None:
|
||||
"""Gets the current viewport size directly from the browser via CDP."""
|
||||
try:
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session()
|
||||
metrics = await cdp_session.cdp_client.send.Page.getLayoutMetrics(session_id=cdp_session.session_id)
|
||||
|
||||
# Use cssVisualViewport for the most accurate representation of the visible area
|
||||
viewport = metrics.get('cssVisualViewport', {})
|
||||
width = viewport.get('clientWidth')
|
||||
height = viewport.get('clientHeight')
|
||||
|
||||
if width and height:
|
||||
self.logger.debug(f'Detected viewport size: {width}x{height}')
|
||||
return ViewportSize(width=int(width), height=int(height))
|
||||
except Exception as e:
|
||||
self.logger.warning(f'Failed to get viewport size from browser: {e}')
|
||||
|
||||
return None
|
||||
|
||||
def on_screencastFrame(self, event: ScreencastFrameEvent, session_id: str | None) -> None:
|
||||
"""
|
||||
Synchronous handler for incoming screencast frames.
|
||||
"""
|
||||
if not self._recorder:
|
||||
return
|
||||
self._recorder.add_frame(event['data'])
|
||||
asyncio.create_task(self._ack_screencast_frame(event, session_id))
|
||||
|
||||
async def _ack_screencast_frame(self, event: ScreencastFrameEvent, session_id: str | None) -> None:
|
||||
"""
|
||||
Asynchronously acknowledges a screencast frame.
|
||||
"""
|
||||
try:
|
||||
await self.browser_session.cdp_client.send.Page.screencastFrameAck(
|
||||
params={'sessionId': event['sessionId']}, session_id=session_id
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.debug(f'Failed to acknowledge screencast frame: {e}')
|
||||
|
||||
async def on_BrowserStopEvent(self, event: BrowserStopEvent) -> None:
|
||||
"""
|
||||
Stops the video recording and finalizes the video file.
|
||||
"""
|
||||
if self._recorder:
|
||||
recorder = self._recorder
|
||||
self._recorder = None
|
||||
|
||||
self.logger.debug('Stopping video recording and saving file...')
|
||||
loop = asyncio.get_event_loop()
|
||||
await loop.run_in_executor(None, recorder.stop_and_save)
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
"""Screenshot watchdog for handling screenshot requests using CDP."""
|
||||
|
||||
from typing import TYPE_CHECKING, Any, ClassVar
|
||||
|
||||
from bubus import BaseEvent
|
||||
from cdp_use.cdp.page import CaptureScreenshotParameters
|
||||
|
||||
from browser_use.browser.events import ScreenshotEvent
|
||||
from browser_use.browser.views import BrowserError
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
from browser_use.observability import observe_debug
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
|
||||
class ScreenshotWatchdog(BaseWatchdog):
|
||||
"""Handles screenshot requests using CDP."""
|
||||
|
||||
# Events this watchdog listens to
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent[Any]]]] = [ScreenshotEvent]
|
||||
|
||||
# Events this watchdog emits
|
||||
EMITS: ClassVar[list[type[BaseEvent[Any]]]] = []
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='screenshot_event_handler')
|
||||
async def on_ScreenshotEvent(self, event: ScreenshotEvent) -> str:
|
||||
"""Handle screenshot request using CDP.
|
||||
|
||||
Args:
|
||||
event: ScreenshotEvent with optional full_page and clip parameters
|
||||
|
||||
Returns:
|
||||
Dict with 'screenshot' key containing base64-encoded screenshot or None
|
||||
"""
|
||||
self.logger.debug('[ScreenshotWatchdog] Handler START - on_ScreenshotEvent called')
|
||||
try:
|
||||
# Get CDP client and session for current target
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session()
|
||||
|
||||
# Prepare screenshot parameters
|
||||
params = CaptureScreenshotParameters(format='png', captureBeyondViewport=False)
|
||||
|
||||
# Take screenshot using CDP
|
||||
self.logger.debug(f'[ScreenshotWatchdog] Taking screenshot with params: {params}')
|
||||
result = await cdp_session.cdp_client.send.Page.captureScreenshot(params=params, session_id=cdp_session.session_id)
|
||||
|
||||
# Return base64-encoded screenshot data
|
||||
if result and 'data' in result:
|
||||
self.logger.debug('[ScreenshotWatchdog] Screenshot captured successfully')
|
||||
return result['data']
|
||||
|
||||
raise BrowserError('[ScreenshotWatchdog] Screenshot result missing data')
|
||||
except Exception as e:
|
||||
self.logger.error(f'[ScreenshotWatchdog] Screenshot failed: {e}')
|
||||
raise
|
||||
finally:
|
||||
# Try to remove highlights even on failure
|
||||
try:
|
||||
await self.browser_session.remove_highlights()
|
||||
except Exception:
|
||||
pass
|
||||
+219
@@ -0,0 +1,219 @@
|
||||
"""Security watchdog for enforcing URL access policies."""
|
||||
|
||||
from typing import TYPE_CHECKING, ClassVar
|
||||
|
||||
from bubus import BaseEvent
|
||||
|
||||
from browser_use.browser.events import (
|
||||
BrowserErrorEvent,
|
||||
NavigateToUrlEvent,
|
||||
NavigationCompleteEvent,
|
||||
TabCreatedEvent,
|
||||
)
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
if TYPE_CHECKING:
|
||||
pass
|
||||
|
||||
# Track if we've shown the glob warning
|
||||
_GLOB_WARNING_SHOWN = False
|
||||
|
||||
|
||||
class SecurityWatchdog(BaseWatchdog):
|
||||
"""Monitors and enforces security policies for URL access."""
|
||||
|
||||
# Event contracts
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [
|
||||
NavigateToUrlEvent,
|
||||
NavigationCompleteEvent,
|
||||
TabCreatedEvent,
|
||||
]
|
||||
EMITS: ClassVar[list[type[BaseEvent]]] = [
|
||||
BrowserErrorEvent,
|
||||
]
|
||||
|
||||
async def on_NavigateToUrlEvent(self, event: NavigateToUrlEvent) -> None:
|
||||
"""Check if navigation URL is allowed before navigation starts."""
|
||||
# Security check BEFORE navigation
|
||||
if not self._is_url_allowed(event.url):
|
||||
self.logger.warning(f'⛔️ Blocking navigation to disallowed URL: {event.url}')
|
||||
self.event_bus.dispatch(
|
||||
BrowserErrorEvent(
|
||||
error_type='NavigationBlocked',
|
||||
message=f'Navigation blocked to disallowed URL: {event.url}',
|
||||
details={'url': event.url, 'reason': 'not_in_allowed_domains'},
|
||||
)
|
||||
)
|
||||
# Stop event propagation by raising exception
|
||||
raise ValueError(f'Navigation to {event.url} blocked by security policy')
|
||||
|
||||
async def on_NavigationCompleteEvent(self, event: NavigationCompleteEvent) -> None:
|
||||
"""Check if navigated URL is allowed and close tab if not."""
|
||||
# Check if the navigated URL is allowed (in case of redirects)
|
||||
if not self._is_url_allowed(event.url):
|
||||
self.logger.warning(f'⛔️ Navigation to non-allowed URL detected: {event.url}')
|
||||
|
||||
# Dispatch browser error
|
||||
self.event_bus.dispatch(
|
||||
BrowserErrorEvent(
|
||||
error_type='NavigationBlocked',
|
||||
message=f'Navigation to non-allowed URL: {event.url}',
|
||||
details={'url': event.url, 'target_id': event.target_id},
|
||||
)
|
||||
)
|
||||
|
||||
# Close the target that navigated to the disallowed URL
|
||||
try:
|
||||
await self.browser_session._cdp_close_page(event.target_id)
|
||||
self.logger.info(f'⛔️ Closed target with non-allowed URL: {event.url}')
|
||||
except Exception as e:
|
||||
self.logger.error(f'⛔️ Failed to close target with non-allowed URL: {type(e).__name__} {e}')
|
||||
|
||||
async def on_TabCreatedEvent(self, event: TabCreatedEvent) -> None:
|
||||
"""Check if new tab URL is allowed."""
|
||||
if not self._is_url_allowed(event.url):
|
||||
self.logger.warning(f'⛔️ New tab created with disallowed URL: {event.url}')
|
||||
|
||||
# Dispatch error and try to close the tab
|
||||
self.event_bus.dispatch(
|
||||
BrowserErrorEvent(
|
||||
error_type='TabCreationBlocked',
|
||||
message=f'Tab created with non-allowed URL: {event.url}',
|
||||
details={'url': event.url, 'target_id': event.target_id},
|
||||
)
|
||||
)
|
||||
|
||||
# Try to close the offending tab
|
||||
try:
|
||||
await self.browser_session._cdp_close_page(event.target_id)
|
||||
self.logger.info(f'⛔️ Closed new tab with non-allowed URL: {event.url}')
|
||||
except Exception as e:
|
||||
self.logger.error(f'⛔️ Failed to close new tab with non-allowed URL: {type(e).__name__} {e}')
|
||||
|
||||
def _is_root_domain(self, domain: str) -> bool:
|
||||
"""Check if a domain is a root domain (no subdomain present).
|
||||
|
||||
Simple heuristic: only add www for domains with exactly 1 dot (domain.tld).
|
||||
For complex cases like country TLDs or subdomains, users should configure explicitly.
|
||||
|
||||
Args:
|
||||
domain: The domain to check
|
||||
|
||||
Returns:
|
||||
True if it's a simple root domain, False otherwise
|
||||
"""
|
||||
# Skip if it contains wildcards or protocol
|
||||
if '*' in domain or '://' in domain:
|
||||
return False
|
||||
|
||||
return domain.count('.') == 1
|
||||
|
||||
def _log_glob_warning(self) -> None:
|
||||
"""Log a warning about glob patterns in allowed_domains."""
|
||||
global _GLOB_WARNING_SHOWN
|
||||
if not _GLOB_WARNING_SHOWN:
|
||||
_GLOB_WARNING_SHOWN = True
|
||||
self.logger.warning(
|
||||
'⚠️ Using glob patterns in allowed_domains. '
|
||||
'Note: Patterns like "*.example.com" will match both subdomains AND the main domain.'
|
||||
)
|
||||
|
||||
def _is_url_allowed(self, url: str) -> bool:
|
||||
"""Check if a URL is allowed based on the allowed_domains configuration.
|
||||
|
||||
Args:
|
||||
url: The URL to check
|
||||
|
||||
Returns:
|
||||
True if the URL is allowed, False otherwise
|
||||
"""
|
||||
|
||||
# If no allowed_domains specified, allow all URLs
|
||||
if (
|
||||
not self.browser_session.browser_profile.allowed_domains
|
||||
and not self.browser_session.browser_profile.prohibited_domains
|
||||
):
|
||||
return True
|
||||
|
||||
# Always allow internal browser targets
|
||||
if url in ['about:blank', 'chrome://new-tab-page/', 'chrome://new-tab-page', 'chrome://newtab/']:
|
||||
return True
|
||||
|
||||
# Parse the URL to extract components
|
||||
from urllib.parse import urlparse
|
||||
|
||||
try:
|
||||
parsed = urlparse(url)
|
||||
except Exception:
|
||||
# Invalid URL
|
||||
return False
|
||||
|
||||
# Get the actual host (domain)
|
||||
host = parsed.hostname
|
||||
if not host:
|
||||
return False
|
||||
|
||||
# Check each allowed domain pattern
|
||||
if self.browser_session.browser_profile.allowed_domains:
|
||||
for pattern in self.browser_session.browser_profile.allowed_domains:
|
||||
if self._is_url_match(url, host, parsed.scheme, pattern):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
# Check each prohibited domain pattern
|
||||
if self.browser_session.browser_profile.prohibited_domains:
|
||||
for pattern in self.browser_session.browser_profile.prohibited_domains:
|
||||
if self._is_url_match(url, host, parsed.scheme, pattern):
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
return True
|
||||
|
||||
def _is_url_match(self, url: str, host: str, scheme: str, pattern: str) -> bool:
|
||||
"""Check if a URL matches a pattern."""
|
||||
|
||||
# Full URL for matching (scheme + host)
|
||||
full_url_pattern = f'{scheme}://{host}'
|
||||
|
||||
# Handle glob patterns
|
||||
if '*' in pattern:
|
||||
self._log_glob_warning()
|
||||
import fnmatch
|
||||
|
||||
# Check if pattern matches the host
|
||||
if pattern.startswith('*.'):
|
||||
# Pattern like *.example.com should match subdomains and main domain
|
||||
domain_part = pattern[2:] # Remove *.
|
||||
if host == domain_part or host.endswith('.' + domain_part):
|
||||
# Only match http/https URLs for domain-only patterns
|
||||
if scheme in ['http', 'https']:
|
||||
return True
|
||||
elif pattern.endswith('/*'):
|
||||
# Pattern like brave://* should match any brave:// URL
|
||||
prefix = pattern[:-1] # Remove the * at the end
|
||||
if url.startswith(prefix):
|
||||
return True
|
||||
else:
|
||||
# Use fnmatch for other glob patterns
|
||||
if fnmatch.fnmatch(
|
||||
full_url_pattern if '://' in pattern else host,
|
||||
pattern,
|
||||
):
|
||||
return True
|
||||
else:
|
||||
# Exact match
|
||||
if '://' in pattern:
|
||||
# Full URL pattern
|
||||
if url.startswith(pattern):
|
||||
return True
|
||||
else:
|
||||
# Domain-only pattern (case-insensitive comparison)
|
||||
if host.lower() == pattern.lower():
|
||||
return True
|
||||
# If pattern is a root domain, also check www subdomain
|
||||
if self._is_root_domain(pattern) and host.lower() == f'www.{pattern.lower()}':
|
||||
return True
|
||||
|
||||
return False
|
||||
+335
@@ -0,0 +1,335 @@
|
||||
"""Storage state watchdog for managing browser cookies and storage persistence."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any, ClassVar
|
||||
|
||||
from bubus import BaseEvent
|
||||
from cdp_use.cdp.network import Cookie
|
||||
from pydantic import Field, PrivateAttr
|
||||
|
||||
from browser_use.browser.events import (
|
||||
BrowserConnectedEvent,
|
||||
BrowserStopEvent,
|
||||
LoadStorageStateEvent,
|
||||
SaveStorageStateEvent,
|
||||
StorageStateLoadedEvent,
|
||||
StorageStateSavedEvent,
|
||||
)
|
||||
from browser_use.browser.watchdog_base import BaseWatchdog
|
||||
|
||||
|
||||
class StorageStateWatchdog(BaseWatchdog):
|
||||
"""Monitors and persists browser storage state including cookies and localStorage."""
|
||||
|
||||
# Event contracts
|
||||
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [
|
||||
BrowserConnectedEvent,
|
||||
BrowserStopEvent,
|
||||
SaveStorageStateEvent,
|
||||
LoadStorageStateEvent,
|
||||
]
|
||||
EMITS: ClassVar[list[type[BaseEvent]]] = [
|
||||
StorageStateSavedEvent,
|
||||
StorageStateLoadedEvent,
|
||||
]
|
||||
|
||||
# Configuration
|
||||
auto_save_interval: float = Field(default=30.0) # Auto-save every 30 seconds
|
||||
save_on_change: bool = Field(default=True) # Save immediately when cookies change
|
||||
|
||||
# Private state
|
||||
_monitoring_task: asyncio.Task | None = PrivateAttr(default=None)
|
||||
_last_cookie_state: list[dict] = PrivateAttr(default_factory=list)
|
||||
_save_lock: asyncio.Lock = PrivateAttr(default_factory=asyncio.Lock)
|
||||
|
||||
async def on_BrowserConnectedEvent(self, event: BrowserConnectedEvent) -> None:
|
||||
"""Start monitoring when browser starts."""
|
||||
self.logger.debug('[StorageStateWatchdog] 🍪 Initializing auth/cookies sync <-> with storage_state.json file')
|
||||
|
||||
# Start monitoring
|
||||
await self._start_monitoring()
|
||||
|
||||
# Automatically load storage state after browser start
|
||||
await self.event_bus.dispatch(LoadStorageStateEvent())
|
||||
|
||||
async def on_BrowserStopEvent(self, event: BrowserStopEvent) -> None:
|
||||
"""Stop monitoring when browser stops."""
|
||||
self.logger.debug('[StorageStateWatchdog] Stopping storage_state monitoring')
|
||||
await self._stop_monitoring()
|
||||
|
||||
async def on_SaveStorageStateEvent(self, event: SaveStorageStateEvent) -> None:
|
||||
"""Handle storage state save request."""
|
||||
# Use provided path or fall back to profile default
|
||||
path = event.path
|
||||
if path is None:
|
||||
# Use profile default path if available
|
||||
if self.browser_session.browser_profile.storage_state:
|
||||
path = str(self.browser_session.browser_profile.storage_state)
|
||||
else:
|
||||
path = None # Skip saving if no path available
|
||||
await self._save_storage_state(path)
|
||||
|
||||
async def on_LoadStorageStateEvent(self, event: LoadStorageStateEvent) -> None:
|
||||
"""Handle storage state load request."""
|
||||
# Use provided path or fall back to profile default
|
||||
path = event.path
|
||||
if path is None:
|
||||
# Use profile default path if available
|
||||
if self.browser_session.browser_profile.storage_state:
|
||||
path = str(self.browser_session.browser_profile.storage_state)
|
||||
else:
|
||||
path = None # Skip loading if no path available
|
||||
await self._load_storage_state(path)
|
||||
|
||||
async def _start_monitoring(self) -> None:
|
||||
"""Start the monitoring task."""
|
||||
if self._monitoring_task and not self._monitoring_task.done():
|
||||
return
|
||||
|
||||
assert self.browser_session.cdp_client is not None
|
||||
|
||||
self._monitoring_task = asyncio.create_task(self._monitor_storage_changes())
|
||||
# self.logger'[StorageStateWatchdog] Started storage monitoring task')
|
||||
|
||||
async def _stop_monitoring(self) -> None:
|
||||
"""Stop the monitoring task."""
|
||||
if self._monitoring_task and not self._monitoring_task.done():
|
||||
self._monitoring_task.cancel()
|
||||
try:
|
||||
await self._monitoring_task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
# self.logger.debug('[StorageStateWatchdog] Stopped storage monitoring task')
|
||||
|
||||
async def _check_for_cookie_changes_cdp(self, event: dict) -> None:
|
||||
"""Check if a CDP network event indicates cookie changes.
|
||||
|
||||
This would be called by Network.responseReceivedExtraInfo events
|
||||
if we set up CDP event listeners.
|
||||
"""
|
||||
try:
|
||||
# Check for Set-Cookie headers in the response
|
||||
headers = event.get('headers', {})
|
||||
if 'set-cookie' in headers or 'Set-Cookie' in headers:
|
||||
self.logger.debug('[StorageStateWatchdog] Cookie change detected via CDP')
|
||||
|
||||
# If save on change is enabled, trigger save immediately
|
||||
if self.save_on_change:
|
||||
await self._save_storage_state()
|
||||
except Exception as e:
|
||||
self.logger.warning(f'[StorageStateWatchdog] Error checking for cookie changes: {e}')
|
||||
|
||||
async def _monitor_storage_changes(self) -> None:
|
||||
"""Periodically check for storage changes and auto-save."""
|
||||
while True:
|
||||
try:
|
||||
await asyncio.sleep(self.auto_save_interval)
|
||||
|
||||
# Check if cookies have changed
|
||||
if await self._have_cookies_changed():
|
||||
self.logger.debug('[StorageStateWatchdog] Detected changes to sync with storage_state.json')
|
||||
await self._save_storage_state()
|
||||
|
||||
except asyncio.CancelledError:
|
||||
break
|
||||
except Exception as e:
|
||||
self.logger.error(f'[StorageStateWatchdog] Error in monitoring loop: {e}')
|
||||
|
||||
async def _have_cookies_changed(self) -> bool:
|
||||
"""Check if cookies have changed since last save."""
|
||||
if not self.browser_session.cdp_client:
|
||||
return False
|
||||
|
||||
try:
|
||||
# Get current cookies using CDP
|
||||
current_cookies = await self.browser_session._cdp_get_cookies()
|
||||
|
||||
# Convert to comparable format, using .get() for optional fields
|
||||
current_cookie_set = {
|
||||
(c.get('name', ''), c.get('domain', ''), c.get('path', '')): c.get('value', '') for c in current_cookies
|
||||
}
|
||||
|
||||
last_cookie_set = {
|
||||
(c.get('name', ''), c.get('domain', ''), c.get('path', '')): c.get('value', '') for c in self._last_cookie_state
|
||||
}
|
||||
|
||||
return current_cookie_set != last_cookie_set
|
||||
except Exception as e:
|
||||
self.logger.debug(f'[StorageStateWatchdog] Error comparing cookies: {e}')
|
||||
return False
|
||||
|
||||
async def _save_storage_state(self, path: str | None = None) -> None:
|
||||
"""Save browser storage state to file."""
|
||||
async with self._save_lock:
|
||||
# Check if CDP client is available
|
||||
assert await self.browser_session.get_or_create_cdp_session(target_id=None, new_socket=False)
|
||||
|
||||
save_path = path or self.browser_session.browser_profile.storage_state
|
||||
if not save_path:
|
||||
return
|
||||
|
||||
# Skip saving if the storage state is already a dict (indicates it was loaded from memory)
|
||||
# We only save to file if it started as a file path
|
||||
if isinstance(save_path, dict):
|
||||
self.logger.debug('[StorageStateWatchdog] Storage state is already a dict, skipping file save')
|
||||
return
|
||||
|
||||
try:
|
||||
# Get current storage state using CDP
|
||||
storage_state = await self.browser_session._cdp_get_storage_state()
|
||||
|
||||
# Update our last known state
|
||||
self._last_cookie_state = storage_state.get('cookies', []).copy()
|
||||
|
||||
# Convert path to Path object
|
||||
json_path = Path(save_path).expanduser().resolve()
|
||||
json_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Merge with existing state if file exists
|
||||
merged_state = storage_state
|
||||
if json_path.exists():
|
||||
try:
|
||||
existing_state = json.loads(json_path.read_text())
|
||||
merged_state = self._merge_storage_states(existing_state, dict(storage_state))
|
||||
except Exception as e:
|
||||
self.logger.error(f'[StorageStateWatchdog] Failed to merge with existing state: {e}')
|
||||
|
||||
# Write atomically
|
||||
temp_path = json_path.with_suffix('.json.tmp')
|
||||
temp_path.write_text(json.dumps(merged_state, indent=4))
|
||||
|
||||
# Backup existing file
|
||||
if json_path.exists():
|
||||
backup_path = json_path.with_suffix('.json.bak')
|
||||
json_path.replace(backup_path)
|
||||
|
||||
# Move temp to final
|
||||
temp_path.replace(json_path)
|
||||
|
||||
# Emit success event
|
||||
self.event_bus.dispatch(
|
||||
StorageStateSavedEvent(
|
||||
path=str(json_path),
|
||||
cookies_count=len(merged_state.get('cookies', [])),
|
||||
origins_count=len(merged_state.get('origins', [])),
|
||||
)
|
||||
)
|
||||
|
||||
self.logger.debug(
|
||||
f'[StorageStateWatchdog] Saved storage state to {json_path} '
|
||||
f'({len(merged_state.get("cookies", []))} cookies, '
|
||||
f'{len(merged_state.get("origins", []))} origins)'
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'[StorageStateWatchdog] Failed to save storage state: {e}')
|
||||
|
||||
async def _load_storage_state(self, path: str | None = None) -> None:
|
||||
"""Load browser storage state from file."""
|
||||
if not self.browser_session.cdp_client:
|
||||
self.logger.warning('[StorageStateWatchdog] No CDP client available for loading')
|
||||
return
|
||||
|
||||
load_path = path or self.browser_session.browser_profile.storage_state
|
||||
if not load_path or not os.path.exists(str(load_path)):
|
||||
return
|
||||
|
||||
try:
|
||||
# Read the storage state file asynchronously
|
||||
import anyio
|
||||
|
||||
content = await anyio.Path(str(load_path)).read_text()
|
||||
storage = json.loads(content)
|
||||
|
||||
# Apply cookies if present
|
||||
if 'cookies' in storage and storage['cookies']:
|
||||
await self.browser_session._cdp_set_cookies(storage['cookies'])
|
||||
self._last_cookie_state = storage['cookies'].copy()
|
||||
self.logger.debug(f'[StorageStateWatchdog] Added {len(storage["cookies"])} cookies from storage state')
|
||||
|
||||
# Apply origins (localStorage/sessionStorage) if present
|
||||
if 'origins' in storage and storage['origins']:
|
||||
for origin in storage['origins']:
|
||||
if 'localStorage' in origin:
|
||||
for item in origin['localStorage']:
|
||||
script = f"""
|
||||
window.localStorage.setItem({json.dumps(item['name'])}, {json.dumps(item['value'])});
|
||||
"""
|
||||
await self.browser_session._cdp_add_init_script(script)
|
||||
if 'sessionStorage' in origin:
|
||||
for item in origin['sessionStorage']:
|
||||
script = f"""
|
||||
window.sessionStorage.setItem({json.dumps(item['name'])}, {json.dumps(item['value'])});
|
||||
"""
|
||||
await self.browser_session._cdp_add_init_script(script)
|
||||
self.logger.debug(
|
||||
f'[StorageStateWatchdog] Applied localStorage/sessionStorage from {len(storage["origins"])} origins'
|
||||
)
|
||||
|
||||
self.event_bus.dispatch(
|
||||
StorageStateLoadedEvent(
|
||||
path=str(load_path),
|
||||
cookies_count=len(storage.get('cookies', [])),
|
||||
origins_count=len(storage.get('origins', [])),
|
||||
)
|
||||
)
|
||||
|
||||
self.logger.debug(f'[StorageStateWatchdog] Loaded storage state from: {load_path}')
|
||||
|
||||
except Exception as e:
|
||||
self.logger.error(f'[StorageStateWatchdog] Failed to load storage state: {e}')
|
||||
|
||||
@staticmethod
|
||||
def _merge_storage_states(existing: dict[str, Any], new: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Merge two storage states, with new values taking precedence."""
|
||||
merged = existing.copy()
|
||||
|
||||
# Merge cookies
|
||||
existing_cookies = {(c['name'], c['domain'], c['path']): c for c in existing.get('cookies', [])}
|
||||
|
||||
for cookie in new.get('cookies', []):
|
||||
key = (cookie['name'], cookie['domain'], cookie['path'])
|
||||
existing_cookies[key] = cookie
|
||||
|
||||
merged['cookies'] = list(existing_cookies.values())
|
||||
|
||||
# Merge origins
|
||||
existing_origins = {origin['origin']: origin for origin in existing.get('origins', [])}
|
||||
|
||||
for origin in new.get('origins', []):
|
||||
existing_origins[origin['origin']] = origin
|
||||
|
||||
merged['origins'] = list(existing_origins.values())
|
||||
|
||||
return merged
|
||||
|
||||
async def get_current_cookies(self) -> list[dict[str, Any]]:
|
||||
"""Get current cookies using CDP."""
|
||||
if not self.browser_session.cdp_client:
|
||||
return []
|
||||
|
||||
try:
|
||||
cookies = await self.browser_session._cdp_get_cookies()
|
||||
# Cookie is a TypedDict, cast to dict for compatibility
|
||||
return [dict(cookie) for cookie in cookies]
|
||||
except Exception as e:
|
||||
self.logger.error(f'[StorageStateWatchdog] Failed to get cookies: {e}')
|
||||
return []
|
||||
|
||||
async def add_cookies(self, cookies: list[dict[str, Any]]) -> None:
|
||||
"""Add cookies using CDP."""
|
||||
if not self.browser_session.cdp_client:
|
||||
self.logger.warning('[StorageStateWatchdog] No CDP client available for adding cookies')
|
||||
return
|
||||
|
||||
try:
|
||||
# Convert dicts to Cookie objects
|
||||
cookie_objects = [Cookie(**cookie_dict) if isinstance(cookie_dict, dict) else cookie_dict for cookie_dict in cookies]
|
||||
# Set cookies using CDP
|
||||
await self.browser_session._cdp_set_cookies(cookie_objects)
|
||||
self.logger.debug(f'[StorageStateWatchdog] Added {len(cookies)} cookies')
|
||||
except Exception as e:
|
||||
self.logger.error(f'[StorageStateWatchdog] Failed to add cookies: {e}')
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,505 @@
|
||||
"""Configuration system for browser-use with automatic migration support."""
|
||||
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
from functools import cache
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from uuid import uuid4
|
||||
|
||||
import psutil
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@cache
|
||||
def is_running_in_docker() -> bool:
|
||||
"""Detect if we are running in a docker container, for the purpose of optimizing chrome launch flags (dev shm usage, gpu settings, etc.)"""
|
||||
try:
|
||||
if Path('/.dockerenv').exists() or 'docker' in Path('/proc/1/cgroup').read_text().lower():
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
# if init proc (PID 1) looks like uvicorn/python/uv/etc. then we're in Docker
|
||||
# if init proc (PID 1) looks like bash/systemd/init/etc. then we're probably NOT in Docker
|
||||
init_cmd = ' '.join(psutil.Process(1).cmdline())
|
||||
if ('py' in init_cmd) or ('uv' in init_cmd) or ('app' in init_cmd):
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
try:
|
||||
# if less than 10 total running procs, then we're almost certainly in a container
|
||||
if len(psutil.pids()) < 10:
|
||||
return True
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return False
|
||||
|
||||
|
||||
class OldConfig:
|
||||
"""Original lazy-loading configuration class for environment variables."""
|
||||
|
||||
# Cache for directory creation tracking
|
||||
_dirs_created = False
|
||||
|
||||
@property
|
||||
def BROWSER_USE_LOGGING_LEVEL(self) -> str:
|
||||
return os.getenv('BROWSER_USE_LOGGING_LEVEL', 'info').lower()
|
||||
|
||||
@property
|
||||
def ANONYMIZED_TELEMETRY(self) -> bool:
|
||||
return os.getenv('ANONYMIZED_TELEMETRY', 'true').lower()[:1] in 'ty1'
|
||||
|
||||
@property
|
||||
def BROWSER_USE_CLOUD_SYNC(self) -> bool:
|
||||
return os.getenv('BROWSER_USE_CLOUD_SYNC', str(self.ANONYMIZED_TELEMETRY)).lower()[:1] in 'ty1'
|
||||
|
||||
@property
|
||||
def BROWSER_USE_CLOUD_API_URL(self) -> str:
|
||||
url = os.getenv('BROWSER_USE_CLOUD_API_URL', 'https://api.browser-use.com')
|
||||
assert '://' in url, 'BROWSER_USE_CLOUD_API_URL must be a valid URL'
|
||||
return url
|
||||
|
||||
@property
|
||||
def BROWSER_USE_CLOUD_UI_URL(self) -> str:
|
||||
url = os.getenv('BROWSER_USE_CLOUD_UI_URL', '')
|
||||
# Allow empty string as default, only validate if set
|
||||
if url and '://' not in url:
|
||||
raise AssertionError('BROWSER_USE_CLOUD_UI_URL must be a valid URL if set')
|
||||
return url
|
||||
|
||||
# Path configuration
|
||||
@property
|
||||
def XDG_CACHE_HOME(self) -> Path:
|
||||
return Path(os.getenv('XDG_CACHE_HOME', '~/.cache')).expanduser().resolve()
|
||||
|
||||
@property
|
||||
def XDG_CONFIG_HOME(self) -> Path:
|
||||
return Path(os.getenv('XDG_CONFIG_HOME', '~/.config')).expanduser().resolve()
|
||||
|
||||
@property
|
||||
def BROWSER_USE_CONFIG_DIR(self) -> Path:
|
||||
path = Path(os.getenv('BROWSER_USE_CONFIG_DIR', str(self.XDG_CONFIG_HOME / 'browseruse'))).expanduser().resolve()
|
||||
self._ensure_dirs()
|
||||
return path
|
||||
|
||||
@property
|
||||
def BROWSER_USE_CONFIG_FILE(self) -> Path:
|
||||
return self.BROWSER_USE_CONFIG_DIR / 'config.json'
|
||||
|
||||
@property
|
||||
def BROWSER_USE_PROFILES_DIR(self) -> Path:
|
||||
path = self.BROWSER_USE_CONFIG_DIR / 'profiles'
|
||||
self._ensure_dirs()
|
||||
return path
|
||||
|
||||
@property
|
||||
def BROWSER_USE_DEFAULT_USER_DATA_DIR(self) -> Path:
|
||||
return self.BROWSER_USE_PROFILES_DIR / 'default'
|
||||
|
||||
@property
|
||||
def BROWSER_USE_EXTENSIONS_DIR(self) -> Path:
|
||||
path = self.BROWSER_USE_CONFIG_DIR / 'extensions'
|
||||
self._ensure_dirs()
|
||||
return path
|
||||
|
||||
def _ensure_dirs(self) -> None:
|
||||
"""Create directories if they don't exist (only once)"""
|
||||
if not self._dirs_created:
|
||||
config_dir = (
|
||||
Path(os.getenv('BROWSER_USE_CONFIG_DIR', str(self.XDG_CONFIG_HOME / 'browseruse'))).expanduser().resolve()
|
||||
)
|
||||
config_dir.mkdir(parents=True, exist_ok=True)
|
||||
(config_dir / 'profiles').mkdir(parents=True, exist_ok=True)
|
||||
(config_dir / 'extensions').mkdir(parents=True, exist_ok=True)
|
||||
self._dirs_created = True
|
||||
|
||||
# LLM API key configuration
|
||||
@property
|
||||
def OPENAI_API_KEY(self) -> str:
|
||||
return os.getenv('OPENAI_API_KEY', '')
|
||||
|
||||
@property
|
||||
def ANTHROPIC_API_KEY(self) -> str:
|
||||
return os.getenv('ANTHROPIC_API_KEY', '')
|
||||
|
||||
@property
|
||||
def GOOGLE_API_KEY(self) -> str:
|
||||
return os.getenv('GOOGLE_API_KEY', '')
|
||||
|
||||
@property
|
||||
def DEEPSEEK_API_KEY(self) -> str:
|
||||
return os.getenv('DEEPSEEK_API_KEY', '')
|
||||
|
||||
@property
|
||||
def GROK_API_KEY(self) -> str:
|
||||
return os.getenv('GROK_API_KEY', '')
|
||||
|
||||
@property
|
||||
def NOVITA_API_KEY(self) -> str:
|
||||
return os.getenv('NOVITA_API_KEY', '')
|
||||
|
||||
@property
|
||||
def AZURE_OPENAI_ENDPOINT(self) -> str:
|
||||
return os.getenv('AZURE_OPENAI_ENDPOINT', '')
|
||||
|
||||
@property
|
||||
def AZURE_OPENAI_KEY(self) -> str:
|
||||
return os.getenv('AZURE_OPENAI_KEY', '')
|
||||
|
||||
@property
|
||||
def SKIP_LLM_API_KEY_VERIFICATION(self) -> bool:
|
||||
return os.getenv('SKIP_LLM_API_KEY_VERIFICATION', 'false').lower()[:1] in 'ty1'
|
||||
|
||||
@property
|
||||
def DEFAULT_LLM(self) -> str:
|
||||
return os.getenv('DEFAULT_LLM', '')
|
||||
|
||||
# Runtime hints
|
||||
@property
|
||||
def IN_DOCKER(self) -> bool:
|
||||
return os.getenv('IN_DOCKER', 'false').lower()[:1] in 'ty1' or is_running_in_docker()
|
||||
|
||||
@property
|
||||
def IS_IN_EVALS(self) -> bool:
|
||||
return os.getenv('IS_IN_EVALS', 'false').lower()[:1] in 'ty1'
|
||||
|
||||
@property
|
||||
def WIN_FONT_DIR(self) -> str:
|
||||
return os.getenv('WIN_FONT_DIR', 'C:\\Windows\\Fonts')
|
||||
|
||||
|
||||
class FlatEnvConfig(BaseSettings):
|
||||
"""All environment variables in a flat namespace."""
|
||||
|
||||
model_config = SettingsConfigDict(env_file='.env', env_file_encoding='utf-8', case_sensitive=True, extra='allow')
|
||||
|
||||
# Logging and telemetry
|
||||
BROWSER_USE_LOGGING_LEVEL: str = Field(default='info')
|
||||
CDP_LOGGING_LEVEL: str = Field(default='WARNING')
|
||||
BROWSER_USE_DEBUG_LOG_FILE: str | None = Field(default=None)
|
||||
BROWSER_USE_INFO_LOG_FILE: str | None = Field(default=None)
|
||||
ANONYMIZED_TELEMETRY: bool = Field(default=True)
|
||||
BROWSER_USE_CLOUD_SYNC: bool | None = Field(default=None)
|
||||
BROWSER_USE_CLOUD_API_URL: str = Field(default='https://api.browser-use.com')
|
||||
BROWSER_USE_CLOUD_UI_URL: str = Field(default='')
|
||||
|
||||
# Path configuration
|
||||
XDG_CACHE_HOME: str = Field(default='~/.cache')
|
||||
XDG_CONFIG_HOME: str = Field(default='~/.config')
|
||||
BROWSER_USE_CONFIG_DIR: str | None = Field(default=None)
|
||||
|
||||
# LLM API keys
|
||||
OPENAI_API_KEY: str = Field(default='')
|
||||
ANTHROPIC_API_KEY: str = Field(default='')
|
||||
GOOGLE_API_KEY: str = Field(default='')
|
||||
DEEPSEEK_API_KEY: str = Field(default='')
|
||||
GROK_API_KEY: str = Field(default='')
|
||||
NOVITA_API_KEY: str = Field(default='')
|
||||
AZURE_OPENAI_ENDPOINT: str = Field(default='')
|
||||
AZURE_OPENAI_KEY: str = Field(default='')
|
||||
SKIP_LLM_API_KEY_VERIFICATION: bool = Field(default=False)
|
||||
DEFAULT_LLM: str = Field(default='')
|
||||
|
||||
# Runtime hints
|
||||
IN_DOCKER: bool | None = Field(default=None)
|
||||
IS_IN_EVALS: bool = Field(default=False)
|
||||
WIN_FONT_DIR: str = Field(default='C:\\Windows\\Fonts')
|
||||
|
||||
# MCP-specific env vars
|
||||
BROWSER_USE_CONFIG_PATH: str | None = Field(default=None)
|
||||
BROWSER_USE_HEADLESS: bool | None = Field(default=None)
|
||||
BROWSER_USE_ALLOWED_DOMAINS: str | None = Field(default=None)
|
||||
BROWSER_USE_LLM_MODEL: str | None = Field(default=None)
|
||||
|
||||
# Proxy env vars
|
||||
BROWSER_USE_PROXY_URL: str | None = Field(default=None)
|
||||
BROWSER_USE_NO_PROXY: str | None = Field(default=None)
|
||||
BROWSER_USE_PROXY_USERNAME: str | None = Field(default=None)
|
||||
BROWSER_USE_PROXY_PASSWORD: str | None = Field(default=None)
|
||||
|
||||
|
||||
class DBStyleEntry(BaseModel):
|
||||
"""Database-style entry with UUID and metadata."""
|
||||
|
||||
id: str = Field(default_factory=lambda: str(uuid4()))
|
||||
default: bool = Field(default=False)
|
||||
created_at: str = Field(default_factory=lambda: datetime.utcnow().isoformat())
|
||||
|
||||
|
||||
class BrowserProfileEntry(DBStyleEntry):
|
||||
"""Browser profile configuration entry - accepts any BrowserProfile fields."""
|
||||
|
||||
model_config = ConfigDict(extra='allow')
|
||||
|
||||
# Common browser profile fields for reference
|
||||
headless: bool | None = None
|
||||
user_data_dir: str | None = None
|
||||
allowed_domains: list[str] | None = None
|
||||
downloads_path: str | None = None
|
||||
|
||||
|
||||
class LLMEntry(DBStyleEntry):
|
||||
"""LLM configuration entry."""
|
||||
|
||||
api_key: str | None = None
|
||||
model: str | None = None
|
||||
temperature: float | None = None
|
||||
max_tokens: int | None = None
|
||||
|
||||
|
||||
class AgentEntry(DBStyleEntry):
|
||||
"""Agent configuration entry."""
|
||||
|
||||
max_steps: int | None = None
|
||||
use_vision: bool | None = None
|
||||
system_prompt: str | None = None
|
||||
|
||||
|
||||
class DBStyleConfigJSON(BaseModel):
|
||||
"""New database-style configuration format."""
|
||||
|
||||
browser_profile: dict[str, BrowserProfileEntry] = Field(default_factory=dict)
|
||||
llm: dict[str, LLMEntry] = Field(default_factory=dict)
|
||||
agent: dict[str, AgentEntry] = Field(default_factory=dict)
|
||||
|
||||
|
||||
def create_default_config() -> DBStyleConfigJSON:
|
||||
"""Create a fresh default configuration."""
|
||||
logger.debug('Creating fresh default config.json')
|
||||
|
||||
new_config = DBStyleConfigJSON()
|
||||
|
||||
# Generate default IDs
|
||||
profile_id = str(uuid4())
|
||||
llm_id = str(uuid4())
|
||||
agent_id = str(uuid4())
|
||||
|
||||
# Create default browser profile entry
|
||||
new_config.browser_profile[profile_id] = BrowserProfileEntry(id=profile_id, default=True, headless=False, user_data_dir=None)
|
||||
|
||||
# Create default LLM entry
|
||||
new_config.llm[llm_id] = LLMEntry(id=llm_id, default=True, model='gpt-4.1-mini', api_key='your-openai-api-key-here')
|
||||
|
||||
# Create default agent entry
|
||||
new_config.agent[agent_id] = AgentEntry(id=agent_id, default=True)
|
||||
|
||||
return new_config
|
||||
|
||||
|
||||
def load_and_migrate_config(config_path: Path) -> DBStyleConfigJSON:
|
||||
"""Load config.json or create fresh one if old format detected."""
|
||||
if not config_path.exists():
|
||||
# Create fresh config with defaults
|
||||
config_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
new_config = create_default_config()
|
||||
with open(config_path, 'w') as f:
|
||||
json.dump(new_config.model_dump(), f, indent=2)
|
||||
return new_config
|
||||
|
||||
try:
|
||||
with open(config_path) as f:
|
||||
data = json.load(f)
|
||||
|
||||
# Check if it's already in DB-style format
|
||||
if all(key in data for key in ['browser_profile', 'llm', 'agent']) and all(
|
||||
isinstance(data.get(key, {}), dict) for key in ['browser_profile', 'llm', 'agent']
|
||||
):
|
||||
# Check if the values are DB-style entries (have UUIDs as keys)
|
||||
if data.get('browser_profile') and all(isinstance(v, dict) and 'id' in v for v in data['browser_profile'].values()):
|
||||
# Already in new format
|
||||
return DBStyleConfigJSON(**data)
|
||||
|
||||
# Old format detected - delete it and create fresh config
|
||||
logger.debug(f'Old config format detected at {config_path}, creating fresh config')
|
||||
new_config = create_default_config()
|
||||
|
||||
# Overwrite with new config
|
||||
with open(config_path, 'w') as f:
|
||||
json.dump(new_config.model_dump(), f, indent=2)
|
||||
|
||||
logger.debug(f'Created fresh config.json at {config_path}')
|
||||
return new_config
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f'Failed to load config from {config_path}: {e}, creating fresh config')
|
||||
# On any error, create fresh config
|
||||
new_config = create_default_config()
|
||||
try:
|
||||
with open(config_path, 'w') as f:
|
||||
json.dump(new_config.model_dump(), f, indent=2)
|
||||
except Exception as write_error:
|
||||
logger.error(f'Failed to write fresh config: {write_error}')
|
||||
return new_config
|
||||
|
||||
|
||||
class Config:
|
||||
"""Backward-compatible configuration class that merges all config sources.
|
||||
|
||||
Re-reads environment variables on every access to maintain compatibility.
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
# Cache for directory creation tracking only
|
||||
self._dirs_created = False
|
||||
|
||||
def __getattr__(self, name: str) -> Any:
|
||||
"""Dynamically proxy all attributes to fresh instances.
|
||||
|
||||
This ensures env vars are re-read on every access.
|
||||
"""
|
||||
# Special handling for internal attributes
|
||||
if name.startswith('_'):
|
||||
raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{name}'")
|
||||
|
||||
# Create fresh instances on every access
|
||||
old_config = OldConfig()
|
||||
|
||||
# Always use old config for all attributes (it handles env vars with proper transformations)
|
||||
if hasattr(old_config, name):
|
||||
return getattr(old_config, name)
|
||||
|
||||
# For new MCP-specific attributes not in old config
|
||||
env_config = FlatEnvConfig()
|
||||
if hasattr(env_config, name):
|
||||
return getattr(env_config, name)
|
||||
|
||||
# Handle special methods
|
||||
if name == 'get_default_profile':
|
||||
return lambda: self._get_default_profile()
|
||||
elif name == 'get_default_llm':
|
||||
return lambda: self._get_default_llm()
|
||||
elif name == 'get_default_agent':
|
||||
return lambda: self._get_default_agent()
|
||||
elif name == 'load_config':
|
||||
return lambda: self._load_config()
|
||||
elif name == '_ensure_dirs':
|
||||
return lambda: old_config._ensure_dirs()
|
||||
|
||||
raise AttributeError(f"'{self.__class__.__name__}' object has no attribute '{name}'")
|
||||
|
||||
def _get_config_path(self) -> Path:
|
||||
"""Get config path from fresh env config."""
|
||||
env_config = FlatEnvConfig()
|
||||
if env_config.BROWSER_USE_CONFIG_PATH:
|
||||
return Path(env_config.BROWSER_USE_CONFIG_PATH).expanduser()
|
||||
elif env_config.BROWSER_USE_CONFIG_DIR:
|
||||
return Path(env_config.BROWSER_USE_CONFIG_DIR).expanduser() / 'config.json'
|
||||
else:
|
||||
xdg_config = Path(env_config.XDG_CONFIG_HOME).expanduser()
|
||||
return xdg_config / 'browseruse' / 'config.json'
|
||||
|
||||
def _get_db_config(self) -> DBStyleConfigJSON:
|
||||
"""Load and migrate config.json."""
|
||||
config_path = self._get_config_path()
|
||||
return load_and_migrate_config(config_path)
|
||||
|
||||
def _get_default_profile(self) -> dict[str, Any]:
|
||||
"""Get the default browser profile configuration."""
|
||||
db_config = self._get_db_config()
|
||||
for profile in db_config.browser_profile.values():
|
||||
if profile.default:
|
||||
return profile.model_dump(exclude_none=True)
|
||||
|
||||
# Return first profile if no default
|
||||
if db_config.browser_profile:
|
||||
return next(iter(db_config.browser_profile.values())).model_dump(exclude_none=True)
|
||||
|
||||
return {}
|
||||
|
||||
def _get_default_llm(self) -> dict[str, Any]:
|
||||
"""Get the default LLM configuration."""
|
||||
db_config = self._get_db_config()
|
||||
for llm in db_config.llm.values():
|
||||
if llm.default:
|
||||
return llm.model_dump(exclude_none=True)
|
||||
|
||||
# Return first LLM if no default
|
||||
if db_config.llm:
|
||||
return next(iter(db_config.llm.values())).model_dump(exclude_none=True)
|
||||
|
||||
return {}
|
||||
|
||||
def _get_default_agent(self) -> dict[str, Any]:
|
||||
"""Get the default agent configuration."""
|
||||
db_config = self._get_db_config()
|
||||
for agent in db_config.agent.values():
|
||||
if agent.default:
|
||||
return agent.model_dump(exclude_none=True)
|
||||
|
||||
# Return first agent if no default
|
||||
if db_config.agent:
|
||||
return next(iter(db_config.agent.values())).model_dump(exclude_none=True)
|
||||
|
||||
return {}
|
||||
|
||||
def _load_config(self) -> dict[str, Any]:
|
||||
"""Load configuration with env var overrides for MCP components."""
|
||||
config = {
|
||||
'browser_profile': self._get_default_profile(),
|
||||
'llm': self._get_default_llm(),
|
||||
'agent': self._get_default_agent(),
|
||||
}
|
||||
|
||||
# Fresh env config for overrides
|
||||
env_config = FlatEnvConfig()
|
||||
|
||||
# Apply MCP-specific env var overrides
|
||||
if env_config.BROWSER_USE_HEADLESS is not None:
|
||||
config['browser_profile']['headless'] = env_config.BROWSER_USE_HEADLESS
|
||||
|
||||
if env_config.BROWSER_USE_ALLOWED_DOMAINS:
|
||||
domains = [d.strip() for d in env_config.BROWSER_USE_ALLOWED_DOMAINS.split(',') if d.strip()]
|
||||
config['browser_profile']['allowed_domains'] = domains
|
||||
|
||||
# Proxy settings (Chromium) -> consolidated `proxy` dict
|
||||
proxy_dict: dict[str, Any] = {}
|
||||
if env_config.BROWSER_USE_PROXY_URL:
|
||||
proxy_dict['server'] = env_config.BROWSER_USE_PROXY_URL
|
||||
if env_config.BROWSER_USE_NO_PROXY:
|
||||
# store bypass as comma-separated string to match Chrome flag
|
||||
proxy_dict['bypass'] = ','.join([d.strip() for d in env_config.BROWSER_USE_NO_PROXY.split(',') if d.strip()])
|
||||
if env_config.BROWSER_USE_PROXY_USERNAME:
|
||||
proxy_dict['username'] = env_config.BROWSER_USE_PROXY_USERNAME
|
||||
if env_config.BROWSER_USE_PROXY_PASSWORD:
|
||||
proxy_dict['password'] = env_config.BROWSER_USE_PROXY_PASSWORD
|
||||
if proxy_dict:
|
||||
# ensure section exists
|
||||
config.setdefault('browser_profile', {})
|
||||
config['browser_profile']['proxy'] = proxy_dict
|
||||
|
||||
if env_config.OPENAI_API_KEY:
|
||||
config['llm']['api_key'] = env_config.OPENAI_API_KEY
|
||||
|
||||
if env_config.BROWSER_USE_LLM_MODEL:
|
||||
config['llm']['model'] = env_config.BROWSER_USE_LLM_MODEL
|
||||
|
||||
return config
|
||||
|
||||
|
||||
# Create singleton instance
|
||||
CONFIG = Config()
|
||||
|
||||
|
||||
# Helper functions for MCP components
|
||||
def load_browser_use_config() -> dict[str, Any]:
|
||||
"""Load browser-use configuration for MCP components."""
|
||||
return CONFIG.load_config()
|
||||
|
||||
|
||||
def get_default_profile(config: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Get default browser profile from config dict."""
|
||||
return config.get('browser_profile', {})
|
||||
|
||||
|
||||
def get_default_llm(config: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Get default LLM config from config dict."""
|
||||
return config.get('llm', {})
|
||||
@@ -0,0 +1,3 @@
|
||||
from browser_use.tools.service import Controller
|
||||
|
||||
__all__ = ['Controller']
|
||||
@@ -0,0 +1,161 @@
|
||||
"""
|
||||
Enhanced snapshot processing for browser-use DOM tree extraction.
|
||||
|
||||
This module provides stateless functions for parsing Chrome DevTools Protocol (CDP) DOMSnapshot data
|
||||
to extract visibility, clickability, cursor styles, and other layout information.
|
||||
"""
|
||||
|
||||
from cdp_use.cdp.domsnapshot.commands import CaptureSnapshotReturns
|
||||
from cdp_use.cdp.domsnapshot.types import (
|
||||
LayoutTreeSnapshot,
|
||||
NodeTreeSnapshot,
|
||||
RareBooleanData,
|
||||
)
|
||||
|
||||
from browser_use.dom.views import DOMRect, EnhancedSnapshotNode
|
||||
|
||||
# Only the ESSENTIAL computed styles for interactivity and visibility detection
|
||||
REQUIRED_COMPUTED_STYLES = [
|
||||
# Only styles actually accessed in the codebase (prevents Chrome crashes on heavy sites)
|
||||
'display', # Used in service.py visibility detection
|
||||
'visibility', # Used in service.py visibility detection
|
||||
'opacity', # Used in service.py visibility detection
|
||||
'overflow', # Used in views.py scrollability detection
|
||||
'overflow-x', # Used in views.py scrollability detection
|
||||
'overflow-y', # Used in views.py scrollability detection
|
||||
'cursor', # Used in enhanced_snapshot.py cursor extraction
|
||||
'pointer-events', # Used for clickability logic
|
||||
'position', # Used for visibility logic
|
||||
'background-color', # Used for visibility logic
|
||||
]
|
||||
|
||||
|
||||
def _parse_rare_boolean_data(rare_data: RareBooleanData, index: int) -> bool | None:
|
||||
"""Parse rare boolean data from snapshot - returns True if index is in the rare data."""
|
||||
return index in rare_data['index']
|
||||
|
||||
|
||||
def _parse_computed_styles(strings: list[str], style_indices: list[int]) -> dict[str, str]:
|
||||
"""Parse computed styles from layout tree using string indices."""
|
||||
styles = {}
|
||||
for i, style_index in enumerate(style_indices):
|
||||
if i < len(REQUIRED_COMPUTED_STYLES) and 0 <= style_index < len(strings):
|
||||
styles[REQUIRED_COMPUTED_STYLES[i]] = strings[style_index]
|
||||
return styles
|
||||
|
||||
|
||||
def build_snapshot_lookup(
|
||||
snapshot: CaptureSnapshotReturns,
|
||||
device_pixel_ratio: float = 1.0,
|
||||
) -> dict[int, EnhancedSnapshotNode]:
|
||||
"""Build a lookup table of backend node ID to enhanced snapshot data with everything calculated upfront."""
|
||||
snapshot_lookup: dict[int, EnhancedSnapshotNode] = {}
|
||||
|
||||
if not snapshot['documents']:
|
||||
return snapshot_lookup
|
||||
|
||||
strings = snapshot['strings']
|
||||
|
||||
for document in snapshot['documents']:
|
||||
nodes: NodeTreeSnapshot = document['nodes']
|
||||
layout: LayoutTreeSnapshot = document['layout']
|
||||
|
||||
# Build backend node id to snapshot index lookup
|
||||
backend_node_to_snapshot_index = {}
|
||||
if 'backendNodeId' in nodes:
|
||||
for i, backend_node_id in enumerate(nodes['backendNodeId']):
|
||||
backend_node_to_snapshot_index[backend_node_id] = i
|
||||
|
||||
# PERFORMANCE: Pre-build layout index map to eliminate O(n²) double lookups
|
||||
# Preserve original behavior: use FIRST occurrence for duplicates
|
||||
layout_index_map = {}
|
||||
if layout and 'nodeIndex' in layout:
|
||||
for layout_idx, node_index in enumerate(layout['nodeIndex']):
|
||||
if node_index not in layout_index_map: # Only store first occurrence
|
||||
layout_index_map[node_index] = layout_idx
|
||||
|
||||
# Build snapshot lookup for each backend node id
|
||||
for backend_node_id, snapshot_index in backend_node_to_snapshot_index.items():
|
||||
is_clickable = None
|
||||
if 'isClickable' in nodes:
|
||||
is_clickable = _parse_rare_boolean_data(nodes['isClickable'], snapshot_index)
|
||||
|
||||
# Find corresponding layout node
|
||||
cursor_style = None
|
||||
is_visible = None
|
||||
bounding_box = None
|
||||
computed_styles = {}
|
||||
|
||||
# Look for layout tree node that corresponds to this snapshot node
|
||||
paint_order = None
|
||||
client_rects = None
|
||||
scroll_rects = None
|
||||
stacking_contexts = None
|
||||
if snapshot_index in layout_index_map:
|
||||
layout_idx = layout_index_map[snapshot_index]
|
||||
if layout_idx < len(layout.get('bounds') or []):
|
||||
# Parse bounding box
|
||||
bounds = layout['bounds'][layout_idx]
|
||||
if len(bounds) >= 4:
|
||||
# IMPORTANT: CDP coordinates are in device pixels, convert to CSS pixels
|
||||
# by dividing by the device pixel ratio
|
||||
raw_x, raw_y, raw_width, raw_height = bounds[0], bounds[1], bounds[2], bounds[3]
|
||||
|
||||
# Apply device pixel ratio scaling to convert device pixels to CSS pixels
|
||||
bounding_box = DOMRect(
|
||||
x=raw_x / device_pixel_ratio,
|
||||
y=raw_y / device_pixel_ratio,
|
||||
width=raw_width / device_pixel_ratio,
|
||||
height=raw_height / device_pixel_ratio,
|
||||
)
|
||||
|
||||
# Parse computed styles for this layout node
|
||||
if layout_idx < len(layout.get('styles') or []):
|
||||
style_indices = layout['styles'][layout_idx]
|
||||
computed_styles = _parse_computed_styles(strings, style_indices)
|
||||
cursor_style = computed_styles.get('cursor')
|
||||
|
||||
# Extract paint order if available
|
||||
if layout_idx < len(layout.get('paintOrders') or []):
|
||||
paint_order = layout.get('paintOrders', [])[layout_idx]
|
||||
|
||||
# Extract client rects if available
|
||||
client_rects_data = layout.get('clientRects') or []
|
||||
if layout_idx < len(client_rects_data):
|
||||
client_rect_data = client_rects_data[layout_idx]
|
||||
if client_rect_data and len(client_rect_data) >= 4:
|
||||
client_rects = DOMRect(
|
||||
x=client_rect_data[0],
|
||||
y=client_rect_data[1],
|
||||
width=client_rect_data[2],
|
||||
height=client_rect_data[3],
|
||||
)
|
||||
|
||||
# Extract scroll rects if available
|
||||
scroll_rects_data = layout.get('scrollRects') or []
|
||||
if layout_idx < len(scroll_rects_data):
|
||||
scroll_rect_data = scroll_rects_data[layout_idx]
|
||||
if scroll_rect_data and len(scroll_rect_data) >= 4:
|
||||
scroll_rects = DOMRect(
|
||||
x=scroll_rect_data[0],
|
||||
y=scroll_rect_data[1],
|
||||
width=scroll_rect_data[2],
|
||||
height=scroll_rect_data[3],
|
||||
)
|
||||
|
||||
# Extract stacking contexts if available
|
||||
if layout_idx < len(layout.get('stackingContexts') or []):
|
||||
stacking_contexts = layout.get('stackingContexts', {}).get('index', [])[layout_idx]
|
||||
|
||||
snapshot_lookup[backend_node_id] = EnhancedSnapshotNode(
|
||||
is_clickable=is_clickable,
|
||||
cursor_style=cursor_style,
|
||||
bounds=bounding_box,
|
||||
clientRects=client_rects,
|
||||
scrollRects=scroll_rects,
|
||||
computed_styles=computed_styles if computed_styles else None,
|
||||
paint_order=paint_order,
|
||||
stacking_contexts=stacking_contexts,
|
||||
)
|
||||
|
||||
return snapshot_lookup
|
||||
@@ -0,0 +1,312 @@
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
|
||||
import anyio
|
||||
import pyperclip
|
||||
import tiktoken
|
||||
|
||||
from browser_use.agent.prompts import AgentMessagePrompt
|
||||
from browser_use.browser import BrowserProfile, BrowserSession
|
||||
from browser_use.browser.events import ClickElementEvent, TypeTextEvent
|
||||
from browser_use.browser.profile import ViewportSize
|
||||
from browser_use.dom.service import DomService
|
||||
from browser_use.dom.views import DEFAULT_INCLUDE_ATTRIBUTES
|
||||
from browser_use.filesystem.file_system import FileSystem
|
||||
|
||||
TIMEOUT = 60
|
||||
|
||||
|
||||
async def test_focus_vs_all_elements():
|
||||
browser_session = BrowserSession(
|
||||
browser_profile=BrowserProfile(
|
||||
# executable_path='/Applications/Google Chrome.app/Contents/MacOS/Google Chrome',
|
||||
window_size=ViewportSize(width=1100, height=1000),
|
||||
disable_security=False,
|
||||
wait_for_network_idle_page_load_time=1,
|
||||
headless=False,
|
||||
args=['--incognito'],
|
||||
paint_order_filtering=True,
|
||||
),
|
||||
)
|
||||
|
||||
# 10 Sample websites with various interactive elements
|
||||
sample_websites = [
|
||||
'https://browser-use.github.io/stress-tests/challenges/iframe-inception-level2.html',
|
||||
'https://www.google.com/travel/flights',
|
||||
'https://v0-simple-ui-test-site.vercel.app',
|
||||
'https://browser-use.github.io/stress-tests/challenges/iframe-inception-level1.html',
|
||||
'https://browser-use.github.io/stress-tests/challenges/angular-form.html',
|
||||
'https://www.google.com/travel/flights',
|
||||
'https://www.amazon.com/s?k=laptop',
|
||||
'https://github.com/trending',
|
||||
'https://www.reddit.com',
|
||||
'https://www.ycombinator.com/companies',
|
||||
'https://www.kayak.com/flights',
|
||||
'https://www.booking.com',
|
||||
'https://www.airbnb.com',
|
||||
'https://www.linkedin.com/jobs',
|
||||
'https://stackoverflow.com/questions',
|
||||
]
|
||||
|
||||
# 5 Difficult websites with complex elements (iframes, canvas, dropdowns, etc.)
|
||||
difficult_websites = [
|
||||
'https://www.w3schools.com/html/tryit.asp?filename=tryhtml_iframe', # Nested iframes
|
||||
'https://semantic-ui.com/modules/dropdown.html', # Complex dropdowns
|
||||
'https://www.dezlearn.com/nested-iframes-example/', # Cross-origin nested iframes
|
||||
'https://codepen.io/towc/pen/mJzOWJ', # Canvas elements with interactions
|
||||
'https://jqueryui.com/accordion/', # Complex accordion/dropdown widgets
|
||||
'https://v0-simple-landing-page-seven-xi.vercel.app/', # Simple landing page with iframe
|
||||
'https://www.unesco.org/en',
|
||||
]
|
||||
|
||||
# Descriptions for difficult websites
|
||||
difficult_descriptions = {
|
||||
'https://www.w3schools.com/html/tryit.asp?filename=tryhtml_iframe': '🔸 NESTED IFRAMES: Multiple iframe layers',
|
||||
'https://semantic-ui.com/modules/dropdown.html': '🔸 COMPLEX DROPDOWNS: Custom dropdown components',
|
||||
'https://www.dezlearn.com/nested-iframes-example/': '🔸 CROSS-ORIGIN IFRAMES: Different domain iframes',
|
||||
'https://codepen.io/towc/pen/mJzOWJ': '🔸 CANVAS ELEMENTS: Interactive canvas graphics',
|
||||
'https://jqueryui.com/accordion/': '🔸 ACCORDION WIDGETS: Collapsible content sections',
|
||||
}
|
||||
|
||||
websites = sample_websites + difficult_websites
|
||||
current_website_index = 0
|
||||
|
||||
def get_website_list_for_prompt() -> str:
|
||||
"""Get a compact website list for the input prompt."""
|
||||
lines = []
|
||||
lines.append('📋 Websites:')
|
||||
|
||||
# Sample websites (1-10)
|
||||
for i, site in enumerate(sample_websites, 1):
|
||||
current_marker = ' ←' if (i - 1) == current_website_index else ''
|
||||
domain = site.replace('https://', '').split('/')[0]
|
||||
lines.append(f' {i:2d}.{domain[:15]:<15}{current_marker}')
|
||||
|
||||
# Difficult websites (11-15)
|
||||
for i, site in enumerate(difficult_websites, len(sample_websites) + 1):
|
||||
current_marker = ' ←' if (i - 1) == current_website_index else ''
|
||||
domain = site.replace('https://', '').split('/')[0]
|
||||
desc = difficult_descriptions.get(site, '')
|
||||
challenge = desc.split(': ')[1][:15] if ': ' in desc else ''
|
||||
lines.append(f' {i:2d}.{domain[:15]:<15} ({challenge}){current_marker}')
|
||||
|
||||
return '\n'.join(lines)
|
||||
|
||||
await browser_session.start()
|
||||
|
||||
# Show startup info
|
||||
print('\n🌐 BROWSER-USE DOM EXTRACTION TESTER')
|
||||
print(f'📊 {len(websites)} websites total: {len(sample_websites)} standard + {len(difficult_websites)} complex')
|
||||
print('🔧 Controls: Type 1-15 to jump | Enter to re-run | "n" next | "q" quit')
|
||||
print('💾 Outputs: tmp/user_message.txt & tmp/element_tree.json\n')
|
||||
|
||||
dom_service = DomService(browser_session)
|
||||
|
||||
while True:
|
||||
# Cycle through websites
|
||||
if current_website_index >= len(websites):
|
||||
current_website_index = 0
|
||||
print('Cycled back to first website!')
|
||||
|
||||
website = websites[current_website_index]
|
||||
# sleep 2
|
||||
await browser_session._cdp_navigate(website)
|
||||
await asyncio.sleep(1)
|
||||
|
||||
last_clicked_index = None # Track the index for text input
|
||||
while True:
|
||||
try:
|
||||
# all_elements_state = await dom_service.get_serialized_dom_tree()
|
||||
|
||||
website_type = 'DIFFICULT' if website in difficult_websites else 'SAMPLE'
|
||||
print(f'\n{"=" * 60}')
|
||||
print(f'[{current_website_index + 1}/{len(websites)}] [{website_type}] Testing: {website}')
|
||||
if website in difficult_descriptions:
|
||||
print(f'{difficult_descriptions[website]}')
|
||||
print(f'{"=" * 60}')
|
||||
|
||||
# Get/refresh the state (includes removing old highlights)
|
||||
print('\nGetting page state...')
|
||||
|
||||
start_time = time.time()
|
||||
all_elements_state = await browser_session.get_browser_state_summary(True)
|
||||
end_time = time.time()
|
||||
get_state_time = end_time - start_time
|
||||
print(f'get_state_summary took {get_state_time:.2f} seconds')
|
||||
|
||||
# Get detailed timing info from DOM service
|
||||
print('\nGetting detailed DOM timing...')
|
||||
serialized_state, _, timing_info = await dom_service.get_serialized_dom_tree()
|
||||
|
||||
# Combine all timing info
|
||||
all_timing = {'get_state_summary_total': get_state_time, **timing_info}
|
||||
|
||||
selector_map = all_elements_state.dom_state.selector_map
|
||||
total_elements = len(selector_map.keys())
|
||||
print(f'Total number of elements: {total_elements}')
|
||||
|
||||
# print(all_elements_state.element_tree.clickable_elements_to_string())
|
||||
prompt = AgentMessagePrompt(
|
||||
browser_state_summary=all_elements_state,
|
||||
file_system=FileSystem(base_dir='./tmp'),
|
||||
include_attributes=DEFAULT_INCLUDE_ATTRIBUTES,
|
||||
step_info=None,
|
||||
)
|
||||
# Write the user message to a file for analysis
|
||||
user_message = prompt.get_user_message(use_vision=False).text
|
||||
|
||||
# clickable_elements_str = all_elements_state.element_tree.clickable_elements_to_string()
|
||||
|
||||
text_to_save = user_message
|
||||
|
||||
os.makedirs('./tmp', exist_ok=True)
|
||||
async with await anyio.open_file('./tmp/user_message.txt', 'w', encoding='utf-8') as f:
|
||||
await f.write(text_to_save)
|
||||
|
||||
# save pure clickable elements to a file
|
||||
if all_elements_state.dom_state._root:
|
||||
async with await anyio.open_file('./tmp/simplified_element_tree.json', 'w', encoding='utf-8') as f:
|
||||
await f.write(json.dumps(all_elements_state.dom_state._root.__json__(), indent=2))
|
||||
|
||||
async with await anyio.open_file('./tmp/original_element_tree.json', 'w', encoding='utf-8') as f:
|
||||
await f.write(json.dumps(all_elements_state.dom_state._root.original_node.__json__(), indent=2))
|
||||
|
||||
# copy the user message to the clipboard
|
||||
# pyperclip.copy(text_to_save)
|
||||
|
||||
encoding = tiktoken.encoding_for_model('gpt-4.1-mini')
|
||||
token_count = len(encoding.encode(text_to_save))
|
||||
print(f'Token count: {token_count}')
|
||||
|
||||
print('User message written to ./tmp/user_message.txt')
|
||||
print('Element tree written to ./tmp/simplified_element_tree.json')
|
||||
print('Original element tree written to ./tmp/original_element_tree.json')
|
||||
|
||||
# Save timing information
|
||||
timing_text = '🔍 DOM EXTRACTION PERFORMANCE ANALYSIS\n'
|
||||
timing_text += f'{"=" * 50}\n\n'
|
||||
timing_text += f'📄 Website: {website}\n'
|
||||
timing_text += f'📊 Total Elements: {total_elements}\n'
|
||||
timing_text += f'🎯 Token Count: {token_count}\n\n'
|
||||
|
||||
timing_text += '⏱️ TIMING BREAKDOWN:\n'
|
||||
timing_text += f'{"─" * 30}\n'
|
||||
for key, value in all_timing.items():
|
||||
timing_text += f'{key:<35}: {value * 1000:>8.2f} ms\n'
|
||||
|
||||
# Calculate percentages
|
||||
total_time = all_timing.get('get_state_summary_total', 0)
|
||||
if total_time > 0 and total_elements > 0:
|
||||
timing_text += '\n📈 PERCENTAGE BREAKDOWN:\n'
|
||||
timing_text += f'{"─" * 30}\n'
|
||||
for key, value in all_timing.items():
|
||||
if key != 'get_state_summary_total':
|
||||
percentage = (value / total_time) * 100
|
||||
timing_text += f'{key:<35}: {percentage:>7.1f}%\n'
|
||||
|
||||
timing_text += '\n🎯 CLICKABLE DETECTION ANALYSIS:\n'
|
||||
timing_text += f'{"─" * 35}\n'
|
||||
clickable_time = all_timing.get('clickable_detection_time', 0)
|
||||
if clickable_time > 0 and total_elements > 0:
|
||||
avg_per_element = (clickable_time / total_elements) * 1000000 # microseconds
|
||||
timing_text += f'Total clickable detection time: {clickable_time * 1000:.2f} ms\n'
|
||||
timing_text += f'Average per element: {avg_per_element:.2f} μs\n'
|
||||
timing_text += f'Clickable detection calls: ~{total_elements} (approx)\n'
|
||||
|
||||
async with await anyio.open_file('./tmp/timing_analysis.txt', 'w', encoding='utf-8') as f:
|
||||
await f.write(timing_text)
|
||||
|
||||
print('Timing analysis written to ./tmp/timing_analysis.txt')
|
||||
|
||||
# also save all_elements_state.element_tree.clickable_elements_to_string() to a file
|
||||
# with open('./tmp/clickable_elements.json', 'w', encoding='utf-8') as f:
|
||||
# f.write(json.dumps(all_elements_state.element_tree.__json__(), indent=2))
|
||||
# print('Clickable elements written to ./tmp/clickable_elements.json')
|
||||
|
||||
website_list = get_website_list_for_prompt()
|
||||
answer = input(
|
||||
"🎮 Enter: element index | 'index' click (clickable) | 'index,text' input | 'c,index' copy | Enter re-run | 'n' next | 'q' quit: "
|
||||
)
|
||||
|
||||
if answer.lower() == 'q':
|
||||
return # Exit completely
|
||||
elif answer.lower() == 'n':
|
||||
print('Moving to next website...')
|
||||
current_website_index += 1
|
||||
break # Break inner loop to go to next website
|
||||
elif answer.strip() == '':
|
||||
print('Re-running extraction on current page state...')
|
||||
continue # Continue inner loop to re-extract DOM without reloading page
|
||||
elif answer.strip().isdigit():
|
||||
# Click element format: index
|
||||
try:
|
||||
clicked_index = int(answer)
|
||||
if clicked_index in selector_map:
|
||||
element_node = selector_map[clicked_index]
|
||||
print(f'Clicking element {clicked_index}: {element_node.tag_name}')
|
||||
event = browser_session.event_bus.dispatch(ClickElementEvent(node=element_node))
|
||||
await event
|
||||
print('Click successful.')
|
||||
except ValueError:
|
||||
print(f"Invalid input: '{answer}'. Enter an index, 'index,text', 'c,index', or 'q'.")
|
||||
continue
|
||||
|
||||
try:
|
||||
if answer.lower().startswith('c,'):
|
||||
# Copy element JSON format: c,index
|
||||
parts = answer.split(',', 1)
|
||||
if len(parts) == 2:
|
||||
try:
|
||||
target_index = int(parts[1].strip())
|
||||
if target_index in selector_map:
|
||||
element_node = selector_map[target_index]
|
||||
element_json = json.dumps(element_node.__json__(), indent=2, default=str)
|
||||
pyperclip.copy(element_json)
|
||||
print(f'Copied element {target_index} JSON to clipboard: {element_node.tag_name}')
|
||||
else:
|
||||
print(f'Invalid index: {target_index}')
|
||||
except ValueError:
|
||||
print(f'Invalid index format: {parts[1]}')
|
||||
else:
|
||||
print("Invalid input format. Use 'c,index'.")
|
||||
elif ',' in answer:
|
||||
# Input text format: index,text
|
||||
parts = answer.split(',', 1)
|
||||
if len(parts) == 2:
|
||||
try:
|
||||
target_index = int(parts[0].strip())
|
||||
text_to_input = parts[1]
|
||||
if target_index in selector_map:
|
||||
element_node = selector_map[target_index]
|
||||
print(
|
||||
f"Inputting text '{text_to_input}' into element {target_index}: {element_node.tag_name}"
|
||||
)
|
||||
|
||||
event = await browser_session.event_bus.dispatch(
|
||||
TypeTextEvent(node=element_node, text=text_to_input)
|
||||
)
|
||||
|
||||
print('Input successful.')
|
||||
else:
|
||||
print(f'Invalid index: {target_index}')
|
||||
except ValueError:
|
||||
print(f'Invalid index format: {parts[0]}')
|
||||
else:
|
||||
print("Invalid input format. Use 'index,text'.")
|
||||
|
||||
except Exception as action_e:
|
||||
print(f'Action failed: {action_e}')
|
||||
|
||||
# No explicit highlight removal here, get_state handles it at the start of the loop
|
||||
|
||||
except Exception as e:
|
||||
print(f'Error in loop: {e}')
|
||||
# Optionally add a small delay before retrying
|
||||
await asyncio.sleep(1)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(test_focus_vs_all_elements())
|
||||
# asyncio.run(test_process_html_file()) # Commented out the other test
|
||||
@@ -0,0 +1,32 @@
|
||||
from browser_use import Agent
|
||||
from browser_use.browser import BrowserProfile, BrowserSession
|
||||
from browser_use.browser.types import ViewportSize
|
||||
from browser_use.llm import ChatAzureOpenAI
|
||||
|
||||
# Initialize the Azure OpenAI client
|
||||
llm = ChatAzureOpenAI(
|
||||
model='gpt-4.1-mini',
|
||||
)
|
||||
|
||||
|
||||
TASK = """
|
||||
Go to https://browser-use.github.io/stress-tests/challenges/react-native-web-form.html and complete the React Native Web form by filling in all required fields and submitting.
|
||||
"""
|
||||
|
||||
|
||||
async def main():
|
||||
browser = BrowserSession(
|
||||
browser_profile=BrowserProfile(
|
||||
window_size=ViewportSize(width=1100, height=1000),
|
||||
)
|
||||
)
|
||||
|
||||
agent = Agent(task=TASK, llm=llm)
|
||||
|
||||
await agent.run()
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
import asyncio
|
||||
|
||||
asyncio.run(main())
|
||||
+199
@@ -0,0 +1,199 @@
|
||||
from browser_use.dom.views import EnhancedDOMTreeNode, NodeType
|
||||
|
||||
|
||||
class ClickableElementDetector:
|
||||
@staticmethod
|
||||
def is_interactive(node: EnhancedDOMTreeNode) -> bool:
|
||||
"""Check if this node is clickable/interactive using enhanced scoring."""
|
||||
|
||||
# Skip non-element nodes
|
||||
if node.node_type != NodeType.ELEMENT_NODE:
|
||||
return False
|
||||
|
||||
# # if ax ignored skip
|
||||
# if node.ax_node and node.ax_node.ignored:
|
||||
# return False
|
||||
|
||||
# remove html and body nodes
|
||||
if node.tag_name in {'html', 'body'}:
|
||||
return False
|
||||
|
||||
# IFRAME elements should be interactive if they're large enough to potentially need scrolling
|
||||
# Small iframes (< 100px width or height) are unlikely to have scrollable content
|
||||
if node.tag_name and node.tag_name.upper() == 'IFRAME' or node.tag_name.upper() == 'FRAME':
|
||||
if node.snapshot_node and node.snapshot_node.bounds:
|
||||
width = node.snapshot_node.bounds.width
|
||||
height = node.snapshot_node.bounds.height
|
||||
# Only include iframes larger than 100x100px
|
||||
if width > 100 and height > 100:
|
||||
return True
|
||||
|
||||
# RELAXED SIZE CHECK: Allow all elements including size 0 (they might be interactive overlays, etc.)
|
||||
# Note: Size 0 elements can still be interactive (e.g., invisible clickable overlays)
|
||||
# Visibility is determined separately by CSS styles, not just bounding box size
|
||||
|
||||
# SEARCH ELEMENT DETECTION: Check for search-related classes and attributes
|
||||
if node.attributes:
|
||||
search_indicators = {
|
||||
'search',
|
||||
'magnify',
|
||||
'glass',
|
||||
'lookup',
|
||||
'find',
|
||||
'query',
|
||||
'search-icon',
|
||||
'search-btn',
|
||||
'search-button',
|
||||
'searchbox',
|
||||
}
|
||||
|
||||
# Check class names for search indicators
|
||||
class_list = node.attributes.get('class', '').lower().split()
|
||||
if any(indicator in ' '.join(class_list) for indicator in search_indicators):
|
||||
return True
|
||||
|
||||
# Check id for search indicators
|
||||
element_id = node.attributes.get('id', '').lower()
|
||||
if any(indicator in element_id for indicator in search_indicators):
|
||||
return True
|
||||
|
||||
# Check data attributes for search functionality
|
||||
for attr_name, attr_value in node.attributes.items():
|
||||
if attr_name.startswith('data-') and any(indicator in attr_value.lower() for indicator in search_indicators):
|
||||
return True
|
||||
|
||||
# Enhanced accessibility property checks - direct clear indicators only
|
||||
if node.ax_node and node.ax_node.properties:
|
||||
for prop in node.ax_node.properties:
|
||||
try:
|
||||
# aria disabled
|
||||
if prop.name == 'disabled' and prop.value:
|
||||
return False
|
||||
|
||||
# aria hidden
|
||||
if prop.name == 'hidden' and prop.value:
|
||||
return False
|
||||
|
||||
# Direct interactiveness indicators
|
||||
if prop.name in ['focusable', 'editable', 'settable'] and prop.value:
|
||||
return True
|
||||
|
||||
# Interactive state properties (presence indicates interactive widget)
|
||||
if prop.name in ['checked', 'expanded', 'pressed', 'selected']:
|
||||
# These properties only exist on interactive elements
|
||||
return True
|
||||
|
||||
# Form-related interactiveness
|
||||
if prop.name in ['required', 'autocomplete'] and prop.value:
|
||||
return True
|
||||
|
||||
# Elements with keyboard shortcuts are interactive
|
||||
if prop.name == 'keyshortcuts' and prop.value:
|
||||
return True
|
||||
except (AttributeError, ValueError):
|
||||
# Skip properties we can't process
|
||||
continue
|
||||
|
||||
# ENHANCED TAG CHECK: Include truly interactive elements
|
||||
# Note: 'label' removed - labels are handled by other attribute checks below - other wise labels with "for" attribute can destroy the real clickable element on apartments.com
|
||||
interactive_tags = {
|
||||
'button',
|
||||
'input',
|
||||
'select',
|
||||
'textarea',
|
||||
'a',
|
||||
'details',
|
||||
'summary',
|
||||
'option',
|
||||
'optgroup',
|
||||
}
|
||||
if node.tag_name in interactive_tags:
|
||||
return True
|
||||
|
||||
# SVG elements need special handling - only interactive if they have explicit handlers
|
||||
# svg_tags = {'svg', 'path', 'circle', 'rect', 'polygon', 'ellipse', 'line', 'polyline', 'g'}
|
||||
# if node.tag_name in svg_tags:
|
||||
# # Only consider SVG elements interactive if they have:
|
||||
# # 1. Explicit event handlers
|
||||
# # 2. Interactive role attributes
|
||||
# # 3. Cursor pointer style
|
||||
# if node.attributes:
|
||||
# # Check for event handlers
|
||||
# if any(attr.startswith('on') for attr in node.attributes):
|
||||
# return True
|
||||
# # Check for interactive roles
|
||||
# if node.attributes.get('role') in {'button', 'link', 'menuitem'}:
|
||||
# return True
|
||||
# # Check for cursor pointer (indicating clickability)
|
||||
# if node.attributes.get('style') and 'cursor: pointer' in node.attributes.get('style', ''):
|
||||
# return True
|
||||
# # Otherwise, SVG elements are decorative
|
||||
# return False
|
||||
|
||||
# Tertiary check: elements with interactive attributes
|
||||
if node.attributes:
|
||||
# Check for event handlers or interactive attributes
|
||||
interactive_attributes = {'onclick', 'onmousedown', 'onmouseup', 'onkeydown', 'onkeyup', 'tabindex'}
|
||||
if any(attr in node.attributes for attr in interactive_attributes):
|
||||
return True
|
||||
|
||||
# Check for interactive ARIA roles
|
||||
if 'role' in node.attributes:
|
||||
interactive_roles = {
|
||||
'button',
|
||||
'link',
|
||||
'menuitem',
|
||||
'option',
|
||||
'radio',
|
||||
'checkbox',
|
||||
'tab',
|
||||
'textbox',
|
||||
'combobox',
|
||||
'slider',
|
||||
'spinbutton',
|
||||
'search',
|
||||
'searchbox',
|
||||
}
|
||||
if node.attributes['role'] in interactive_roles:
|
||||
return True
|
||||
|
||||
# Quaternary check: accessibility tree roles
|
||||
if node.ax_node and node.ax_node.role:
|
||||
interactive_ax_roles = {
|
||||
'button',
|
||||
'link',
|
||||
'menuitem',
|
||||
'option',
|
||||
'radio',
|
||||
'checkbox',
|
||||
'tab',
|
||||
'textbox',
|
||||
'combobox',
|
||||
'slider',
|
||||
'spinbutton',
|
||||
'listbox',
|
||||
'search',
|
||||
'searchbox',
|
||||
}
|
||||
if node.ax_node.role in interactive_ax_roles:
|
||||
return True
|
||||
|
||||
# ICON AND SMALL ELEMENT CHECK: Elements that might be icons
|
||||
if (
|
||||
node.snapshot_node
|
||||
and node.snapshot_node.bounds
|
||||
and 10 <= node.snapshot_node.bounds.width <= 50 # Icon-sized elements
|
||||
and 10 <= node.snapshot_node.bounds.height <= 50
|
||||
):
|
||||
# Check if this small element has interactive properties
|
||||
if node.attributes:
|
||||
# Small elements with these attributes are likely interactive icons
|
||||
icon_attributes = {'class', 'role', 'onclick', 'data-action', 'aria-label'}
|
||||
if any(attr in node.attributes for attr in icon_attributes):
|
||||
return True
|
||||
|
||||
# Final fallback: cursor style indicates interactivity (for cases Chrome missed)
|
||||
if node.snapshot_node and node.snapshot_node.cursor_style and node.snapshot_node.cursor_style == 'pointer':
|
||||
return True
|
||||
|
||||
return False
|
||||
@@ -0,0 +1,197 @@
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
|
||||
from browser_use.dom.views import SimplifiedNode
|
||||
|
||||
"""
|
||||
Helper class for maintaining a union of rectangles (used for order of elements calculation)
|
||||
"""
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Rect:
|
||||
"""Closed axis-aligned rectangle with (x1,y1) bottom-left, (x2,y2) top-right."""
|
||||
|
||||
x1: float
|
||||
y1: float
|
||||
x2: float
|
||||
y2: float
|
||||
|
||||
def __post_init__(self):
|
||||
if not (self.x1 <= self.x2 and self.y1 <= self.y2):
|
||||
return False
|
||||
|
||||
# --- fast relations ----------------------------------------------------
|
||||
def area(self) -> float:
|
||||
return (self.x2 - self.x1) * (self.y2 - self.y1)
|
||||
|
||||
def intersects(self, other: 'Rect') -> bool:
|
||||
return not (self.x2 <= other.x1 or other.x2 <= self.x1 or self.y2 <= other.y1 or other.y2 <= self.y1)
|
||||
|
||||
def contains(self, other: 'Rect') -> bool:
|
||||
return self.x1 <= other.x1 and self.y1 <= other.y1 and self.x2 >= other.x2 and self.y2 >= other.y2
|
||||
|
||||
|
||||
class RectUnionPure:
|
||||
"""
|
||||
Maintains a *disjoint* set of rectangles.
|
||||
No external dependencies - fine for a few thousand rectangles.
|
||||
"""
|
||||
|
||||
__slots__ = ('_rects',)
|
||||
|
||||
def __init__(self):
|
||||
self._rects: list[Rect] = []
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
def _split_diff(self, a: Rect, b: Rect) -> list[Rect]:
|
||||
r"""
|
||||
Return list of up to 4 rectangles = a \ b.
|
||||
Assumes a intersects b.
|
||||
"""
|
||||
parts = []
|
||||
|
||||
# Bottom slice
|
||||
if a.y1 < b.y1:
|
||||
parts.append(Rect(a.x1, a.y1, a.x2, b.y1))
|
||||
# Top slice
|
||||
if b.y2 < a.y2:
|
||||
parts.append(Rect(a.x1, b.y2, a.x2, a.y2))
|
||||
|
||||
# Middle (vertical) strip: y overlap is [max(a.y1,b.y1), min(a.y2,b.y2)]
|
||||
y_lo = max(a.y1, b.y1)
|
||||
y_hi = min(a.y2, b.y2)
|
||||
|
||||
# Left slice
|
||||
if a.x1 < b.x1:
|
||||
parts.append(Rect(a.x1, y_lo, b.x1, y_hi))
|
||||
# Right slice
|
||||
if b.x2 < a.x2:
|
||||
parts.append(Rect(b.x2, y_lo, a.x2, y_hi))
|
||||
|
||||
return parts
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
def contains(self, r: Rect) -> bool:
|
||||
"""
|
||||
True iff r is fully covered by the current union.
|
||||
"""
|
||||
if not self._rects:
|
||||
return False
|
||||
|
||||
stack = [r]
|
||||
for s in self._rects:
|
||||
new_stack = []
|
||||
for piece in stack:
|
||||
if s.contains(piece):
|
||||
# piece completely gone
|
||||
continue
|
||||
if piece.intersects(s):
|
||||
new_stack.extend(self._split_diff(piece, s))
|
||||
else:
|
||||
new_stack.append(piece)
|
||||
if not new_stack: # everything eaten – covered
|
||||
return True
|
||||
stack = new_stack
|
||||
return False # something survived
|
||||
|
||||
# -----------------------------------------------------------------
|
||||
def add(self, r: Rect) -> bool:
|
||||
"""
|
||||
Insert r unless it is already covered.
|
||||
Returns True if the union grew.
|
||||
"""
|
||||
if self.contains(r):
|
||||
return False
|
||||
|
||||
pending = [r]
|
||||
i = 0
|
||||
while i < len(self._rects):
|
||||
s = self._rects[i]
|
||||
new_pending = []
|
||||
changed = False
|
||||
for piece in pending:
|
||||
if piece.intersects(s):
|
||||
new_pending.extend(self._split_diff(piece, s))
|
||||
changed = True
|
||||
else:
|
||||
new_pending.append(piece)
|
||||
pending = new_pending
|
||||
if changed:
|
||||
# s unchanged; proceed with next existing rectangle
|
||||
i += 1
|
||||
else:
|
||||
i += 1
|
||||
|
||||
# Any left‑over pieces are new, non‑overlapping areas
|
||||
self._rects.extend(pending)
|
||||
return True
|
||||
|
||||
|
||||
class PaintOrderRemover:
|
||||
"""
|
||||
Calculates which elements should be removed based on the paint order parameter.
|
||||
"""
|
||||
|
||||
def __init__(self, root: SimplifiedNode):
|
||||
self.root = root
|
||||
|
||||
def calculate_paint_order(self) -> None:
|
||||
all_simplified_nodes_with_paint_order: list[SimplifiedNode] = []
|
||||
|
||||
def collect_paint_order(node: SimplifiedNode) -> None:
|
||||
if (
|
||||
node.original_node.snapshot_node
|
||||
and node.original_node.snapshot_node.paint_order is not None
|
||||
and node.original_node.snapshot_node.bounds is not None
|
||||
):
|
||||
all_simplified_nodes_with_paint_order.append(node)
|
||||
|
||||
for child in node.children:
|
||||
collect_paint_order(child)
|
||||
|
||||
collect_paint_order(self.root)
|
||||
|
||||
grouped_by_paint_order: defaultdict[int, list[SimplifiedNode]] = defaultdict(list)
|
||||
|
||||
for node in all_simplified_nodes_with_paint_order:
|
||||
if node.original_node.snapshot_node and node.original_node.snapshot_node.paint_order is not None:
|
||||
grouped_by_paint_order[node.original_node.snapshot_node.paint_order].append(node)
|
||||
|
||||
rect_union = RectUnionPure()
|
||||
|
||||
for paint_order, nodes in sorted(grouped_by_paint_order.items(), key=lambda x: -x[0]):
|
||||
rects_to_add = []
|
||||
|
||||
for node in nodes:
|
||||
if not node.original_node.snapshot_node or not node.original_node.snapshot_node.bounds:
|
||||
continue # shouldn't happen by how we filter them out in the first place
|
||||
|
||||
rect = Rect(
|
||||
x1=node.original_node.snapshot_node.bounds.x,
|
||||
y1=node.original_node.snapshot_node.bounds.y,
|
||||
x2=node.original_node.snapshot_node.bounds.x + node.original_node.snapshot_node.bounds.width,
|
||||
y2=node.original_node.snapshot_node.bounds.y + node.original_node.snapshot_node.bounds.height,
|
||||
)
|
||||
|
||||
if rect_union.contains(rect):
|
||||
node.ignored_by_paint_order = True
|
||||
|
||||
# don't add to the nodes if opacity is less then 0.95 or background-color is transparent
|
||||
if (
|
||||
node.original_node.snapshot_node.computed_styles
|
||||
and node.original_node.snapshot_node.computed_styles.get('background-color', 'rgba(0, 0, 0, 0)')
|
||||
== 'rgba(0, 0, 0, 0)'
|
||||
) or (
|
||||
node.original_node.snapshot_node.computed_styles
|
||||
and float(node.original_node.snapshot_node.computed_styles.get('opacity', '1'))
|
||||
< 0.8 # this is highly vibes based number
|
||||
):
|
||||
continue
|
||||
|
||||
rects_to_add.append(rect)
|
||||
|
||||
for rect in rects_to_add:
|
||||
rect_union.add(rect)
|
||||
|
||||
return None
|
||||
@@ -0,0 +1,954 @@
|
||||
# @file purpose: Serializes enhanced DOM trees to string format for LLM consumption
|
||||
|
||||
from typing import Any
|
||||
|
||||
from browser_use.dom.serializer.clickable_elements import ClickableElementDetector
|
||||
from browser_use.dom.serializer.paint_order import PaintOrderRemover
|
||||
from browser_use.dom.utils import cap_text_length
|
||||
from browser_use.dom.views import (
|
||||
DOMRect,
|
||||
DOMSelectorMap,
|
||||
EnhancedDOMTreeNode,
|
||||
NodeType,
|
||||
PropagatingBounds,
|
||||
SerializedDOMState,
|
||||
SimplifiedNode,
|
||||
)
|
||||
|
||||
DISABLED_ELEMENTS = {'style', 'script', 'head', 'meta', 'link', 'title'}
|
||||
|
||||
|
||||
class DOMTreeSerializer:
|
||||
"""Serializes enhanced DOM trees to string format."""
|
||||
|
||||
# Configuration - elements that propagate bounds to their children
|
||||
PROPAGATING_ELEMENTS = [
|
||||
{'tag': 'a', 'role': None}, # Any <a> tag
|
||||
{'tag': 'button', 'role': None}, # Any <button> tag
|
||||
{'tag': 'div', 'role': 'button'}, # <div role="button">
|
||||
{'tag': 'div', 'role': 'combobox'}, # <div role="combobox"> - dropdowns/selects
|
||||
{'tag': 'span', 'role': 'button'}, # <span role="button">
|
||||
{'tag': 'span', 'role': 'combobox'}, # <span role="combobox">
|
||||
{'tag': 'input', 'role': 'combobox'}, # <input role="combobox"> - autocomplete inputs
|
||||
{'tag': 'input', 'role': 'combobox'}, # <input type="text"> - text inputs with suggestions
|
||||
# {'tag': 'div', 'role': 'link'}, # <div role="link">
|
||||
# {'tag': 'span', 'role': 'link'}, # <span role="link">
|
||||
]
|
||||
DEFAULT_CONTAINMENT_THRESHOLD = 0.99 # 99% containment by default
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
root_node: EnhancedDOMTreeNode,
|
||||
previous_cached_state: SerializedDOMState | None = None,
|
||||
enable_bbox_filtering: bool = True,
|
||||
containment_threshold: float | None = None,
|
||||
paint_order_filtering: bool = True,
|
||||
):
|
||||
self.root_node = root_node
|
||||
self._interactive_counter = 1
|
||||
self._selector_map: DOMSelectorMap = {}
|
||||
self._previous_cached_selector_map = previous_cached_state.selector_map if previous_cached_state else None
|
||||
# Add timing tracking
|
||||
self.timing_info: dict[str, float] = {}
|
||||
# Cache for clickable element detection to avoid redundant calls
|
||||
self._clickable_cache: dict[int, bool] = {}
|
||||
# Bounding box filtering configuration
|
||||
self.enable_bbox_filtering = enable_bbox_filtering
|
||||
self.containment_threshold = containment_threshold or self.DEFAULT_CONTAINMENT_THRESHOLD
|
||||
# Paint order filtering configuration
|
||||
self.paint_order_filtering = paint_order_filtering
|
||||
|
||||
def _safe_parse_number(self, value_str: str, default: float) -> float:
|
||||
"""Parse string to float, handling negatives and decimals."""
|
||||
try:
|
||||
return float(value_str)
|
||||
except (ValueError, TypeError):
|
||||
return default
|
||||
|
||||
def _safe_parse_optional_number(self, value_str: str | None) -> float | None:
|
||||
"""Parse string to float, returning None for invalid values."""
|
||||
if not value_str:
|
||||
return None
|
||||
try:
|
||||
return float(value_str)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
def serialize_accessible_elements(self) -> tuple[SerializedDOMState, dict[str, float]]:
|
||||
import time
|
||||
|
||||
start_total = time.time()
|
||||
|
||||
# Reset state
|
||||
self._interactive_counter = 1
|
||||
self._selector_map = {}
|
||||
self._semantic_groups = []
|
||||
self._clickable_cache = {} # Clear cache for new serialization
|
||||
|
||||
# Step 1: Create simplified tree (includes clickable element detection)
|
||||
start_step1 = time.time()
|
||||
simplified_tree = self._create_simplified_tree(self.root_node)
|
||||
end_step1 = time.time()
|
||||
self.timing_info['create_simplified_tree'] = end_step1 - start_step1
|
||||
|
||||
# Step 2: Remove elements based on paint order
|
||||
start_step3 = time.time()
|
||||
if self.paint_order_filtering and simplified_tree:
|
||||
PaintOrderRemover(simplified_tree).calculate_paint_order()
|
||||
end_step3 = time.time()
|
||||
self.timing_info['calculate_paint_order'] = end_step3 - start_step3
|
||||
|
||||
# Step 3: Optimize tree (remove unnecessary parents)
|
||||
start_step2 = time.time()
|
||||
optimized_tree = self._optimize_tree(simplified_tree)
|
||||
end_step2 = time.time()
|
||||
self.timing_info['optimize_tree'] = end_step2 - start_step2
|
||||
|
||||
# Step 3: Apply bounding box filtering (NEW)
|
||||
if self.enable_bbox_filtering and optimized_tree:
|
||||
start_step3 = time.time()
|
||||
filtered_tree = self._apply_bounding_box_filtering(optimized_tree)
|
||||
end_step3 = time.time()
|
||||
self.timing_info['bbox_filtering'] = end_step3 - start_step3
|
||||
else:
|
||||
filtered_tree = optimized_tree
|
||||
|
||||
# Step 4: Assign interactive indices to clickable elements
|
||||
start_step4 = time.time()
|
||||
self._assign_interactive_indices_and_mark_new_nodes(filtered_tree)
|
||||
end_step4 = time.time()
|
||||
self.timing_info['assign_interactive_indices'] = end_step4 - start_step4
|
||||
|
||||
end_total = time.time()
|
||||
self.timing_info['serialize_accessible_elements_total'] = end_total - start_total
|
||||
|
||||
return SerializedDOMState(_root=filtered_tree, selector_map=self._selector_map), self.timing_info
|
||||
|
||||
def _add_compound_components(self, simplified: SimplifiedNode, node: EnhancedDOMTreeNode) -> None:
|
||||
"""Enhance compound controls with information from their child components."""
|
||||
# Only process elements that might have compound components
|
||||
if node.tag_name not in ['input', 'select', 'details', 'audio', 'video']:
|
||||
return
|
||||
|
||||
# For input elements, check for compound input types
|
||||
if node.tag_name == 'input':
|
||||
if not node.attributes or node.attributes.get('type') not in [
|
||||
'date',
|
||||
'time',
|
||||
'datetime-local',
|
||||
'month',
|
||||
'week',
|
||||
'range',
|
||||
'number',
|
||||
'color',
|
||||
'file',
|
||||
]:
|
||||
return
|
||||
# For other elements, check if they have AX child indicators
|
||||
elif not node.ax_node or not node.ax_node.child_ids:
|
||||
return
|
||||
|
||||
# Add compound component information based on element type
|
||||
element_type = node.tag_name
|
||||
input_type = node.attributes.get('type', '') if node.attributes else ''
|
||||
|
||||
if element_type == 'input':
|
||||
if input_type == 'date':
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'spinbutton', 'name': 'Day', 'valuemin': 1, 'valuemax': 31, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Month', 'valuemin': 1, 'valuemax': 12, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Year', 'valuemin': 1, 'valuemax': 275760, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'time':
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'spinbutton', 'name': 'Hour', 'valuemin': 0, 'valuemax': 23, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Minute', 'valuemin': 0, 'valuemax': 59, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'datetime-local':
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'spinbutton', 'name': 'Day', 'valuemin': 1, 'valuemax': 31, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Month', 'valuemin': 1, 'valuemax': 12, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Year', 'valuemin': 1, 'valuemax': 275760, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Hour', 'valuemin': 0, 'valuemax': 23, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Minute', 'valuemin': 0, 'valuemax': 59, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'month':
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'spinbutton', 'name': 'Month', 'valuemin': 1, 'valuemax': 12, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Year', 'valuemin': 1, 'valuemax': 275760, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'week':
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'spinbutton', 'name': 'Week', 'valuemin': 1, 'valuemax': 53, 'valuenow': None},
|
||||
{'role': 'spinbutton', 'name': 'Year', 'valuemin': 1, 'valuemax': 275760, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'range':
|
||||
# Range slider with value indicator
|
||||
min_val = node.attributes.get('min', '0') if node.attributes else '0'
|
||||
max_val = node.attributes.get('max', '100') if node.attributes else '100'
|
||||
|
||||
node._compound_children.append(
|
||||
{
|
||||
'role': 'slider',
|
||||
'name': 'Value',
|
||||
'valuemin': self._safe_parse_number(min_val, 0.0),
|
||||
'valuemax': self._safe_parse_number(max_val, 100.0),
|
||||
'valuenow': None,
|
||||
}
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'number':
|
||||
# Number input with increment/decrement buttons
|
||||
min_val = node.attributes.get('min') if node.attributes else None
|
||||
max_val = node.attributes.get('max') if node.attributes else None
|
||||
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'button', 'name': 'Increment', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{'role': 'button', 'name': 'Decrement', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{
|
||||
'role': 'textbox',
|
||||
'name': 'Value',
|
||||
'valuemin': self._safe_parse_optional_number(min_val),
|
||||
'valuemax': self._safe_parse_optional_number(max_val),
|
||||
'valuenow': None,
|
||||
},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'color':
|
||||
# Color picker with components
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'textbox', 'name': 'Hex Value', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{'role': 'button', 'name': 'Color Picker', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
elif input_type == 'file':
|
||||
# File input with browse button
|
||||
multiple = 'multiple' in node.attributes if node.attributes else False
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'button', 'name': 'Browse Files', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{
|
||||
'role': 'textbox',
|
||||
'name': f'{"Files" if multiple else "File"} Selected',
|
||||
'valuemin': None,
|
||||
'valuemax': None,
|
||||
'valuenow': None,
|
||||
},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
|
||||
elif element_type == 'select':
|
||||
# Select dropdown with option list and detailed option information
|
||||
base_components = [
|
||||
{'role': 'button', 'name': 'Dropdown Toggle', 'valuemin': None, 'valuemax': None, 'valuenow': None}
|
||||
]
|
||||
|
||||
# Extract option information from child nodes
|
||||
options_info = self._extract_select_options(node)
|
||||
if options_info:
|
||||
options_component = {
|
||||
'role': 'listbox',
|
||||
'name': 'Options',
|
||||
'valuemin': None,
|
||||
'valuemax': None,
|
||||
'valuenow': None,
|
||||
'options_count': options_info['count'],
|
||||
'first_options': options_info['first_options'],
|
||||
}
|
||||
if options_info['format_hint']:
|
||||
options_component['format_hint'] = options_info['format_hint']
|
||||
base_components.append(options_component)
|
||||
else:
|
||||
base_components.append(
|
||||
{'role': 'listbox', 'name': 'Options', 'valuemin': None, 'valuemax': None, 'valuenow': None}
|
||||
)
|
||||
|
||||
node._compound_children.extend(base_components)
|
||||
simplified.is_compound_component = True
|
||||
|
||||
elif element_type == 'details':
|
||||
# Details/summary disclosure widget
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'button', 'name': 'Toggle Disclosure', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{'role': 'region', 'name': 'Content Area', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
|
||||
elif element_type == 'audio':
|
||||
# Audio player controls
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'button', 'name': 'Play/Pause', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{'role': 'slider', 'name': 'Progress', 'valuemin': 0, 'valuemax': 100, 'valuenow': None},
|
||||
{'role': 'button', 'name': 'Mute', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{'role': 'slider', 'name': 'Volume', 'valuemin': 0, 'valuemax': 100, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
|
||||
elif element_type == 'video':
|
||||
# Video player controls
|
||||
node._compound_children.extend(
|
||||
[
|
||||
{'role': 'button', 'name': 'Play/Pause', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{'role': 'slider', 'name': 'Progress', 'valuemin': 0, 'valuemax': 100, 'valuenow': None},
|
||||
{'role': 'button', 'name': 'Mute', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
{'role': 'slider', 'name': 'Volume', 'valuemin': 0, 'valuemax': 100, 'valuenow': None},
|
||||
{'role': 'button', 'name': 'Fullscreen', 'valuemin': None, 'valuemax': None, 'valuenow': None},
|
||||
]
|
||||
)
|
||||
simplified.is_compound_component = True
|
||||
|
||||
def _extract_select_options(self, select_node: EnhancedDOMTreeNode) -> dict[str, Any] | None:
|
||||
"""Extract option information from a select element."""
|
||||
if not select_node.children:
|
||||
return None
|
||||
|
||||
options = []
|
||||
option_values = []
|
||||
|
||||
def extract_options_recursive(node: EnhancedDOMTreeNode) -> None:
|
||||
"""Recursively extract option elements, including from optgroups."""
|
||||
if node.tag_name.lower() == 'option':
|
||||
# Extract option text and value
|
||||
option_text = ''
|
||||
option_value = ''
|
||||
|
||||
# Get value attribute if present
|
||||
if node.attributes and 'value' in node.attributes:
|
||||
option_value = str(node.attributes['value']).strip()
|
||||
|
||||
# Get text content from direct child text nodes only to avoid duplication
|
||||
def get_direct_text_content(n: EnhancedDOMTreeNode) -> str:
|
||||
text = ''
|
||||
for child in n.children:
|
||||
if child.node_type == NodeType.TEXT_NODE and child.node_value:
|
||||
text += child.node_value.strip() + ' '
|
||||
return text.strip()
|
||||
|
||||
option_text = get_direct_text_content(node)
|
||||
|
||||
# Use text as value if no explicit value
|
||||
if not option_value and option_text:
|
||||
option_value = option_text
|
||||
|
||||
if option_text or option_value:
|
||||
options.append({'text': option_text, 'value': option_value})
|
||||
option_values.append(option_value)
|
||||
|
||||
elif node.tag_name.lower() == 'optgroup':
|
||||
# Process optgroup children
|
||||
for child in node.children:
|
||||
extract_options_recursive(child)
|
||||
else:
|
||||
# Process other children that might contain options
|
||||
for child in node.children:
|
||||
extract_options_recursive(child)
|
||||
|
||||
# Extract all options from select children
|
||||
for child in select_node.children:
|
||||
extract_options_recursive(child)
|
||||
|
||||
if not options:
|
||||
return None
|
||||
|
||||
# Prepare first 4 options for display
|
||||
first_options = []
|
||||
for option in options[:4]:
|
||||
if option['text'] and option['value'] and option['text'] != option['value']:
|
||||
# Limit individual option text to avoid overly long attributes
|
||||
text = option['text'][:20] + ('...' if len(option['text']) > 20 else '')
|
||||
value = option['value'][:10] + ('...' if len(option['value']) > 10 else '')
|
||||
first_options.append(f'{text} ({value})')
|
||||
elif option['text']:
|
||||
text = option['text'][:25] + ('...' if len(option['text']) > 25 else '')
|
||||
first_options.append(text)
|
||||
elif option['value']:
|
||||
value = option['value'][:25] + ('...' if len(option['value']) > 25 else '')
|
||||
first_options.append(value)
|
||||
|
||||
# Try to infer format hint from option values
|
||||
format_hint = None
|
||||
if len(option_values) >= 2:
|
||||
# Check for common patterns
|
||||
if all(val.isdigit() for val in option_values[:5] if val):
|
||||
format_hint = 'numeric'
|
||||
elif all(len(val) == 2 and val.isupper() for val in option_values[:5] if val):
|
||||
format_hint = 'country/state codes'
|
||||
elif all('/' in val or '-' in val for val in option_values[:5] if val):
|
||||
format_hint = 'date/path format'
|
||||
elif any('@' in val for val in option_values[:5] if val):
|
||||
format_hint = 'email addresses'
|
||||
|
||||
return {'count': len(options), 'first_options': first_options, 'format_hint': format_hint}
|
||||
|
||||
def _is_interactive_cached(self, node: EnhancedDOMTreeNode) -> bool:
|
||||
"""Cached version of clickable element detection to avoid redundant calls."""
|
||||
if node.node_id not in self._clickable_cache:
|
||||
import time
|
||||
|
||||
start_time = time.time()
|
||||
result = ClickableElementDetector.is_interactive(node)
|
||||
end_time = time.time()
|
||||
|
||||
if 'clickable_detection_time' not in self.timing_info:
|
||||
self.timing_info['clickable_detection_time'] = 0
|
||||
self.timing_info['clickable_detection_time'] += end_time - start_time
|
||||
|
||||
self._clickable_cache[node.node_id] = result
|
||||
|
||||
return self._clickable_cache[node.node_id]
|
||||
|
||||
def _create_simplified_tree(self, node: EnhancedDOMTreeNode, depth: int = 0) -> SimplifiedNode | None:
|
||||
"""Step 1: Create a simplified tree with enhanced element detection."""
|
||||
|
||||
if node.node_type == NodeType.DOCUMENT_NODE:
|
||||
# for all cldren including shadow roots
|
||||
for child in node.children_and_shadow_roots:
|
||||
simplified_child = self._create_simplified_tree(child, depth + 1)
|
||||
if simplified_child:
|
||||
return simplified_child
|
||||
|
||||
return None
|
||||
|
||||
if node.node_type == NodeType.DOCUMENT_FRAGMENT_NODE:
|
||||
# ENHANCED shadow DOM processing - always include shadow content
|
||||
simplified = SimplifiedNode(original_node=node, children=[])
|
||||
for child in node.children_and_shadow_roots:
|
||||
simplified_child = self._create_simplified_tree(child, depth + 1)
|
||||
if simplified_child:
|
||||
simplified.children.append(simplified_child)
|
||||
|
||||
# Always return shadow DOM fragments, even if children seem empty
|
||||
# Shadow DOM often contains the actual interactive content in SPAs
|
||||
return simplified if simplified.children else SimplifiedNode(original_node=node, children=[])
|
||||
|
||||
elif node.node_type == NodeType.ELEMENT_NODE:
|
||||
# Skip non-content elements
|
||||
if node.node_name.lower() in DISABLED_ELEMENTS:
|
||||
return None
|
||||
|
||||
if node.node_name == 'IFRAME' or node.node_name == 'FRAME':
|
||||
if node.content_document:
|
||||
simplified = SimplifiedNode(original_node=node, children=[])
|
||||
for child in node.content_document.children_nodes or []:
|
||||
simplified_child = self._create_simplified_tree(child, depth + 1)
|
||||
if simplified_child is not None:
|
||||
simplified.children.append(simplified_child)
|
||||
return simplified
|
||||
|
||||
is_visible = node.is_visible
|
||||
is_scrollable = node.is_actually_scrollable
|
||||
has_shadow_content = bool(node.children_and_shadow_roots)
|
||||
|
||||
# ENHANCED SHADOW DOM DETECTION: Include shadow hosts even if not visible
|
||||
is_shadow_host = any(child.node_type == NodeType.DOCUMENT_FRAGMENT_NODE for child in node.children_and_shadow_roots)
|
||||
|
||||
# Override visibility for elements with validation attributes
|
||||
if not is_visible and node.attributes:
|
||||
has_validation_attrs = any(attr.startswith(('aria-', 'pseudo')) for attr in node.attributes.keys())
|
||||
if has_validation_attrs:
|
||||
is_visible = True # Force visibility for validation elements
|
||||
|
||||
# Include if visible, scrollable, has children, or is shadow host
|
||||
if is_visible or is_scrollable or has_shadow_content or is_shadow_host:
|
||||
simplified = SimplifiedNode(original_node=node, children=[], is_shadow_host=is_shadow_host)
|
||||
|
||||
# Process ALL children including shadow roots with enhanced logging
|
||||
for child in node.children_and_shadow_roots:
|
||||
simplified_child = self._create_simplified_tree(child, depth + 1)
|
||||
if simplified_child:
|
||||
simplified.children.append(simplified_child)
|
||||
|
||||
# COMPOUND CONTROL PROCESSING: Add virtual components for compound controls
|
||||
self._add_compound_components(simplified, node)
|
||||
|
||||
# SHADOW DOM SPECIAL CASE: Always include shadow hosts even if not visible
|
||||
# Many SPA frameworks (React, Vue) render content in shadow DOM
|
||||
if is_shadow_host and simplified.children:
|
||||
return simplified
|
||||
|
||||
# Return if meaningful or has meaningful children
|
||||
if is_visible or is_scrollable or simplified.children:
|
||||
return simplified
|
||||
|
||||
elif node.node_type == NodeType.TEXT_NODE:
|
||||
# Include meaningful text nodes
|
||||
is_visible = node.snapshot_node and node.is_visible
|
||||
if is_visible and node.node_value and node.node_value.strip() and len(node.node_value.strip()) > 1:
|
||||
return SimplifiedNode(original_node=node, children=[])
|
||||
|
||||
return None
|
||||
|
||||
def _optimize_tree(self, node: SimplifiedNode | None) -> SimplifiedNode | None:
|
||||
"""Step 2: Optimize tree structure."""
|
||||
if not node:
|
||||
return None
|
||||
|
||||
# Process children
|
||||
optimized_children = []
|
||||
for child in node.children:
|
||||
optimized_child = self._optimize_tree(child)
|
||||
if optimized_child:
|
||||
optimized_children.append(optimized_child)
|
||||
|
||||
node.children = optimized_children
|
||||
|
||||
# Keep meaningful nodes
|
||||
is_visible = node.original_node.snapshot_node and node.original_node.is_visible
|
||||
|
||||
if (
|
||||
is_visible # Keep all visible nodes
|
||||
or node.original_node.is_actually_scrollable
|
||||
or node.original_node.node_type == NodeType.TEXT_NODE
|
||||
or node.children
|
||||
):
|
||||
return node
|
||||
|
||||
return None
|
||||
|
||||
def _collect_interactive_elements(self, node: SimplifiedNode, elements: list[SimplifiedNode]) -> None:
|
||||
"""Recursively collect interactive elements that are also visible."""
|
||||
is_interactive = self._is_interactive_cached(node.original_node)
|
||||
is_visible = node.original_node.snapshot_node and node.original_node.is_visible
|
||||
|
||||
# Only collect elements that are both interactive AND visible
|
||||
if is_interactive and is_visible:
|
||||
elements.append(node)
|
||||
|
||||
for child in node.children:
|
||||
self._collect_interactive_elements(child, elements)
|
||||
|
||||
def _assign_interactive_indices_and_mark_new_nodes(self, node: SimplifiedNode | None) -> None:
|
||||
"""Assign interactive indices to clickable elements that are also visible."""
|
||||
if not node:
|
||||
return
|
||||
|
||||
# Skip assigning index to excluded nodes, or ignored by paint order
|
||||
if not node.excluded_by_parent and not node.ignored_by_paint_order:
|
||||
# Regular interactive element assignment (including enhanced compound controls)
|
||||
is_interactive_assign = self._is_interactive_cached(node.original_node)
|
||||
is_visible = node.original_node.snapshot_node and node.original_node.is_visible
|
||||
|
||||
# Only add to selector map if element is both interactive AND visible
|
||||
if is_interactive_assign and is_visible:
|
||||
node.interactive_index = self._interactive_counter
|
||||
node.original_node.element_index = self._interactive_counter
|
||||
self._selector_map[self._interactive_counter] = node.original_node
|
||||
self._interactive_counter += 1
|
||||
|
||||
# Mark compound components as new for visibility
|
||||
if node.is_compound_component:
|
||||
node.is_new = True
|
||||
elif self._previous_cached_selector_map:
|
||||
# Check if node is new for regular elements
|
||||
previous_backend_node_ids = {node.backend_node_id for node in self._previous_cached_selector_map.values()}
|
||||
if node.original_node.backend_node_id not in previous_backend_node_ids:
|
||||
node.is_new = True
|
||||
|
||||
# Process children
|
||||
for child in node.children:
|
||||
self._assign_interactive_indices_and_mark_new_nodes(child)
|
||||
|
||||
def _apply_bounding_box_filtering(self, node: SimplifiedNode | None) -> SimplifiedNode | None:
|
||||
"""Filter children contained within propagating parent bounds."""
|
||||
if not node:
|
||||
return None
|
||||
|
||||
# Start with no active bounds
|
||||
self._filter_tree_recursive(node, active_bounds=None, depth=0)
|
||||
|
||||
# Log statistics
|
||||
excluded_count = self._count_excluded_nodes(node)
|
||||
if excluded_count > 0:
|
||||
import logging
|
||||
|
||||
logging.debug(f'BBox filtering excluded {excluded_count} nodes')
|
||||
|
||||
return node
|
||||
|
||||
def _filter_tree_recursive(self, node: SimplifiedNode, active_bounds: PropagatingBounds | None = None, depth: int = 0):
|
||||
"""
|
||||
Recursively filter tree with bounding box propagation.
|
||||
Bounds propagate to ALL descendants until overridden.
|
||||
"""
|
||||
|
||||
# Check if this node should be excluded by active bounds
|
||||
if active_bounds and self._should_exclude_child(node, active_bounds):
|
||||
node.excluded_by_parent = True
|
||||
# Important: Still check if this node starts NEW propagation
|
||||
|
||||
# Check if this node starts new propagation (even if excluded!)
|
||||
new_bounds = None
|
||||
tag = node.original_node.tag_name.lower()
|
||||
role = node.original_node.attributes.get('role') if node.original_node.attributes else None
|
||||
attributes = {
|
||||
'tag': tag,
|
||||
'role': role,
|
||||
}
|
||||
# Check if this element matches any propagating element pattern
|
||||
if self._is_propagating_element(attributes):
|
||||
# This node propagates bounds to ALL its descendants
|
||||
if node.original_node.snapshot_node and node.original_node.snapshot_node.bounds:
|
||||
new_bounds = PropagatingBounds(
|
||||
tag=tag,
|
||||
bounds=node.original_node.snapshot_node.bounds,
|
||||
node_id=node.original_node.node_id,
|
||||
depth=depth,
|
||||
)
|
||||
|
||||
# Propagate to ALL children
|
||||
# Use new_bounds if this node starts propagation, otherwise continue with active_bounds
|
||||
propagate_bounds = new_bounds if new_bounds else active_bounds
|
||||
|
||||
for child in node.children:
|
||||
self._filter_tree_recursive(child, propagate_bounds, depth + 1)
|
||||
|
||||
def _should_exclude_child(self, node: SimplifiedNode, active_bounds: PropagatingBounds) -> bool:
|
||||
"""
|
||||
Determine if child should be excluded based on propagating bounds.
|
||||
"""
|
||||
|
||||
# Never exclude text nodes - we always want to preserve text content
|
||||
if node.original_node.node_type == NodeType.TEXT_NODE:
|
||||
return False
|
||||
|
||||
# Get child bounds
|
||||
if not node.original_node.snapshot_node or not node.original_node.snapshot_node.bounds:
|
||||
return False # No bounds = can't determine containment
|
||||
|
||||
child_bounds = node.original_node.snapshot_node.bounds
|
||||
|
||||
# Check containment with configured threshold
|
||||
if not self._is_contained(child_bounds, active_bounds.bounds, self.containment_threshold):
|
||||
return False # Not sufficiently contained
|
||||
|
||||
# EXCEPTION RULES - Keep these even if contained:
|
||||
|
||||
child_tag = node.original_node.tag_name.lower()
|
||||
child_role = node.original_node.attributes.get('role') if node.original_node.attributes else None
|
||||
child_attributes = {
|
||||
'tag': child_tag,
|
||||
'role': child_role,
|
||||
}
|
||||
|
||||
# 1. Never exclude form elements (they need individual interaction)
|
||||
if child_tag in ['input', 'select', 'textarea', 'label']:
|
||||
return False
|
||||
|
||||
# 2. Keep if child is also a propagating element
|
||||
# (might have stopPropagation, e.g., button in button)
|
||||
if self._is_propagating_element(child_attributes):
|
||||
return False
|
||||
|
||||
# 3. Keep if has explicit onclick handler
|
||||
if node.original_node.attributes and 'onclick' in node.original_node.attributes:
|
||||
return False
|
||||
|
||||
# 4. Keep if has aria-label suggesting it's independently interactive
|
||||
if node.original_node.attributes:
|
||||
aria_label = node.original_node.attributes.get('aria-label')
|
||||
if aria_label and aria_label.strip():
|
||||
# Has meaningful aria-label, likely interactive
|
||||
return False
|
||||
|
||||
# 5. Keep if has role suggesting interactivity
|
||||
if node.original_node.attributes:
|
||||
role = node.original_node.attributes.get('role')
|
||||
if role in ['button', 'link', 'checkbox', 'radio', 'tab', 'menuitem']:
|
||||
return False
|
||||
|
||||
# Default: exclude this child
|
||||
return True
|
||||
|
||||
def _is_contained(self, child: DOMRect, parent: DOMRect, threshold: float) -> bool:
|
||||
"""
|
||||
Check if child is contained within parent bounds.
|
||||
|
||||
Args:
|
||||
threshold: Percentage (0.0-1.0) of child that must be within parent
|
||||
"""
|
||||
# Calculate intersection
|
||||
x_overlap = max(0, min(child.x + child.width, parent.x + parent.width) - max(child.x, parent.x))
|
||||
y_overlap = max(0, min(child.y + child.height, parent.y + parent.height) - max(child.y, parent.y))
|
||||
|
||||
intersection_area = x_overlap * y_overlap
|
||||
child_area = child.width * child.height
|
||||
|
||||
if child_area == 0:
|
||||
return False # Zero-area element
|
||||
|
||||
containment_ratio = intersection_area / child_area
|
||||
return containment_ratio >= threshold
|
||||
|
||||
def _count_excluded_nodes(self, node: SimplifiedNode, count: int = 0) -> int:
|
||||
"""Count how many nodes were excluded (for debugging)."""
|
||||
if hasattr(node, 'excluded_by_parent') and node.excluded_by_parent:
|
||||
count += 1
|
||||
for child in node.children:
|
||||
count = self._count_excluded_nodes(child, count)
|
||||
return count
|
||||
|
||||
def _is_propagating_element(self, attributes: dict[str, str | None]) -> bool:
|
||||
"""
|
||||
Check if an element should propagate bounds based on attributes.
|
||||
If the element satisfies one of the patterns, it propagates bounds to all its children.
|
||||
"""
|
||||
keys_to_check = ['tag', 'role']
|
||||
for pattern in self.PROPAGATING_ELEMENTS:
|
||||
# Check if the element satisfies the pattern
|
||||
check = [pattern.get(key) is None or pattern.get(key) == attributes.get(key) for key in keys_to_check]
|
||||
if all(check):
|
||||
return True
|
||||
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def serialize_tree(node: SimplifiedNode | None, include_attributes: list[str], depth: int = 0) -> str:
|
||||
"""Serialize the optimized tree to string format."""
|
||||
if not node:
|
||||
return ''
|
||||
|
||||
# Skip rendering excluded nodes, but process their children
|
||||
if hasattr(node, 'excluded_by_parent') and node.excluded_by_parent:
|
||||
formatted_text = []
|
||||
for child in node.children:
|
||||
child_text = DOMTreeSerializer.serialize_tree(child, include_attributes, depth)
|
||||
if child_text:
|
||||
formatted_text.append(child_text)
|
||||
return '\n'.join(formatted_text)
|
||||
|
||||
formatted_text = []
|
||||
depth_str = depth * '\t'
|
||||
next_depth = depth
|
||||
|
||||
if node.original_node.node_type == NodeType.ELEMENT_NODE:
|
||||
# Skip displaying nodes marked as should_display=False
|
||||
if not node.should_display:
|
||||
for child in node.children:
|
||||
child_text = DOMTreeSerializer.serialize_tree(child, include_attributes, depth)
|
||||
if child_text:
|
||||
formatted_text.append(child_text)
|
||||
return '\n'.join(formatted_text)
|
||||
|
||||
# Add element with interactive_index if clickable, scrollable, or iframe
|
||||
is_any_scrollable = node.original_node.is_actually_scrollable or node.original_node.is_scrollable
|
||||
should_show_scroll = node.original_node.should_show_scroll_info
|
||||
if (
|
||||
node.interactive_index is not None
|
||||
or is_any_scrollable
|
||||
or node.original_node.tag_name.upper() == 'IFRAME'
|
||||
or node.original_node.tag_name.upper() == 'FRAME'
|
||||
):
|
||||
next_depth += 1
|
||||
|
||||
# Build attributes string with compound component info
|
||||
text_content = ''
|
||||
attributes_html_str = DOMTreeSerializer._build_attributes_string(
|
||||
node.original_node, include_attributes, text_content
|
||||
)
|
||||
|
||||
# Add compound component information to attributes if present
|
||||
if node.original_node._compound_children:
|
||||
compound_info = []
|
||||
for child_info in node.original_node._compound_children:
|
||||
parts = []
|
||||
if child_info['name']:
|
||||
parts.append(f'name={child_info["name"]}')
|
||||
if child_info['role']:
|
||||
parts.append(f'role={child_info["role"]}')
|
||||
if child_info['valuemin'] is not None:
|
||||
parts.append(f'min={child_info["valuemin"]}')
|
||||
if child_info['valuemax'] is not None:
|
||||
parts.append(f'max={child_info["valuemax"]}')
|
||||
if child_info['valuenow'] is not None:
|
||||
parts.append(f'current={child_info["valuenow"]}')
|
||||
|
||||
# Add select-specific information
|
||||
if 'options_count' in child_info and child_info['options_count'] is not None:
|
||||
parts.append(f'count={child_info["options_count"]}')
|
||||
if 'first_options' in child_info and child_info['first_options']:
|
||||
options_str = '|'.join(child_info['first_options'][:4]) # Limit to 4 options
|
||||
parts.append(f'options={options_str}')
|
||||
if 'format_hint' in child_info and child_info['format_hint']:
|
||||
parts.append(f'format={child_info["format_hint"]}')
|
||||
|
||||
if parts:
|
||||
compound_info.append(f'({",".join(parts)})')
|
||||
|
||||
if compound_info:
|
||||
compound_attr = f'compound_components={",".join(compound_info)}'
|
||||
if attributes_html_str:
|
||||
attributes_html_str += f' {compound_attr}'
|
||||
else:
|
||||
attributes_html_str = compound_attr
|
||||
|
||||
# Build the line with shadow host indicator
|
||||
shadow_prefix = ''
|
||||
if node.is_shadow_host:
|
||||
# Check if any shadow children are closed
|
||||
has_closed_shadow = any(
|
||||
child.original_node.node_type == NodeType.DOCUMENT_FRAGMENT_NODE
|
||||
and child.original_node.shadow_root_type
|
||||
and child.original_node.shadow_root_type.lower() == 'closed'
|
||||
for child in node.children
|
||||
)
|
||||
shadow_prefix = '|SHADOW(closed)|' if has_closed_shadow else '|SHADOW(open)|'
|
||||
|
||||
if should_show_scroll and node.interactive_index is None:
|
||||
# Scrollable container but not clickable
|
||||
line = f'{depth_str}{shadow_prefix}|SCROLL|<{node.original_node.tag_name}'
|
||||
elif node.interactive_index is not None:
|
||||
# Clickable (and possibly scrollable)
|
||||
new_prefix = '*' if node.is_new else ''
|
||||
scroll_prefix = '|SCROLL+' if should_show_scroll else '['
|
||||
line = f'{depth_str}{shadow_prefix}{new_prefix}{scroll_prefix}{node.interactive_index}]<{node.original_node.tag_name}'
|
||||
elif node.original_node.tag_name.upper() == 'IFRAME':
|
||||
# Iframe element (not interactive)
|
||||
line = f'{depth_str}{shadow_prefix}|IFRAME|<{node.original_node.tag_name}'
|
||||
elif node.original_node.tag_name.upper() == 'FRAME':
|
||||
# Frame element (not interactive)
|
||||
line = f'{depth_str}{shadow_prefix}|FRAME|<{node.original_node.tag_name}'
|
||||
else:
|
||||
line = f'{depth_str}{shadow_prefix}<{node.original_node.tag_name}'
|
||||
|
||||
if attributes_html_str:
|
||||
line += f' {attributes_html_str}'
|
||||
|
||||
line += ' />'
|
||||
|
||||
# Add scroll information only when we should show it
|
||||
if should_show_scroll:
|
||||
scroll_info_text = node.original_node.get_scroll_info_text()
|
||||
if scroll_info_text:
|
||||
line += f' ({scroll_info_text})'
|
||||
|
||||
formatted_text.append(line)
|
||||
|
||||
elif node.original_node.node_type == NodeType.DOCUMENT_FRAGMENT_NODE:
|
||||
# Shadow DOM representation - show clearly to LLM
|
||||
if node.original_node.shadow_root_type and node.original_node.shadow_root_type.lower() == 'closed':
|
||||
formatted_text.append(f'{depth_str}▼ Shadow Content (Closed)')
|
||||
else:
|
||||
formatted_text.append(f'{depth_str}▼ Shadow Content (Open)')
|
||||
|
||||
next_depth += 1
|
||||
|
||||
# Process shadow DOM children
|
||||
for child in node.children:
|
||||
child_text = DOMTreeSerializer.serialize_tree(child, include_attributes, next_depth)
|
||||
if child_text:
|
||||
formatted_text.append(child_text)
|
||||
|
||||
# Close shadow DOM indicator
|
||||
if node.children: # Only show close if we had content
|
||||
formatted_text.append(f'{depth_str}▲ Shadow Content End')
|
||||
|
||||
elif node.original_node.node_type == NodeType.TEXT_NODE:
|
||||
# Include visible text
|
||||
is_visible = node.original_node.snapshot_node and node.original_node.is_visible
|
||||
if (
|
||||
is_visible
|
||||
and node.original_node.node_value
|
||||
and node.original_node.node_value.strip()
|
||||
and len(node.original_node.node_value.strip()) > 1
|
||||
):
|
||||
clean_text = node.original_node.node_value.strip()
|
||||
formatted_text.append(f'{depth_str}{clean_text}')
|
||||
|
||||
# Process children (for non-shadow elements)
|
||||
if node.original_node.node_type != NodeType.DOCUMENT_FRAGMENT_NODE:
|
||||
for child in node.children:
|
||||
child_text = DOMTreeSerializer.serialize_tree(child, include_attributes, next_depth)
|
||||
if child_text:
|
||||
formatted_text.append(child_text)
|
||||
|
||||
return '\n'.join(formatted_text)
|
||||
|
||||
@staticmethod
|
||||
def _build_attributes_string(node: EnhancedDOMTreeNode, include_attributes: list[str], text: str) -> str:
|
||||
"""Build the attributes string for an element."""
|
||||
attributes_to_include = {}
|
||||
|
||||
# Include HTML attributes
|
||||
if node.attributes:
|
||||
attributes_to_include.update(
|
||||
{
|
||||
key: str(value).strip()
|
||||
for key, value in node.attributes.items()
|
||||
if key in include_attributes and str(value).strip() != ''
|
||||
}
|
||||
)
|
||||
|
||||
# Include accessibility properties
|
||||
if node.ax_node and node.ax_node.properties:
|
||||
for prop in node.ax_node.properties:
|
||||
try:
|
||||
if prop.name in include_attributes and prop.value is not None:
|
||||
# Convert boolean to lowercase string, keep others as-is
|
||||
if isinstance(prop.value, bool):
|
||||
attributes_to_include[prop.name] = str(prop.value).lower()
|
||||
else:
|
||||
prop_value_str = str(prop.value).strip()
|
||||
if prop_value_str:
|
||||
attributes_to_include[prop.name] = prop_value_str
|
||||
except (AttributeError, ValueError):
|
||||
continue
|
||||
|
||||
if not attributes_to_include:
|
||||
return ''
|
||||
|
||||
# Remove duplicate values
|
||||
ordered_keys = [key for key in include_attributes if key in attributes_to_include]
|
||||
|
||||
if len(ordered_keys) > 1:
|
||||
keys_to_remove = set()
|
||||
seen_values = {}
|
||||
|
||||
for key in ordered_keys:
|
||||
value = attributes_to_include[key]
|
||||
if len(value) > 5:
|
||||
if value in seen_values:
|
||||
keys_to_remove.add(key)
|
||||
else:
|
||||
seen_values[value] = key
|
||||
|
||||
for key in keys_to_remove:
|
||||
del attributes_to_include[key]
|
||||
|
||||
# Remove attributes that duplicate accessibility data
|
||||
role = node.ax_node.role if node.ax_node else None
|
||||
if role and node.node_name == role:
|
||||
attributes_to_include.pop('role', None)
|
||||
|
||||
attrs_to_remove_if_text_matches = ['aria-label', 'placeholder', 'title']
|
||||
for attr in attrs_to_remove_if_text_matches:
|
||||
if attributes_to_include.get(attr) and attributes_to_include.get(attr, '').strip().lower() == text.strip().lower():
|
||||
del attributes_to_include[attr]
|
||||
|
||||
if attributes_to_include:
|
||||
return ' '.join(f'{key}={cap_text_length(value, 100)}' for key, value in attributes_to_include.items())
|
||||
|
||||
return ''
|
||||
@@ -0,0 +1,741 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import time
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from cdp_use.cdp.accessibility.commands import GetFullAXTreeReturns
|
||||
from cdp_use.cdp.accessibility.types import AXNode
|
||||
from cdp_use.cdp.dom.types import Node
|
||||
from cdp_use.cdp.target import TargetID
|
||||
|
||||
from browser_use.dom.enhanced_snapshot import (
|
||||
REQUIRED_COMPUTED_STYLES,
|
||||
build_snapshot_lookup,
|
||||
)
|
||||
from browser_use.dom.serializer.serializer import DOMTreeSerializer
|
||||
from browser_use.dom.views import (
|
||||
CurrentPageTargets,
|
||||
DOMRect,
|
||||
EnhancedAXNode,
|
||||
EnhancedAXProperty,
|
||||
EnhancedDOMTreeNode,
|
||||
NodeType,
|
||||
SerializedDOMState,
|
||||
TargetAllTrees,
|
||||
)
|
||||
from browser_use.observability import observe_debug
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from browser_use.browser.session import BrowserSession
|
||||
|
||||
# Note: iframe limits are now configurable via BrowserProfile.max_iframes and BrowserProfile.max_iframe_depth
|
||||
|
||||
|
||||
class DomService:
|
||||
"""
|
||||
Service for getting the DOM tree and other DOM-related information.
|
||||
|
||||
Either browser or page must be provided.
|
||||
|
||||
TODO: currently we start a new websocket connection PER STEP, we should definitely keep this persistent
|
||||
"""
|
||||
|
||||
logger: logging.Logger
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
browser_session: 'BrowserSession',
|
||||
logger: logging.Logger | None = None,
|
||||
cross_origin_iframes: bool = False,
|
||||
paint_order_filtering: bool = True,
|
||||
max_iframes: int = 100,
|
||||
max_iframe_depth: int = 5,
|
||||
):
|
||||
self.browser_session = browser_session
|
||||
self.logger = logger or browser_session.logger
|
||||
self.cross_origin_iframes = cross_origin_iframes
|
||||
self.paint_order_filtering = paint_order_filtering
|
||||
self.max_iframes = max_iframes
|
||||
self.max_iframe_depth = max_iframe_depth
|
||||
|
||||
async def __aenter__(self):
|
||||
return self
|
||||
|
||||
async def __aexit__(self, exc_type, exc_value, traceback):
|
||||
pass # no need to cleanup anything, browser_session auto handles cleaning up session cache
|
||||
|
||||
async def _get_targets_for_page(self, target_id: TargetID | None = None) -> CurrentPageTargets:
|
||||
"""Get the target info for a specific page.
|
||||
|
||||
Args:
|
||||
target_id: The target ID to get info for. If None, uses current_target_id.
|
||||
"""
|
||||
targets = await self.browser_session.cdp_client.send.Target.getTargets()
|
||||
|
||||
# Use provided target_id or fall back to current_target_id
|
||||
if target_id is None:
|
||||
target_id = self.browser_session.current_target_id
|
||||
if not target_id:
|
||||
raise ValueError('No current target ID set in browser session')
|
||||
|
||||
# Find main page target by ID
|
||||
main_target = next((t for t in targets['targetInfos'] if t['targetId'] == target_id), None)
|
||||
|
||||
if not main_target:
|
||||
raise ValueError(f'No target found for target ID: {target_id}')
|
||||
|
||||
# Get all frames using the new method to find iframe targets for this page
|
||||
all_frames, _ = await self.browser_session.get_all_frames()
|
||||
|
||||
# Find iframe targets that are children of this target
|
||||
iframe_targets = []
|
||||
for frame_info in all_frames.values():
|
||||
# Check if this frame is a cross-origin iframe with its own target
|
||||
if frame_info.get('isCrossOrigin') and frame_info.get('frameTargetId'):
|
||||
# Check if this frame belongs to our target
|
||||
parent_target = frame_info.get('parentTargetId', frame_info.get('frameTargetId'))
|
||||
if parent_target == target_id:
|
||||
# Find the target info for this iframe
|
||||
iframe_target = next(
|
||||
(t for t in targets['targetInfos'] if t['targetId'] == frame_info['frameTargetId']), None
|
||||
)
|
||||
if iframe_target:
|
||||
iframe_targets.append(iframe_target)
|
||||
|
||||
return CurrentPageTargets(
|
||||
page_session=main_target,
|
||||
iframe_sessions=iframe_targets,
|
||||
)
|
||||
|
||||
def _build_enhanced_ax_node(self, ax_node: AXNode) -> EnhancedAXNode:
|
||||
properties: list[EnhancedAXProperty] | None = None
|
||||
if 'properties' in ax_node and ax_node['properties']:
|
||||
properties = []
|
||||
for property in ax_node['properties']:
|
||||
try:
|
||||
# test whether property name can go into the enum (sometimes Chrome returns some random properties)
|
||||
properties.append(
|
||||
EnhancedAXProperty(
|
||||
name=property['name'],
|
||||
value=property.get('value', {}).get('value', None),
|
||||
# related_nodes=[], # TODO: add related nodes
|
||||
)
|
||||
)
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
enhanced_ax_node = EnhancedAXNode(
|
||||
ax_node_id=ax_node['nodeId'],
|
||||
ignored=ax_node['ignored'],
|
||||
role=ax_node.get('role', {}).get('value', None),
|
||||
name=ax_node.get('name', {}).get('value', None),
|
||||
description=ax_node.get('description', {}).get('value', None),
|
||||
properties=properties,
|
||||
child_ids=ax_node.get('childIds', []) if ax_node.get('childIds') else None,
|
||||
)
|
||||
return enhanced_ax_node
|
||||
|
||||
async def _get_viewport_ratio(self, target_id: TargetID) -> float:
|
||||
"""Get viewport dimensions, device pixel ratio, and scroll position using CDP."""
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session(target_id=target_id, focus=True)
|
||||
|
||||
try:
|
||||
# Get the layout metrics which includes the visual viewport
|
||||
metrics = await cdp_session.cdp_client.send.Page.getLayoutMetrics(session_id=cdp_session.session_id)
|
||||
|
||||
visual_viewport = metrics.get('visualViewport', {})
|
||||
|
||||
# IMPORTANT: Use CSS viewport instead of device pixel viewport
|
||||
# This fixes the coordinate mismatch on high-DPI displays
|
||||
css_visual_viewport = metrics.get('cssVisualViewport', {})
|
||||
css_layout_viewport = metrics.get('cssLayoutViewport', {})
|
||||
|
||||
# Use CSS pixels (what JavaScript sees) instead of device pixels
|
||||
width = css_visual_viewport.get('clientWidth', css_layout_viewport.get('clientWidth', 1920.0))
|
||||
|
||||
# Calculate device pixel ratio
|
||||
device_width = visual_viewport.get('clientWidth', width)
|
||||
css_width = css_visual_viewport.get('clientWidth', width)
|
||||
device_pixel_ratio = device_width / css_width if css_width > 0 else 1.0
|
||||
|
||||
return float(device_pixel_ratio)
|
||||
except Exception as e:
|
||||
self.logger.debug(f'Viewport size detection failed: {e}')
|
||||
# Fallback to default viewport size
|
||||
return 1.0
|
||||
|
||||
@classmethod
|
||||
def is_element_visible_according_to_all_parents(
|
||||
cls, node: EnhancedDOMTreeNode, html_frames: list[EnhancedDOMTreeNode]
|
||||
) -> bool:
|
||||
"""Check if the element is visible according to all its parent HTML frames."""
|
||||
|
||||
if not node.snapshot_node:
|
||||
return False
|
||||
|
||||
computed_styles = node.snapshot_node.computed_styles or {}
|
||||
|
||||
display = computed_styles.get('display', '').lower()
|
||||
visibility = computed_styles.get('visibility', '').lower()
|
||||
opacity = computed_styles.get('opacity', '1')
|
||||
|
||||
if display == 'none' or visibility == 'hidden':
|
||||
return False
|
||||
|
||||
try:
|
||||
if float(opacity) <= 0:
|
||||
return False
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
# Start with the element's local bounds (in its own frame's coordinate system)
|
||||
current_bounds = node.snapshot_node.bounds
|
||||
|
||||
if not current_bounds:
|
||||
return False # If there are no bounds, the element is not visible
|
||||
|
||||
"""
|
||||
Reverse iterate through the html frames (that can be either iframe or document -> if it's a document frame compare if the current bounds interest with it (taking scroll into account) otherwise move the current bounds by the iframe offset)
|
||||
"""
|
||||
for frame in reversed(html_frames):
|
||||
if (
|
||||
frame.node_type == NodeType.ELEMENT_NODE
|
||||
and (frame.node_name.upper() == 'IFRAME' or frame.node_name.upper() == 'FRAME')
|
||||
and frame.snapshot_node
|
||||
and frame.snapshot_node.bounds
|
||||
):
|
||||
iframe_bounds = frame.snapshot_node.bounds
|
||||
|
||||
# negate the values added in `_construct_enhanced_node`
|
||||
current_bounds.x += iframe_bounds.x
|
||||
current_bounds.y += iframe_bounds.y
|
||||
|
||||
if (
|
||||
frame.node_type == NodeType.ELEMENT_NODE
|
||||
and frame.node_name == 'HTML'
|
||||
and frame.snapshot_node
|
||||
and frame.snapshot_node.scrollRects
|
||||
and frame.snapshot_node.clientRects
|
||||
):
|
||||
# For iframe content, we need to check visibility within the iframe's viewport
|
||||
# The scrollRects represent the current scroll position
|
||||
# The clientRects represent the viewport size
|
||||
# Elements are visible if they fall within the viewport after accounting for scroll
|
||||
|
||||
# The viewport of the frame (what's actually visible)
|
||||
viewport_left = 0 # Viewport always starts at 0 in frame coordinates
|
||||
viewport_top = 0
|
||||
viewport_right = frame.snapshot_node.clientRects.width
|
||||
viewport_bottom = frame.snapshot_node.clientRects.height
|
||||
|
||||
# Adjust element bounds by the scroll offset to get position relative to viewport
|
||||
# When scrolled down, scrollRects.y is positive, so we subtract it from element's y
|
||||
adjusted_x = current_bounds.x - frame.snapshot_node.scrollRects.x
|
||||
adjusted_y = current_bounds.y - frame.snapshot_node.scrollRects.y
|
||||
|
||||
frame_intersects = (
|
||||
adjusted_x < viewport_right
|
||||
and adjusted_x + current_bounds.width > viewport_left
|
||||
and adjusted_y < viewport_bottom + 1000
|
||||
and adjusted_y + current_bounds.height > viewport_top - 1000
|
||||
)
|
||||
|
||||
if not frame_intersects:
|
||||
return False
|
||||
|
||||
# Keep the original coordinate adjustment to maintain consistency
|
||||
# This adjustment is needed for proper coordinate transformation
|
||||
current_bounds.x -= frame.snapshot_node.scrollRects.x
|
||||
current_bounds.y -= frame.snapshot_node.scrollRects.y
|
||||
|
||||
# If we reach here, element is visible in main viewport and all containing iframes
|
||||
return True
|
||||
|
||||
async def _get_ax_tree_for_all_frames(self, target_id: TargetID) -> GetFullAXTreeReturns:
|
||||
"""Recursively collect all frames and merge their accessibility trees into a single array."""
|
||||
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session(target_id=target_id, focus=False)
|
||||
frame_tree = await cdp_session.cdp_client.send.Page.getFrameTree(session_id=cdp_session.session_id)
|
||||
|
||||
def collect_all_frame_ids(frame_tree_node) -> list[str]:
|
||||
"""Recursively collect all frame IDs from the frame tree."""
|
||||
frame_ids = [frame_tree_node['frame']['id']]
|
||||
|
||||
if 'childFrames' in frame_tree_node and frame_tree_node['childFrames']:
|
||||
for child_frame in frame_tree_node['childFrames']:
|
||||
frame_ids.extend(collect_all_frame_ids(child_frame))
|
||||
|
||||
return frame_ids
|
||||
|
||||
# Collect all frame IDs recursively
|
||||
all_frame_ids = collect_all_frame_ids(frame_tree['frameTree'])
|
||||
|
||||
# Get accessibility tree for each frame
|
||||
ax_tree_requests = []
|
||||
for frame_id in all_frame_ids:
|
||||
ax_tree_request = cdp_session.cdp_client.send.Accessibility.getFullAXTree(
|
||||
params={'frameId': frame_id}, session_id=cdp_session.session_id
|
||||
)
|
||||
ax_tree_requests.append(ax_tree_request)
|
||||
|
||||
# Wait for all requests to complete
|
||||
ax_trees = await asyncio.gather(*ax_tree_requests)
|
||||
|
||||
# Merge all AX nodes into a single array
|
||||
merged_nodes: list[AXNode] = []
|
||||
for ax_tree in ax_trees:
|
||||
merged_nodes.extend(ax_tree['nodes'])
|
||||
|
||||
return {'nodes': merged_nodes}
|
||||
|
||||
async def _get_all_trees(self, target_id: TargetID) -> TargetAllTrees:
|
||||
cdp_session = await self.browser_session.get_or_create_cdp_session(target_id=target_id, focus=False)
|
||||
|
||||
# Wait for the page to be ready first
|
||||
try:
|
||||
ready_state = await cdp_session.cdp_client.send.Runtime.evaluate(
|
||||
params={'expression': 'document.readyState'}, session_id=cdp_session.session_id
|
||||
)
|
||||
except Exception as e:
|
||||
pass # Page might not be ready yet
|
||||
# DEBUG: Log before capturing snapshot
|
||||
self.logger.debug(f'🔍 DEBUG: Capturing DOM snapshot for target {target_id}')
|
||||
|
||||
# Get actual scroll positions for all iframes before capturing snapshot
|
||||
iframe_scroll_positions = {}
|
||||
try:
|
||||
scroll_result = await cdp_session.cdp_client.send.Runtime.evaluate(
|
||||
params={
|
||||
'expression': """
|
||||
(() => {
|
||||
const scrollData = {};
|
||||
const iframes = document.querySelectorAll('iframe');
|
||||
iframes.forEach((iframe, index) => {
|
||||
try {
|
||||
const doc = iframe.contentDocument || iframe.contentWindow.document;
|
||||
if (doc) {
|
||||
scrollData[index] = {
|
||||
scrollTop: doc.documentElement.scrollTop || doc.body.scrollTop || 0,
|
||||
scrollLeft: doc.documentElement.scrollLeft || doc.body.scrollLeft || 0
|
||||
};
|
||||
}
|
||||
} catch (e) {
|
||||
// Cross-origin iframe, can't access
|
||||
}
|
||||
});
|
||||
return scrollData;
|
||||
})()
|
||||
""",
|
||||
'returnByValue': True,
|
||||
},
|
||||
session_id=cdp_session.session_id,
|
||||
)
|
||||
if scroll_result and 'result' in scroll_result and 'value' in scroll_result['result']:
|
||||
iframe_scroll_positions = scroll_result['result']['value']
|
||||
for idx, scroll_data in iframe_scroll_positions.items():
|
||||
self.logger.debug(
|
||||
f'🔍 DEBUG: Iframe {idx} actual scroll position - scrollTop={scroll_data.get("scrollTop", 0)}, scrollLeft={scroll_data.get("scrollLeft", 0)}'
|
||||
)
|
||||
except Exception as e:
|
||||
self.logger.debug(f'Failed to get iframe scroll positions: {e}')
|
||||
|
||||
# Define CDP request factories to avoid duplication
|
||||
def create_snapshot_request():
|
||||
return cdp_session.cdp_client.send.DOMSnapshot.captureSnapshot(
|
||||
params={
|
||||
'computedStyles': REQUIRED_COMPUTED_STYLES,
|
||||
'includePaintOrder': True,
|
||||
'includeDOMRects': True,
|
||||
'includeBlendedBackgroundColors': False,
|
||||
'includeTextColorOpacities': False,
|
||||
},
|
||||
session_id=cdp_session.session_id,
|
||||
)
|
||||
|
||||
def create_dom_tree_request():
|
||||
return cdp_session.cdp_client.send.DOM.getDocument(
|
||||
params={'depth': -1, 'pierce': True}, session_id=cdp_session.session_id
|
||||
)
|
||||
|
||||
start = time.time()
|
||||
|
||||
# Create initial tasks
|
||||
tasks = {
|
||||
'snapshot': asyncio.create_task(create_snapshot_request()),
|
||||
'dom_tree': asyncio.create_task(create_dom_tree_request()),
|
||||
'ax_tree': asyncio.create_task(self._get_ax_tree_for_all_frames(target_id)),
|
||||
'device_pixel_ratio': asyncio.create_task(self._get_viewport_ratio(target_id)),
|
||||
}
|
||||
|
||||
# Wait for all tasks with timeout
|
||||
done, pending = await asyncio.wait(tasks.values(), timeout=10.0)
|
||||
|
||||
# Retry any failed or timed out tasks
|
||||
if pending:
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
|
||||
# Retry mapping for pending tasks
|
||||
retry_map = {
|
||||
tasks['snapshot']: lambda: asyncio.create_task(create_snapshot_request()),
|
||||
tasks['dom_tree']: lambda: asyncio.create_task(create_dom_tree_request()),
|
||||
tasks['ax_tree']: lambda: asyncio.create_task(self._get_ax_tree_for_all_frames(target_id)),
|
||||
tasks['device_pixel_ratio']: lambda: asyncio.create_task(self._get_viewport_ratio(target_id)),
|
||||
}
|
||||
|
||||
# Create new tasks only for the ones that didn't complete
|
||||
for key, task in tasks.items():
|
||||
if task in pending and task in retry_map:
|
||||
tasks[key] = retry_map[task]()
|
||||
|
||||
# Wait again with shorter timeout
|
||||
done2, pending2 = await asyncio.wait([t for t in tasks.values() if not t.done()], timeout=2.0)
|
||||
|
||||
if pending2:
|
||||
for task in pending2:
|
||||
task.cancel()
|
||||
|
||||
# Extract results, tracking which ones failed
|
||||
results = {}
|
||||
failed = []
|
||||
for key, task in tasks.items():
|
||||
if task.done() and not task.cancelled():
|
||||
try:
|
||||
results[key] = task.result()
|
||||
except Exception as e:
|
||||
self.logger.warning(f'CDP request {key} failed with exception: {e}')
|
||||
failed.append(key)
|
||||
else:
|
||||
self.logger.warning(f'CDP request {key} timed out')
|
||||
failed.append(key)
|
||||
|
||||
# If any required tasks failed, raise an exception
|
||||
if failed:
|
||||
raise TimeoutError(f'CDP requests failed or timed out: {", ".join(failed)}')
|
||||
|
||||
snapshot = results['snapshot']
|
||||
dom_tree = results['dom_tree']
|
||||
ax_tree = results['ax_tree']
|
||||
device_pixel_ratio = results['device_pixel_ratio']
|
||||
end = time.time()
|
||||
cdp_timing = {'cdp_calls_total': end - start}
|
||||
|
||||
# DEBUG: Log snapshot info and limit documents to prevent explosion
|
||||
if snapshot and 'documents' in snapshot:
|
||||
original_doc_count = len(snapshot['documents'])
|
||||
# Limit to max_iframes documents to prevent iframe explosion
|
||||
if original_doc_count > self.max_iframes:
|
||||
self.logger.warning(
|
||||
f'⚠️ Limiting processing of {original_doc_count} iframes on page to only first {self.max_iframes} to prevent crashes!'
|
||||
)
|
||||
snapshot['documents'] = snapshot['documents'][: self.max_iframes]
|
||||
|
||||
total_nodes = sum(len(doc.get('nodes', [])) for doc in snapshot['documents'])
|
||||
self.logger.debug(f'🔍 DEBUG: Snapshot contains {len(snapshot["documents"])} frames with {total_nodes} total nodes')
|
||||
# Log iframe-specific info
|
||||
for doc_idx, doc in enumerate(snapshot['documents']):
|
||||
if doc_idx > 0: # Not the main document
|
||||
self.logger.debug(
|
||||
f'🔍 DEBUG: Iframe #{doc_idx} {doc.get("frameId", "no-frame-id")} {doc.get("url", "no-url")} has {len(doc.get("nodes", []))} nodes'
|
||||
)
|
||||
|
||||
return TargetAllTrees(
|
||||
snapshot=snapshot,
|
||||
dom_tree=dom_tree,
|
||||
ax_tree=ax_tree,
|
||||
device_pixel_ratio=device_pixel_ratio,
|
||||
cdp_timing=cdp_timing,
|
||||
)
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='get_dom_tree')
|
||||
async def get_dom_tree(
|
||||
self,
|
||||
target_id: TargetID,
|
||||
initial_html_frames: list[EnhancedDOMTreeNode] | None = None,
|
||||
initial_total_frame_offset: DOMRect | None = None,
|
||||
iframe_depth: int = 0,
|
||||
) -> EnhancedDOMTreeNode:
|
||||
"""Get the DOM tree for a specific target.
|
||||
|
||||
Args:
|
||||
target_id: Target ID of the page to get the DOM tree for.
|
||||
initial_html_frames: List of HTML frame nodes encountered so far
|
||||
initial_total_frame_offset: Accumulated coordinate offset
|
||||
iframe_depth: Current depth of iframe nesting to prevent infinite recursion
|
||||
"""
|
||||
|
||||
trees = await self._get_all_trees(target_id)
|
||||
|
||||
dom_tree = trees.dom_tree
|
||||
ax_tree = trees.ax_tree
|
||||
snapshot = trees.snapshot
|
||||
device_pixel_ratio = trees.device_pixel_ratio
|
||||
|
||||
ax_tree_lookup: dict[int, AXNode] = {
|
||||
ax_node['backendDOMNodeId']: ax_node for ax_node in ax_tree['nodes'] if 'backendDOMNodeId' in ax_node
|
||||
}
|
||||
|
||||
enhanced_dom_tree_node_lookup: dict[int, EnhancedDOMTreeNode] = {}
|
||||
""" NodeId (NOT backend node id) -> enhanced dom tree node""" # way to get the parent/content node
|
||||
|
||||
# Parse snapshot data with everything calculated upfront
|
||||
snapshot_lookup = build_snapshot_lookup(snapshot, device_pixel_ratio)
|
||||
|
||||
async def _construct_enhanced_node(
|
||||
node: Node, html_frames: list[EnhancedDOMTreeNode] | None, total_frame_offset: DOMRect | None
|
||||
) -> EnhancedDOMTreeNode:
|
||||
"""
|
||||
Recursively construct enhanced DOM tree nodes.
|
||||
|
||||
Args:
|
||||
node: The DOM node to construct
|
||||
html_frames: List of HTML frame nodes encountered so far
|
||||
accumulated_iframe_offset: Accumulated coordinate translation from parent iframes (includes scroll corrections)
|
||||
"""
|
||||
|
||||
# Initialize lists if not provided
|
||||
if html_frames is None:
|
||||
html_frames = []
|
||||
|
||||
# to get rid of the pointer references
|
||||
if total_frame_offset is None:
|
||||
total_frame_offset = DOMRect(x=0.0, y=0.0, width=0.0, height=0.0)
|
||||
else:
|
||||
total_frame_offset = DOMRect(
|
||||
total_frame_offset.x, total_frame_offset.y, total_frame_offset.width, total_frame_offset.height
|
||||
)
|
||||
|
||||
# memoize the mf (I don't know if some nodes are duplicated)
|
||||
if node['nodeId'] in enhanced_dom_tree_node_lookup:
|
||||
return enhanced_dom_tree_node_lookup[node['nodeId']]
|
||||
|
||||
ax_node = ax_tree_lookup.get(node['backendNodeId'])
|
||||
if ax_node:
|
||||
enhanced_ax_node = self._build_enhanced_ax_node(ax_node)
|
||||
else:
|
||||
enhanced_ax_node = None
|
||||
|
||||
# To make attributes more readable
|
||||
attributes: dict[str, str] | None = None
|
||||
if 'attributes' in node and node['attributes']:
|
||||
attributes = {}
|
||||
for i in range(0, len(node['attributes']), 2):
|
||||
attributes[node['attributes'][i]] = node['attributes'][i + 1]
|
||||
|
||||
shadow_root_type = None
|
||||
if 'shadowRootType' in node and node['shadowRootType']:
|
||||
try:
|
||||
shadow_root_type = node['shadowRootType']
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
# Get snapshot data and calculate absolute position
|
||||
snapshot_data = snapshot_lookup.get(node['backendNodeId'], None)
|
||||
absolute_position = None
|
||||
if snapshot_data and snapshot_data.bounds:
|
||||
absolute_position = DOMRect(
|
||||
x=snapshot_data.bounds.x + total_frame_offset.x,
|
||||
y=snapshot_data.bounds.y + total_frame_offset.y,
|
||||
width=snapshot_data.bounds.width,
|
||||
height=snapshot_data.bounds.height,
|
||||
)
|
||||
|
||||
dom_tree_node = EnhancedDOMTreeNode(
|
||||
node_id=node['nodeId'],
|
||||
backend_node_id=node['backendNodeId'],
|
||||
node_type=NodeType(node['nodeType']),
|
||||
node_name=node['nodeName'],
|
||||
node_value=node['nodeValue'],
|
||||
attributes=attributes or {},
|
||||
is_scrollable=node.get('isScrollable', None),
|
||||
frame_id=node.get('frameId', None),
|
||||
session_id=self.browser_session.agent_focus.session_id if self.browser_session.agent_focus else None,
|
||||
target_id=target_id,
|
||||
content_document=None,
|
||||
shadow_root_type=shadow_root_type,
|
||||
shadow_roots=None,
|
||||
parent_node=None,
|
||||
children_nodes=None,
|
||||
ax_node=enhanced_ax_node,
|
||||
snapshot_node=snapshot_data,
|
||||
is_visible=None,
|
||||
absolute_position=absolute_position,
|
||||
element_index=None,
|
||||
)
|
||||
|
||||
enhanced_dom_tree_node_lookup[node['nodeId']] = dom_tree_node
|
||||
|
||||
if 'parentId' in node and node['parentId']:
|
||||
dom_tree_node.parent_node = enhanced_dom_tree_node_lookup[
|
||||
node['parentId']
|
||||
] # parents should always be in the lookup
|
||||
|
||||
# Check if this is an HTML frame node and add it to the list
|
||||
updated_html_frames = html_frames.copy()
|
||||
if node['nodeType'] == NodeType.ELEMENT_NODE.value and node['nodeName'] == 'HTML' and node.get('frameId') is not None:
|
||||
updated_html_frames.append(dom_tree_node)
|
||||
|
||||
# and adjust the total frame offset by scroll
|
||||
if snapshot_data and snapshot_data.scrollRects:
|
||||
total_frame_offset.x -= snapshot_data.scrollRects.x
|
||||
total_frame_offset.y -= snapshot_data.scrollRects.y
|
||||
# DEBUG: Log iframe scroll information
|
||||
self.logger.debug(
|
||||
f'🔍 DEBUG: HTML frame scroll - scrollY={snapshot_data.scrollRects.y}, scrollX={snapshot_data.scrollRects.x}, frameId={node.get("frameId")}, nodeId={node["nodeId"]}'
|
||||
)
|
||||
|
||||
# Calculate new iframe offset for content documents, accounting for iframe scroll
|
||||
if (
|
||||
(node['nodeName'].upper() == 'IFRAME' or node['nodeName'].upper() == 'FRAME')
|
||||
and snapshot_data
|
||||
and snapshot_data.bounds
|
||||
):
|
||||
if snapshot_data.bounds:
|
||||
updated_html_frames.append(dom_tree_node)
|
||||
|
||||
total_frame_offset.x += snapshot_data.bounds.x
|
||||
total_frame_offset.y += snapshot_data.bounds.y
|
||||
|
||||
if 'contentDocument' in node and node['contentDocument']:
|
||||
dom_tree_node.content_document = await _construct_enhanced_node(
|
||||
node['contentDocument'], updated_html_frames, total_frame_offset
|
||||
)
|
||||
dom_tree_node.content_document.parent_node = dom_tree_node
|
||||
# forcefully set the parent node to the content document node (helps traverse the tree)
|
||||
|
||||
if 'shadowRoots' in node and node['shadowRoots']:
|
||||
dom_tree_node.shadow_roots = []
|
||||
for shadow_root in node['shadowRoots']:
|
||||
shadow_root_node = await _construct_enhanced_node(shadow_root, updated_html_frames, total_frame_offset)
|
||||
# forcefully set the parent node to the shadow root node (helps traverse the tree)
|
||||
shadow_root_node.parent_node = dom_tree_node
|
||||
dom_tree_node.shadow_roots.append(shadow_root_node)
|
||||
|
||||
if 'children' in node and node['children']:
|
||||
dom_tree_node.children_nodes = []
|
||||
for child in node['children']:
|
||||
dom_tree_node.children_nodes.append(
|
||||
await _construct_enhanced_node(child, updated_html_frames, total_frame_offset)
|
||||
)
|
||||
|
||||
# Set visibility using the collected HTML frames
|
||||
dom_tree_node.is_visible = self.is_element_visible_according_to_all_parents(dom_tree_node, updated_html_frames)
|
||||
|
||||
# DEBUG: Log visibility info for form elements in iframes
|
||||
if dom_tree_node.tag_name and dom_tree_node.tag_name.upper() in ['INPUT', 'SELECT', 'TEXTAREA', 'LABEL']:
|
||||
attrs = dom_tree_node.attributes or {}
|
||||
elem_id = attrs.get('id', '')
|
||||
elem_name = attrs.get('name', '')
|
||||
if (
|
||||
'city' in elem_id.lower()
|
||||
or 'city' in elem_name.lower()
|
||||
or 'state' in elem_id.lower()
|
||||
or 'state' in elem_name.lower()
|
||||
or 'zip' in elem_id.lower()
|
||||
or 'zip' in elem_name.lower()
|
||||
):
|
||||
self.logger.debug(
|
||||
f"🔍 DEBUG: Form element {dom_tree_node.tag_name} id='{elem_id}' name='{elem_name}' - visible={dom_tree_node.is_visible}, bounds={dom_tree_node.snapshot_node.bounds if dom_tree_node.snapshot_node else 'NO_SNAPSHOT'}"
|
||||
)
|
||||
|
||||
# handle cross origin iframe (just recursively call the main function with the proper target if it exists in iframes)
|
||||
# only do this if the iframe is visible (otherwise it's not worth it)
|
||||
|
||||
if (
|
||||
# TODO: hacky way to disable cross origin iframes for now
|
||||
self.cross_origin_iframes and node['nodeName'].upper() == 'IFRAME' and node.get('contentDocument', None) is None
|
||||
): # None meaning there is no content
|
||||
# Check iframe depth to prevent infinite recursion
|
||||
if iframe_depth >= self.max_iframe_depth:
|
||||
self.logger.debug(
|
||||
f'Skipping iframe at depth {iframe_depth} to prevent infinite recursion (max depth: {self.max_iframe_depth})'
|
||||
)
|
||||
else:
|
||||
# Check if iframe is visible and large enough (>= 200px in both dimensions)
|
||||
should_process_iframe = False
|
||||
|
||||
# First check if the iframe element itself is visible
|
||||
if dom_tree_node.is_visible:
|
||||
# Check iframe dimensions
|
||||
if dom_tree_node.snapshot_node and dom_tree_node.snapshot_node.bounds:
|
||||
bounds = dom_tree_node.snapshot_node.bounds
|
||||
width = bounds.width
|
||||
height = bounds.height
|
||||
|
||||
# Only process if iframe is at least 200px in both dimensions
|
||||
if width >= 200 and height >= 200:
|
||||
should_process_iframe = True
|
||||
self.logger.debug(f'Processing cross-origin iframe: visible=True, width={width}, height={height}')
|
||||
else:
|
||||
self.logger.debug(
|
||||
f'Skipping small cross-origin iframe: width={width}, height={height} (needs >= 200px)'
|
||||
)
|
||||
else:
|
||||
self.logger.debug('Skipping cross-origin iframe: no bounds available')
|
||||
else:
|
||||
self.logger.debug('Skipping invisible cross-origin iframe')
|
||||
|
||||
if should_process_iframe:
|
||||
# Use get_all_frames to find the iframe's target
|
||||
frame_id = node.get('frameId', None)
|
||||
if frame_id:
|
||||
all_frames, _ = await self.browser_session.get_all_frames()
|
||||
frame_info = all_frames.get(frame_id)
|
||||
iframe_document_target = None
|
||||
if frame_info and frame_info.get('frameTargetId'):
|
||||
# Get the target info for this iframe
|
||||
targets = await self.browser_session.cdp_client.send.Target.getTargets()
|
||||
iframe_document_target = next(
|
||||
(t for t in targets['targetInfos'] if t['targetId'] == frame_info['frameTargetId']), None
|
||||
)
|
||||
else:
|
||||
iframe_document_target = None
|
||||
# if target actually exists in one of the frames, just recursively build the dom tree for it
|
||||
if iframe_document_target:
|
||||
self.logger.debug(
|
||||
f'Getting content document for iframe {node.get("frameId", None)} at depth {iframe_depth + 1}'
|
||||
)
|
||||
content_document = await self.get_dom_tree(
|
||||
target_id=iframe_document_target.get('targetId'),
|
||||
# TODO: experiment with this values -> not sure whether the whole cross origin iframe should be ALWAYS included as soon as some part of it is visible or not.
|
||||
# Current config: if the cross origin iframe is AT ALL visible, then just include everything inside of it!
|
||||
# initial_html_frames=updated_html_frames,
|
||||
initial_total_frame_offset=total_frame_offset,
|
||||
iframe_depth=iframe_depth + 1,
|
||||
)
|
||||
|
||||
dom_tree_node.content_document = content_document
|
||||
dom_tree_node.content_document.parent_node = dom_tree_node
|
||||
|
||||
return dom_tree_node
|
||||
|
||||
enhanced_dom_tree_node = await _construct_enhanced_node(dom_tree['root'], initial_html_frames, initial_total_frame_offset)
|
||||
|
||||
return enhanced_dom_tree_node
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='get_serialized_dom_tree')
|
||||
async def get_serialized_dom_tree(
|
||||
self, previous_cached_state: SerializedDOMState | None = None
|
||||
) -> tuple[SerializedDOMState, EnhancedDOMTreeNode, dict[str, float]]:
|
||||
"""Get the serialized DOM tree representation for LLM consumption.
|
||||
|
||||
Returns:
|
||||
Tuple of (serialized_dom_state, enhanced_dom_tree_root, timing_info)
|
||||
"""
|
||||
|
||||
# Use current target (None means use current)
|
||||
assert self.browser_session.current_target_id is not None
|
||||
enhanced_dom_tree = await self.get_dom_tree(target_id=self.browser_session.current_target_id)
|
||||
|
||||
start = time.time()
|
||||
serialized_dom_state, serializer_timing = DOMTreeSerializer(
|
||||
enhanced_dom_tree, previous_cached_state, paint_order_filtering=self.paint_order_filtering
|
||||
).serialize_accessible_elements()
|
||||
|
||||
end = time.time()
|
||||
serialize_total_timing = {'serialize_dom_tree_total': end - start}
|
||||
|
||||
# Combine all timing info
|
||||
all_timing = {**serializer_timing, **serialize_total_timing}
|
||||
|
||||
return serialized_dom_state, enhanced_dom_tree, all_timing
|
||||
+38
@@ -0,0 +1,38 @@
|
||||
"""
|
||||
Test suite locking out TypeError in _build_dom_tree
|
||||
when layout dictionary contains None values for array properties.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
|
||||
|
||||
|
||||
def test_layout_index_map_handles_null_layout_arrays():
|
||||
"""
|
||||
Ensure layout array checks tolerate None for bounds, styles, paintOrders, etc.
|
||||
"""
|
||||
layout = {
|
||||
'bounds': None,
|
||||
'styles': None,
|
||||
'paintOrders': None,
|
||||
'clientRects': None,
|
||||
'scrollRects': None,
|
||||
'stackingContexts': None
|
||||
}
|
||||
|
||||
layout_idx = 0
|
||||
bounds_len = len(layout.get('bounds') or [])
|
||||
styles_len = len(layout.get('styles') or [])
|
||||
paint_len = len(layout.get('paintOrders') or [])
|
||||
client_len = len(layout.get('clientRects') or [])
|
||||
scroll_len = len(layout.get('scrollRects') or [])
|
||||
stacking_len = len(layout.get('stackingContexts') or [])
|
||||
|
||||
assert bounds_len == 0
|
||||
assert styles_len == 0
|
||||
assert paint_len == 0
|
||||
assert client_len == 0
|
||||
assert scroll_len == 0
|
||||
assert stacking_len == 0
|
||||
@@ -0,0 +1,5 @@
|
||||
def cap_text_length(text: str, max_length: int) -> str:
|
||||
"""Cap text length for display."""
|
||||
if len(text) <= max_length:
|
||||
return text
|
||||
return text[:max_length] + '...'
|
||||
@@ -0,0 +1,873 @@
|
||||
import hashlib
|
||||
from dataclasses import asdict, dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
from cdp_use.cdp.accessibility.commands import GetFullAXTreeReturns
|
||||
from cdp_use.cdp.accessibility.types import AXPropertyName
|
||||
from cdp_use.cdp.dom.commands import GetDocumentReturns
|
||||
from cdp_use.cdp.dom.types import ShadowRootType
|
||||
from cdp_use.cdp.domsnapshot.commands import CaptureSnapshotReturns
|
||||
from cdp_use.cdp.target.types import SessionID, TargetID, TargetInfo
|
||||
from uuid_extensions import uuid7str
|
||||
|
||||
from browser_use.dom.utils import cap_text_length
|
||||
from browser_use.observability import observe_debug
|
||||
|
||||
# Serializer types
|
||||
DEFAULT_INCLUDE_ATTRIBUTES = [
|
||||
'title',
|
||||
'type',
|
||||
'checked',
|
||||
# 'class',
|
||||
'id',
|
||||
'name',
|
||||
'role',
|
||||
'value',
|
||||
'placeholder',
|
||||
'data-date-format',
|
||||
'alt',
|
||||
'aria-label',
|
||||
'aria-expanded',
|
||||
'data-state',
|
||||
'aria-checked',
|
||||
# ARIA value attributes for datetime/range inputs
|
||||
'aria-valuemin',
|
||||
'aria-valuemax',
|
||||
'aria-valuenow',
|
||||
'aria-placeholder',
|
||||
# Validation attributes - help agents avoid brute force attempts
|
||||
'pattern',
|
||||
'min',
|
||||
'max',
|
||||
'minlength',
|
||||
'maxlength',
|
||||
'step',
|
||||
# Webkit shadow DOM identifiers
|
||||
'pseudo',
|
||||
# Accessibility properties from ax_node (ordered by importance for automation)
|
||||
'checked',
|
||||
'selected',
|
||||
'expanded',
|
||||
'pressed',
|
||||
'disabled',
|
||||
'invalid', # Current validation state from AX node
|
||||
'valuemin', # Min value from AX node (for datetime/range)
|
||||
'valuemax', # Max value from AX node (for datetime/range)
|
||||
'valuenow',
|
||||
'keyshortcuts',
|
||||
'haspopup',
|
||||
'multiselectable',
|
||||
# Less commonly needed (uncomment if required):
|
||||
# 'readonly',
|
||||
'required',
|
||||
'valuetext',
|
||||
'level',
|
||||
'busy',
|
||||
'live',
|
||||
# Accessibility name (contains text content for StaticText elements)
|
||||
'ax_name',
|
||||
]
|
||||
|
||||
STATIC_ATTRIBUTES = {
|
||||
'class',
|
||||
'id',
|
||||
'name',
|
||||
'type',
|
||||
'placeholder',
|
||||
'aria-label',
|
||||
'title',
|
||||
# 'aria-expanded',
|
||||
'role',
|
||||
'data-testid',
|
||||
'data-test',
|
||||
'data-cy',
|
||||
'data-selenium',
|
||||
'for',
|
||||
'required',
|
||||
'disabled',
|
||||
'readonly',
|
||||
'checked',
|
||||
'selected',
|
||||
'multiple',
|
||||
'href',
|
||||
'target',
|
||||
'rel',
|
||||
'aria-describedby',
|
||||
'aria-labelledby',
|
||||
'aria-controls',
|
||||
'aria-owns',
|
||||
'aria-live',
|
||||
'aria-atomic',
|
||||
'aria-busy',
|
||||
'aria-disabled',
|
||||
'aria-hidden',
|
||||
'aria-pressed',
|
||||
'aria-checked',
|
||||
'aria-selected',
|
||||
'tabindex',
|
||||
'alt',
|
||||
'src',
|
||||
'lang',
|
||||
'itemscope',
|
||||
'itemtype',
|
||||
'itemprop',
|
||||
# Webkit shadow DOM attributes
|
||||
'pseudo',
|
||||
'aria-valuemin',
|
||||
'aria-valuemax',
|
||||
'aria-valuenow',
|
||||
'aria-placeholder',
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class CurrentPageTargets:
|
||||
page_session: TargetInfo
|
||||
iframe_sessions: list[TargetInfo]
|
||||
"""
|
||||
Iframe sessions are ALL the iframes sessions of all the pages (not just the current page)
|
||||
"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class TargetAllTrees:
|
||||
snapshot: CaptureSnapshotReturns
|
||||
dom_tree: GetDocumentReturns
|
||||
ax_tree: GetFullAXTreeReturns
|
||||
device_pixel_ratio: float
|
||||
cdp_timing: dict[str, float]
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class PropagatingBounds:
|
||||
"""Track bounds that propagate from parent elements to filter children."""
|
||||
|
||||
tag: str # The tag that started propagation ('a' or 'button')
|
||||
bounds: 'DOMRect' # The bounding box
|
||||
node_id: int # Node ID for debugging
|
||||
depth: int # How deep in tree this started (for debugging)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class SimplifiedNode:
|
||||
"""Simplified tree node for optimization."""
|
||||
|
||||
original_node: 'EnhancedDOMTreeNode'
|
||||
children: list['SimplifiedNode']
|
||||
should_display: bool = True
|
||||
interactive_index: int | None = None
|
||||
|
||||
is_new: bool = False
|
||||
|
||||
ignored_by_paint_order: bool = False # More info in dom/serializer/paint_order.py
|
||||
excluded_by_parent: bool = False # New field for bbox filtering
|
||||
is_shadow_host: bool = False # New field for shadow DOM hosts
|
||||
is_compound_component: bool = False # True for virtual components of compound controls
|
||||
|
||||
def _clean_original_node_json(self, node_json: dict) -> dict:
|
||||
"""Recursively remove children_nodes and shadow_roots from original_node JSON."""
|
||||
# Remove the fields we don't want in SimplifiedNode serialization
|
||||
if 'children_nodes' in node_json:
|
||||
del node_json['children_nodes']
|
||||
if 'shadow_roots' in node_json:
|
||||
del node_json['shadow_roots']
|
||||
|
||||
# Clean nested content_document if it exists
|
||||
if node_json.get('content_document'):
|
||||
node_json['content_document'] = self._clean_original_node_json(node_json['content_document'])
|
||||
|
||||
return node_json
|
||||
|
||||
def __json__(self) -> dict:
|
||||
original_node_json = self.original_node.__json__()
|
||||
# Remove children_nodes and shadow_roots to avoid duplication with SimplifiedNode.children
|
||||
cleaned_original_node_json = self._clean_original_node_json(original_node_json)
|
||||
return {
|
||||
'should_display': self.should_display,
|
||||
'interactive_index': self.interactive_index,
|
||||
'ignored_by_paint_order': self.ignored_by_paint_order,
|
||||
'excluded_by_parent': self.excluded_by_parent,
|
||||
'original_node': cleaned_original_node_json,
|
||||
'children': [c.__json__() for c in self.children],
|
||||
}
|
||||
|
||||
|
||||
class NodeType(int, Enum):
|
||||
"""DOM node types based on the DOM specification."""
|
||||
|
||||
ELEMENT_NODE = 1
|
||||
ATTRIBUTE_NODE = 2
|
||||
TEXT_NODE = 3
|
||||
CDATA_SECTION_NODE = 4
|
||||
ENTITY_REFERENCE_NODE = 5
|
||||
ENTITY_NODE = 6
|
||||
PROCESSING_INSTRUCTION_NODE = 7
|
||||
COMMENT_NODE = 8
|
||||
DOCUMENT_NODE = 9
|
||||
DOCUMENT_TYPE_NODE = 10
|
||||
DOCUMENT_FRAGMENT_NODE = 11
|
||||
NOTATION_NODE = 12
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class DOMRect:
|
||||
x: float
|
||||
y: float
|
||||
width: float
|
||||
height: float
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
'x': self.x,
|
||||
'y': self.y,
|
||||
'width': self.width,
|
||||
'height': self.height,
|
||||
}
|
||||
|
||||
def __json__(self) -> dict:
|
||||
return self.to_dict()
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class EnhancedAXProperty:
|
||||
"""we don't need `sources` and `related_nodes` for now (not sure how to use them)
|
||||
|
||||
TODO: there is probably some way to determine whether it has a value or related nodes or not, but for now it's kinda fine idk
|
||||
"""
|
||||
|
||||
name: AXPropertyName
|
||||
value: str | bool | None
|
||||
# related_nodes: list[EnhancedAXRelatedNode] | None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class EnhancedAXNode:
|
||||
ax_node_id: str
|
||||
"""Not to be confused the DOM node_id. Only useful for AX node tree"""
|
||||
ignored: bool
|
||||
# we don't need ignored_reasons as we anyway ignore the node otherwise
|
||||
role: str | None
|
||||
name: str | None
|
||||
description: str | None
|
||||
|
||||
properties: list[EnhancedAXProperty] | None
|
||||
child_ids: list[str] | None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class EnhancedSnapshotNode:
|
||||
"""Snapshot data extracted from DOMSnapshot for enhanced functionality."""
|
||||
|
||||
is_clickable: bool | None
|
||||
cursor_style: str | None
|
||||
bounds: DOMRect | None
|
||||
"""
|
||||
Document coordinates (origin = top-left of the page, ignores current scroll).
|
||||
Equivalent JS API: layoutNode.boundingBox in the older API.
|
||||
Typical use: Quick hit-test that doesn't care about scroll position.
|
||||
"""
|
||||
|
||||
clientRects: DOMRect | None
|
||||
"""
|
||||
Viewport coordinates (origin = top-left of the visible scrollport).
|
||||
Equivalent JS API: element.getClientRects() / getBoundingClientRect().
|
||||
Typical use: Pixel-perfect hit-testing on screen, taking current scroll into account.
|
||||
"""
|
||||
|
||||
scrollRects: DOMRect | None
|
||||
"""
|
||||
Scrollable area of the element.
|
||||
"""
|
||||
|
||||
computed_styles: dict[str, str] | None
|
||||
"""Computed styles from the layout tree"""
|
||||
paint_order: int | None
|
||||
"""Paint order from the layout tree"""
|
||||
stacking_contexts: int | None
|
||||
"""Stacking contexts from the layout tree"""
|
||||
|
||||
|
||||
# @dataclass(slots=True)
|
||||
# class SuperSelector:
|
||||
# node_id: int
|
||||
# backend_node_id: int
|
||||
# frame_id: str | None
|
||||
# target_id: TargetID
|
||||
|
||||
# node_type: NodeType
|
||||
# node_name: str
|
||||
|
||||
# # is_visible: bool | None
|
||||
# # is_scrollable: bool | None
|
||||
|
||||
# element_index: int | None
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class EnhancedDOMTreeNode:
|
||||
"""
|
||||
Enhanced DOM tree node that contains information from AX, DOM, and Snapshot trees. It's mostly based on the types on DOM node type with enhanced data from AX and Snapshot trees.
|
||||
|
||||
@dev when serializing check if the value is a valid value first!
|
||||
|
||||
Learn more about the fields:
|
||||
- (DOM node) https://chromedevtools.github.io/devtools-protocol/tot/DOM/#type-BackendNode
|
||||
- (AX node) https://chromedevtools.github.io/devtools-protocol/tot/Accessibility/#type-AXNode
|
||||
- (Snapshot node) https://chromedevtools.github.io/devtools-protocol/tot/DOMSnapshot/#type-DOMNode
|
||||
"""
|
||||
|
||||
# region - DOM Node data
|
||||
|
||||
node_id: int
|
||||
backend_node_id: int
|
||||
|
||||
node_type: NodeType
|
||||
"""Node types, defined in `NodeType` enum."""
|
||||
node_name: str
|
||||
"""Only applicable for `NodeType.ELEMENT_NODE`"""
|
||||
node_value: str
|
||||
"""this is where the value from `NodeType.TEXT_NODE` is stored usually"""
|
||||
attributes: dict[str, str]
|
||||
"""slightly changed from the original attributes to be more readable"""
|
||||
is_scrollable: bool | None
|
||||
"""
|
||||
Whether the node is scrollable.
|
||||
"""
|
||||
is_visible: bool | None
|
||||
"""
|
||||
Whether the node is visible according to the upper most frame node.
|
||||
"""
|
||||
|
||||
absolute_position: DOMRect | None
|
||||
"""
|
||||
Absolute position of the node in the document according to the top-left of the page.
|
||||
"""
|
||||
|
||||
# frames
|
||||
target_id: TargetID
|
||||
frame_id: str | None
|
||||
session_id: SessionID | None
|
||||
content_document: 'EnhancedDOMTreeNode | None'
|
||||
"""
|
||||
Content document is the document inside a new iframe.
|
||||
"""
|
||||
# Shadow DOM
|
||||
shadow_root_type: ShadowRootType | None
|
||||
shadow_roots: list['EnhancedDOMTreeNode'] | None
|
||||
"""
|
||||
Shadow roots are the shadow DOMs of the element.
|
||||
"""
|
||||
|
||||
# Navigation
|
||||
parent_node: 'EnhancedDOMTreeNode | None'
|
||||
children_nodes: list['EnhancedDOMTreeNode'] | None
|
||||
|
||||
# endregion - DOM Node data
|
||||
|
||||
# region - AX Node data
|
||||
ax_node: EnhancedAXNode | None
|
||||
|
||||
# endregion - AX Node data
|
||||
|
||||
# region - Snapshot Node data
|
||||
snapshot_node: EnhancedSnapshotNode | None
|
||||
|
||||
# endregion - Snapshot Node data
|
||||
|
||||
# Interactive element index
|
||||
element_index: int | None = None
|
||||
|
||||
# Compound control child components information
|
||||
_compound_children: list[dict[str, Any]] = field(default_factory=list)
|
||||
|
||||
uuid: str = field(default_factory=uuid7str)
|
||||
|
||||
@property
|
||||
def parent(self) -> 'EnhancedDOMTreeNode | None':
|
||||
return self.parent_node
|
||||
|
||||
@property
|
||||
def children(self) -> list['EnhancedDOMTreeNode']:
|
||||
return self.children_nodes or []
|
||||
|
||||
@property
|
||||
def children_and_shadow_roots(self) -> list['EnhancedDOMTreeNode']:
|
||||
"""
|
||||
Returns all children nodes, including shadow roots
|
||||
"""
|
||||
children = self.children_nodes or []
|
||||
if self.shadow_roots:
|
||||
children.extend(self.shadow_roots)
|
||||
return children
|
||||
|
||||
@property
|
||||
def tag_name(self) -> str:
|
||||
return self.node_name.lower()
|
||||
|
||||
@property
|
||||
def xpath(self) -> str:
|
||||
"""Generate XPath for this DOM node, stopping at shadow boundaries or iframes."""
|
||||
segments = []
|
||||
current_element = self
|
||||
|
||||
while current_element and (
|
||||
current_element.node_type == NodeType.ELEMENT_NODE or current_element.node_type == NodeType.DOCUMENT_FRAGMENT_NODE
|
||||
):
|
||||
# just pass through shadow roots
|
||||
if current_element.node_type == NodeType.DOCUMENT_FRAGMENT_NODE:
|
||||
current_element = current_element.parent_node
|
||||
continue
|
||||
|
||||
# stop ONLY if we hit iframe
|
||||
if current_element.parent_node and current_element.parent_node.node_name.lower() == 'iframe':
|
||||
break
|
||||
|
||||
position = self._get_element_position(current_element)
|
||||
tag_name = current_element.node_name.lower()
|
||||
xpath_index = f'[{position}]' if position > 0 else ''
|
||||
segments.insert(0, f'{tag_name}{xpath_index}')
|
||||
|
||||
current_element = current_element.parent_node
|
||||
|
||||
return '/'.join(segments)
|
||||
|
||||
def _get_element_position(self, element: 'EnhancedDOMTreeNode') -> int:
|
||||
"""Get the position of an element among its siblings with the same tag name.
|
||||
Returns 0 if it's the only element of its type, otherwise returns 1-based index."""
|
||||
if not element.parent_node or not element.parent_node.children_nodes:
|
||||
return 0
|
||||
|
||||
same_tag_siblings = [
|
||||
child
|
||||
for child in element.parent_node.children_nodes
|
||||
if child.node_type == NodeType.ELEMENT_NODE and child.node_name.lower() == element.node_name.lower()
|
||||
]
|
||||
|
||||
if len(same_tag_siblings) <= 1:
|
||||
return 0 # No index needed if it's the only one
|
||||
|
||||
try:
|
||||
# XPath is 1-indexed
|
||||
position = same_tag_siblings.index(element) + 1
|
||||
return position
|
||||
except ValueError:
|
||||
return 0
|
||||
|
||||
def __json__(self) -> dict:
|
||||
"""Serializes the node and its descendants to a dictionary, omitting parent references."""
|
||||
return {
|
||||
'node_id': self.node_id,
|
||||
'backend_node_id': self.backend_node_id,
|
||||
'node_type': self.node_type.name,
|
||||
'node_name': self.node_name,
|
||||
'node_value': self.node_value,
|
||||
'is_visible': self.is_visible,
|
||||
'attributes': self.attributes,
|
||||
'is_scrollable': self.is_scrollable,
|
||||
'session_id': self.session_id,
|
||||
'target_id': self.target_id,
|
||||
'frame_id': self.frame_id,
|
||||
'content_document': self.content_document.__json__() if self.content_document else None,
|
||||
'shadow_root_type': self.shadow_root_type,
|
||||
'ax_node': asdict(self.ax_node) if self.ax_node else None,
|
||||
'snapshot_node': asdict(self.snapshot_node) if self.snapshot_node else None,
|
||||
# these two in the end, so it's easier to read json
|
||||
'shadow_roots': [r.__json__() for r in self.shadow_roots] if self.shadow_roots else [],
|
||||
'children_nodes': [c.__json__() for c in self.children_nodes] if self.children_nodes else [],
|
||||
}
|
||||
|
||||
def get_all_children_text(self, max_depth: int = -1) -> str:
|
||||
text_parts = []
|
||||
|
||||
def collect_text(node: EnhancedDOMTreeNode, current_depth: int) -> None:
|
||||
if max_depth != -1 and current_depth > max_depth:
|
||||
return
|
||||
|
||||
# Skip this branch if we hit a highlighted element (except for the current node)
|
||||
# TODO: think whether if makese sense to add text until the next clickable element or everything from children
|
||||
# if node.node_type == NodeType.ELEMENT_NODE
|
||||
# if isinstance(node, DOMElementNode) and node != self and node.highlight_index is not None:
|
||||
# return
|
||||
|
||||
if node.node_type == NodeType.TEXT_NODE:
|
||||
text_parts.append(node.node_value)
|
||||
elif node.node_type == NodeType.ELEMENT_NODE:
|
||||
for child in node.children:
|
||||
collect_text(child, current_depth + 1)
|
||||
|
||||
collect_text(self, 0)
|
||||
return '\n'.join(text_parts).strip()
|
||||
|
||||
def __repr__(self) -> str:
|
||||
"""
|
||||
@DEV ! don't display this to the LLM, it's SUPER long
|
||||
"""
|
||||
attributes = ', '.join([f'{k}={v}' for k, v in self.attributes.items()])
|
||||
is_scrollable = getattr(self, 'is_scrollable', False)
|
||||
num_children = len(self.children_nodes or [])
|
||||
return (
|
||||
f'<{self.tag_name} {attributes} is_scrollable={is_scrollable} '
|
||||
f'num_children={num_children} >{self.node_value}</{self.tag_name}>'
|
||||
)
|
||||
|
||||
def llm_representation(self, max_text_length: int = 100) -> str:
|
||||
"""
|
||||
Token friendly representation of the node, used in the LLM
|
||||
"""
|
||||
|
||||
return f'<{self.tag_name}>{cap_text_length(self.get_all_children_text(), max_text_length) or ""}'
|
||||
|
||||
def get_meaningful_text_for_llm(self) -> str:
|
||||
"""
|
||||
Get the meaningful text content that the LLM actually sees for this element.
|
||||
This matches exactly what goes into the DOMTreeSerializer output.
|
||||
"""
|
||||
meaningful_text = ''
|
||||
if hasattr(self, 'attributes') and self.attributes:
|
||||
# Priority order: value, aria-label, title, placeholder, alt, text content
|
||||
for attr in ['value', 'aria-label', 'title', 'placeholder', 'alt']:
|
||||
if attr in self.attributes and self.attributes[attr]:
|
||||
meaningful_text = self.attributes[attr]
|
||||
break
|
||||
|
||||
# Fallback to text content if no meaningful attributes
|
||||
if not meaningful_text:
|
||||
meaningful_text = self.get_all_children_text()
|
||||
|
||||
return meaningful_text.strip()
|
||||
|
||||
@property
|
||||
def is_actually_scrollable(self) -> bool:
|
||||
"""
|
||||
Enhanced scroll detection that combines CDP detection with CSS analysis.
|
||||
|
||||
This detects scrollable elements that Chrome's CDP might miss, which is common
|
||||
in iframes and dynamically sized containers.
|
||||
"""
|
||||
# First check if CDP already detected it as scrollable
|
||||
if self.is_scrollable:
|
||||
return True
|
||||
|
||||
# Enhanced detection for elements CDP missed
|
||||
if not self.snapshot_node:
|
||||
return False
|
||||
|
||||
# Check scroll vs client rects - this is the most reliable indicator
|
||||
scroll_rects = self.snapshot_node.scrollRects
|
||||
client_rects = self.snapshot_node.clientRects
|
||||
|
||||
if scroll_rects and client_rects:
|
||||
# Content is larger than visible area = scrollable
|
||||
has_vertical_scroll = scroll_rects.height > client_rects.height + 1 # +1 for rounding
|
||||
has_horizontal_scroll = scroll_rects.width > client_rects.width + 1
|
||||
|
||||
if has_vertical_scroll or has_horizontal_scroll:
|
||||
# Also check CSS to make sure scrolling is allowed
|
||||
if self.snapshot_node.computed_styles:
|
||||
styles = self.snapshot_node.computed_styles
|
||||
|
||||
overflow = styles.get('overflow', 'visible').lower()
|
||||
overflow_x = styles.get('overflow-x', overflow).lower()
|
||||
overflow_y = styles.get('overflow-y', overflow).lower()
|
||||
|
||||
# Only allow scrolling if overflow is explicitly set to auto, scroll, or overlay
|
||||
# Do NOT consider 'visible' overflow as scrollable - this was causing the issue
|
||||
allows_scroll = (
|
||||
overflow in ['auto', 'scroll', 'overlay']
|
||||
or overflow_x in ['auto', 'scroll', 'overlay']
|
||||
or overflow_y in ['auto', 'scroll', 'overlay']
|
||||
)
|
||||
|
||||
return allows_scroll
|
||||
else:
|
||||
# No CSS info, but content overflows - be more conservative
|
||||
# Only consider it scrollable if it's a common scrollable container element
|
||||
scrollable_tags = {'div', 'main', 'section', 'article', 'aside', 'body', 'html'}
|
||||
return self.tag_name.lower() in scrollable_tags
|
||||
|
||||
return False
|
||||
|
||||
@property
|
||||
def should_show_scroll_info(self) -> bool:
|
||||
"""
|
||||
Simple check: show scroll info only if this element is scrollable
|
||||
and doesn't have a scrollable parent (to avoid nested scroll spam).
|
||||
|
||||
Special case for iframes: Always show scroll info since Chrome might not
|
||||
always detect iframe scrollability correctly (scrollHeight: 0 issue).
|
||||
"""
|
||||
# Special case: Always show scroll info for iframe elements
|
||||
# Even if not detected as scrollable, they might have scrollable content
|
||||
if self.tag_name.lower() == 'iframe':
|
||||
return True
|
||||
|
||||
# Must be scrollable first for non-iframe elements
|
||||
if not (self.is_scrollable or self.is_actually_scrollable):
|
||||
return False
|
||||
|
||||
# Always show for iframe content documents (body/html)
|
||||
if self.tag_name.lower() in {'body', 'html'}:
|
||||
return True
|
||||
|
||||
# Don't show if parent is already scrollable (avoid nested spam)
|
||||
if self.parent_node and (self.parent_node.is_scrollable or self.parent_node.is_actually_scrollable):
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
def _find_html_in_content_document(self) -> 'EnhancedDOMTreeNode | None':
|
||||
"""Find HTML element in iframe content document."""
|
||||
if not self.content_document:
|
||||
return None
|
||||
|
||||
# Check if content document itself is HTML
|
||||
if self.content_document.tag_name.lower() == 'html':
|
||||
return self.content_document
|
||||
|
||||
# Look through children for HTML element
|
||||
if self.content_document.children_nodes:
|
||||
for child in self.content_document.children_nodes:
|
||||
if child.tag_name.lower() == 'html':
|
||||
return child
|
||||
|
||||
return None
|
||||
|
||||
@property
|
||||
def scroll_info(self) -> dict[str, Any] | None:
|
||||
"""Calculate scroll information for this element if it's scrollable."""
|
||||
if not self.is_actually_scrollable or not self.snapshot_node:
|
||||
return None
|
||||
|
||||
# Get scroll and client rects from snapshot data
|
||||
scroll_rects = self.snapshot_node.scrollRects
|
||||
client_rects = self.snapshot_node.clientRects
|
||||
bounds = self.snapshot_node.bounds
|
||||
|
||||
if not scroll_rects or not client_rects:
|
||||
return None
|
||||
|
||||
# Calculate scroll position and percentages
|
||||
scroll_top = scroll_rects.y
|
||||
scroll_left = scroll_rects.x
|
||||
|
||||
# Total scrollable height and width
|
||||
scrollable_height = scroll_rects.height
|
||||
scrollable_width = scroll_rects.width
|
||||
|
||||
# Visible (client) dimensions
|
||||
visible_height = client_rects.height
|
||||
visible_width = client_rects.width
|
||||
|
||||
# Calculate how much content is above/below/left/right of current view
|
||||
content_above = max(0, scroll_top)
|
||||
content_below = max(0, scrollable_height - visible_height - scroll_top)
|
||||
content_left = max(0, scroll_left)
|
||||
content_right = max(0, scrollable_width - visible_width - scroll_left)
|
||||
|
||||
# Calculate scroll percentages
|
||||
vertical_scroll_percentage = 0
|
||||
horizontal_scroll_percentage = 0
|
||||
|
||||
if scrollable_height > visible_height:
|
||||
max_scroll_top = scrollable_height - visible_height
|
||||
vertical_scroll_percentage = (scroll_top / max_scroll_top) * 100 if max_scroll_top > 0 else 0
|
||||
|
||||
if scrollable_width > visible_width:
|
||||
max_scroll_left = scrollable_width - visible_width
|
||||
horizontal_scroll_percentage = (scroll_left / max_scroll_left) * 100 if max_scroll_left > 0 else 0
|
||||
|
||||
# Calculate pages equivalent (using visible height as page unit)
|
||||
pages_above = content_above / visible_height if visible_height > 0 else 0
|
||||
pages_below = content_below / visible_height if visible_height > 0 else 0
|
||||
total_pages = scrollable_height / visible_height if visible_height > 0 else 1
|
||||
|
||||
return {
|
||||
'scroll_top': scroll_top,
|
||||
'scroll_left': scroll_left,
|
||||
'scrollable_height': scrollable_height,
|
||||
'scrollable_width': scrollable_width,
|
||||
'visible_height': visible_height,
|
||||
'visible_width': visible_width,
|
||||
'content_above': content_above,
|
||||
'content_below': content_below,
|
||||
'content_left': content_left,
|
||||
'content_right': content_right,
|
||||
'vertical_scroll_percentage': round(vertical_scroll_percentage, 1),
|
||||
'horizontal_scroll_percentage': round(horizontal_scroll_percentage, 1),
|
||||
'pages_above': round(pages_above, 1),
|
||||
'pages_below': round(pages_below, 1),
|
||||
'total_pages': round(total_pages, 1),
|
||||
'can_scroll_up': content_above > 0,
|
||||
'can_scroll_down': content_below > 0,
|
||||
'can_scroll_left': content_left > 0,
|
||||
'can_scroll_right': content_right > 0,
|
||||
}
|
||||
|
||||
def get_scroll_info_text(self) -> str:
|
||||
"""Get human-readable scroll information text for this element."""
|
||||
# Special case for iframes: check content document for scroll info
|
||||
if self.tag_name.lower() == 'iframe':
|
||||
# Try to get scroll info from the HTML document inside the iframe
|
||||
if self.content_document:
|
||||
# Look for HTML element in content document
|
||||
html_element = self._find_html_in_content_document()
|
||||
if html_element and html_element.scroll_info:
|
||||
info = html_element.scroll_info
|
||||
# Provide minimal but useful scroll info
|
||||
pages_below = info.get('pages_below', 0)
|
||||
pages_above = info.get('pages_above', 0)
|
||||
v_pct = int(info.get('vertical_scroll_percentage', 0))
|
||||
|
||||
if pages_below > 0 or pages_above > 0:
|
||||
return f'scroll: {pages_above:.1f}↑ {pages_below:.1f}↓ {v_pct}%'
|
||||
|
||||
return 'scroll'
|
||||
|
||||
scroll_info = self.scroll_info
|
||||
if not scroll_info:
|
||||
return ''
|
||||
|
||||
parts = []
|
||||
|
||||
# Vertical scroll info (concise format)
|
||||
if scroll_info['scrollable_height'] > scroll_info['visible_height']:
|
||||
parts.append(f'{scroll_info["pages_above"]:.1f} pages above, {scroll_info["pages_below"]:.1f} pages below')
|
||||
|
||||
# Horizontal scroll info (concise format)
|
||||
if scroll_info['scrollable_width'] > scroll_info['visible_width']:
|
||||
parts.append(f'horizontal {scroll_info["horizontal_scroll_percentage"]:.0f}%')
|
||||
|
||||
return ' '.join(parts)
|
||||
|
||||
@property
|
||||
def element_hash(self) -> int:
|
||||
return hash(self)
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f'[<{self.tag_name}>#{self.frame_id[-4:] if self.frame_id else "?"}:{self.element_index}]'
|
||||
|
||||
def __hash__(self) -> int:
|
||||
"""
|
||||
Hash the element based on its parent branch path and attributes.
|
||||
|
||||
TODO: migrate this to use only backendNodeId + current SessionId
|
||||
"""
|
||||
|
||||
# Get parent branch path
|
||||
parent_branch_path = self._get_parent_branch_path()
|
||||
parent_branch_path_string = '/'.join(parent_branch_path)
|
||||
|
||||
attributes_string = ''.join(
|
||||
f'{k}={v}' for k, v in sorted((k, v) for k, v in self.attributes.items() if k in STATIC_ATTRIBUTES)
|
||||
)
|
||||
|
||||
# Combine both for final hash
|
||||
combined_string = f'{parent_branch_path_string}|{attributes_string}'
|
||||
element_hash = hashlib.sha256(combined_string.encode()).hexdigest()
|
||||
|
||||
# Convert to int for __hash__ return type - use first 16 chars and convert from hex to int
|
||||
return int(element_hash[:16], 16)
|
||||
|
||||
def parent_branch_hash(self) -> int:
|
||||
"""
|
||||
Hash the element based on its parent branch path and attributes.
|
||||
"""
|
||||
parent_branch_path = self._get_parent_branch_path()
|
||||
parent_branch_path_string = '/'.join(parent_branch_path)
|
||||
element_hash = hashlib.sha256(parent_branch_path_string.encode()).hexdigest()
|
||||
|
||||
return int(element_hash[:16], 16)
|
||||
|
||||
def _get_parent_branch_path(self) -> list[str]:
|
||||
"""Get the parent branch path as a list of tag names from root to current element."""
|
||||
parents: list['EnhancedDOMTreeNode'] = []
|
||||
current_element: 'EnhancedDOMTreeNode | None' = self
|
||||
|
||||
while current_element is not None:
|
||||
if current_element.node_type == NodeType.ELEMENT_NODE:
|
||||
parents.append(current_element)
|
||||
current_element = current_element.parent_node
|
||||
|
||||
parents.reverse()
|
||||
return [parent.tag_name for parent in parents]
|
||||
|
||||
|
||||
DOMSelectorMap = dict[int, EnhancedDOMTreeNode]
|
||||
|
||||
|
||||
@dataclass
|
||||
class SerializedDOMState:
|
||||
_root: SimplifiedNode | None
|
||||
"""Not meant to be used directly, use `llm_representation` instead"""
|
||||
|
||||
selector_map: DOMSelectorMap
|
||||
|
||||
@observe_debug(ignore_input=True, ignore_output=True, name='llm_representation')
|
||||
def llm_representation(
|
||||
self,
|
||||
include_attributes: list[str] | None = None,
|
||||
) -> str:
|
||||
"""Kinda ugly, but leaving this as an internal method because include_attributes are a parameter on the agent, so we need to leave it as a 2 step process"""
|
||||
from browser_use.dom.serializer.serializer import DOMTreeSerializer
|
||||
|
||||
if not self._root:
|
||||
return 'Empty DOM tree (you might have to wait for the page to load)'
|
||||
|
||||
include_attributes = include_attributes or DEFAULT_INCLUDE_ATTRIBUTES
|
||||
|
||||
return DOMTreeSerializer.serialize_tree(self._root, include_attributes)
|
||||
|
||||
|
||||
@dataclass
|
||||
class DOMInteractedElement:
|
||||
"""
|
||||
DOMInteractedElement is a class that represents a DOM element that has been interacted with.
|
||||
It is used to store the DOM element that has been interacted with and to store the DOM element that has been interacted with.
|
||||
|
||||
TODO: this is a bit of a hack, we should probably have a better way to do this
|
||||
"""
|
||||
|
||||
node_id: int
|
||||
backend_node_id: int
|
||||
frame_id: str | None
|
||||
|
||||
node_type: NodeType
|
||||
node_value: str
|
||||
node_name: str
|
||||
attributes: dict[str, str] | None
|
||||
|
||||
bounds: DOMRect | None
|
||||
|
||||
x_path: str
|
||||
|
||||
element_hash: int
|
||||
|
||||
def to_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
'node_id': self.node_id,
|
||||
'backend_node_id': self.backend_node_id,
|
||||
'frame_id': self.frame_id,
|
||||
'node_type': self.node_type.value,
|
||||
'node_value': self.node_value,
|
||||
'node_name': self.node_name,
|
||||
'attributes': self.attributes,
|
||||
'x_path': self.x_path,
|
||||
'element_hash': self.element_hash,
|
||||
'bounds': self.bounds.to_dict() if self.bounds else None,
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def load_from_enhanced_dom_tree(cls, enhanced_dom_tree: EnhancedDOMTreeNode) -> 'DOMInteractedElement':
|
||||
return cls(
|
||||
node_id=enhanced_dom_tree.node_id,
|
||||
backend_node_id=enhanced_dom_tree.backend_node_id,
|
||||
frame_id=enhanced_dom_tree.frame_id,
|
||||
node_type=enhanced_dom_tree.node_type,
|
||||
node_value=enhanced_dom_tree.node_value,
|
||||
node_name=enhanced_dom_tree.node_name,
|
||||
attributes=enhanced_dom_tree.attributes,
|
||||
bounds=enhanced_dom_tree.snapshot_node.bounds if enhanced_dom_tree.snapshot_node else None,
|
||||
x_path=enhanced_dom_tree.xpath,
|
||||
element_hash=hash(enhanced_dom_tree),
|
||||
)
|
||||
@@ -0,0 +1,5 @@
|
||||
class LLMException(Exception):
|
||||
def __init__(self, status_code, message):
|
||||
self.status_code = status_code
|
||||
self.message = message
|
||||
super().__init__(f'Error {status_code}: {message}')
|
||||
@@ -0,0 +1,504 @@
|
||||
import asyncio
|
||||
import re
|
||||
import shutil
|
||||
from abc import ABC, abstractmethod
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
from reportlab.lib.pagesizes import letter
|
||||
from reportlab.lib.styles import getSampleStyleSheet
|
||||
from reportlab.platypus import Paragraph, SimpleDocTemplate, Spacer
|
||||
|
||||
INVALID_FILENAME_ERROR_MESSAGE = 'Error: Invalid filename format. Must be alphanumeric with supported extension.'
|
||||
DEFAULT_FILE_SYSTEM_PATH = 'browseruse_agent_data'
|
||||
|
||||
|
||||
class FileSystemError(Exception):
|
||||
"""Custom exception for file system operations that should be shown to LLM"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class BaseFile(BaseModel, ABC):
|
||||
"""Base class for all file types"""
|
||||
|
||||
name: str
|
||||
content: str = ''
|
||||
|
||||
# --- Subclass must define this ---
|
||||
@property
|
||||
@abstractmethod
|
||||
def extension(self) -> str:
|
||||
"""File extension (e.g. 'txt', 'md')"""
|
||||
pass
|
||||
|
||||
def write_file_content(self, content: str) -> None:
|
||||
"""Update internal content (formatted)"""
|
||||
self.update_content(content)
|
||||
|
||||
def append_file_content(self, content: str) -> None:
|
||||
"""Append content to internal content"""
|
||||
self.update_content(self.content + content)
|
||||
|
||||
# --- These are shared and implemented here ---
|
||||
|
||||
def update_content(self, content: str) -> None:
|
||||
self.content = content
|
||||
|
||||
def sync_to_disk_sync(self, path: Path) -> None:
|
||||
file_path = path / self.full_name
|
||||
file_path.write_text(self.content)
|
||||
|
||||
async def sync_to_disk(self, path: Path) -> None:
|
||||
file_path = path / self.full_name
|
||||
with ThreadPoolExecutor() as executor:
|
||||
await asyncio.get_event_loop().run_in_executor(executor, lambda: file_path.write_text(self.content))
|
||||
|
||||
async def write(self, content: str, path: Path) -> None:
|
||||
self.write_file_content(content)
|
||||
await self.sync_to_disk(path)
|
||||
|
||||
async def append(self, content: str, path: Path) -> None:
|
||||
self.append_file_content(content)
|
||||
await self.sync_to_disk(path)
|
||||
|
||||
def read(self) -> str:
|
||||
return self.content
|
||||
|
||||
@property
|
||||
def full_name(self) -> str:
|
||||
return f'{self.name}.{self.extension}'
|
||||
|
||||
@property
|
||||
def get_size(self) -> int:
|
||||
return len(self.content)
|
||||
|
||||
@property
|
||||
def get_line_count(self) -> int:
|
||||
return len(self.content.splitlines())
|
||||
|
||||
|
||||
class MarkdownFile(BaseFile):
|
||||
"""Markdown file implementation"""
|
||||
|
||||
@property
|
||||
def extension(self) -> str:
|
||||
return 'md'
|
||||
|
||||
|
||||
class TxtFile(BaseFile):
|
||||
"""Plain text file implementation"""
|
||||
|
||||
@property
|
||||
def extension(self) -> str:
|
||||
return 'txt'
|
||||
|
||||
|
||||
class JsonFile(BaseFile):
|
||||
"""JSON file implementation"""
|
||||
|
||||
@property
|
||||
def extension(self) -> str:
|
||||
return 'json'
|
||||
|
||||
|
||||
class CsvFile(BaseFile):
|
||||
"""CSV file implementation"""
|
||||
|
||||
@property
|
||||
def extension(self) -> str:
|
||||
return 'csv'
|
||||
|
||||
|
||||
class PdfFile(BaseFile):
|
||||
"""PDF file implementation"""
|
||||
|
||||
@property
|
||||
def extension(self) -> str:
|
||||
return 'pdf'
|
||||
|
||||
def sync_to_disk_sync(self, path: Path) -> None:
|
||||
file_path = path / self.full_name
|
||||
try:
|
||||
# Create PDF document
|
||||
doc = SimpleDocTemplate(str(file_path), pagesize=letter)
|
||||
styles = getSampleStyleSheet()
|
||||
story = []
|
||||
|
||||
# Convert markdown content to simple text and add to PDF
|
||||
# For basic implementation, we'll treat content as plain text
|
||||
# This avoids the AGPL license issue while maintaining functionality
|
||||
content_lines = self.content.split('\n')
|
||||
|
||||
for line in content_lines:
|
||||
if line.strip():
|
||||
# Handle basic markdown headers
|
||||
if line.startswith('# '):
|
||||
para = Paragraph(line[2:], styles['Title'])
|
||||
elif line.startswith('## '):
|
||||
para = Paragraph(line[3:], styles['Heading1'])
|
||||
elif line.startswith('### '):
|
||||
para = Paragraph(line[4:], styles['Heading2'])
|
||||
else:
|
||||
para = Paragraph(line, styles['Normal'])
|
||||
story.append(para)
|
||||
else:
|
||||
story.append(Spacer(1, 6))
|
||||
|
||||
doc.build(story)
|
||||
except Exception as e:
|
||||
raise FileSystemError(f"Error: Could not write to file '{self.full_name}'. {str(e)}")
|
||||
|
||||
async def sync_to_disk(self, path: Path) -> None:
|
||||
with ThreadPoolExecutor() as executor:
|
||||
await asyncio.get_event_loop().run_in_executor(executor, lambda: self.sync_to_disk_sync(path))
|
||||
|
||||
|
||||
class FileSystemState(BaseModel):
|
||||
"""Serializable state of the file system"""
|
||||
|
||||
files: dict[str, dict[str, Any]] = Field(default_factory=dict) # full filename -> file data
|
||||
base_dir: str
|
||||
extracted_content_count: int = 0
|
||||
|
||||
|
||||
class FileSystem:
|
||||
"""Enhanced file system with in-memory storage and multiple file type support"""
|
||||
|
||||
def __init__(self, base_dir: str | Path, create_default_files: bool = True):
|
||||
# Handle the Path conversion before calling super().__init__
|
||||
self.base_dir = Path(base_dir) if isinstance(base_dir, str) else base_dir
|
||||
self.base_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Create and use a dedicated subfolder for all operations
|
||||
self.data_dir = self.base_dir / DEFAULT_FILE_SYSTEM_PATH
|
||||
if self.data_dir.exists():
|
||||
# clean the data directory
|
||||
shutil.rmtree(self.data_dir)
|
||||
self.data_dir.mkdir(exist_ok=True)
|
||||
|
||||
self._file_types: dict[str, type[BaseFile]] = {
|
||||
'md': MarkdownFile,
|
||||
'txt': TxtFile,
|
||||
'json': JsonFile,
|
||||
'csv': CsvFile,
|
||||
'pdf': PdfFile,
|
||||
}
|
||||
|
||||
self.files = {}
|
||||
if create_default_files:
|
||||
self.default_files = ['todo.md']
|
||||
self._create_default_files()
|
||||
|
||||
self.extracted_content_count = 0
|
||||
|
||||
def get_allowed_extensions(self) -> list[str]:
|
||||
"""Get allowed extensions"""
|
||||
return list(self._file_types.keys())
|
||||
|
||||
def _get_file_type_class(self, extension: str) -> type[BaseFile] | None:
|
||||
"""Get the appropriate file class for an extension."""
|
||||
return self._file_types.get(extension.lower(), None)
|
||||
|
||||
def _create_default_files(self) -> None:
|
||||
"""Create default results and todo files"""
|
||||
for full_filename in self.default_files:
|
||||
name_without_ext, extension = self._parse_filename(full_filename)
|
||||
file_class = self._get_file_type_class(extension)
|
||||
if not file_class:
|
||||
raise ValueError(f"Error: Invalid file extension '{extension}' for file '{full_filename}'.")
|
||||
|
||||
file_obj = file_class(name=name_without_ext)
|
||||
self.files[full_filename] = file_obj # Use full filename as key
|
||||
file_obj.sync_to_disk_sync(self.data_dir)
|
||||
|
||||
def _is_valid_filename(self, file_name: str) -> bool:
|
||||
"""Check if filename matches the required pattern: name.extension"""
|
||||
# Build extensions pattern from _file_types
|
||||
extensions = '|'.join(self._file_types.keys())
|
||||
pattern = rf'^[a-zA-Z0-9_\-]+\.({extensions})$'
|
||||
return bool(re.match(pattern, file_name))
|
||||
|
||||
def _parse_filename(self, filename: str) -> tuple[str, str]:
|
||||
"""Parse filename into name and extension. Always check _is_valid_filename first."""
|
||||
name, extension = filename.rsplit('.', 1)
|
||||
return name, extension.lower()
|
||||
|
||||
def get_dir(self) -> Path:
|
||||
"""Get the file system directory"""
|
||||
return self.data_dir
|
||||
|
||||
def get_file(self, full_filename: str) -> BaseFile | None:
|
||||
"""Get a file object by full filename"""
|
||||
if not self._is_valid_filename(full_filename):
|
||||
return None
|
||||
|
||||
# Use full filename as key
|
||||
return self.files.get(full_filename)
|
||||
|
||||
def list_files(self) -> list[str]:
|
||||
"""List all files in the system"""
|
||||
return [file_obj.full_name for file_obj in self.files.values()]
|
||||
|
||||
def display_file(self, full_filename: str) -> str | None:
|
||||
"""Display file content using file-specific display method"""
|
||||
if not self._is_valid_filename(full_filename):
|
||||
return None
|
||||
|
||||
file_obj = self.get_file(full_filename)
|
||||
if not file_obj:
|
||||
return None
|
||||
|
||||
return file_obj.read()
|
||||
|
||||
async def read_file(self, full_filename: str, external_file: bool = False) -> str:
|
||||
"""Read file content using file-specific read method and return appropriate message to LLM"""
|
||||
if external_file:
|
||||
try:
|
||||
try:
|
||||
_, extension = self._parse_filename(full_filename)
|
||||
except Exception:
|
||||
return f'Error: Invalid filename format {full_filename}. Must be alphanumeric with a supported extension.'
|
||||
if extension in ['md', 'txt', 'json', 'csv']:
|
||||
import anyio
|
||||
|
||||
async with await anyio.open_file(full_filename, 'r') as f:
|
||||
content = await f.read()
|
||||
return f'Read from file {full_filename}.\n<content>\n{content}\n</content>'
|
||||
elif extension == 'pdf':
|
||||
import pypdf
|
||||
|
||||
reader = pypdf.PdfReader(full_filename)
|
||||
num_pages = len(reader.pages)
|
||||
MAX_PDF_PAGES = 10
|
||||
extra_pages = num_pages - MAX_PDF_PAGES
|
||||
extracted_text = ''
|
||||
for page in reader.pages[:MAX_PDF_PAGES]:
|
||||
extracted_text += page.extract_text()
|
||||
extra_pages_text = f'{extra_pages} more pages...' if extra_pages > 0 else ''
|
||||
return f'Read from file {full_filename}.\n<content>\n{extracted_text}\n{extra_pages_text}</content>'
|
||||
else:
|
||||
return f'Error: Cannot read file {full_filename} as {extension} extension is not supported.'
|
||||
except FileNotFoundError:
|
||||
return f"Error: File '{full_filename}' not found."
|
||||
except PermissionError:
|
||||
return f"Error: Permission denied to read file '{full_filename}'."
|
||||
except Exception as e:
|
||||
return f"Error: Could not read file '{full_filename}'."
|
||||
|
||||
if not self._is_valid_filename(full_filename):
|
||||
return INVALID_FILENAME_ERROR_MESSAGE
|
||||
|
||||
file_obj = self.get_file(full_filename)
|
||||
if not file_obj:
|
||||
return f"File '{full_filename}' not found."
|
||||
|
||||
try:
|
||||
content = file_obj.read()
|
||||
return f'Read from file {full_filename}.\n<content>\n{content}\n</content>'
|
||||
except FileSystemError as e:
|
||||
return str(e)
|
||||
except Exception:
|
||||
return f"Error: Could not read file '{full_filename}'."
|
||||
|
||||
async def write_file(self, full_filename: str, content: str) -> str:
|
||||
"""Write content to file using file-specific write method"""
|
||||
if not self._is_valid_filename(full_filename):
|
||||
return INVALID_FILENAME_ERROR_MESSAGE
|
||||
|
||||
try:
|
||||
name_without_ext, extension = self._parse_filename(full_filename)
|
||||
file_class = self._get_file_type_class(extension)
|
||||
if not file_class:
|
||||
raise ValueError(f"Error: Invalid file extension '{extension}' for file '{full_filename}'.")
|
||||
|
||||
# Create or get existing file using full filename as key
|
||||
if full_filename in self.files:
|
||||
file_obj = self.files[full_filename]
|
||||
else:
|
||||
file_obj = file_class(name=name_without_ext)
|
||||
self.files[full_filename] = file_obj # Use full filename as key
|
||||
|
||||
# Use file-specific write method
|
||||
await file_obj.write(content, self.data_dir)
|
||||
return f'Data written to file {full_filename} successfully.'
|
||||
except FileSystemError as e:
|
||||
return str(e)
|
||||
except Exception as e:
|
||||
return f"Error: Could not write to file '{full_filename}'. {str(e)}"
|
||||
|
||||
async def append_file(self, full_filename: str, content: str) -> str:
|
||||
"""Append content to file using file-specific append method"""
|
||||
if not self._is_valid_filename(full_filename):
|
||||
return INVALID_FILENAME_ERROR_MESSAGE
|
||||
|
||||
file_obj = self.get_file(full_filename)
|
||||
if not file_obj:
|
||||
return f"File '{full_filename}' not found."
|
||||
|
||||
try:
|
||||
await file_obj.append(content, self.data_dir)
|
||||
return f'Data appended to file {full_filename} successfully.'
|
||||
except FileSystemError as e:
|
||||
return str(e)
|
||||
except Exception as e:
|
||||
return f"Error: Could not append to file '{full_filename}'. {str(e)}"
|
||||
|
||||
async def replace_file_str(self, full_filename: str, old_str: str, new_str: str) -> str:
|
||||
"""Replace old_str with new_str in file_name"""
|
||||
if not self._is_valid_filename(full_filename):
|
||||
return INVALID_FILENAME_ERROR_MESSAGE
|
||||
|
||||
if not old_str:
|
||||
return 'Error: Cannot replace empty string. Please provide a non-empty string to replace.'
|
||||
|
||||
file_obj = self.get_file(full_filename)
|
||||
if not file_obj:
|
||||
return f"File '{full_filename}' not found."
|
||||
|
||||
try:
|
||||
content = file_obj.read()
|
||||
content = content.replace(old_str, new_str)
|
||||
await file_obj.write(content, self.data_dir)
|
||||
return f'Successfully replaced all occurrences of "{old_str}" with "{new_str}" in file {full_filename}'
|
||||
except FileSystemError as e:
|
||||
return str(e)
|
||||
except Exception as e:
|
||||
return f"Error: Could not replace string in file '{full_filename}'. {str(e)}"
|
||||
|
||||
async def save_extracted_content(self, content: str) -> str:
|
||||
"""Save extracted content to a numbered file"""
|
||||
initial_filename = f'extracted_content_{self.extracted_content_count}'
|
||||
extracted_filename = f'{initial_filename}.md'
|
||||
file_obj = MarkdownFile(name=initial_filename)
|
||||
await file_obj.write(content, self.data_dir)
|
||||
self.files[extracted_filename] = file_obj
|
||||
self.extracted_content_count += 1
|
||||
return f'Extracted content saved to file {extracted_filename} successfully.'
|
||||
|
||||
def describe(self) -> str:
|
||||
"""List all files with their content information using file-specific display methods"""
|
||||
DISPLAY_CHARS = 400
|
||||
description = ''
|
||||
|
||||
for file_obj in self.files.values():
|
||||
# Skip todo.md from description
|
||||
if file_obj.full_name == 'todo.md':
|
||||
continue
|
||||
|
||||
content = file_obj.read()
|
||||
|
||||
# Handle empty files
|
||||
if not content:
|
||||
description += f'<file>\n{file_obj.full_name} - [empty file]\n</file>\n'
|
||||
continue
|
||||
|
||||
lines = content.splitlines()
|
||||
line_count = len(lines)
|
||||
|
||||
# For small files, display the entire content
|
||||
whole_file_description = (
|
||||
f'<file>\n{file_obj.full_name} - {line_count} lines\n<content>\n{content}\n</content>\n</file>\n'
|
||||
)
|
||||
if len(content) < int(1.5 * DISPLAY_CHARS):
|
||||
description += whole_file_description
|
||||
continue
|
||||
|
||||
# For larger files, display start and end previews
|
||||
half_display_chars = DISPLAY_CHARS // 2
|
||||
|
||||
# Get start preview
|
||||
start_preview = ''
|
||||
start_line_count = 0
|
||||
chars_count = 0
|
||||
for line in lines:
|
||||
if chars_count + len(line) + 1 > half_display_chars:
|
||||
break
|
||||
start_preview += line + '\n'
|
||||
chars_count += len(line) + 1
|
||||
start_line_count += 1
|
||||
|
||||
# Get end preview
|
||||
end_preview = ''
|
||||
end_line_count = 0
|
||||
chars_count = 0
|
||||
for line in reversed(lines):
|
||||
if chars_count + len(line) + 1 > half_display_chars:
|
||||
break
|
||||
end_preview = line + '\n' + end_preview
|
||||
chars_count += len(line) + 1
|
||||
end_line_count += 1
|
||||
|
||||
# Calculate lines in between
|
||||
middle_line_count = line_count - start_line_count - end_line_count
|
||||
if middle_line_count <= 0:
|
||||
description += whole_file_description
|
||||
continue
|
||||
|
||||
start_preview = start_preview.strip('\n').rstrip()
|
||||
end_preview = end_preview.strip('\n').rstrip()
|
||||
|
||||
# Format output
|
||||
if not (start_preview or end_preview):
|
||||
description += f'<file>\n{file_obj.full_name} - {line_count} lines\n<content>\n{middle_line_count} lines...\n</content>\n</file>\n'
|
||||
else:
|
||||
description += f'<file>\n{file_obj.full_name} - {line_count} lines\n<content>\n{start_preview}\n'
|
||||
description += f'... {middle_line_count} more lines ...\n'
|
||||
description += f'{end_preview}\n'
|
||||
description += '</content>\n</file>\n'
|
||||
|
||||
return description.strip('\n')
|
||||
|
||||
def get_todo_contents(self) -> str:
|
||||
"""Get todo file contents"""
|
||||
todo_file = self.get_file('todo.md')
|
||||
return todo_file.read() if todo_file else ''
|
||||
|
||||
def get_state(self) -> FileSystemState:
|
||||
"""Get serializable state of the file system"""
|
||||
files_data = {}
|
||||
for full_filename, file_obj in self.files.items():
|
||||
files_data[full_filename] = {'type': file_obj.__class__.__name__, 'data': file_obj.model_dump()}
|
||||
|
||||
return FileSystemState(
|
||||
files=files_data, base_dir=str(self.base_dir), extracted_content_count=self.extracted_content_count
|
||||
)
|
||||
|
||||
def nuke(self) -> None:
|
||||
"""Delete the file system directory"""
|
||||
shutil.rmtree(self.data_dir)
|
||||
|
||||
@classmethod
|
||||
def from_state(cls, state: FileSystemState) -> 'FileSystem':
|
||||
"""Restore file system from serializable state at the exact same location"""
|
||||
# Create file system without default files
|
||||
fs = cls(base_dir=Path(state.base_dir), create_default_files=False)
|
||||
fs.extracted_content_count = state.extracted_content_count
|
||||
|
||||
# Restore all files
|
||||
for full_filename, file_data in state.files.items():
|
||||
file_type = file_data['type']
|
||||
file_info = file_data['data']
|
||||
|
||||
# Create the appropriate file object based on type
|
||||
if file_type == 'MarkdownFile':
|
||||
file_obj = MarkdownFile(**file_info)
|
||||
elif file_type == 'TxtFile':
|
||||
file_obj = TxtFile(**file_info)
|
||||
elif file_type == 'JsonFile':
|
||||
file_obj = JsonFile(**file_info)
|
||||
elif file_type == 'CsvFile':
|
||||
file_obj = CsvFile(**file_info)
|
||||
elif file_type == 'PdfFile':
|
||||
file_obj = PdfFile(**file_info)
|
||||
else:
|
||||
# Skip unknown file types
|
||||
continue
|
||||
|
||||
# Add to files dict and sync to disk
|
||||
fs.files[full_filename] = file_obj
|
||||
file_obj.sync_to_disk_sync(fs.data_dir)
|
||||
|
||||
return fs
|
||||
@@ -0,0 +1,24 @@
|
||||
"""
|
||||
Gmail Integration for Browser Use
|
||||
Provides Gmail API integration for email reading and verification code extraction.
|
||||
This integration enables agents to read email content and extract verification codes themselves.
|
||||
Usage:
|
||||
from browser_use.integrations.gmail import GmailService, register_gmail_actions
|
||||
# Option 1: Register Gmail actions with file-based authentication
|
||||
tools = Tools()
|
||||
register_gmail_actions(tools)
|
||||
# Option 2: Register Gmail actions with direct access token (recommended for production)
|
||||
tools = Tools()
|
||||
register_gmail_actions(tools, access_token="your_access_token_here")
|
||||
# Option 3: Use the service directly
|
||||
gmail = GmailService(access_token="your_access_token_here")
|
||||
await gmail.authenticate()
|
||||
emails = await gmail.get_recent_emails()
|
||||
"""
|
||||
|
||||
# @file purpose: Gmail integration for 2FA email authentication and email reading
|
||||
|
||||
from .actions import register_gmail_actions
|
||||
from .service import GmailService
|
||||
|
||||
__all__ = ['GmailService', 'register_gmail_actions']
|
||||
@@ -0,0 +1,115 @@
|
||||
"""
|
||||
Gmail Actions for Browser Use
|
||||
Defines agent actions for Gmail integration including 2FA code retrieval,
|
||||
email reading, and authentication management.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from browser_use.agent.views import ActionResult
|
||||
from browser_use.tools.service import Tools
|
||||
|
||||
from .service import GmailService
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Global Gmail service instance - initialized when actions are registered
|
||||
_gmail_service: GmailService | None = None
|
||||
|
||||
|
||||
class GetRecentEmailsParams(BaseModel):
|
||||
"""Parameters for getting recent emails"""
|
||||
|
||||
keyword: str = Field(default='', description='A single keyword for search, e.g. github, airbnb, etc.')
|
||||
max_results: int = Field(default=3, ge=1, le=50, description='Maximum number of emails to retrieve (1-50, default: 3)')
|
||||
|
||||
|
||||
def register_gmail_actions(tools: Tools, gmail_service: GmailService | None = None, access_token: str | None = None) -> Tools:
|
||||
"""
|
||||
Register Gmail actions with the provided tools
|
||||
Args:
|
||||
tools: The browser-use tools to register actions with
|
||||
gmail_service: Optional pre-configured Gmail service instance
|
||||
access_token: Optional direct access token (alternative to file-based auth)
|
||||
"""
|
||||
global _gmail_service
|
||||
|
||||
# Use provided service or create a new one with access token if provided
|
||||
if gmail_service:
|
||||
_gmail_service = gmail_service
|
||||
elif access_token:
|
||||
_gmail_service = GmailService(access_token=access_token)
|
||||
else:
|
||||
_gmail_service = GmailService()
|
||||
|
||||
@tools.registry.action(
|
||||
description='Get recent emails from the mailbox with a keyword to retrieve verification codes, OTP, 2FA tokens, magic links, or any recent email content. Keep your query a single keyword.',
|
||||
param_model=GetRecentEmailsParams,
|
||||
)
|
||||
async def get_recent_emails(params: GetRecentEmailsParams) -> ActionResult:
|
||||
"""Get recent emails from the last 5 minutes with full content"""
|
||||
try:
|
||||
if _gmail_service is None:
|
||||
raise RuntimeError('Gmail service not initialized')
|
||||
|
||||
# Ensure authentication
|
||||
if not _gmail_service.is_authenticated():
|
||||
logger.info('📧 Gmail not authenticated, attempting authentication...')
|
||||
authenticated = await _gmail_service.authenticate()
|
||||
if not authenticated:
|
||||
return ActionResult(
|
||||
extracted_content='Failed to authenticate with Gmail. Please ensure Gmail credentials are set up properly.',
|
||||
long_term_memory='Gmail authentication failed',
|
||||
)
|
||||
|
||||
# Use specified max_results (1-50, default 10), last 5 minutes
|
||||
max_results = params.max_results
|
||||
time_filter = '5m'
|
||||
|
||||
# Build query with time filter and optional user query
|
||||
query_parts = [f'newer_than:{time_filter}']
|
||||
if params.keyword.strip():
|
||||
query_parts.append(params.keyword.strip())
|
||||
|
||||
query = ' '.join(query_parts)
|
||||
logger.info(f'🔍 Gmail search query: {query}')
|
||||
|
||||
# Get emails
|
||||
emails = await _gmail_service.get_recent_emails(max_results=max_results, query=query, time_filter=time_filter)
|
||||
|
||||
if not emails:
|
||||
query_info = f" matching '{params.keyword}'" if params.keyword.strip() else ''
|
||||
memory = f'No recent emails found from last {time_filter}{query_info}'
|
||||
return ActionResult(
|
||||
extracted_content=memory,
|
||||
long_term_memory=memory,
|
||||
)
|
||||
|
||||
# Format with full email content for large display
|
||||
content = f'Found {len(emails)} recent email{"s" if len(emails) > 1 else ""} from the last {time_filter}:\n\n'
|
||||
|
||||
for i, email in enumerate(emails, 1):
|
||||
content += f'Email {i}:\n'
|
||||
content += f'From: {email["from"]}\n'
|
||||
content += f'Subject: {email["subject"]}\n'
|
||||
content += f'Date: {email["date"]}\n'
|
||||
content += f'Content:\n{email["body"]}\n'
|
||||
content += '-' * 50 + '\n\n'
|
||||
|
||||
logger.info(f'📧 Retrieved {len(emails)} recent emails')
|
||||
return ActionResult(
|
||||
extracted_content=content,
|
||||
include_extracted_content_only_once=True,
|
||||
long_term_memory=f'Retrieved {len(emails)} recent emails from last {time_filter} for query {query}.',
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f'Error getting recent emails: {e}')
|
||||
return ActionResult(
|
||||
error=f'Error getting recent emails: {str(e)}',
|
||||
long_term_memory='Failed to get recent emails due to error',
|
||||
)
|
||||
|
||||
return tools
|
||||
@@ -0,0 +1,226 @@
|
||||
"""
|
||||
Gmail API Service for Browser Use
|
||||
Handles Gmail API authentication, email reading, and 2FA code extraction.
|
||||
This service provides a clean interface for agents to interact with Gmail.
|
||||
"""
|
||||
|
||||
import base64
|
||||
import logging
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import aiofiles
|
||||
from google.auth.transport.requests import Request
|
||||
from google.oauth2.credentials import Credentials
|
||||
from google_auth_oauthlib.flow import InstalledAppFlow
|
||||
from googleapiclient.discovery import build
|
||||
from googleapiclient.errors import HttpError
|
||||
|
||||
from browser_use.config import CONFIG
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class GmailService:
|
||||
"""
|
||||
Gmail API service for email reading.
|
||||
Provides functionality to:
|
||||
- Authenticate with Gmail API using OAuth2
|
||||
- Read recent emails with filtering
|
||||
- Return full email content for agent analysis
|
||||
"""
|
||||
|
||||
# Gmail API scopes
|
||||
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
credentials_file: str | None = None,
|
||||
token_file: str | None = None,
|
||||
config_dir: str | None = None,
|
||||
access_token: str | None = None,
|
||||
):
|
||||
"""
|
||||
Initialize Gmail Service
|
||||
Args:
|
||||
credentials_file: Path to OAuth credentials JSON from Google Cloud Console
|
||||
token_file: Path to store/load access tokens
|
||||
config_dir: Directory to store config files (defaults to browser-use config directory)
|
||||
access_token: Direct access token (skips file-based auth if provided)
|
||||
"""
|
||||
# Set up configuration directory using browser-use's config system
|
||||
if config_dir is None:
|
||||
self.config_dir = CONFIG.BROWSER_USE_CONFIG_DIR
|
||||
else:
|
||||
self.config_dir = Path(config_dir).expanduser().resolve()
|
||||
|
||||
# Ensure config directory exists (only if not using direct token)
|
||||
if access_token is None:
|
||||
self.config_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Set up credential paths
|
||||
self.credentials_file = credentials_file or self.config_dir / 'gmail_credentials.json'
|
||||
self.token_file = token_file or self.config_dir / 'gmail_token.json'
|
||||
|
||||
# Direct access token support
|
||||
self.access_token = access_token
|
||||
|
||||
self.service = None
|
||||
self.creds = None
|
||||
self._authenticated = False
|
||||
|
||||
def is_authenticated(self) -> bool:
|
||||
"""Check if Gmail service is authenticated"""
|
||||
return self._authenticated and self.service is not None
|
||||
|
||||
async def authenticate(self) -> bool:
|
||||
"""
|
||||
Handle OAuth authentication and token management
|
||||
Returns:
|
||||
bool: True if authentication successful, False otherwise
|
||||
"""
|
||||
try:
|
||||
logger.info('🔐 Authenticating with Gmail API...')
|
||||
|
||||
# Check if using direct access token
|
||||
if self.access_token:
|
||||
logger.info('🔑 Using provided access token')
|
||||
# Create credentials from access token
|
||||
self.creds = Credentials(token=self.access_token, scopes=self.SCOPES)
|
||||
# Test token validity by building service
|
||||
self.service = build('gmail', 'v1', credentials=self.creds)
|
||||
self._authenticated = True
|
||||
logger.info('✅ Gmail API ready with access token!')
|
||||
return True
|
||||
|
||||
# Original file-based authentication flow
|
||||
# Try to load existing tokens
|
||||
if os.path.exists(self.token_file):
|
||||
self.creds = Credentials.from_authorized_user_file(str(self.token_file), self.SCOPES)
|
||||
logger.debug('📁 Loaded existing tokens')
|
||||
|
||||
# If no valid credentials, run OAuth flow
|
||||
if not self.creds or not self.creds.valid:
|
||||
if self.creds and self.creds.expired and self.creds.refresh_token:
|
||||
logger.info('🔄 Refreshing expired tokens...')
|
||||
self.creds.refresh(Request())
|
||||
else:
|
||||
logger.info('🌐 Starting OAuth flow...')
|
||||
if not os.path.exists(self.credentials_file):
|
||||
logger.error(
|
||||
f'❌ Gmail credentials file not found: {self.credentials_file}\n'
|
||||
'Please download it from Google Cloud Console:\n'
|
||||
'1. Go to https://console.cloud.google.com/\n'
|
||||
'2. APIs & Services > Credentials\n'
|
||||
'3. Download OAuth 2.0 Client JSON\n'
|
||||
f"4. Save as 'gmail_credentials.json' in {self.config_dir}/"
|
||||
)
|
||||
return False
|
||||
|
||||
flow = InstalledAppFlow.from_client_secrets_file(str(self.credentials_file), self.SCOPES)
|
||||
# Use specific redirect URI to match OAuth credentials
|
||||
self.creds = flow.run_local_server(port=4242, open_browser=True)
|
||||
|
||||
# Save tokens for next time
|
||||
async with aiofiles.open(self.token_file, 'w') as token:
|
||||
await token.write(self.creds.to_json())
|
||||
logger.info(f'💾 Tokens saved to {self.token_file}')
|
||||
|
||||
# Build Gmail service
|
||||
self.service = build('gmail', 'v1', credentials=self.creds)
|
||||
self._authenticated = True
|
||||
logger.info('✅ Gmail API ready!')
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f'❌ Gmail authentication failed: {e}')
|
||||
return False
|
||||
|
||||
async def get_recent_emails(self, max_results: int = 10, query: str = '', time_filter: str = '1h') -> list[dict[str, Any]]:
|
||||
"""
|
||||
Get recent emails with optional query filter
|
||||
Args:
|
||||
max_results: Maximum number of emails to fetch
|
||||
query: Gmail search query (e.g., 'from:noreply@example.com')
|
||||
time_filter: Time filter (e.g., '5m', '1h', '1d')
|
||||
Returns:
|
||||
List of email dictionaries with parsed content
|
||||
"""
|
||||
if not self.is_authenticated():
|
||||
logger.error('❌ Gmail service not authenticated. Call authenticate() first.')
|
||||
return []
|
||||
|
||||
try:
|
||||
# Add time filter to query if provided
|
||||
if time_filter and 'newer_than:' not in query:
|
||||
query = f'newer_than:{time_filter} {query}'.strip()
|
||||
|
||||
logger.info(f'📧 Fetching {max_results} recent emails...')
|
||||
if query:
|
||||
logger.debug(f'🔍 Query: {query}')
|
||||
|
||||
# Get message list
|
||||
assert self.service is not None
|
||||
results = self.service.users().messages().list(userId='me', maxResults=max_results, q=query).execute()
|
||||
|
||||
messages = results.get('messages', [])
|
||||
if not messages:
|
||||
logger.info('📭 No messages found')
|
||||
return []
|
||||
|
||||
logger.info(f'📨 Found {len(messages)} messages, fetching details...')
|
||||
|
||||
# Get full message details
|
||||
emails = []
|
||||
for i, message in enumerate(messages, 1):
|
||||
logger.debug(f'📖 Reading email {i}/{len(messages)}...')
|
||||
|
||||
full_message = self.service.users().messages().get(userId='me', id=message['id'], format='full').execute()
|
||||
|
||||
email_data = self._parse_email(full_message)
|
||||
emails.append(email_data)
|
||||
|
||||
return emails
|
||||
|
||||
except HttpError as error:
|
||||
logger.error(f'❌ Gmail API error: {error}')
|
||||
return []
|
||||
except Exception as e:
|
||||
logger.error(f'❌ Unexpected error fetching emails: {e}')
|
||||
return []
|
||||
|
||||
def _parse_email(self, message: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Parse Gmail message into readable format"""
|
||||
headers = {h['name']: h['value'] for h in message['payload']['headers']}
|
||||
|
||||
return {
|
||||
'id': message['id'],
|
||||
'thread_id': message['threadId'],
|
||||
'subject': headers.get('Subject', ''),
|
||||
'from': headers.get('From', ''),
|
||||
'to': headers.get('To', ''),
|
||||
'date': headers.get('Date', ''),
|
||||
'timestamp': int(message['internalDate']),
|
||||
'body': self._extract_body(message['payload']),
|
||||
'raw_message': message,
|
||||
}
|
||||
|
||||
def _extract_body(self, payload: dict[str, Any]) -> str:
|
||||
"""Extract email body from payload"""
|
||||
body = ''
|
||||
|
||||
if payload.get('body', {}).get('data'):
|
||||
# Simple email body
|
||||
body = base64.urlsafe_b64decode(payload['body']['data']).decode('utf-8')
|
||||
elif payload.get('parts'):
|
||||
# Multi-part email
|
||||
for part in payload['parts']:
|
||||
if part['mimeType'] == 'text/plain' and part.get('body', {}).get('data'):
|
||||
part_body = base64.urlsafe_b64decode(part['body']['data']).decode('utf-8')
|
||||
body += part_body
|
||||
elif part['mimeType'] == 'text/html' and not body and part.get('body', {}).get('data'):
|
||||
# Fallback to HTML if no plain text
|
||||
body = base64.urlsafe_b64decode(part['body']['data']).decode('utf-8')
|
||||
|
||||
return body
|
||||
@@ -0,0 +1,16 @@
|
||||
# Browser Use LLMs
|
||||
|
||||
We officially support the following LLMs:
|
||||
|
||||
- OpenAI
|
||||
- Anthropic
|
||||
- Google
|
||||
- Groq
|
||||
- Ollama
|
||||
- DeepSeek
|
||||
|
||||
## Migrating from LangChain
|
||||
|
||||
Because of how we implemented the LLMs, we can technically support anything. If you want to use a LangChain model, you can use the `ChatLangchain` (NOT OFFICIALLY SUPPORTED) class.
|
||||
|
||||
You can find all the details in the [LangChain example](examples/models/langchain/example.py). We suggest you grab that code and use it as a reference.
|
||||
@@ -0,0 +1,146 @@
|
||||
"""
|
||||
We have switched all of our code from langchain to openai.types.chat.chat_completion_message_param.
|
||||
|
||||
For easier transition we have
|
||||
"""
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
# Lightweight imports that are commonly used
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
SystemMessage,
|
||||
UserMessage,
|
||||
)
|
||||
from browser_use.llm.messages import (
|
||||
ContentPartImageParam as ContentImage,
|
||||
)
|
||||
from browser_use.llm.messages import (
|
||||
ContentPartRefusalParam as ContentRefusal,
|
||||
)
|
||||
from browser_use.llm.messages import (
|
||||
ContentPartTextParam as ContentText,
|
||||
)
|
||||
|
||||
# Type stubs for lazy imports
|
||||
if TYPE_CHECKING:
|
||||
from browser_use.llm.anthropic.chat import ChatAnthropic
|
||||
from browser_use.llm.aws.chat_anthropic import ChatAnthropicBedrock
|
||||
from browser_use.llm.aws.chat_bedrock import ChatAWSBedrock
|
||||
from browser_use.llm.azure.chat import ChatAzureOpenAI
|
||||
from browser_use.llm.deepseek.chat import ChatDeepSeek
|
||||
from browser_use.llm.google.chat import ChatGoogle
|
||||
from browser_use.llm.groq.chat import ChatGroq
|
||||
from browser_use.llm.ollama.chat import ChatOllama
|
||||
from browser_use.llm.openai.chat import ChatOpenAI
|
||||
from browser_use.llm.openrouter.chat import ChatOpenRouter
|
||||
|
||||
# Type stubs for model instances - enables IDE autocomplete
|
||||
openai_gpt_4o: ChatOpenAI
|
||||
openai_gpt_4o_mini: ChatOpenAI
|
||||
openai_gpt_4_1_mini: ChatOpenAI
|
||||
openai_o1: ChatOpenAI
|
||||
openai_o1_mini: ChatOpenAI
|
||||
openai_o1_pro: ChatOpenAI
|
||||
openai_o3: ChatOpenAI
|
||||
openai_o3_mini: ChatOpenAI
|
||||
openai_o3_pro: ChatOpenAI
|
||||
openai_o4_mini: ChatOpenAI
|
||||
openai_gpt_5: ChatOpenAI
|
||||
openai_gpt_5_mini: ChatOpenAI
|
||||
openai_gpt_5_nano: ChatOpenAI
|
||||
|
||||
azure_gpt_4o: ChatAzureOpenAI
|
||||
azure_gpt_4o_mini: ChatAzureOpenAI
|
||||
azure_gpt_4_1_mini: ChatAzureOpenAI
|
||||
azure_o1: ChatAzureOpenAI
|
||||
azure_o1_mini: ChatAzureOpenAI
|
||||
azure_o1_pro: ChatAzureOpenAI
|
||||
azure_o3: ChatAzureOpenAI
|
||||
azure_o3_mini: ChatAzureOpenAI
|
||||
azure_o3_pro: ChatAzureOpenAI
|
||||
azure_gpt_5: ChatAzureOpenAI
|
||||
azure_gpt_5_mini: ChatAzureOpenAI
|
||||
|
||||
google_gemini_2_0_flash: ChatGoogle
|
||||
google_gemini_2_0_pro: ChatGoogle
|
||||
google_gemini_2_5_pro: ChatGoogle
|
||||
google_gemini_2_5_flash: ChatGoogle
|
||||
google_gemini_2_5_flash_lite: ChatGoogle
|
||||
|
||||
# Models are imported on-demand via __getattr__
|
||||
|
||||
# Lazy imports mapping for heavy chat models
|
||||
_LAZY_IMPORTS = {
|
||||
'ChatAnthropic': ('browser_use.llm.anthropic.chat', 'ChatAnthropic'),
|
||||
'ChatAnthropicBedrock': ('browser_use.llm.aws.chat_anthropic', 'ChatAnthropicBedrock'),
|
||||
'ChatAWSBedrock': ('browser_use.llm.aws.chat_bedrock', 'ChatAWSBedrock'),
|
||||
'ChatAzureOpenAI': ('browser_use.llm.azure.chat', 'ChatAzureOpenAI'),
|
||||
'ChatDeepSeek': ('browser_use.llm.deepseek.chat', 'ChatDeepSeek'),
|
||||
'ChatGoogle': ('browser_use.llm.google.chat', 'ChatGoogle'),
|
||||
'ChatGroq': ('browser_use.llm.groq.chat', 'ChatGroq'),
|
||||
'ChatOllama': ('browser_use.llm.ollama.chat', 'ChatOllama'),
|
||||
'ChatOpenAI': ('browser_use.llm.openai.chat', 'ChatOpenAI'),
|
||||
'ChatOpenRouter': ('browser_use.llm.openrouter.chat', 'ChatOpenRouter'),
|
||||
}
|
||||
|
||||
# Cache for model instances - only created when accessed
|
||||
_model_cache: dict[str, 'BaseChatModel'] = {}
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
"""Lazy import mechanism for heavy chat model imports and model instances."""
|
||||
if name in _LAZY_IMPORTS:
|
||||
module_path, attr_name = _LAZY_IMPORTS[name]
|
||||
try:
|
||||
from importlib import import_module
|
||||
|
||||
module = import_module(module_path)
|
||||
attr = getattr(module, attr_name)
|
||||
return attr
|
||||
except ImportError as e:
|
||||
raise ImportError(f'Failed to import {name} from {module_path}: {e}') from e
|
||||
|
||||
# Check cache first for model instances
|
||||
if name in _model_cache:
|
||||
return _model_cache[name]
|
||||
|
||||
# Try to get model instances from models module on-demand
|
||||
try:
|
||||
from browser_use.llm.models import __getattr__ as models_getattr
|
||||
|
||||
attr = models_getattr(name)
|
||||
# Cache in our clean cache dict
|
||||
_model_cache[name] = attr
|
||||
return attr
|
||||
except (AttributeError, ImportError):
|
||||
pass
|
||||
|
||||
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
|
||||
|
||||
|
||||
__all__ = [
|
||||
# Message types -> for easier transition from langchain
|
||||
'BaseMessage',
|
||||
'UserMessage',
|
||||
'SystemMessage',
|
||||
'AssistantMessage',
|
||||
# Content parts with better names
|
||||
'ContentText',
|
||||
'ContentRefusal',
|
||||
'ContentImage',
|
||||
# Chat models
|
||||
'BaseChatModel',
|
||||
'ChatOpenAI',
|
||||
'ChatDeepSeek',
|
||||
'ChatGoogle',
|
||||
'ChatAnthropic',
|
||||
'ChatAnthropicBedrock',
|
||||
'ChatAWSBedrock',
|
||||
'ChatGroq',
|
||||
'ChatAzureOpenAI',
|
||||
'ChatOllama',
|
||||
'ChatOpenRouter',
|
||||
]
|
||||
@@ -0,0 +1,236 @@
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, TypeVar, overload
|
||||
|
||||
import httpx
|
||||
from anthropic import (
|
||||
NOT_GIVEN,
|
||||
APIConnectionError,
|
||||
APIStatusError,
|
||||
AsyncAnthropic,
|
||||
NotGiven,
|
||||
RateLimitError,
|
||||
)
|
||||
from anthropic.types import CacheControlEphemeralParam, Message, ToolParam
|
||||
from anthropic.types.model_param import ModelParam
|
||||
from anthropic.types.text_block import TextBlock
|
||||
from anthropic.types.tool_choice_tool_param import ToolChoiceToolParam
|
||||
from httpx import Timeout
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.anthropic.serializer import AnthropicMessageSerializer
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.schema import SchemaOptimizer
|
||||
from browser_use.llm.views import ChatInvokeCompletion, ChatInvokeUsage
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatAnthropic(BaseChatModel):
|
||||
"""
|
||||
A wrapper around Anthropic's chat model.
|
||||
"""
|
||||
|
||||
# Model configuration
|
||||
model: str | ModelParam
|
||||
max_tokens: int = 8192
|
||||
temperature: float | None = None
|
||||
top_p: float | None = None
|
||||
seed: int | None = None
|
||||
|
||||
# Client initialization parameters
|
||||
api_key: str | None = None
|
||||
auth_token: str | None = None
|
||||
base_url: str | httpx.URL | None = None
|
||||
timeout: float | Timeout | None | NotGiven = NotGiven()
|
||||
max_retries: int = 10
|
||||
default_headers: Mapping[str, str] | None = None
|
||||
default_query: Mapping[str, object] | None = None
|
||||
|
||||
# Static
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'anthropic'
|
||||
|
||||
def _get_client_params(self) -> dict[str, Any]:
|
||||
"""Prepare client parameters dictionary."""
|
||||
# Define base client params
|
||||
base_params = {
|
||||
'api_key': self.api_key,
|
||||
'auth_token': self.auth_token,
|
||||
'base_url': self.base_url,
|
||||
'timeout': self.timeout,
|
||||
'max_retries': self.max_retries,
|
||||
'default_headers': self.default_headers,
|
||||
'default_query': self.default_query,
|
||||
}
|
||||
|
||||
# Create client_params dict with non-None values and non-NotGiven values
|
||||
client_params = {}
|
||||
for k, v in base_params.items():
|
||||
if v is not None and v is not NotGiven():
|
||||
client_params[k] = v
|
||||
|
||||
return client_params
|
||||
|
||||
def _get_client_params_for_invoke(self):
|
||||
"""Prepare client parameters dictionary for invoke."""
|
||||
|
||||
client_params = {}
|
||||
|
||||
if self.temperature is not None:
|
||||
client_params['temperature'] = self.temperature
|
||||
|
||||
if self.max_tokens is not None:
|
||||
client_params['max_tokens'] = self.max_tokens
|
||||
|
||||
if self.top_p is not None:
|
||||
client_params['top_p'] = self.top_p
|
||||
|
||||
if self.seed is not None:
|
||||
client_params['seed'] = self.seed
|
||||
|
||||
return client_params
|
||||
|
||||
def get_client(self) -> AsyncAnthropic:
|
||||
"""
|
||||
Returns an AsyncAnthropic client.
|
||||
|
||||
Returns:
|
||||
AsyncAnthropic: An instance of the AsyncAnthropic client.
|
||||
"""
|
||||
client_params = self._get_client_params()
|
||||
return AsyncAnthropic(**client_params)
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return str(self.model)
|
||||
|
||||
def _get_usage(self, response: Message) -> ChatInvokeUsage | None:
|
||||
usage = ChatInvokeUsage(
|
||||
prompt_tokens=response.usage.input_tokens
|
||||
+ (
|
||||
response.usage.cache_read_input_tokens or 0
|
||||
), # Total tokens in Anthropic are a bit fucked, you have to add cached tokens to the prompt tokens
|
||||
completion_tokens=response.usage.output_tokens,
|
||||
total_tokens=response.usage.input_tokens + response.usage.output_tokens,
|
||||
prompt_cached_tokens=response.usage.cache_read_input_tokens,
|
||||
prompt_cache_creation_tokens=response.usage.cache_creation_input_tokens,
|
||||
prompt_image_tokens=None,
|
||||
)
|
||||
return usage
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
anthropic_messages, system_prompt = AnthropicMessageSerializer.serialize_messages(messages)
|
||||
|
||||
try:
|
||||
if output_format is None:
|
||||
# Normal completion without structured output
|
||||
response = await self.get_client().messages.create(
|
||||
model=self.model,
|
||||
messages=anthropic_messages,
|
||||
system=system_prompt or NOT_GIVEN,
|
||||
**self._get_client_params_for_invoke(),
|
||||
)
|
||||
|
||||
# Ensure we have a valid Message object before accessing attributes
|
||||
if not isinstance(response, Message):
|
||||
raise ModelProviderError(
|
||||
message=f'Unexpected response type from Anthropic API: {type(response).__name__}. Response: {str(response)[:200]}',
|
||||
status_code=502,
|
||||
model=self.name,
|
||||
)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
# Extract text from the first content block
|
||||
first_content = response.content[0]
|
||||
if isinstance(first_content, TextBlock):
|
||||
response_text = first_content.text
|
||||
else:
|
||||
# If it's not a text block, convert to string
|
||||
response_text = str(first_content)
|
||||
|
||||
return ChatInvokeCompletion(
|
||||
completion=response_text,
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
else:
|
||||
# Use tool calling for structured output
|
||||
# Create a tool that represents the output format
|
||||
tool_name = output_format.__name__
|
||||
schema = SchemaOptimizer.create_optimized_json_schema(output_format)
|
||||
|
||||
# Remove title from schema if present (Anthropic doesn't like it in parameters)
|
||||
if 'title' in schema:
|
||||
del schema['title']
|
||||
|
||||
tool = ToolParam(
|
||||
name=tool_name,
|
||||
description=f'Extract information in the format of {tool_name}',
|
||||
input_schema=schema,
|
||||
cache_control=CacheControlEphemeralParam(type='ephemeral'),
|
||||
)
|
||||
|
||||
# Force the model to use this tool
|
||||
tool_choice = ToolChoiceToolParam(type='tool', name=tool_name)
|
||||
|
||||
response = await self.get_client().messages.create(
|
||||
model=self.model,
|
||||
messages=anthropic_messages,
|
||||
tools=[tool],
|
||||
system=system_prompt or NOT_GIVEN,
|
||||
tool_choice=tool_choice,
|
||||
**self._get_client_params_for_invoke(),
|
||||
)
|
||||
|
||||
# Ensure we have a valid Message object before accessing attributes
|
||||
if not isinstance(response, Message):
|
||||
raise ModelProviderError(
|
||||
message=f'Unexpected response type from Anthropic API: {type(response).__name__}. Response: {str(response)[:200]}',
|
||||
status_code=502,
|
||||
model=self.name,
|
||||
)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
# Extract the tool use block
|
||||
for content_block in response.content:
|
||||
if hasattr(content_block, 'type') and content_block.type == 'tool_use':
|
||||
# Parse the tool input as the structured output
|
||||
try:
|
||||
return ChatInvokeCompletion(completion=output_format.model_validate(content_block.input), usage=usage)
|
||||
except Exception as e:
|
||||
# If validation fails, try to parse it as JSON first
|
||||
if isinstance(content_block.input, str):
|
||||
data = json.loads(content_block.input)
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(data),
|
||||
usage=usage,
|
||||
)
|
||||
raise e
|
||||
|
||||
# If no tool use block found, raise an error
|
||||
raise ValueError('Expected tool use in response but none found')
|
||||
|
||||
except APIConnectionError as e:
|
||||
raise ModelProviderError(message=e.message, model=self.name) from e
|
||||
except RateLimitError as e:
|
||||
raise ModelRateLimitError(message=e.message, model=self.name) from e
|
||||
except APIStatusError as e:
|
||||
raise ModelProviderError(message=e.message, status_code=e.status_code, model=self.name) from e
|
||||
except Exception as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
@@ -0,0 +1,312 @@
|
||||
import json
|
||||
from typing import overload
|
||||
|
||||
from anthropic.types import (
|
||||
Base64ImageSourceParam,
|
||||
CacheControlEphemeralParam,
|
||||
ImageBlockParam,
|
||||
MessageParam,
|
||||
TextBlockParam,
|
||||
ToolUseBlockParam,
|
||||
URLImageSourceParam,
|
||||
)
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
ContentPartImageParam,
|
||||
ContentPartTextParam,
|
||||
SupportedImageMediaType,
|
||||
SystemMessage,
|
||||
UserMessage,
|
||||
)
|
||||
|
||||
NonSystemMessage = UserMessage | AssistantMessage
|
||||
|
||||
|
||||
class AnthropicMessageSerializer:
|
||||
"""Serializer for converting between custom message types and Anthropic message param types."""
|
||||
|
||||
@staticmethod
|
||||
def _is_base64_image(url: str) -> bool:
|
||||
"""Check if the URL is a base64 encoded image."""
|
||||
return url.startswith('data:image/')
|
||||
|
||||
@staticmethod
|
||||
def _parse_base64_url(url: str) -> tuple[SupportedImageMediaType, str]:
|
||||
"""Parse a base64 data URL to extract media type and data."""
|
||||
# Format: data:image/jpeg;base64,<data>
|
||||
if not url.startswith('data:'):
|
||||
raise ValueError(f'Invalid base64 URL: {url}')
|
||||
|
||||
header, data = url.split(',', 1)
|
||||
media_type = header.split(';')[0].replace('data:', '')
|
||||
|
||||
# Ensure it's a supported media type
|
||||
supported_types = ['image/jpeg', 'image/png', 'image/gif', 'image/webp']
|
||||
if media_type not in supported_types:
|
||||
# Default to png if not recognized
|
||||
media_type = 'image/png'
|
||||
|
||||
return media_type, data # type: ignore
|
||||
|
||||
@staticmethod
|
||||
def _serialize_cache_control(use_cache: bool) -> CacheControlEphemeralParam | None:
|
||||
"""Serialize cache control."""
|
||||
if use_cache:
|
||||
return CacheControlEphemeralParam(type='ephemeral')
|
||||
return None
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_text(part: ContentPartTextParam, use_cache: bool) -> TextBlockParam:
|
||||
"""Convert a text content part to Anthropic's TextBlockParam."""
|
||||
return TextBlockParam(
|
||||
text=part.text, type='text', cache_control=AnthropicMessageSerializer._serialize_cache_control(use_cache)
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_image(part: ContentPartImageParam) -> ImageBlockParam:
|
||||
"""Convert an image content part to Anthropic's ImageBlockParam."""
|
||||
url = part.image_url.url
|
||||
|
||||
if AnthropicMessageSerializer._is_base64_image(url):
|
||||
# Handle base64 encoded images
|
||||
media_type, data = AnthropicMessageSerializer._parse_base64_url(url)
|
||||
return ImageBlockParam(
|
||||
source=Base64ImageSourceParam(
|
||||
data=data,
|
||||
media_type=media_type,
|
||||
type='base64',
|
||||
),
|
||||
type='image',
|
||||
)
|
||||
else:
|
||||
# Handle URL images
|
||||
return ImageBlockParam(source=URLImageSourceParam(url=url, type='url'), type='image')
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_to_str(
|
||||
content: str | list[ContentPartTextParam], use_cache: bool = False
|
||||
) -> list[TextBlockParam] | str:
|
||||
"""Serialize content to a string."""
|
||||
cache_control = AnthropicMessageSerializer._serialize_cache_control(use_cache)
|
||||
|
||||
if isinstance(content, str):
|
||||
if cache_control:
|
||||
return [TextBlockParam(text=content, type='text', cache_control=cache_control)]
|
||||
else:
|
||||
return content
|
||||
|
||||
serialized_blocks: list[TextBlockParam] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_blocks.append(AnthropicMessageSerializer._serialize_content_part_text(part, use_cache))
|
||||
|
||||
return serialized_blocks
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content(
|
||||
content: str | list[ContentPartTextParam | ContentPartImageParam],
|
||||
use_cache: bool = False,
|
||||
) -> str | list[TextBlockParam | ImageBlockParam]:
|
||||
"""Serialize content to Anthropic format."""
|
||||
if isinstance(content, str):
|
||||
if use_cache:
|
||||
return [TextBlockParam(text=content, type='text', cache_control=CacheControlEphemeralParam(type='ephemeral'))]
|
||||
else:
|
||||
return content
|
||||
|
||||
serialized_blocks: list[TextBlockParam | ImageBlockParam] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_blocks.append(AnthropicMessageSerializer._serialize_content_part_text(part, use_cache))
|
||||
elif part.type == 'image_url':
|
||||
serialized_blocks.append(AnthropicMessageSerializer._serialize_content_part_image(part))
|
||||
|
||||
return serialized_blocks
|
||||
|
||||
@staticmethod
|
||||
def _serialize_tool_calls_to_content(tool_calls, use_cache: bool = False) -> list[ToolUseBlockParam]:
|
||||
"""Convert tool calls to Anthropic's ToolUseBlockParam format."""
|
||||
blocks: list[ToolUseBlockParam] = []
|
||||
for tool_call in tool_calls:
|
||||
# Parse the arguments JSON string to object
|
||||
|
||||
try:
|
||||
input_obj = json.loads(tool_call.function.arguments)
|
||||
except json.JSONDecodeError:
|
||||
# If arguments aren't valid JSON, use as string
|
||||
input_obj = {'arguments': tool_call.function.arguments}
|
||||
|
||||
blocks.append(
|
||||
ToolUseBlockParam(
|
||||
id=tool_call.id,
|
||||
input=input_obj,
|
||||
name=tool_call.function.name,
|
||||
type='tool_use',
|
||||
cache_control=AnthropicMessageSerializer._serialize_cache_control(use_cache),
|
||||
)
|
||||
)
|
||||
return blocks
|
||||
|
||||
# region - Serialize overloads
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: UserMessage) -> MessageParam: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: SystemMessage) -> SystemMessage: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: AssistantMessage) -> MessageParam: ...
|
||||
|
||||
@staticmethod
|
||||
def serialize(message: BaseMessage) -> MessageParam | SystemMessage:
|
||||
"""Serialize a custom message to an Anthropic MessageParam.
|
||||
|
||||
Note: Anthropic doesn't have a 'system' role. System messages should be
|
||||
handled separately as the system parameter in the API call, not as a message.
|
||||
If a SystemMessage is passed here, it will be converted to a user message.
|
||||
"""
|
||||
if isinstance(message, UserMessage):
|
||||
content = AnthropicMessageSerializer._serialize_content(message.content, use_cache=message.cache)
|
||||
return MessageParam(role='user', content=content)
|
||||
|
||||
elif isinstance(message, SystemMessage):
|
||||
# Anthropic doesn't have system messages in the messages array
|
||||
# System prompts are passed separately. Convert to user message.
|
||||
return message
|
||||
|
||||
elif isinstance(message, AssistantMessage):
|
||||
# Handle content and tool calls
|
||||
blocks: list[TextBlockParam | ToolUseBlockParam] = []
|
||||
|
||||
# Add content blocks if present
|
||||
if message.content is not None:
|
||||
if isinstance(message.content, str):
|
||||
blocks.append(
|
||||
TextBlockParam(
|
||||
text=message.content,
|
||||
type='text',
|
||||
cache_control=AnthropicMessageSerializer._serialize_cache_control(message.cache),
|
||||
)
|
||||
)
|
||||
else:
|
||||
# Process content parts (text and refusal)
|
||||
for part in message.content:
|
||||
if part.type == 'text':
|
||||
blocks.append(AnthropicMessageSerializer._serialize_content_part_text(part, use_cache=message.cache))
|
||||
# # Note: Anthropic doesn't have a specific refusal block type,
|
||||
# # so we convert refusals to text blocks
|
||||
# elif part.type == 'refusal':
|
||||
# blocks.append(TextBlockParam(text=f'[Refusal] {part.refusal}', type='text'))
|
||||
|
||||
# Add tool use blocks if present
|
||||
if message.tool_calls:
|
||||
tool_blocks = AnthropicMessageSerializer._serialize_tool_calls_to_content(
|
||||
message.tool_calls, use_cache=message.cache
|
||||
)
|
||||
blocks.extend(tool_blocks)
|
||||
|
||||
# If no content or tool calls, add empty text block
|
||||
# (Anthropic requires at least one content block)
|
||||
if not blocks:
|
||||
blocks.append(
|
||||
TextBlockParam(
|
||||
text='', type='text', cache_control=AnthropicMessageSerializer._serialize_cache_control(message.cache)
|
||||
)
|
||||
)
|
||||
|
||||
# If caching is enabled or we have multiple blocks, return blocks as-is
|
||||
# Otherwise, simplify single text blocks to plain string
|
||||
if message.cache or len(blocks) > 1:
|
||||
content = blocks
|
||||
else:
|
||||
# Only simplify when no caching and single block
|
||||
single_block = blocks[0]
|
||||
if single_block['type'] == 'text' and not single_block.get('cache_control'):
|
||||
content = single_block['text']
|
||||
else:
|
||||
content = blocks
|
||||
|
||||
return MessageParam(
|
||||
role='assistant',
|
||||
content=content,
|
||||
)
|
||||
|
||||
else:
|
||||
raise ValueError(f'Unknown message type: {type(message)}')
|
||||
|
||||
@staticmethod
|
||||
def _clean_cache_messages(messages: list[NonSystemMessage]) -> list[NonSystemMessage]:
|
||||
"""Clean cache settings so only the last cache=True message remains cached.
|
||||
|
||||
Because of how Claude caching works, only the last cache message matters.
|
||||
This method automatically removes cache=True from all messages except the last one.
|
||||
|
||||
Args:
|
||||
messages: List of non-system messages to clean
|
||||
|
||||
Returns:
|
||||
List of messages with cleaned cache settings
|
||||
"""
|
||||
if not messages:
|
||||
return messages
|
||||
|
||||
# Create a copy to avoid modifying the original
|
||||
cleaned_messages = [msg.model_copy(deep=True) for msg in messages]
|
||||
|
||||
# Find the last message with cache=True
|
||||
last_cache_index = -1
|
||||
for i in range(len(cleaned_messages) - 1, -1, -1):
|
||||
if cleaned_messages[i].cache:
|
||||
last_cache_index = i
|
||||
break
|
||||
|
||||
# If we found a cached message, disable cache for all others
|
||||
if last_cache_index != -1:
|
||||
for i, msg in enumerate(cleaned_messages):
|
||||
if i != last_cache_index and msg.cache:
|
||||
# Set cache to False for all messages except the last cached one
|
||||
msg.cache = False
|
||||
|
||||
return cleaned_messages
|
||||
|
||||
@staticmethod
|
||||
def serialize_messages(messages: list[BaseMessage]) -> tuple[list[MessageParam], list[TextBlockParam] | str | None]:
|
||||
"""Serialize a list of messages, extracting any system message.
|
||||
|
||||
Returns:
|
||||
A tuple of (messages, system_message) where system_message is extracted
|
||||
from any SystemMessage in the list.
|
||||
"""
|
||||
messages = [m.model_copy(deep=True) for m in messages]
|
||||
|
||||
# Separate system messages from normal messages
|
||||
normal_messages: list[NonSystemMessage] = []
|
||||
system_message: SystemMessage | None = None
|
||||
|
||||
for message in messages:
|
||||
if isinstance(message, SystemMessage):
|
||||
system_message = message
|
||||
else:
|
||||
normal_messages.append(message)
|
||||
|
||||
# Clean cache messages so only the last cache=True message remains cached
|
||||
normal_messages = AnthropicMessageSerializer._clean_cache_messages(normal_messages)
|
||||
|
||||
# Serialize normal messages
|
||||
serialized_messages: list[MessageParam] = []
|
||||
for message in normal_messages:
|
||||
serialized_messages.append(AnthropicMessageSerializer.serialize(message))
|
||||
|
||||
# Serialize system message
|
||||
serialized_system_message: list[TextBlockParam] | str | None = None
|
||||
if system_message:
|
||||
serialized_system_message = AnthropicMessageSerializer._serialize_content_to_str(
|
||||
system_message.content, use_cache=system_message.cache
|
||||
)
|
||||
|
||||
return serialized_messages, serialized_system_message
|
||||
@@ -0,0 +1,36 @@
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
# Type stubs for lazy imports
|
||||
if TYPE_CHECKING:
|
||||
from browser_use.llm.aws.chat_anthropic import ChatAnthropicBedrock
|
||||
from browser_use.llm.aws.chat_bedrock import ChatAWSBedrock
|
||||
|
||||
# Lazy imports mapping for AWS chat models
|
||||
_LAZY_IMPORTS = {
|
||||
'ChatAnthropicBedrock': ('browser_use.llm.aws.chat_anthropic', 'ChatAnthropicBedrock'),
|
||||
'ChatAWSBedrock': ('browser_use.llm.aws.chat_bedrock', 'ChatAWSBedrock'),
|
||||
}
|
||||
|
||||
|
||||
def __getattr__(name: str):
|
||||
"""Lazy import mechanism for AWS chat models."""
|
||||
if name in _LAZY_IMPORTS:
|
||||
module_path, attr_name = _LAZY_IMPORTS[name]
|
||||
try:
|
||||
from importlib import import_module
|
||||
|
||||
module = import_module(module_path)
|
||||
attr = getattr(module, attr_name)
|
||||
# Cache the imported attribute in the module's globals
|
||||
globals()[name] = attr
|
||||
return attr
|
||||
except ImportError as e:
|
||||
raise ImportError(f'Failed to import {name} from {module_path}: {e}') from e
|
||||
|
||||
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
|
||||
|
||||
|
||||
__all__ = [
|
||||
'ChatAWSBedrock',
|
||||
'ChatAnthropicBedrock',
|
||||
]
|
||||
@@ -0,0 +1,242 @@
|
||||
import json
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Any, TypeVar, overload
|
||||
|
||||
from anthropic import (
|
||||
NOT_GIVEN,
|
||||
APIConnectionError,
|
||||
APIStatusError,
|
||||
AsyncAnthropicBedrock,
|
||||
RateLimitError,
|
||||
)
|
||||
from anthropic.types import CacheControlEphemeralParam, Message, ToolParam
|
||||
from anthropic.types.text_block import TextBlock
|
||||
from anthropic.types.tool_choice_tool_param import ToolChoiceToolParam
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.anthropic.serializer import AnthropicMessageSerializer
|
||||
from browser_use.llm.aws.chat_bedrock import ChatAWSBedrock
|
||||
from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.views import ChatInvokeCompletion, ChatInvokeUsage
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from boto3.session import Session # pyright: ignore
|
||||
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatAnthropicBedrock(ChatAWSBedrock):
|
||||
"""
|
||||
AWS Bedrock Anthropic Claude chat model.
|
||||
|
||||
This is a convenience class that provides Claude-specific defaults
|
||||
for the AWS Bedrock service. It inherits all functionality from
|
||||
ChatAWSBedrock but sets Anthropic Claude as the default model.
|
||||
"""
|
||||
|
||||
# Anthropic Claude specific defaults
|
||||
model: str = 'anthropic.claude-3-5-sonnet-20240620-v1:0'
|
||||
max_tokens: int = 8192
|
||||
temperature: float | None = None
|
||||
top_p: float | None = None
|
||||
top_k: int | None = None
|
||||
stop_sequences: list[str] | None = None
|
||||
seed: int | None = None
|
||||
|
||||
# AWS credentials and configuration
|
||||
aws_access_key: str | None = None
|
||||
aws_secret_key: str | None = None
|
||||
aws_session_token: str | None = None
|
||||
aws_region: str | None = None
|
||||
session: 'Session | None' = None
|
||||
|
||||
# Client initialization parameters
|
||||
max_retries: int = 10
|
||||
default_headers: Mapping[str, str] | None = None
|
||||
default_query: Mapping[str, object] | None = None
|
||||
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'anthropic_bedrock'
|
||||
|
||||
def _get_client_params(self) -> dict[str, Any]:
|
||||
"""Prepare client parameters dictionary for Bedrock."""
|
||||
client_params: dict[str, Any] = {}
|
||||
|
||||
if self.session:
|
||||
credentials = self.session.get_credentials()
|
||||
client_params.update(
|
||||
{
|
||||
'aws_access_key': credentials.access_key,
|
||||
'aws_secret_key': credentials.secret_key,
|
||||
'aws_session_token': credentials.token,
|
||||
'aws_region': self.session.region_name,
|
||||
}
|
||||
)
|
||||
else:
|
||||
# Use individual credentials
|
||||
if self.aws_access_key:
|
||||
client_params['aws_access_key'] = self.aws_access_key
|
||||
if self.aws_secret_key:
|
||||
client_params['aws_secret_key'] = self.aws_secret_key
|
||||
if self.aws_region:
|
||||
client_params['aws_region'] = self.aws_region
|
||||
if self.aws_session_token:
|
||||
client_params['aws_session_token'] = self.aws_session_token
|
||||
|
||||
# Add optional parameters
|
||||
if self.max_retries:
|
||||
client_params['max_retries'] = self.max_retries
|
||||
if self.default_headers:
|
||||
client_params['default_headers'] = self.default_headers
|
||||
if self.default_query:
|
||||
client_params['default_query'] = self.default_query
|
||||
|
||||
return client_params
|
||||
|
||||
def _get_client_params_for_invoke(self) -> dict[str, Any]:
|
||||
"""Prepare client parameters dictionary for invoke."""
|
||||
client_params = {}
|
||||
|
||||
if self.temperature is not None:
|
||||
client_params['temperature'] = self.temperature
|
||||
if self.max_tokens is not None:
|
||||
client_params['max_tokens'] = self.max_tokens
|
||||
if self.top_p is not None:
|
||||
client_params['top_p'] = self.top_p
|
||||
if self.top_k is not None:
|
||||
client_params['top_k'] = self.top_k
|
||||
if self.seed is not None:
|
||||
client_params['seed'] = self.seed
|
||||
if self.stop_sequences is not None:
|
||||
client_params['stop_sequences'] = self.stop_sequences
|
||||
|
||||
return client_params
|
||||
|
||||
def get_client(self) -> AsyncAnthropicBedrock:
|
||||
"""
|
||||
Returns an AsyncAnthropicBedrock client.
|
||||
|
||||
Returns:
|
||||
AsyncAnthropicBedrock: An instance of the AsyncAnthropicBedrock client.
|
||||
"""
|
||||
client_params = self._get_client_params()
|
||||
return AsyncAnthropicBedrock(**client_params)
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return str(self.model)
|
||||
|
||||
def _get_usage(self, response: Message) -> ChatInvokeUsage | None:
|
||||
"""Extract usage information from the response."""
|
||||
usage = ChatInvokeUsage(
|
||||
prompt_tokens=response.usage.input_tokens
|
||||
+ (
|
||||
response.usage.cache_read_input_tokens or 0
|
||||
), # Total tokens in Anthropic are a bit fucked, you have to add cached tokens to the prompt tokens
|
||||
completion_tokens=response.usage.output_tokens,
|
||||
total_tokens=response.usage.input_tokens + response.usage.output_tokens,
|
||||
prompt_cached_tokens=response.usage.cache_read_input_tokens,
|
||||
prompt_cache_creation_tokens=response.usage.cache_creation_input_tokens,
|
||||
prompt_image_tokens=None,
|
||||
)
|
||||
return usage
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
anthropic_messages, system_prompt = AnthropicMessageSerializer.serialize_messages(messages)
|
||||
|
||||
try:
|
||||
if output_format is None:
|
||||
# Normal completion without structured output
|
||||
response = await self.get_client().messages.create(
|
||||
model=self.model,
|
||||
messages=anthropic_messages,
|
||||
system=system_prompt or NOT_GIVEN,
|
||||
**self._get_client_params_for_invoke(),
|
||||
)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
# Extract text from the first content block
|
||||
first_content = response.content[0]
|
||||
if isinstance(first_content, TextBlock):
|
||||
response_text = first_content.text
|
||||
else:
|
||||
# If it's not a text block, convert to string
|
||||
response_text = str(first_content)
|
||||
|
||||
return ChatInvokeCompletion(
|
||||
completion=response_text,
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
else:
|
||||
# Use tool calling for structured output
|
||||
# Create a tool that represents the output format
|
||||
tool_name = output_format.__name__
|
||||
schema = output_format.model_json_schema()
|
||||
|
||||
# Remove title from schema if present (Anthropic doesn't like it in parameters)
|
||||
if 'title' in schema:
|
||||
del schema['title']
|
||||
|
||||
tool = ToolParam(
|
||||
name=tool_name,
|
||||
description=f'Extract information in the format of {tool_name}',
|
||||
input_schema=schema,
|
||||
cache_control=CacheControlEphemeralParam(type='ephemeral'),
|
||||
)
|
||||
|
||||
# Force the model to use this tool
|
||||
tool_choice = ToolChoiceToolParam(type='tool', name=tool_name)
|
||||
|
||||
response = await self.get_client().messages.create(
|
||||
model=self.model,
|
||||
messages=anthropic_messages,
|
||||
tools=[tool],
|
||||
system=system_prompt or NOT_GIVEN,
|
||||
tool_choice=tool_choice,
|
||||
**self._get_client_params_for_invoke(),
|
||||
)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
# Extract the tool use block
|
||||
for content_block in response.content:
|
||||
if hasattr(content_block, 'type') and content_block.type == 'tool_use':
|
||||
# Parse the tool input as the structured output
|
||||
try:
|
||||
return ChatInvokeCompletion(completion=output_format.model_validate(content_block.input), usage=usage)
|
||||
except Exception as e:
|
||||
# If validation fails, try to parse it as JSON first
|
||||
if isinstance(content_block.input, str):
|
||||
data = json.loads(content_block.input)
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(data),
|
||||
usage=usage,
|
||||
)
|
||||
raise e
|
||||
|
||||
# If no tool use block found, raise an error
|
||||
raise ValueError('Expected tool use in response but none found')
|
||||
|
||||
except APIConnectionError as e:
|
||||
raise ModelProviderError(message=e.message, model=self.name) from e
|
||||
except RateLimitError as e:
|
||||
raise ModelRateLimitError(message=e.message, model=self.name) from e
|
||||
except APIStatusError as e:
|
||||
raise ModelProviderError(message=e.message, status_code=e.status_code, model=self.name) from e
|
||||
except Exception as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
@@ -0,0 +1,289 @@
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from os import getenv
|
||||
from typing import TYPE_CHECKING, Any, TypeVar, overload
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.aws.serializer import AWSBedrockMessageSerializer
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.views import ChatInvokeCompletion, ChatInvokeUsage
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from boto3 import client as AwsClient # type: ignore
|
||||
from boto3.session import Session # type: ignore
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatAWSBedrock(BaseChatModel):
|
||||
"""
|
||||
AWS Bedrock chat model supporting multiple providers (Anthropic, Meta, etc.).
|
||||
|
||||
This class provides access to various models via AWS Bedrock,
|
||||
supporting both text generation and structured output via tool calling.
|
||||
|
||||
To use this model, you need to either:
|
||||
1. Set the following environment variables:
|
||||
- AWS_ACCESS_KEY_ID
|
||||
- AWS_SECRET_ACCESS_KEY
|
||||
- AWS_SESSION_TOKEN (only required when using temporary credentials)
|
||||
- AWS_REGION
|
||||
2. Or provide a boto3 Session object
|
||||
3. Or use AWS SSO authentication
|
||||
"""
|
||||
|
||||
# Model configuration
|
||||
model: str = 'anthropic.claude-3-5-sonnet-20240620-v1:0'
|
||||
max_tokens: int | None = 4096
|
||||
temperature: float | None = None
|
||||
top_p: float | None = None
|
||||
seed: int | None = None
|
||||
stop_sequences: list[str] | None = None
|
||||
|
||||
# AWS credentials and configuration
|
||||
aws_access_key_id: str | None = None
|
||||
aws_secret_access_key: str | None = None
|
||||
aws_session_token: str | None = None
|
||||
aws_region: str | None = None
|
||||
aws_sso_auth: bool = False
|
||||
session: 'Session | None' = None
|
||||
|
||||
# Request parameters
|
||||
request_params: dict[str, Any] | None = None
|
||||
|
||||
# Static
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'aws_bedrock'
|
||||
|
||||
def _get_client(self) -> 'AwsClient': # type: ignore
|
||||
"""Get the AWS Bedrock client."""
|
||||
try:
|
||||
from boto3 import client as AwsClient # type: ignore
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
'`boto3` not installed. Please install using `pip install browser-use[aws] or pip install browser-use[all]`'
|
||||
)
|
||||
|
||||
if self.session:
|
||||
return self.session.client('bedrock-runtime')
|
||||
|
||||
# Get credentials from environment or instance parameters
|
||||
access_key = self.aws_access_key_id or getenv('AWS_ACCESS_KEY_ID')
|
||||
secret_key = self.aws_secret_access_key or getenv('AWS_SECRET_ACCESS_KEY')
|
||||
session_token = self.aws_session_token or getenv('AWS_SESSION_TOKEN')
|
||||
region = self.aws_region or getenv('AWS_REGION') or getenv('AWS_DEFAULT_REGION')
|
||||
|
||||
if self.aws_sso_auth:
|
||||
return AwsClient(service_name='bedrock-runtime', region_name=region)
|
||||
else:
|
||||
if not access_key or not secret_key:
|
||||
raise ModelProviderError(
|
||||
message='AWS credentials not found. Please set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables (and AWS_SESSION_TOKEN if using temporary credentials) or provide a boto3 session.',
|
||||
model=self.name,
|
||||
)
|
||||
|
||||
return AwsClient(
|
||||
service_name='bedrock-runtime',
|
||||
region_name=region,
|
||||
aws_access_key_id=access_key,
|
||||
aws_secret_access_key=secret_key,
|
||||
aws_session_token=session_token,
|
||||
)
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return str(self.model)
|
||||
|
||||
def _get_inference_config(self) -> dict[str, Any]:
|
||||
"""Get the inference configuration for the request."""
|
||||
config = {}
|
||||
if self.max_tokens is not None:
|
||||
config['maxTokens'] = self.max_tokens
|
||||
if self.temperature is not None:
|
||||
config['temperature'] = self.temperature
|
||||
if self.top_p is not None:
|
||||
config['topP'] = self.top_p
|
||||
if self.stop_sequences is not None:
|
||||
config['stopSequences'] = self.stop_sequences
|
||||
if self.seed is not None:
|
||||
config['seed'] = self.seed
|
||||
return config
|
||||
|
||||
def _format_tools_for_request(self, output_format: type[BaseModel]) -> list[dict[str, Any]]:
|
||||
"""Format a Pydantic model as a tool for structured output."""
|
||||
schema = output_format.model_json_schema()
|
||||
|
||||
# Convert Pydantic schema to Bedrock tool format
|
||||
properties = {}
|
||||
required = []
|
||||
|
||||
for prop_name, prop_info in schema.get('properties', {}).items():
|
||||
properties[prop_name] = {
|
||||
'type': prop_info.get('type', 'string'),
|
||||
'description': prop_info.get('description', ''),
|
||||
}
|
||||
|
||||
# Add required fields
|
||||
required = schema.get('required', [])
|
||||
|
||||
return [
|
||||
{
|
||||
'toolSpec': {
|
||||
'name': f'extract_{output_format.__name__.lower()}',
|
||||
'description': f'Extract information in the format of {output_format.__name__}',
|
||||
'inputSchema': {'json': {'type': 'object', 'properties': properties, 'required': required}},
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
def _get_usage(self, response: dict[str, Any]) -> ChatInvokeUsage | None:
|
||||
"""Extract usage information from the response."""
|
||||
if 'usage' not in response:
|
||||
return None
|
||||
|
||||
usage_data = response['usage']
|
||||
return ChatInvokeUsage(
|
||||
prompt_tokens=usage_data.get('inputTokens', 0),
|
||||
completion_tokens=usage_data.get('outputTokens', 0),
|
||||
total_tokens=usage_data.get('totalTokens', 0),
|
||||
prompt_cached_tokens=None, # Bedrock doesn't provide this
|
||||
prompt_cache_creation_tokens=None,
|
||||
prompt_image_tokens=None,
|
||||
)
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
"""
|
||||
Invoke the AWS Bedrock model with the given messages.
|
||||
|
||||
Args:
|
||||
messages: List of chat messages
|
||||
output_format: Optional Pydantic model class for structured output
|
||||
|
||||
Returns:
|
||||
Either a string response or an instance of output_format
|
||||
"""
|
||||
try:
|
||||
from botocore.exceptions import ClientError # type: ignore
|
||||
except ImportError:
|
||||
raise ImportError(
|
||||
'`boto3` not installed. Please install using `pip install browser-use[aws] or pip install browser-use[all]`'
|
||||
)
|
||||
|
||||
bedrock_messages, system_message = AWSBedrockMessageSerializer.serialize_messages(messages)
|
||||
|
||||
try:
|
||||
# Prepare the request body
|
||||
body: dict[str, Any] = {}
|
||||
|
||||
if system_message:
|
||||
body['system'] = system_message
|
||||
|
||||
inference_config = self._get_inference_config()
|
||||
if inference_config:
|
||||
body['inferenceConfig'] = inference_config
|
||||
|
||||
# Handle structured output via tool calling
|
||||
if output_format is not None:
|
||||
tools = self._format_tools_for_request(output_format)
|
||||
body['toolConfig'] = {'tools': tools}
|
||||
|
||||
# Add any additional request parameters
|
||||
if self.request_params:
|
||||
body.update(self.request_params)
|
||||
|
||||
# Filter out None values
|
||||
body = {k: v for k, v in body.items() if v is not None}
|
||||
|
||||
# Make the API call
|
||||
client = self._get_client()
|
||||
response = client.converse(modelId=self.model, messages=bedrock_messages, **body)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
# Extract the response content
|
||||
if 'output' in response and 'message' in response['output']:
|
||||
message = response['output']['message']
|
||||
content = message.get('content', [])
|
||||
|
||||
if output_format is None:
|
||||
# Return text response
|
||||
text_content = []
|
||||
for item in content:
|
||||
if 'text' in item:
|
||||
text_content.append(item['text'])
|
||||
|
||||
response_text = '\n'.join(text_content) if text_content else ''
|
||||
return ChatInvokeCompletion(
|
||||
completion=response_text,
|
||||
usage=usage,
|
||||
)
|
||||
else:
|
||||
# Handle structured output from tool calls
|
||||
for item in content:
|
||||
if 'toolUse' in item:
|
||||
tool_use = item['toolUse']
|
||||
tool_input = tool_use.get('input', {})
|
||||
|
||||
try:
|
||||
# Validate and return the structured output
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(tool_input),
|
||||
usage=usage,
|
||||
)
|
||||
except Exception as e:
|
||||
# If validation fails, try to parse as JSON first
|
||||
if isinstance(tool_input, str):
|
||||
try:
|
||||
data = json.loads(tool_input)
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(data),
|
||||
usage=usage,
|
||||
)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
raise ModelProviderError(
|
||||
message=f'Failed to validate structured output: {str(e)}',
|
||||
model=self.name,
|
||||
) from e
|
||||
|
||||
# If no tool use found but output_format was requested
|
||||
raise ModelProviderError(
|
||||
message='Expected structured output but no tool use found in response',
|
||||
model=self.name,
|
||||
)
|
||||
|
||||
# If no valid content found
|
||||
if output_format is None:
|
||||
return ChatInvokeCompletion(
|
||||
completion='',
|
||||
usage=usage,
|
||||
)
|
||||
else:
|
||||
raise ModelProviderError(
|
||||
message='No valid content found in response',
|
||||
model=self.name,
|
||||
)
|
||||
|
||||
except ClientError as e:
|
||||
error_code = e.response.get('Error', {}).get('Code', 'Unknown')
|
||||
error_message = e.response.get('Error', {}).get('Message', str(e))
|
||||
|
||||
if error_code in ['ThrottlingException', 'TooManyRequestsException']:
|
||||
raise ModelRateLimitError(message=error_message, model=self.name) from e
|
||||
else:
|
||||
raise ModelProviderError(message=error_message, model=self.name) from e
|
||||
except Exception as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
@@ -0,0 +1,257 @@
|
||||
import base64
|
||||
import json
|
||||
import re
|
||||
from typing import Any, overload
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
ContentPartImageParam,
|
||||
ContentPartRefusalParam,
|
||||
ContentPartTextParam,
|
||||
SystemMessage,
|
||||
ToolCall,
|
||||
UserMessage,
|
||||
)
|
||||
|
||||
|
||||
class AWSBedrockMessageSerializer:
|
||||
"""Serializer for converting between custom message types and AWS Bedrock message format."""
|
||||
|
||||
@staticmethod
|
||||
def _is_base64_image(url: str) -> bool:
|
||||
"""Check if the URL is a base64 encoded image."""
|
||||
return url.startswith('data:image/')
|
||||
|
||||
@staticmethod
|
||||
def _is_url_image(url: str) -> bool:
|
||||
"""Check if the URL is a regular HTTP/HTTPS image URL."""
|
||||
return url.startswith(('http://', 'https://')) and any(
|
||||
url.lower().endswith(ext) for ext in ['.jpg', '.jpeg', '.png', '.gif', '.webp', '.bmp']
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _parse_base64_url(url: str) -> tuple[str, bytes]:
|
||||
"""Parse a base64 data URL to extract format and raw bytes."""
|
||||
# Format: data:image/jpeg;base64,<data>
|
||||
if not url.startswith('data:'):
|
||||
raise ValueError(f'Invalid base64 URL: {url}')
|
||||
|
||||
header, data = url.split(',', 1)
|
||||
|
||||
# Extract format from mime type
|
||||
mime_match = re.search(r'image/(\w+)', header)
|
||||
if mime_match:
|
||||
format_name = mime_match.group(1).lower()
|
||||
# Map common formats
|
||||
format_mapping = {'jpg': 'jpeg', 'jpeg': 'jpeg', 'png': 'png', 'gif': 'gif', 'webp': 'webp'}
|
||||
image_format = format_mapping.get(format_name, 'jpeg')
|
||||
else:
|
||||
image_format = 'jpeg' # Default format
|
||||
|
||||
# Decode base64 data
|
||||
try:
|
||||
image_bytes = base64.b64decode(data)
|
||||
except Exception as e:
|
||||
raise ValueError(f'Failed to decode base64 image data: {e}')
|
||||
|
||||
return image_format, image_bytes
|
||||
|
||||
@staticmethod
|
||||
def _download_and_convert_image(url: str) -> tuple[str, bytes]:
|
||||
"""Download an image from URL and convert to base64 bytes."""
|
||||
try:
|
||||
import httpx
|
||||
except ImportError:
|
||||
raise ImportError('httpx not available. Please install it to use URL images with AWS Bedrock.')
|
||||
|
||||
try:
|
||||
response = httpx.get(url, timeout=30)
|
||||
response.raise_for_status()
|
||||
|
||||
# Detect format from content type or URL
|
||||
content_type = response.headers.get('content-type', '').lower()
|
||||
if 'jpeg' in content_type or url.lower().endswith(('.jpg', '.jpeg')):
|
||||
image_format = 'jpeg'
|
||||
elif 'png' in content_type or url.lower().endswith('.png'):
|
||||
image_format = 'png'
|
||||
elif 'gif' in content_type or url.lower().endswith('.gif'):
|
||||
image_format = 'gif'
|
||||
elif 'webp' in content_type or url.lower().endswith('.webp'):
|
||||
image_format = 'webp'
|
||||
else:
|
||||
image_format = 'jpeg' # Default format
|
||||
|
||||
return image_format, response.content
|
||||
|
||||
except Exception as e:
|
||||
raise ValueError(f'Failed to download image from {url}: {e}')
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_text(part: ContentPartTextParam) -> dict[str, Any]:
|
||||
"""Convert a text content part to AWS Bedrock format."""
|
||||
return {'text': part.text}
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_image(part: ContentPartImageParam) -> dict[str, Any]:
|
||||
"""Convert an image content part to AWS Bedrock format."""
|
||||
url = part.image_url.url
|
||||
|
||||
if AWSBedrockMessageSerializer._is_base64_image(url):
|
||||
# Handle base64 encoded images
|
||||
image_format, image_bytes = AWSBedrockMessageSerializer._parse_base64_url(url)
|
||||
elif AWSBedrockMessageSerializer._is_url_image(url):
|
||||
# Download and convert URL images
|
||||
image_format, image_bytes = AWSBedrockMessageSerializer._download_and_convert_image(url)
|
||||
else:
|
||||
raise ValueError(f'Unsupported image URL format: {url}')
|
||||
|
||||
return {
|
||||
'image': {
|
||||
'format': image_format,
|
||||
'source': {
|
||||
'bytes': image_bytes,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _serialize_user_content(
|
||||
content: str | list[ContentPartTextParam | ContentPartImageParam],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Serialize content for user messages."""
|
||||
if isinstance(content, str):
|
||||
return [{'text': content}]
|
||||
|
||||
content_blocks: list[dict[str, Any]] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
content_blocks.append(AWSBedrockMessageSerializer._serialize_content_part_text(part))
|
||||
elif part.type == 'image_url':
|
||||
content_blocks.append(AWSBedrockMessageSerializer._serialize_content_part_image(part))
|
||||
|
||||
return content_blocks
|
||||
|
||||
@staticmethod
|
||||
def _serialize_system_content(
|
||||
content: str | list[ContentPartTextParam],
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Serialize content for system messages."""
|
||||
if isinstance(content, str):
|
||||
return [{'text': content}]
|
||||
|
||||
content_blocks: list[dict[str, Any]] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
content_blocks.append(AWSBedrockMessageSerializer._serialize_content_part_text(part))
|
||||
|
||||
return content_blocks
|
||||
|
||||
@staticmethod
|
||||
def _serialize_assistant_content(
|
||||
content: str | list[ContentPartTextParam | ContentPartRefusalParam] | None,
|
||||
) -> list[dict[str, Any]]:
|
||||
"""Serialize content for assistant messages."""
|
||||
if content is None:
|
||||
return []
|
||||
if isinstance(content, str):
|
||||
return [{'text': content}]
|
||||
|
||||
content_blocks: list[dict[str, Any]] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
content_blocks.append(AWSBedrockMessageSerializer._serialize_content_part_text(part))
|
||||
# Skip refusal content parts - AWS Bedrock doesn't need them
|
||||
|
||||
return content_blocks
|
||||
|
||||
@staticmethod
|
||||
def _serialize_tool_call(tool_call: ToolCall) -> dict[str, Any]:
|
||||
"""Convert a tool call to AWS Bedrock format."""
|
||||
try:
|
||||
arguments = json.loads(tool_call.function.arguments)
|
||||
except json.JSONDecodeError:
|
||||
# If arguments aren't valid JSON, wrap them
|
||||
arguments = {'arguments': tool_call.function.arguments}
|
||||
|
||||
return {
|
||||
'toolUse': {
|
||||
'toolUseId': tool_call.id,
|
||||
'name': tool_call.function.name,
|
||||
'input': arguments,
|
||||
}
|
||||
}
|
||||
|
||||
# region - Serialize overloads
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: UserMessage) -> dict[str, Any]: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: SystemMessage) -> SystemMessage: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: AssistantMessage) -> dict[str, Any]: ...
|
||||
|
||||
@staticmethod
|
||||
def serialize(message: BaseMessage) -> dict[str, Any] | SystemMessage:
|
||||
"""Serialize a custom message to AWS Bedrock format."""
|
||||
|
||||
if isinstance(message, UserMessage):
|
||||
return {
|
||||
'role': 'user',
|
||||
'content': AWSBedrockMessageSerializer._serialize_user_content(message.content),
|
||||
}
|
||||
|
||||
elif isinstance(message, SystemMessage):
|
||||
# System messages are handled separately in AWS Bedrock
|
||||
return message
|
||||
|
||||
elif isinstance(message, AssistantMessage):
|
||||
content_blocks: list[dict[str, Any]] = []
|
||||
|
||||
# Add content blocks if present
|
||||
if message.content is not None:
|
||||
content_blocks.extend(AWSBedrockMessageSerializer._serialize_assistant_content(message.content))
|
||||
|
||||
# Add tool use blocks if present
|
||||
if message.tool_calls:
|
||||
for tool_call in message.tool_calls:
|
||||
content_blocks.append(AWSBedrockMessageSerializer._serialize_tool_call(tool_call))
|
||||
|
||||
# AWS Bedrock requires at least one content block
|
||||
if not content_blocks:
|
||||
content_blocks = [{'text': ''}]
|
||||
|
||||
return {
|
||||
'role': 'assistant',
|
||||
'content': content_blocks,
|
||||
}
|
||||
|
||||
else:
|
||||
raise ValueError(f'Unknown message type: {type(message)}')
|
||||
|
||||
@staticmethod
|
||||
def serialize_messages(messages: list[BaseMessage]) -> tuple[list[dict[str, Any]], list[dict[str, Any]] | None]:
|
||||
"""
|
||||
Serialize a list of messages, extracting any system message.
|
||||
|
||||
Returns:
|
||||
Tuple of (bedrock_messages, system_message) where system_message is extracted
|
||||
from any SystemMessage in the list.
|
||||
"""
|
||||
bedrock_messages: list[dict[str, Any]] = []
|
||||
system_message: list[dict[str, Any]] | None = None
|
||||
|
||||
for message in messages:
|
||||
if isinstance(message, SystemMessage):
|
||||
# Extract system message content
|
||||
system_message = AWSBedrockMessageSerializer._serialize_system_content(message.content)
|
||||
else:
|
||||
# Serialize and add to regular messages
|
||||
serialized = AWSBedrockMessageSerializer.serialize(message)
|
||||
bedrock_messages.append(serialized)
|
||||
|
||||
return bedrock_messages, system_message
|
||||
@@ -0,0 +1,91 @@
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
import httpx
|
||||
from openai import AsyncAzureOpenAI as AsyncAzureOpenAIClient
|
||||
from openai.types.shared import ChatModel
|
||||
|
||||
from browser_use.llm.openai.like import ChatOpenAILike
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatAzureOpenAI(ChatOpenAILike):
|
||||
"""
|
||||
A class for to interact with any provider using the OpenAI API schema.
|
||||
|
||||
Args:
|
||||
model (str): The name of the OpenAI model to use. Defaults to "not-provided".
|
||||
api_key (Optional[str]): The API key to use. Defaults to "not-provided".
|
||||
"""
|
||||
|
||||
# Model configuration
|
||||
model: str | ChatModel
|
||||
|
||||
# Client initialization parameters
|
||||
api_key: str | None = None
|
||||
api_version: str | None = '2024-12-01-preview'
|
||||
azure_endpoint: str | None = None
|
||||
azure_deployment: str | None = None
|
||||
base_url: str | None = None
|
||||
azure_ad_token: str | None = None
|
||||
azure_ad_token_provider: Any | None = None
|
||||
|
||||
default_headers: dict[str, str] | None = None
|
||||
default_query: dict[str, Any] | None = None
|
||||
|
||||
client: AsyncAzureOpenAIClient | None = None
|
||||
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'azure'
|
||||
|
||||
def _get_client_params(self) -> dict[str, Any]:
|
||||
_client_params: dict[str, Any] = {}
|
||||
|
||||
self.api_key = self.api_key or os.getenv('AZURE_OPENAI_API_KEY')
|
||||
self.azure_endpoint = self.azure_endpoint or os.getenv('AZURE_OPENAI_ENDPOINT')
|
||||
self.azure_deployment = self.azure_deployment or os.getenv('AZURE_OPENAI_DEPLOYMENT')
|
||||
params_mapping = {
|
||||
'api_key': self.api_key,
|
||||
'api_version': self.api_version,
|
||||
'organization': self.organization,
|
||||
'azure_endpoint': self.azure_endpoint,
|
||||
'azure_deployment': self.azure_deployment,
|
||||
'base_url': self.base_url,
|
||||
'azure_ad_token': self.azure_ad_token,
|
||||
'azure_ad_token_provider': self.azure_ad_token_provider,
|
||||
'http_client': self.http_client,
|
||||
}
|
||||
if self.default_headers is not None:
|
||||
_client_params['default_headers'] = self.default_headers
|
||||
if self.default_query is not None:
|
||||
_client_params['default_query'] = self.default_query
|
||||
|
||||
_client_params.update({k: v for k, v in params_mapping.items() if v is not None})
|
||||
|
||||
return _client_params
|
||||
|
||||
def get_client(self) -> AsyncAzureOpenAIClient:
|
||||
"""
|
||||
Returns an asynchronous OpenAI client.
|
||||
|
||||
Returns:
|
||||
AsyncAzureOpenAIClient: An instance of the asynchronous OpenAI client.
|
||||
"""
|
||||
if self.client:
|
||||
return self.client
|
||||
|
||||
_client_params: dict[str, Any] = self._get_client_params()
|
||||
|
||||
if self.http_client:
|
||||
_client_params['http_client'] = self.http_client
|
||||
else:
|
||||
# Create a new async HTTP client with custom limits
|
||||
_client_params['http_client'] = httpx.AsyncClient(
|
||||
limits=httpx.Limits(max_connections=20, max_keepalive_connections=6)
|
||||
)
|
||||
|
||||
self.client = AsyncAzureOpenAIClient(**_client_params)
|
||||
|
||||
return self.client
|
||||
@@ -0,0 +1,57 @@
|
||||
"""
|
||||
We have switched all of our code from langchain to openai.types.chat.chat_completion_message_param.
|
||||
|
||||
For easier transition we have
|
||||
"""
|
||||
|
||||
from typing import Any, Protocol, TypeVar, overload, runtime_checkable
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.views import ChatInvokeCompletion
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@runtime_checkable
|
||||
class BaseChatModel(Protocol):
|
||||
_verified_api_keys: bool = False
|
||||
|
||||
model: str
|
||||
|
||||
@property
|
||||
def provider(self) -> str: ...
|
||||
|
||||
@property
|
||||
def name(self) -> str: ...
|
||||
|
||||
@property
|
||||
def model_name(self) -> str:
|
||||
# for legacy support
|
||||
return self.model
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]: ...
|
||||
|
||||
@classmethod
|
||||
def __get_pydantic_core_schema__(
|
||||
cls,
|
||||
source_type: type,
|
||||
handler: Any,
|
||||
) -> Any:
|
||||
"""
|
||||
Allow this Protocol to be used in Pydantic models -> very useful to typesafe the agent settings for example.
|
||||
Returns a schema that allows any object (since this is a Protocol).
|
||||
"""
|
||||
from pydantic_core import core_schema
|
||||
|
||||
# Return a schema that accepts any object for Protocol types
|
||||
return core_schema.any_schema()
|
||||
@@ -0,0 +1,212 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, TypeVar, overload
|
||||
|
||||
import httpx
|
||||
from openai import (
|
||||
APIConnectionError,
|
||||
APIError,
|
||||
APIStatusError,
|
||||
APITimeoutError,
|
||||
AsyncOpenAI,
|
||||
RateLimitError,
|
||||
)
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.deepseek.serializer import DeepSeekMessageSerializer
|
||||
from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.schema import SchemaOptimizer
|
||||
from browser_use.llm.views import ChatInvokeCompletion
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatDeepSeek(BaseChatModel):
|
||||
"""DeepSeek /chat/completions wrapper (OpenAI-compatible)."""
|
||||
|
||||
model: str = 'deepseek-chat'
|
||||
|
||||
# Generation parameters
|
||||
max_tokens: int | None = None
|
||||
temperature: float | None = None
|
||||
top_p: float | None = None
|
||||
seed: int | None = None
|
||||
|
||||
# Connection parameters
|
||||
api_key: str | None = None
|
||||
base_url: str | httpx.URL | None = 'https://api.deepseek.com/v1'
|
||||
timeout: float | httpx.Timeout | None = None
|
||||
client_params: dict[str, Any] | None = None
|
||||
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'deepseek'
|
||||
|
||||
def _client(self) -> AsyncOpenAI:
|
||||
return AsyncOpenAI(
|
||||
api_key=self.api_key,
|
||||
base_url=self.base_url,
|
||||
timeout=self.timeout,
|
||||
**(self.client_params or {}),
|
||||
)
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return self.model
|
||||
|
||||
@overload
|
||||
async def ainvoke(
|
||||
self,
|
||||
messages: list[BaseMessage],
|
||||
output_format: None = None,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
stop: list[str] | None = None,
|
||||
) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(
|
||||
self,
|
||||
messages: list[BaseMessage],
|
||||
output_format: type[T],
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
stop: list[str] | None = None,
|
||||
) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self,
|
||||
messages: list[BaseMessage],
|
||||
output_format: type[T] | None = None,
|
||||
tools: list[dict[str, Any]] | None = None,
|
||||
stop: list[str] | None = None,
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
"""
|
||||
DeepSeek ainvoke supports:
|
||||
1. Regular text/multi-turn conversation
|
||||
2. Function Calling
|
||||
3. JSON Output (response_format)
|
||||
4. Conversation prefix continuation (beta, prefix, stop)
|
||||
"""
|
||||
client = self._client()
|
||||
ds_messages = DeepSeekMessageSerializer.serialize_messages(messages)
|
||||
common: dict[str, Any] = {}
|
||||
|
||||
if self.temperature is not None:
|
||||
common['temperature'] = self.temperature
|
||||
if self.max_tokens is not None:
|
||||
common['max_tokens'] = self.max_tokens
|
||||
if self.top_p is not None:
|
||||
common['top_p'] = self.top_p
|
||||
if self.seed is not None:
|
||||
common['seed'] = self.seed
|
||||
|
||||
# Beta conversation prefix continuation (see official documentation)
|
||||
if self.base_url and str(self.base_url).endswith('/beta'):
|
||||
# The last assistant message must have prefix
|
||||
if ds_messages and isinstance(ds_messages[-1], dict) and ds_messages[-1].get('role') == 'assistant':
|
||||
ds_messages[-1]['prefix'] = True
|
||||
if stop:
|
||||
common['stop'] = stop
|
||||
|
||||
# ① Regular multi-turn conversation/text output
|
||||
if output_format is None and not tools:
|
||||
try:
|
||||
resp = await client.chat.completions.create( # type: ignore
|
||||
model=self.model,
|
||||
messages=ds_messages, # type: ignore
|
||||
**common,
|
||||
)
|
||||
return ChatInvokeCompletion(
|
||||
completion=resp.choices[0].message.content or '',
|
||||
usage=None,
|
||||
)
|
||||
except RateLimitError as e:
|
||||
raise ModelRateLimitError(str(e), model=self.name) from e
|
||||
except (APIError, APIConnectionError, APITimeoutError, APIStatusError) as e:
|
||||
raise ModelProviderError(str(e), model=self.name) from e
|
||||
except Exception as e:
|
||||
raise ModelProviderError(str(e), model=self.name) from e
|
||||
|
||||
# ② Function Calling path (with tools or output_format)
|
||||
if tools or (output_format is not None and hasattr(output_format, 'model_json_schema')):
|
||||
try:
|
||||
call_tools = tools
|
||||
tool_choice = None
|
||||
if output_format is not None and hasattr(output_format, 'model_json_schema'):
|
||||
tool_name = output_format.__name__
|
||||
schema = SchemaOptimizer.create_optimized_json_schema(output_format)
|
||||
schema.pop('title', None)
|
||||
call_tools = [
|
||||
{
|
||||
'type': 'function',
|
||||
'function': {
|
||||
'name': tool_name,
|
||||
'description': f'Return a JSON object of type {tool_name}',
|
||||
'parameters': schema,
|
||||
},
|
||||
}
|
||||
]
|
||||
tool_choice = {'type': 'function', 'function': {'name': tool_name}}
|
||||
resp = await client.chat.completions.create( # type: ignore
|
||||
model=self.model,
|
||||
messages=ds_messages, # type: ignore
|
||||
tools=call_tools, # type: ignore
|
||||
tool_choice=tool_choice, # type: ignore
|
||||
**common,
|
||||
)
|
||||
msg = resp.choices[0].message
|
||||
if not msg.tool_calls:
|
||||
raise ValueError('Expected tool_calls in response but got none')
|
||||
raw_args = msg.tool_calls[0].function.arguments
|
||||
if isinstance(raw_args, str):
|
||||
parsed = json.loads(raw_args)
|
||||
else:
|
||||
parsed = raw_args
|
||||
# --------- Fix: only use model_validate when output_format is not None ----------
|
||||
if output_format is not None:
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(parsed),
|
||||
usage=None,
|
||||
)
|
||||
else:
|
||||
# If no output_format, return dict directly
|
||||
return ChatInvokeCompletion(
|
||||
completion=parsed,
|
||||
usage=None,
|
||||
)
|
||||
except RateLimitError as e:
|
||||
raise ModelRateLimitError(str(e), model=self.name) from e
|
||||
except (APIError, APIConnectionError, APITimeoutError, APIStatusError) as e:
|
||||
raise ModelProviderError(str(e), model=self.name) from e
|
||||
except Exception as e:
|
||||
raise ModelProviderError(str(e), model=self.name) from e
|
||||
|
||||
# ③ JSON Output path (official response_format)
|
||||
if output_format is not None and hasattr(output_format, 'model_json_schema'):
|
||||
try:
|
||||
resp = await client.chat.completions.create( # type: ignore
|
||||
model=self.model,
|
||||
messages=ds_messages, # type: ignore
|
||||
response_format={'type': 'json_object'},
|
||||
**common,
|
||||
)
|
||||
content = resp.choices[0].message.content
|
||||
if not content:
|
||||
raise ModelProviderError('Empty JSON content in DeepSeek response', model=self.name)
|
||||
parsed = output_format.model_validate_json(content)
|
||||
return ChatInvokeCompletion(
|
||||
completion=parsed,
|
||||
usage=None,
|
||||
)
|
||||
except RateLimitError as e:
|
||||
raise ModelRateLimitError(str(e), model=self.name) from e
|
||||
except (APIError, APIConnectionError, APITimeoutError, APIStatusError) as e:
|
||||
raise ModelProviderError(str(e), model=self.name) from e
|
||||
except Exception as e:
|
||||
raise ModelProviderError(str(e), model=self.name) from e
|
||||
|
||||
raise ModelProviderError('No valid ainvoke execution path for DeepSeek LLM', model=self.name)
|
||||
@@ -0,0 +1,109 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from typing import Any, overload
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
ContentPartImageParam,
|
||||
ContentPartTextParam,
|
||||
SystemMessage,
|
||||
ToolCall,
|
||||
UserMessage,
|
||||
)
|
||||
|
||||
MessageDict = dict[str, Any]
|
||||
|
||||
|
||||
class DeepSeekMessageSerializer:
|
||||
"""Serializer for converting browser-use messages to DeepSeek messages."""
|
||||
|
||||
# -------- content 处理 --------------------------------------------------
|
||||
@staticmethod
|
||||
def _serialize_text_part(part: ContentPartTextParam) -> str:
|
||||
return part.text
|
||||
|
||||
@staticmethod
|
||||
def _serialize_image_part(part: ContentPartImageParam) -> dict[str, Any]:
|
||||
url = part.image_url.url
|
||||
if url.startswith('data:'):
|
||||
return {'type': 'image_url', 'image_url': {'url': url}}
|
||||
return {'type': 'image_url', 'image_url': {'url': url}}
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content(content: Any) -> str | list[dict[str, Any]]:
|
||||
if content is None:
|
||||
return ''
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
serialized: list[dict[str, Any]] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized.append({'type': 'text', 'text': DeepSeekMessageSerializer._serialize_text_part(part)})
|
||||
elif part.type == 'image_url':
|
||||
serialized.append(DeepSeekMessageSerializer._serialize_image_part(part))
|
||||
elif part.type == 'refusal':
|
||||
serialized.append({'type': 'text', 'text': f'[Refusal] {part.refusal}'})
|
||||
return serialized
|
||||
|
||||
# -------- Tool-call 处理 -------------------------------------------------
|
||||
@staticmethod
|
||||
def _serialize_tool_calls(tool_calls: list[ToolCall]) -> list[dict[str, Any]]:
|
||||
deepseek_tool_calls: list[dict[str, Any]] = []
|
||||
for tc in tool_calls:
|
||||
try:
|
||||
arguments = json.loads(tc.function.arguments)
|
||||
except json.JSONDecodeError:
|
||||
arguments = {'arguments': tc.function.arguments}
|
||||
deepseek_tool_calls.append(
|
||||
{
|
||||
'id': tc.id,
|
||||
'type': 'function',
|
||||
'function': {
|
||||
'name': tc.function.name,
|
||||
'arguments': arguments,
|
||||
},
|
||||
}
|
||||
)
|
||||
return deepseek_tool_calls
|
||||
|
||||
# -------- 单条消息序列化 -------------------------------------------------
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: UserMessage) -> MessageDict: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: SystemMessage) -> MessageDict: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: AssistantMessage) -> MessageDict: ...
|
||||
|
||||
@staticmethod
|
||||
def serialize(message: BaseMessage) -> MessageDict:
|
||||
if isinstance(message, UserMessage):
|
||||
return {
|
||||
'role': 'user',
|
||||
'content': DeepSeekMessageSerializer._serialize_content(message.content),
|
||||
}
|
||||
if isinstance(message, SystemMessage):
|
||||
return {
|
||||
'role': 'system',
|
||||
'content': DeepSeekMessageSerializer._serialize_content(message.content),
|
||||
}
|
||||
if isinstance(message, AssistantMessage):
|
||||
msg: MessageDict = {
|
||||
'role': 'assistant',
|
||||
'content': DeepSeekMessageSerializer._serialize_content(message.content),
|
||||
}
|
||||
if message.tool_calls:
|
||||
msg['tool_calls'] = DeepSeekMessageSerializer._serialize_tool_calls(message.tool_calls)
|
||||
return msg
|
||||
raise ValueError(f'Unknown message type: {type(message)}')
|
||||
|
||||
# -------- 列表序列化 -----------------------------------------------------
|
||||
@staticmethod
|
||||
def serialize_messages(messages: list[BaseMessage]) -> list[MessageDict]:
|
||||
return [DeepSeekMessageSerializer.serialize(m) for m in messages]
|
||||
@@ -0,0 +1,27 @@
|
||||
class ModelError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ModelProviderError(ModelError):
|
||||
"""Exception raised when a model provider returns an error."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
status_code: int = 502,
|
||||
model: str | None = None,
|
||||
):
|
||||
super().__init__(message, status_code)
|
||||
self.model = model
|
||||
|
||||
|
||||
class ModelRateLimitError(ModelProviderError):
|
||||
"""Exception raised when a model provider returns a rate limit error."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
message: str,
|
||||
status_code: int = 429,
|
||||
model: str | None = None,
|
||||
):
|
||||
super().__init__(message, status_code, model)
|
||||
@@ -0,0 +1,3 @@
|
||||
from browser_use.llm.google.chat import ChatGoogle
|
||||
|
||||
__all__ = ['ChatGoogle']
|
||||
@@ -0,0 +1,506 @@
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Literal, TypeVar, overload
|
||||
|
||||
from google import genai
|
||||
from google.auth.credentials import Credentials
|
||||
from google.genai import types
|
||||
from google.genai.types import MediaModality
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.exceptions import ModelProviderError
|
||||
from browser_use.llm.google.serializer import GoogleMessageSerializer
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.schema import SchemaOptimizer
|
||||
from browser_use.llm.views import ChatInvokeCompletion, ChatInvokeUsage
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
VerifiedGeminiModels = Literal[
|
||||
'gemini-2.0-flash',
|
||||
'gemini-2.0-flash-exp',
|
||||
'gemini-2.0-flash-lite-preview-02-05',
|
||||
'Gemini-2.0-exp',
|
||||
'gemini-2.5-flash',
|
||||
'gemini-2.5-flash-lite',
|
||||
'gemini-2.5-pro',
|
||||
'gemma-3-27b-it',
|
||||
'gemma-3-4b',
|
||||
'gemma-3-12b',
|
||||
'gemma-3n-e2b',
|
||||
'gemma-3n-e4b',
|
||||
]
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatGoogle(BaseChatModel):
|
||||
"""
|
||||
A wrapper around Google's Gemini chat model using the genai client.
|
||||
|
||||
This class accepts all genai.Client parameters while adding model,
|
||||
temperature, and config parameters for the LLM interface.
|
||||
|
||||
Args:
|
||||
model: The Gemini model to use
|
||||
temperature: Temperature for response generation
|
||||
config: Additional configuration parameters to pass to generate_content
|
||||
(e.g., tools, safety_settings, etc.).
|
||||
api_key: Google API key
|
||||
vertexai: Whether to use Vertex AI
|
||||
credentials: Google credentials object
|
||||
project: Google Cloud project ID
|
||||
location: Google Cloud location
|
||||
http_options: HTTP options for the client
|
||||
include_system_in_user: If True, system messages are included in the first user message
|
||||
supports_structured_output: If True, uses native JSON mode; if False, uses prompt-based fallback
|
||||
|
||||
Example:
|
||||
from google.genai import types
|
||||
|
||||
llm = ChatGoogle(
|
||||
model='gemini-2.0-flash-exp',
|
||||
config={
|
||||
'tools': [types.Tool(code_execution=types.ToolCodeExecution())]
|
||||
}
|
||||
)
|
||||
"""
|
||||
|
||||
# Model configuration
|
||||
model: VerifiedGeminiModels | str
|
||||
temperature: float | None = 0.2
|
||||
top_p: float | None = None
|
||||
seed: int | None = None
|
||||
thinking_budget: int | None = None
|
||||
max_output_tokens: int | None = 4096
|
||||
config: types.GenerateContentConfigDict | None = None
|
||||
include_system_in_user: bool = False
|
||||
supports_structured_output: bool = True # New flag
|
||||
|
||||
# Client initialization parameters
|
||||
api_key: str | None = None
|
||||
vertexai: bool | None = None
|
||||
credentials: Credentials | None = None
|
||||
project: str | None = None
|
||||
location: str | None = None
|
||||
http_options: types.HttpOptions | types.HttpOptionsDict | None = None
|
||||
|
||||
# Static
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'google'
|
||||
|
||||
@property
|
||||
def logger(self) -> logging.Logger:
|
||||
"""Get logger for this chat instance"""
|
||||
return logging.getLogger(f'browser_use.llm.google.{self.model}')
|
||||
|
||||
def _get_client_params(self) -> dict[str, Any]:
|
||||
"""Prepare client parameters dictionary."""
|
||||
# Define base client params
|
||||
base_params = {
|
||||
'api_key': self.api_key,
|
||||
'vertexai': self.vertexai,
|
||||
'credentials': self.credentials,
|
||||
'project': self.project,
|
||||
'location': self.location,
|
||||
'http_options': self.http_options,
|
||||
}
|
||||
|
||||
# Create client_params dict with non-None values
|
||||
client_params = {k: v for k, v in base_params.items() if v is not None}
|
||||
|
||||
return client_params
|
||||
|
||||
def get_client(self) -> genai.Client:
|
||||
"""
|
||||
Returns a genai.Client instance.
|
||||
|
||||
Returns:
|
||||
genai.Client: An instance of the Google genai client.
|
||||
"""
|
||||
client_params = self._get_client_params()
|
||||
return genai.Client(**client_params)
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return str(self.model)
|
||||
|
||||
def _get_usage(self, response: types.GenerateContentResponse) -> ChatInvokeUsage | None:
|
||||
usage: ChatInvokeUsage | None = None
|
||||
|
||||
if response.usage_metadata is not None:
|
||||
image_tokens = 0
|
||||
if response.usage_metadata.prompt_tokens_details is not None:
|
||||
image_tokens = sum(
|
||||
detail.token_count or 0
|
||||
for detail in response.usage_metadata.prompt_tokens_details
|
||||
if detail.modality == MediaModality.IMAGE
|
||||
)
|
||||
|
||||
usage = ChatInvokeUsage(
|
||||
prompt_tokens=response.usage_metadata.prompt_token_count or 0,
|
||||
completion_tokens=(response.usage_metadata.candidates_token_count or 0)
|
||||
+ (response.usage_metadata.thoughts_token_count or 0),
|
||||
total_tokens=response.usage_metadata.total_token_count or 0,
|
||||
prompt_cached_tokens=response.usage_metadata.cached_content_token_count,
|
||||
prompt_cache_creation_tokens=None,
|
||||
prompt_image_tokens=image_tokens,
|
||||
)
|
||||
|
||||
return usage
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
"""
|
||||
Invoke the model with the given messages.
|
||||
|
||||
Args:
|
||||
messages: List of chat messages
|
||||
output_format: Optional Pydantic model class for structured output
|
||||
|
||||
Returns:
|
||||
Either a string response or an instance of output_format
|
||||
"""
|
||||
|
||||
# Serialize messages to Google format with the include_system_in_user flag
|
||||
contents, system_instruction = GoogleMessageSerializer.serialize_messages(
|
||||
messages, include_system_in_user=self.include_system_in_user
|
||||
)
|
||||
|
||||
# Build config dictionary starting with user-provided config
|
||||
config: types.GenerateContentConfigDict = {}
|
||||
if self.config:
|
||||
config = self.config.copy()
|
||||
|
||||
# Apply model-specific configuration (these can override config)
|
||||
if self.temperature is not None:
|
||||
config['temperature'] = self.temperature
|
||||
|
||||
# Add system instruction if present
|
||||
if system_instruction:
|
||||
config['system_instruction'] = system_instruction
|
||||
|
||||
if self.top_p is not None:
|
||||
config['top_p'] = self.top_p
|
||||
|
||||
if self.seed is not None:
|
||||
config['seed'] = self.seed
|
||||
|
||||
if self.thinking_budget is None and 'gemini-2.5-flash' in self.model:
|
||||
self.thinking_budget = 0
|
||||
|
||||
if self.thinking_budget is not None:
|
||||
thinking_config_dict: types.ThinkingConfigDict = {'thinking_budget': self.thinking_budget}
|
||||
config['thinking_config'] = thinking_config_dict
|
||||
|
||||
if self.max_output_tokens is not None:
|
||||
config['max_output_tokens'] = self.max_output_tokens
|
||||
|
||||
async def _make_api_call():
|
||||
start_time = time.time()
|
||||
self.logger.debug(f'🚀 Starting API call to {self.model}')
|
||||
|
||||
try:
|
||||
if output_format is None:
|
||||
# Return string response
|
||||
self.logger.debug('📄 Requesting text response')
|
||||
|
||||
response = await self.get_client().aio.models.generate_content(
|
||||
model=self.model,
|
||||
contents=contents, # type: ignore
|
||||
config=config,
|
||||
)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
self.logger.debug(f'✅ Got text response in {elapsed:.2f}s')
|
||||
|
||||
# Handle case where response.text might be None
|
||||
text = response.text or ''
|
||||
if not text:
|
||||
self.logger.warning('⚠️ Empty text response received')
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
return ChatInvokeCompletion(
|
||||
completion=text,
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
else:
|
||||
# Handle structured output
|
||||
if self.supports_structured_output:
|
||||
# Use native JSON mode
|
||||
self.logger.debug(f'🔧 Requesting structured output for {output_format.__name__}')
|
||||
config['response_mime_type'] = 'application/json'
|
||||
# Convert Pydantic model to Gemini-compatible schema
|
||||
optimized_schema = SchemaOptimizer.create_optimized_json_schema(output_format)
|
||||
|
||||
gemini_schema = self._fix_gemini_schema(optimized_schema)
|
||||
config['response_schema'] = gemini_schema
|
||||
|
||||
response = await self.get_client().aio.models.generate_content(
|
||||
model=self.model,
|
||||
contents=contents,
|
||||
config=config,
|
||||
)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
self.logger.debug(f'✅ Got structured response in {elapsed:.2f}s')
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
# Handle case where response.parsed might be None
|
||||
if response.parsed is None:
|
||||
self.logger.debug('📝 Parsing JSON from text response')
|
||||
# When using response_schema, Gemini returns JSON as text
|
||||
if response.text:
|
||||
try:
|
||||
# Handle JSON wrapped in markdown code blocks (common Gemini behavior)
|
||||
text = response.text.strip()
|
||||
if text.startswith('```json') and text.endswith('```'):
|
||||
text = text[7:-3].strip()
|
||||
self.logger.debug('🔧 Stripped ```json``` wrapper from response')
|
||||
elif text.startswith('```') and text.endswith('```'):
|
||||
text = text[3:-3].strip()
|
||||
self.logger.debug('🔧 Stripped ``` wrapper from response')
|
||||
|
||||
# Parse the JSON text and validate with the Pydantic model
|
||||
parsed_data = json.loads(text)
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(parsed_data),
|
||||
usage=usage,
|
||||
)
|
||||
except (json.JSONDecodeError, ValueError) as e:
|
||||
self.logger.error(f'❌ Failed to parse JSON response: {str(e)}')
|
||||
self.logger.debug(f'Raw response text: {response.text[:200]}...')
|
||||
raise ModelProviderError(
|
||||
message=f'Failed to parse or validate response {response}: {str(e)}',
|
||||
status_code=500,
|
||||
model=self.model,
|
||||
) from e
|
||||
else:
|
||||
self.logger.error('❌ No response text received')
|
||||
raise ModelProviderError(
|
||||
message=f'No response from model {response}',
|
||||
status_code=500,
|
||||
model=self.model,
|
||||
)
|
||||
|
||||
# Ensure we return the correct type
|
||||
if isinstance(response.parsed, output_format):
|
||||
return ChatInvokeCompletion(
|
||||
completion=response.parsed,
|
||||
usage=usage,
|
||||
)
|
||||
else:
|
||||
# If it's not the expected type, try to validate it
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(response.parsed),
|
||||
usage=usage,
|
||||
)
|
||||
else:
|
||||
# Fallback: Request JSON in the prompt for models without native JSON mode
|
||||
self.logger.debug(f'🔄 Using fallback JSON mode for {output_format.__name__}')
|
||||
# Create a copy of messages to modify
|
||||
modified_messages = [m.model_copy(deep=True) for m in messages]
|
||||
|
||||
# Add JSON instruction to the last message
|
||||
if modified_messages and isinstance(modified_messages[-1].content, str):
|
||||
json_instruction = f'\n\nPlease respond with a valid JSON object that matches this schema: {SchemaOptimizer.create_optimized_json_schema(output_format)}'
|
||||
modified_messages[-1].content += json_instruction
|
||||
|
||||
# Re-serialize with modified messages
|
||||
fallback_contents, fallback_system = GoogleMessageSerializer.serialize_messages(
|
||||
modified_messages, include_system_in_user=self.include_system_in_user
|
||||
)
|
||||
|
||||
# Update config with fallback system instruction if present
|
||||
fallback_config = config.copy()
|
||||
if fallback_system:
|
||||
fallback_config['system_instruction'] = fallback_system
|
||||
|
||||
response = await self.get_client().aio.models.generate_content(
|
||||
model=self.model,
|
||||
contents=fallback_contents, # type: ignore
|
||||
config=fallback_config,
|
||||
)
|
||||
|
||||
elapsed = time.time() - start_time
|
||||
self.logger.debug(f'✅ Got fallback response in {elapsed:.2f}s')
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
# Try to extract JSON from the text response
|
||||
if response.text:
|
||||
try:
|
||||
# Try to find JSON in the response
|
||||
text = response.text.strip()
|
||||
|
||||
# Common patterns: JSON wrapped in markdown code blocks
|
||||
if text.startswith('```json') and text.endswith('```'):
|
||||
text = text[7:-3].strip()
|
||||
elif text.startswith('```') and text.endswith('```'):
|
||||
text = text[3:-3].strip()
|
||||
|
||||
# Parse and validate
|
||||
parsed_data = json.loads(text)
|
||||
return ChatInvokeCompletion(
|
||||
completion=output_format.model_validate(parsed_data),
|
||||
usage=usage,
|
||||
)
|
||||
except (json.JSONDecodeError, ValueError) as e:
|
||||
self.logger.error(f'❌ Failed to parse fallback JSON: {str(e)}')
|
||||
self.logger.debug(f'Raw response text: {response.text[:200]}...')
|
||||
raise ModelProviderError(
|
||||
message=f'Model does not support JSON mode and failed to parse JSON from text response: {str(e)}',
|
||||
status_code=500,
|
||||
model=self.model,
|
||||
) from e
|
||||
else:
|
||||
self.logger.error('❌ No response text in fallback mode')
|
||||
raise ModelProviderError(
|
||||
message='No response from model',
|
||||
status_code=500,
|
||||
model=self.model,
|
||||
)
|
||||
except Exception as e:
|
||||
elapsed = time.time() - start_time
|
||||
self.logger.error(f'💥 API call failed after {elapsed:.2f}s: {type(e).__name__}: {e}')
|
||||
# Re-raise the exception
|
||||
raise
|
||||
|
||||
try:
|
||||
# Let Google client handle retries internally with proper connection management
|
||||
self.logger.debug(f'🔄 Making API call to {self.model} (using built-in retry)')
|
||||
return await _make_api_call()
|
||||
|
||||
except Exception as e:
|
||||
# Handle specific Google API errors with enhanced diagnostics
|
||||
error_message = str(e)
|
||||
status_code: int | None = None
|
||||
|
||||
# Enhanced timeout error handling
|
||||
if 'timeout' in error_message.lower() or 'cancelled' in error_message.lower():
|
||||
if isinstance(e, asyncio.CancelledError) or 'CancelledError' in str(type(e)):
|
||||
enhanced_message = 'Gemini API request was cancelled (likely timeout). '
|
||||
enhanced_message += 'This suggests the API is taking too long to respond. '
|
||||
enhanced_message += (
|
||||
'Consider: 1) Reducing input size, 2) Using a different model, 3) Checking network connectivity.'
|
||||
)
|
||||
error_message = enhanced_message
|
||||
status_code = 504 # Gateway timeout
|
||||
self.logger.error(f'🕐 Timeout diagnosis: Model: {self.model}')
|
||||
else:
|
||||
status_code = 408 # Request timeout
|
||||
# Check if this is a rate limit error
|
||||
elif any(
|
||||
indicator in error_message.lower()
|
||||
for indicator in ['rate limit', 'resource exhausted', 'quota exceeded', 'too many requests', '429']
|
||||
):
|
||||
status_code = 429
|
||||
elif any(
|
||||
indicator in error_message.lower()
|
||||
for indicator in ['service unavailable', 'internal server error', 'bad gateway', '503', '502', '500']
|
||||
):
|
||||
status_code = 503
|
||||
|
||||
# Try to extract status code if available
|
||||
if hasattr(e, 'response'):
|
||||
response_obj = getattr(e, 'response', None)
|
||||
if response_obj and hasattr(response_obj, 'status_code'):
|
||||
status_code = getattr(response_obj, 'status_code', None)
|
||||
|
||||
raise ModelProviderError(
|
||||
message=error_message,
|
||||
status_code=status_code or 502, # Use default if None
|
||||
model=self.name,
|
||||
) from e
|
||||
|
||||
def _fix_gemini_schema(self, schema: dict[str, Any]) -> dict[str, Any]:
|
||||
"""
|
||||
Convert a Pydantic model to a Gemini-compatible schema.
|
||||
|
||||
This function removes unsupported properties like 'additionalProperties' and resolves
|
||||
$ref references that Gemini doesn't support.
|
||||
"""
|
||||
|
||||
# Handle $defs and $ref resolution
|
||||
if '$defs' in schema:
|
||||
defs = schema.pop('$defs')
|
||||
|
||||
def resolve_refs(obj: Any) -> Any:
|
||||
if isinstance(obj, dict):
|
||||
if '$ref' in obj:
|
||||
ref = obj.pop('$ref')
|
||||
ref_name = ref.split('/')[-1]
|
||||
if ref_name in defs:
|
||||
# Replace the reference with the actual definition
|
||||
resolved = defs[ref_name].copy()
|
||||
# Merge any additional properties from the reference
|
||||
for key, value in obj.items():
|
||||
if key != '$ref':
|
||||
resolved[key] = value
|
||||
return resolve_refs(resolved)
|
||||
return obj
|
||||
else:
|
||||
# Recursively process all dictionary values
|
||||
return {k: resolve_refs(v) for k, v in obj.items()}
|
||||
elif isinstance(obj, list):
|
||||
return [resolve_refs(item) for item in obj]
|
||||
return obj
|
||||
|
||||
schema = resolve_refs(schema)
|
||||
|
||||
# Remove unsupported properties
|
||||
def clean_schema(obj: Any) -> Any:
|
||||
if isinstance(obj, dict):
|
||||
# Remove unsupported properties
|
||||
cleaned = {}
|
||||
for key, value in obj.items():
|
||||
if key not in ['additionalProperties', 'title', 'default']:
|
||||
cleaned_value = clean_schema(value)
|
||||
# Handle empty object properties - Gemini doesn't allow empty OBJECT types
|
||||
if (
|
||||
key == 'properties'
|
||||
and isinstance(cleaned_value, dict)
|
||||
and len(cleaned_value) == 0
|
||||
and isinstance(obj.get('type', ''), str)
|
||||
and obj.get('type', '').upper() == 'OBJECT'
|
||||
):
|
||||
# Convert empty object to have at least one property
|
||||
cleaned['properties'] = {'_placeholder': {'type': 'string'}}
|
||||
else:
|
||||
cleaned[key] = cleaned_value
|
||||
|
||||
# If this is an object type with empty properties, add a placeholder
|
||||
if (
|
||||
isinstance(cleaned.get('type', ''), str)
|
||||
and cleaned.get('type', '').upper() == 'OBJECT'
|
||||
and 'properties' in cleaned
|
||||
and isinstance(cleaned['properties'], dict)
|
||||
and len(cleaned['properties']) == 0
|
||||
):
|
||||
cleaned['properties'] = {'_placeholder': {'type': 'string'}}
|
||||
|
||||
# Also remove 'title' from the required list if it exists
|
||||
if 'required' in cleaned and isinstance(cleaned.get('required'), list):
|
||||
cleaned['required'] = [p for p in cleaned['required'] if p != 'title']
|
||||
|
||||
return cleaned
|
||||
elif isinstance(obj, list):
|
||||
return [clean_schema(item) for item in obj]
|
||||
return obj
|
||||
|
||||
return clean_schema(schema)
|
||||
@@ -0,0 +1,120 @@
|
||||
import base64
|
||||
|
||||
from google.genai.types import Content, ContentListUnion, Part
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
SystemMessage,
|
||||
UserMessage,
|
||||
)
|
||||
|
||||
|
||||
class GoogleMessageSerializer:
|
||||
"""Serializer for converting messages to Google Gemini format."""
|
||||
|
||||
@staticmethod
|
||||
def serialize_messages(
|
||||
messages: list[BaseMessage], include_system_in_user: bool = False
|
||||
) -> tuple[ContentListUnion, str | None]:
|
||||
"""
|
||||
Convert a list of BaseMessages to Google format, extracting system message.
|
||||
|
||||
Google handles system instructions separately from the conversation, so we need to:
|
||||
1. Extract any system messages and return them separately as a string (or include in first user message if flag is set)
|
||||
2. Convert the remaining messages to Content objects
|
||||
|
||||
Args:
|
||||
messages: List of messages to convert
|
||||
include_system_in_user: If True, system/developer messages are prepended to the first user message
|
||||
|
||||
Returns:
|
||||
A tuple of (formatted_messages, system_message) where:
|
||||
- formatted_messages: List of Content objects for the conversation
|
||||
- system_message: System instruction string or None
|
||||
"""
|
||||
|
||||
messages = [m.model_copy(deep=True) for m in messages]
|
||||
|
||||
formatted_messages: ContentListUnion = []
|
||||
system_message: str | None = None
|
||||
system_parts: list[str] = []
|
||||
|
||||
for i, message in enumerate(messages):
|
||||
role = message.role if hasattr(message, 'role') else None
|
||||
|
||||
# Handle system/developer messages
|
||||
if isinstance(message, SystemMessage) or role in ['system', 'developer']:
|
||||
# Extract system message content as string
|
||||
if isinstance(message.content, str):
|
||||
if include_system_in_user:
|
||||
system_parts.append(message.content)
|
||||
else:
|
||||
system_message = message.content
|
||||
elif message.content is not None:
|
||||
# Handle Iterable of content parts
|
||||
parts = []
|
||||
for part in message.content:
|
||||
if part.type == 'text':
|
||||
parts.append(part.text)
|
||||
combined_text = '\n'.join(parts)
|
||||
if include_system_in_user:
|
||||
system_parts.append(combined_text)
|
||||
else:
|
||||
system_message = combined_text
|
||||
continue
|
||||
|
||||
# Determine the role for non-system messages
|
||||
if isinstance(message, UserMessage):
|
||||
role = 'user'
|
||||
elif isinstance(message, AssistantMessage):
|
||||
role = 'model'
|
||||
else:
|
||||
# Default to user for any unknown message types
|
||||
role = 'user'
|
||||
|
||||
# Initialize message parts
|
||||
message_parts: list[Part] = []
|
||||
|
||||
# If this is the first user message and we have system parts, prepend them
|
||||
if include_system_in_user and system_parts and role == 'user' and not formatted_messages:
|
||||
system_text = '\n\n'.join(system_parts)
|
||||
if isinstance(message.content, str):
|
||||
message_parts.append(Part.from_text(text=f'{system_text}\n\n{message.content}'))
|
||||
else:
|
||||
# Add system text as the first part
|
||||
message_parts.append(Part.from_text(text=system_text))
|
||||
system_parts = [] # Clear after using
|
||||
else:
|
||||
# Extract content and create parts normally
|
||||
if isinstance(message.content, str):
|
||||
# Regular text content
|
||||
message_parts = [Part.from_text(text=message.content)]
|
||||
elif message.content is not None:
|
||||
# Handle Iterable of content parts
|
||||
for part in message.content:
|
||||
if part.type == 'text':
|
||||
message_parts.append(Part.from_text(text=part.text))
|
||||
elif part.type == 'refusal':
|
||||
message_parts.append(Part.from_text(text=f'[Refusal] {part.refusal}'))
|
||||
elif part.type == 'image_url':
|
||||
# Handle images
|
||||
url = part.image_url.url
|
||||
|
||||
# Format: data:image/png;base64,<data>
|
||||
header, data = url.split(',', 1)
|
||||
# Decode base64 to bytes
|
||||
image_bytes = base64.b64decode(data)
|
||||
|
||||
# Add image part
|
||||
image_part = Part.from_bytes(data=image_bytes, mime_type='image/png')
|
||||
|
||||
message_parts.append(image_part)
|
||||
|
||||
# Create the Content object
|
||||
if message_parts:
|
||||
final_message = Content(role=role, parts=message_parts)
|
||||
# for some reason, the type checker is not able to infer the type of formatted_messages
|
||||
formatted_messages.append(final_message) # type: ignore
|
||||
|
||||
return formatted_messages, system_message
|
||||
@@ -0,0 +1,229 @@
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Literal, TypeVar, overload
|
||||
|
||||
from groq import (
|
||||
APIError,
|
||||
APIResponseValidationError,
|
||||
APIStatusError,
|
||||
AsyncGroq,
|
||||
NotGiven,
|
||||
RateLimitError,
|
||||
Timeout,
|
||||
)
|
||||
from groq.types.chat import ChatCompletion, ChatCompletionToolChoiceOptionParam, ChatCompletionToolParam
|
||||
from groq.types.chat.completion_create_params import (
|
||||
ResponseFormatResponseFormatJsonSchema,
|
||||
ResponseFormatResponseFormatJsonSchemaJsonSchema,
|
||||
)
|
||||
from httpx import URL
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.base import BaseChatModel, ChatInvokeCompletion
|
||||
from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError
|
||||
from browser_use.llm.groq.parser import try_parse_groq_failed_generation
|
||||
from browser_use.llm.groq.serializer import GroqMessageSerializer
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.schema import SchemaOptimizer
|
||||
from browser_use.llm.views import ChatInvokeUsage
|
||||
|
||||
GroqVerifiedModels = Literal[
|
||||
'meta-llama/llama-4-maverick-17b-128e-instruct',
|
||||
'meta-llama/llama-4-scout-17b-16e-instruct',
|
||||
'qwen/qwen3-32b',
|
||||
'moonshotai/kimi-k2-instruct',
|
||||
'openai/gpt-oss-20b',
|
||||
'openai/gpt-oss-120b',
|
||||
]
|
||||
|
||||
JsonSchemaModels = [
|
||||
'meta-llama/llama-4-maverick-17b-128e-instruct',
|
||||
'meta-llama/llama-4-scout-17b-16e-instruct',
|
||||
'openai/gpt-oss-20b',
|
||||
'openai/gpt-oss-120b',
|
||||
]
|
||||
|
||||
ToolCallingModels = [
|
||||
'moonshotai/kimi-k2-instruct',
|
||||
]
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatGroq(BaseChatModel):
|
||||
"""
|
||||
A wrapper around AsyncGroq that implements the BaseLLM protocol.
|
||||
"""
|
||||
|
||||
# Model configuration
|
||||
model: GroqVerifiedModels | str
|
||||
|
||||
# Model params
|
||||
temperature: float | None = None
|
||||
service_tier: Literal['auto', 'on_demand', 'flex'] | None = None
|
||||
top_p: float | None = None
|
||||
seed: int | None = None
|
||||
|
||||
# Client initialization parameters
|
||||
api_key: str | None = None
|
||||
base_url: str | URL | None = None
|
||||
timeout: float | Timeout | NotGiven | None = None
|
||||
max_retries: int = 10 # Increase default retries for automation reliability
|
||||
|
||||
def get_client(self) -> AsyncGroq:
|
||||
return AsyncGroq(api_key=self.api_key, base_url=self.base_url, timeout=self.timeout, max_retries=self.max_retries)
|
||||
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'groq'
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return str(self.model)
|
||||
|
||||
def _get_usage(self, response: ChatCompletion) -> ChatInvokeUsage | None:
|
||||
usage = (
|
||||
ChatInvokeUsage(
|
||||
prompt_tokens=response.usage.prompt_tokens,
|
||||
completion_tokens=response.usage.completion_tokens,
|
||||
total_tokens=response.usage.total_tokens,
|
||||
prompt_cached_tokens=None, # Groq doesn't support cached tokens
|
||||
prompt_cache_creation_tokens=None,
|
||||
prompt_image_tokens=None,
|
||||
)
|
||||
if response.usage is not None
|
||||
else None
|
||||
)
|
||||
return usage
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
groq_messages = GroqMessageSerializer.serialize_messages(messages)
|
||||
|
||||
try:
|
||||
if output_format is None:
|
||||
return await self._invoke_regular_completion(groq_messages)
|
||||
else:
|
||||
return await self._invoke_structured_output(groq_messages, output_format)
|
||||
|
||||
except RateLimitError as e:
|
||||
raise ModelRateLimitError(message=e.response.text, status_code=e.response.status_code, model=self.name) from e
|
||||
|
||||
except APIResponseValidationError as e:
|
||||
raise ModelProviderError(message=e.response.text, status_code=e.response.status_code, model=self.name) from e
|
||||
|
||||
except APIStatusError as e:
|
||||
if output_format is None:
|
||||
raise ModelProviderError(message=e.response.text, status_code=e.response.status_code, model=self.name) from e
|
||||
else:
|
||||
try:
|
||||
logger.debug(f'Groq failed generation: {e.response.text}; fallback to manual parsing')
|
||||
|
||||
parsed_response = try_parse_groq_failed_generation(e, output_format)
|
||||
|
||||
logger.debug('Manual error parsing successful ✅')
|
||||
|
||||
return ChatInvokeCompletion(
|
||||
completion=parsed_response,
|
||||
usage=None, # because this is a hacky way to get the outputs
|
||||
# TODO: @groq needs to fix their parsers and validators
|
||||
)
|
||||
except Exception as _:
|
||||
raise ModelProviderError(message=str(e), status_code=e.response.status_code, model=self.name) from e
|
||||
|
||||
except APIError as e:
|
||||
raise ModelProviderError(message=e.message, model=self.name) from e
|
||||
except Exception as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
|
||||
async def _invoke_regular_completion(self, groq_messages) -> ChatInvokeCompletion[str]:
|
||||
"""Handle regular completion without structured output."""
|
||||
chat_completion = await self.get_client().chat.completions.create(
|
||||
messages=groq_messages,
|
||||
model=self.model,
|
||||
service_tier=self.service_tier,
|
||||
temperature=self.temperature,
|
||||
top_p=self.top_p,
|
||||
seed=self.seed,
|
||||
)
|
||||
usage = self._get_usage(chat_completion)
|
||||
return ChatInvokeCompletion(
|
||||
completion=chat_completion.choices[0].message.content or '',
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
async def _invoke_structured_output(self, groq_messages, output_format: type[T]) -> ChatInvokeCompletion[T]:
|
||||
"""Handle structured output using either tool calling or JSON schema."""
|
||||
schema = SchemaOptimizer.create_optimized_json_schema(output_format)
|
||||
|
||||
if self.model in ToolCallingModels:
|
||||
response = await self._invoke_with_tool_calling(groq_messages, output_format, schema)
|
||||
else:
|
||||
response = await self._invoke_with_json_schema(groq_messages, output_format, schema)
|
||||
|
||||
if not response.choices[0].message.content:
|
||||
raise ModelProviderError(
|
||||
message='No content in response',
|
||||
status_code=500,
|
||||
model=self.name,
|
||||
)
|
||||
|
||||
parsed_response = output_format.model_validate_json(response.choices[0].message.content)
|
||||
usage = self._get_usage(response)
|
||||
|
||||
return ChatInvokeCompletion(
|
||||
completion=parsed_response,
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
async def _invoke_with_tool_calling(self, groq_messages, output_format: type[T], schema) -> ChatCompletion:
|
||||
"""Handle structured output using tool calling."""
|
||||
tool = ChatCompletionToolParam(
|
||||
function={
|
||||
'name': output_format.__name__,
|
||||
'description': f'Extract information in the format of {output_format.__name__}',
|
||||
'parameters': schema,
|
||||
},
|
||||
type='function',
|
||||
)
|
||||
tool_choice: ChatCompletionToolChoiceOptionParam = 'required'
|
||||
|
||||
return await self.get_client().chat.completions.create(
|
||||
model=self.model,
|
||||
messages=groq_messages,
|
||||
temperature=self.temperature,
|
||||
top_p=self.top_p,
|
||||
seed=self.seed,
|
||||
tools=[tool],
|
||||
tool_choice=tool_choice,
|
||||
service_tier=self.service_tier,
|
||||
)
|
||||
|
||||
async def _invoke_with_json_schema(self, groq_messages, output_format: type[T], schema) -> ChatCompletion:
|
||||
"""Handle structured output using JSON schema."""
|
||||
return await self.get_client().chat.completions.create(
|
||||
model=self.model,
|
||||
messages=groq_messages,
|
||||
temperature=self.temperature,
|
||||
top_p=self.top_p,
|
||||
seed=self.seed,
|
||||
response_format=ResponseFormatResponseFormatJsonSchema(
|
||||
json_schema=ResponseFormatResponseFormatJsonSchemaJsonSchema(
|
||||
name=output_format.__name__,
|
||||
description='Model output schema',
|
||||
schema=schema,
|
||||
),
|
||||
type='json_schema',
|
||||
),
|
||||
service_tier=self.service_tier,
|
||||
)
|
||||
@@ -0,0 +1,158 @@
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
from typing import TypeVar
|
||||
|
||||
from groq import APIStatusError
|
||||
from pydantic import BaseModel
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
class ParseFailedGenerationError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def try_parse_groq_failed_generation(
|
||||
error: APIStatusError,
|
||||
output_format: type[T],
|
||||
) -> T:
|
||||
"""Extract JSON from model output, handling both plain JSON and code-block-wrapped JSON."""
|
||||
try:
|
||||
content = error.body['error']['failed_generation'] # type: ignore
|
||||
|
||||
# If content is wrapped in code blocks, extract just the JSON part
|
||||
if '```' in content:
|
||||
# Find the JSON content between code blocks
|
||||
content = content.split('```')[1]
|
||||
# Remove language identifier if present (e.g., 'json\n')
|
||||
if '\n' in content:
|
||||
content = content.split('\n', 1)[1]
|
||||
|
||||
# remove html-like tags before the first { and after the last }
|
||||
# This handles cases like <|header_start|>assistant<|header_end|> and <function=AgentOutput>
|
||||
# Only remove content before { if content doesn't already start with {
|
||||
if not content.strip().startswith('{'):
|
||||
content = re.sub(r'^.*?(?=\{)', '', content, flags=re.DOTALL)
|
||||
|
||||
# Remove common HTML-like tags and patterns at the end, but be more conservative
|
||||
# Look for patterns like </function>, <|header_start|>, etc. after the JSON
|
||||
content = re.sub(r'\}(\s*<[^>]*>.*?$)', '}', content, flags=re.DOTALL)
|
||||
content = re.sub(r'\}(\s*<\|[^|]*\|>.*?$)', '}', content, flags=re.DOTALL)
|
||||
|
||||
# Handle extra characters after the JSON, including stray braces
|
||||
# Find the position of the last } that would close the main JSON object
|
||||
content = content.strip()
|
||||
|
||||
if content.endswith('}'):
|
||||
# Try to parse and see if we get valid JSON
|
||||
try:
|
||||
json.loads(content)
|
||||
except json.JSONDecodeError:
|
||||
# If parsing fails, try to find the correct end of the JSON
|
||||
# by counting braces and removing anything after the balanced JSON
|
||||
brace_count = 0
|
||||
last_valid_pos = -1
|
||||
for i, char in enumerate(content):
|
||||
if char == '{':
|
||||
brace_count += 1
|
||||
elif char == '}':
|
||||
brace_count -= 1
|
||||
if brace_count == 0:
|
||||
last_valid_pos = i + 1
|
||||
break
|
||||
|
||||
if last_valid_pos > 0:
|
||||
content = content[:last_valid_pos]
|
||||
|
||||
# Fix control characters in JSON strings before parsing
|
||||
# This handles cases where literal control characters appear in JSON values
|
||||
content = _fix_control_characters_in_json(content)
|
||||
|
||||
# Parse the cleaned content
|
||||
result_dict = json.loads(content)
|
||||
|
||||
# some models occasionally respond with a list containing one dict: https://github.com/browser-use/browser-use/issues/1458
|
||||
if isinstance(result_dict, list) and len(result_dict) == 1 and isinstance(result_dict[0], dict):
|
||||
result_dict = result_dict[0]
|
||||
|
||||
logger.debug(f'Successfully parsed model output: {result_dict}')
|
||||
return output_format.model_validate(result_dict)
|
||||
|
||||
except KeyError as e:
|
||||
raise ParseFailedGenerationError(e) from e
|
||||
|
||||
except json.JSONDecodeError as e:
|
||||
logger.warning(f'Failed to parse model output: {content} {str(e)}')
|
||||
raise ValueError(f'Could not parse response. {str(e)}')
|
||||
|
||||
except Exception as e:
|
||||
raise ParseFailedGenerationError(error.response.text) from e
|
||||
|
||||
|
||||
def _fix_control_characters_in_json(content: str) -> str:
|
||||
"""Fix control characters in JSON string values to make them valid JSON."""
|
||||
try:
|
||||
# First try to parse as-is to see if it's already valid
|
||||
json.loads(content)
|
||||
return content
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
# More sophisticated approach: only escape control characters inside string values
|
||||
# while preserving JSON structure formatting
|
||||
|
||||
result = []
|
||||
i = 0
|
||||
in_string = False
|
||||
escaped = False
|
||||
|
||||
while i < len(content):
|
||||
char = content[i]
|
||||
|
||||
if not in_string:
|
||||
# Outside of string - check if we're entering a string
|
||||
if char == '"':
|
||||
in_string = True
|
||||
result.append(char)
|
||||
else:
|
||||
# Inside string - handle escaping and control characters
|
||||
if escaped:
|
||||
# Previous character was backslash, so this character is escaped
|
||||
result.append(char)
|
||||
escaped = False
|
||||
elif char == '\\':
|
||||
# This is an escape character
|
||||
result.append(char)
|
||||
escaped = True
|
||||
elif char == '"':
|
||||
# End of string
|
||||
result.append(char)
|
||||
in_string = False
|
||||
elif char == '\n':
|
||||
# Literal newline inside string - escape it
|
||||
result.append('\\n')
|
||||
elif char == '\r':
|
||||
# Literal carriage return inside string - escape it
|
||||
result.append('\\r')
|
||||
elif char == '\t':
|
||||
# Literal tab inside string - escape it
|
||||
result.append('\\t')
|
||||
elif char == '\b':
|
||||
# Literal backspace inside string - escape it
|
||||
result.append('\\b')
|
||||
elif char == '\f':
|
||||
# Literal form feed inside string - escape it
|
||||
result.append('\\f')
|
||||
elif ord(char) < 32:
|
||||
# Other control characters inside string - convert to unicode escape
|
||||
result.append(f'\\u{ord(char):04x}')
|
||||
else:
|
||||
# Normal character inside string
|
||||
result.append(char)
|
||||
|
||||
i += 1
|
||||
|
||||
return ''.join(result)
|
||||
@@ -0,0 +1,159 @@
|
||||
from typing import overload
|
||||
|
||||
from groq.types.chat import (
|
||||
ChatCompletionAssistantMessageParam,
|
||||
ChatCompletionContentPartImageParam,
|
||||
ChatCompletionContentPartTextParam,
|
||||
ChatCompletionMessageParam,
|
||||
ChatCompletionMessageToolCallParam,
|
||||
ChatCompletionSystemMessageParam,
|
||||
ChatCompletionUserMessageParam,
|
||||
)
|
||||
from groq.types.chat.chat_completion_content_part_image_param import ImageURL
|
||||
from groq.types.chat.chat_completion_message_tool_call_param import Function
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
ContentPartImageParam,
|
||||
ContentPartRefusalParam,
|
||||
ContentPartTextParam,
|
||||
SystemMessage,
|
||||
ToolCall,
|
||||
UserMessage,
|
||||
)
|
||||
|
||||
|
||||
class GroqMessageSerializer:
|
||||
"""Serializer for converting between custom message types and OpenAI message param types."""
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_text(part: ContentPartTextParam) -> ChatCompletionContentPartTextParam:
|
||||
return ChatCompletionContentPartTextParam(text=part.text, type='text')
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_image(part: ContentPartImageParam) -> ChatCompletionContentPartImageParam:
|
||||
return ChatCompletionContentPartImageParam(
|
||||
image_url=ImageURL(url=part.image_url.url, detail=part.image_url.detail),
|
||||
type='image_url',
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _serialize_user_content(
|
||||
content: str | list[ContentPartTextParam | ContentPartImageParam],
|
||||
) -> str | list[ChatCompletionContentPartTextParam | ChatCompletionContentPartImageParam]:
|
||||
"""Serialize content for user messages (text and images allowed)."""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
serialized_parts: list[ChatCompletionContentPartTextParam | ChatCompletionContentPartImageParam] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_parts.append(GroqMessageSerializer._serialize_content_part_text(part))
|
||||
elif part.type == 'image_url':
|
||||
serialized_parts.append(GroqMessageSerializer._serialize_content_part_image(part))
|
||||
return serialized_parts
|
||||
|
||||
@staticmethod
|
||||
def _serialize_system_content(
|
||||
content: str | list[ContentPartTextParam],
|
||||
) -> str:
|
||||
"""Serialize content for system messages (text only)."""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
serialized_parts: list[str] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_parts.append(GroqMessageSerializer._serialize_content_part_text(part)['text'])
|
||||
|
||||
return '\n'.join(serialized_parts)
|
||||
|
||||
@staticmethod
|
||||
def _serialize_assistant_content(
|
||||
content: str | list[ContentPartTextParam | ContentPartRefusalParam] | None,
|
||||
) -> str | None:
|
||||
"""Serialize content for assistant messages (text and refusal allowed)."""
|
||||
if content is None:
|
||||
return None
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
serialized_parts: list[str] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_parts.append(GroqMessageSerializer._serialize_content_part_text(part)['text'])
|
||||
|
||||
return '\n'.join(serialized_parts)
|
||||
|
||||
@staticmethod
|
||||
def _serialize_tool_call(tool_call: ToolCall) -> ChatCompletionMessageToolCallParam:
|
||||
return ChatCompletionMessageToolCallParam(
|
||||
id=tool_call.id,
|
||||
function=Function(name=tool_call.function.name, arguments=tool_call.function.arguments),
|
||||
type='function',
|
||||
)
|
||||
|
||||
# endregion
|
||||
|
||||
# region - Serialize overloads
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: UserMessage) -> ChatCompletionUserMessageParam: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: SystemMessage) -> ChatCompletionSystemMessageParam: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: AssistantMessage) -> ChatCompletionAssistantMessageParam: ...
|
||||
|
||||
@staticmethod
|
||||
def serialize(message: BaseMessage) -> ChatCompletionMessageParam:
|
||||
"""Serialize a custom message to an OpenAI message param."""
|
||||
|
||||
if isinstance(message, UserMessage):
|
||||
user_result: ChatCompletionUserMessageParam = {
|
||||
'role': 'user',
|
||||
'content': GroqMessageSerializer._serialize_user_content(message.content),
|
||||
}
|
||||
if message.name is not None:
|
||||
user_result['name'] = message.name
|
||||
return user_result
|
||||
|
||||
elif isinstance(message, SystemMessage):
|
||||
system_result: ChatCompletionSystemMessageParam = {
|
||||
'role': 'system',
|
||||
'content': GroqMessageSerializer._serialize_system_content(message.content),
|
||||
}
|
||||
if message.name is not None:
|
||||
system_result['name'] = message.name
|
||||
return system_result
|
||||
|
||||
elif isinstance(message, AssistantMessage):
|
||||
# Handle content serialization
|
||||
content = None
|
||||
if message.content is not None:
|
||||
content = GroqMessageSerializer._serialize_assistant_content(message.content)
|
||||
|
||||
assistant_result: ChatCompletionAssistantMessageParam = {'role': 'assistant'}
|
||||
|
||||
# Only add content if it's not None
|
||||
if content is not None:
|
||||
assistant_result['content'] = content
|
||||
|
||||
if message.name is not None:
|
||||
assistant_result['name'] = message.name
|
||||
|
||||
if message.tool_calls:
|
||||
assistant_result['tool_calls'] = [GroqMessageSerializer._serialize_tool_call(tc) for tc in message.tool_calls]
|
||||
|
||||
return assistant_result
|
||||
|
||||
else:
|
||||
raise ValueError(f'Unknown message type: {type(message)}')
|
||||
|
||||
@staticmethod
|
||||
def serialize_messages(messages: list[BaseMessage]) -> list[ChatCompletionMessageParam]:
|
||||
return [GroqMessageSerializer.serialize(m) for m in messages]
|
||||
@@ -0,0 +1,238 @@
|
||||
"""
|
||||
This implementation is based on the OpenAI types, while removing all the parts that are not needed for Browser Use.
|
||||
"""
|
||||
|
||||
# region - Content parts
|
||||
from typing import Literal, Union
|
||||
|
||||
from openai import BaseModel
|
||||
|
||||
|
||||
def _truncate(text: str, max_length: int = 50) -> str:
|
||||
"""Truncate text to max_length characters, adding ellipsis if truncated."""
|
||||
if len(text) <= max_length:
|
||||
return text
|
||||
return text[: max_length - 3] + '...'
|
||||
|
||||
|
||||
def _format_image_url(url: str, max_length: int = 50) -> str:
|
||||
"""Format image URL for display, truncating if necessary."""
|
||||
if url.startswith('data:'):
|
||||
# Base64 image
|
||||
media_type = url.split(';')[0].split(':')[1] if ';' in url else 'image'
|
||||
return f'<base64 {media_type}>'
|
||||
else:
|
||||
# Regular URL
|
||||
return _truncate(url, max_length)
|
||||
|
||||
|
||||
class ContentPartTextParam(BaseModel):
|
||||
text: str
|
||||
type: Literal['text'] = 'text'
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f'Text: {_truncate(self.text)}'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'ContentPartTextParam(text={_truncate(self.text)})'
|
||||
|
||||
|
||||
class ContentPartRefusalParam(BaseModel):
|
||||
refusal: str
|
||||
type: Literal['refusal'] = 'refusal'
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f'Refusal: {_truncate(self.refusal)}'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'ContentPartRefusalParam(refusal={_truncate(repr(self.refusal), 50)})'
|
||||
|
||||
|
||||
SupportedImageMediaType = Literal['image/jpeg', 'image/png', 'image/gif', 'image/webp']
|
||||
|
||||
|
||||
class ImageURL(BaseModel):
|
||||
url: str
|
||||
"""Either a URL of the image or the base64 encoded image data."""
|
||||
detail: Literal['auto', 'low', 'high'] = 'auto'
|
||||
"""Specifies the detail level of the image.
|
||||
|
||||
Learn more in the
|
||||
[Vision guide](https://platform.openai.com/docs/guides/vision#low-or-high-fidelity-image-understanding).
|
||||
"""
|
||||
# needed for Anthropic
|
||||
media_type: SupportedImageMediaType = 'image/png'
|
||||
|
||||
def __str__(self) -> str:
|
||||
url_display = _format_image_url(self.url)
|
||||
return f'🖼️ Image[{self.media_type}, detail={self.detail}]: {url_display}'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
url_repr = _format_image_url(self.url, 30)
|
||||
return f'ImageURL(url={repr(url_repr)}, detail={repr(self.detail)}, media_type={repr(self.media_type)})'
|
||||
|
||||
|
||||
class ContentPartImageParam(BaseModel):
|
||||
image_url: ImageURL
|
||||
type: Literal['image_url'] = 'image_url'
|
||||
|
||||
def __str__(self) -> str:
|
||||
return str(self.image_url)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'ContentPartImageParam(image_url={repr(self.image_url)})'
|
||||
|
||||
|
||||
class Function(BaseModel):
|
||||
arguments: str
|
||||
"""
|
||||
The arguments to call the function with, as generated by the model in JSON
|
||||
format. Note that the model does not always generate valid JSON, and may
|
||||
hallucinate parameters not defined by your function schema. Validate the
|
||||
arguments in your code before calling your function.
|
||||
"""
|
||||
name: str
|
||||
"""The name of the function to call."""
|
||||
|
||||
def __str__(self) -> str:
|
||||
args_preview = _truncate(self.arguments, 80)
|
||||
return f'{self.name}({args_preview})'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
args_repr = _truncate(repr(self.arguments), 50)
|
||||
return f'Function(name={repr(self.name)}, arguments={args_repr})'
|
||||
|
||||
|
||||
class ToolCall(BaseModel):
|
||||
id: str
|
||||
"""The ID of the tool call."""
|
||||
function: Function
|
||||
"""The function that the model called."""
|
||||
type: Literal['function'] = 'function'
|
||||
"""The type of the tool. Currently, only `function` is supported."""
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f'ToolCall[{self.id}]: {self.function}'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'ToolCall(id={repr(self.id)}, function={repr(self.function)})'
|
||||
|
||||
|
||||
# endregion
|
||||
|
||||
|
||||
# region - Message types
|
||||
class _MessageBase(BaseModel):
|
||||
"""Base class for all message types"""
|
||||
|
||||
role: Literal['user', 'system', 'assistant']
|
||||
|
||||
cache: bool = False
|
||||
"""Whether to cache this message. This is only applicable when using Anthropic models.
|
||||
"""
|
||||
|
||||
|
||||
class UserMessage(_MessageBase):
|
||||
role: Literal['user'] = 'user'
|
||||
"""The role of the messages author, in this case `user`."""
|
||||
|
||||
content: str | list[ContentPartTextParam | ContentPartImageParam]
|
||||
"""The contents of the user message."""
|
||||
|
||||
name: str | None = None
|
||||
"""An optional name for the participant.
|
||||
|
||||
Provides the model information to differentiate between participants of the same
|
||||
role.
|
||||
"""
|
||||
|
||||
@property
|
||||
def text(self) -> str:
|
||||
"""
|
||||
Automatically parse the text inside content, whether it's a string or a list of content parts.
|
||||
"""
|
||||
if isinstance(self.content, str):
|
||||
return self.content
|
||||
elif isinstance(self.content, list):
|
||||
return '\n'.join([part.text for part in self.content if part.type == 'text'])
|
||||
else:
|
||||
return ''
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f'UserMessage(content={self.text})'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'UserMessage(content={repr(self.text)})'
|
||||
|
||||
|
||||
class SystemMessage(_MessageBase):
|
||||
role: Literal['system'] = 'system'
|
||||
"""The role of the messages author, in this case `system`."""
|
||||
|
||||
content: str | list[ContentPartTextParam]
|
||||
"""The contents of the system message."""
|
||||
|
||||
name: str | None = None
|
||||
|
||||
@property
|
||||
def text(self) -> str:
|
||||
"""
|
||||
Automatically parse the text inside content, whether it's a string or a list of content parts.
|
||||
"""
|
||||
if isinstance(self.content, str):
|
||||
return self.content
|
||||
elif isinstance(self.content, list):
|
||||
return '\n'.join([part.text for part in self.content if part.type == 'text'])
|
||||
else:
|
||||
return ''
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f'SystemMessage(content={self.text})'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'SystemMessage(content={repr(self.text)})'
|
||||
|
||||
|
||||
class AssistantMessage(_MessageBase):
|
||||
role: Literal['assistant'] = 'assistant'
|
||||
"""The role of the messages author, in this case `assistant`."""
|
||||
|
||||
content: str | list[ContentPartTextParam | ContentPartRefusalParam] | None
|
||||
"""The contents of the assistant message."""
|
||||
|
||||
name: str | None = None
|
||||
|
||||
refusal: str | None = None
|
||||
"""The refusal message by the assistant."""
|
||||
|
||||
tool_calls: list[ToolCall] = []
|
||||
"""The tool calls generated by the model, such as function calls."""
|
||||
|
||||
@property
|
||||
def text(self) -> str:
|
||||
"""
|
||||
Automatically parse the text inside content, whether it's a string or a list of content parts.
|
||||
"""
|
||||
if isinstance(self.content, str):
|
||||
return self.content
|
||||
elif isinstance(self.content, list):
|
||||
text = ''
|
||||
for part in self.content:
|
||||
if part.type == 'text':
|
||||
text += part.text
|
||||
elif part.type == 'refusal':
|
||||
text += f'[Refusal] {part.refusal}'
|
||||
return text
|
||||
else:
|
||||
return ''
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f'AssistantMessage(content={self.text})'
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f'AssistantMessage(content={repr(self.text)})'
|
||||
|
||||
|
||||
BaseMessage = Union[UserMessage, SystemMessage, AssistantMessage]
|
||||
|
||||
# endregion
|
||||
@@ -0,0 +1,171 @@
|
||||
"""
|
||||
Convenient access to LLM models.
|
||||
|
||||
Usage:
|
||||
from browser_use import llm
|
||||
|
||||
# Simple model access
|
||||
model = llm.azure_gpt_4_1_mini
|
||||
model = llm.openai_gpt_4o
|
||||
model = llm.google_gemini_2_5_pro
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from browser_use.llm.azure.chat import ChatAzureOpenAI
|
||||
from browser_use.llm.google.chat import ChatGoogle
|
||||
from browser_use.llm.openai.chat import ChatOpenAI
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
|
||||
# Type stubs for IDE autocomplete
|
||||
openai_gpt_4o: 'BaseChatModel'
|
||||
openai_gpt_4o_mini: 'BaseChatModel'
|
||||
openai_gpt_4_1_mini: 'BaseChatModel'
|
||||
openai_o1: 'BaseChatModel'
|
||||
openai_o1_mini: 'BaseChatModel'
|
||||
openai_o1_pro: 'BaseChatModel'
|
||||
openai_o3: 'BaseChatModel'
|
||||
openai_o3_mini: 'BaseChatModel'
|
||||
openai_o3_pro: 'BaseChatModel'
|
||||
openai_o4_mini: 'BaseChatModel'
|
||||
openai_gpt_5: 'BaseChatModel'
|
||||
openai_gpt_5_mini: 'BaseChatModel'
|
||||
openai_gpt_5_nano: 'BaseChatModel'
|
||||
|
||||
azure_gpt_4o: 'BaseChatModel'
|
||||
azure_gpt_4o_mini: 'BaseChatModel'
|
||||
azure_gpt_4_1_mini: 'BaseChatModel'
|
||||
azure_o1: 'BaseChatModel'
|
||||
azure_o1_mini: 'BaseChatModel'
|
||||
azure_o1_pro: 'BaseChatModel'
|
||||
azure_o3: 'BaseChatModel'
|
||||
azure_o3_mini: 'BaseChatModel'
|
||||
azure_o3_pro: 'BaseChatModel'
|
||||
azure_gpt_5: 'BaseChatModel'
|
||||
azure_gpt_5_mini: 'BaseChatModel'
|
||||
|
||||
google_gemini_2_0_flash: 'BaseChatModel'
|
||||
google_gemini_2_0_pro: 'BaseChatModel'
|
||||
google_gemini_2_5_pro: 'BaseChatModel'
|
||||
google_gemini_2_5_flash: 'BaseChatModel'
|
||||
google_gemini_2_5_flash_lite: 'BaseChatModel'
|
||||
|
||||
|
||||
def get_llm_by_name(model_name: str):
|
||||
"""
|
||||
Factory function to create LLM instances from string names with API keys from environment.
|
||||
|
||||
Args:
|
||||
model_name: String name like 'azure_gpt_4_1_mini', 'openai_gpt_4o', etc.
|
||||
|
||||
Returns:
|
||||
LLM instance with API keys from environment variables
|
||||
|
||||
Raises:
|
||||
ValueError: If model_name is not recognized
|
||||
"""
|
||||
if not model_name:
|
||||
raise ValueError('Model name cannot be empty')
|
||||
|
||||
# Parse model name
|
||||
parts = model_name.split('_', 1)
|
||||
if len(parts) < 2:
|
||||
raise ValueError(f"Invalid model name format: '{model_name}'. Expected format: 'provider_model_name'")
|
||||
|
||||
provider = parts[0]
|
||||
model_part = parts[1]
|
||||
|
||||
# Convert underscores back to dots/dashes for actual model names
|
||||
if 'gpt_4_1_mini' in model_part:
|
||||
model = model_part.replace('gpt_4_1_mini', 'gpt-4.1-mini')
|
||||
elif 'gpt_4o_mini' in model_part:
|
||||
model = model_part.replace('gpt_4o_mini', 'gpt-4o-mini')
|
||||
elif 'gpt_4o' in model_part:
|
||||
model = model_part.replace('gpt_4o', 'gpt-4o')
|
||||
elif 'gemini_2_0' in model_part:
|
||||
model = model_part.replace('gemini_2_0', 'gemini-2.0').replace('_', '-')
|
||||
elif 'gemini_2_5' in model_part:
|
||||
model = model_part.replace('gemini_2_5', 'gemini-2.5').replace('_', '-')
|
||||
else:
|
||||
model = model_part.replace('_', '-')
|
||||
|
||||
# OpenAI Models
|
||||
if provider == 'openai':
|
||||
api_key = os.getenv('OPENAI_API_KEY')
|
||||
return ChatOpenAI(model=model, api_key=api_key)
|
||||
|
||||
# Azure OpenAI Models
|
||||
elif provider == 'azure':
|
||||
api_key = os.getenv('AZURE_OPENAI_KEY') or os.getenv('AZURE_OPENAI_API_KEY')
|
||||
azure_endpoint = os.getenv('AZURE_OPENAI_ENDPOINT')
|
||||
return ChatAzureOpenAI(model=model, api_key=api_key, azure_endpoint=azure_endpoint)
|
||||
|
||||
# Google Models
|
||||
elif provider == 'google':
|
||||
api_key = os.getenv('GOOGLE_API_KEY')
|
||||
return ChatGoogle(model=model, api_key=api_key)
|
||||
|
||||
else:
|
||||
available_providers = ['openai', 'azure', 'google']
|
||||
raise ValueError(f"Unknown provider: '{provider}'. Available providers: {', '.join(available_providers)}")
|
||||
|
||||
|
||||
# Pre-configured model instances (lazy loaded via __getattr__)
|
||||
def __getattr__(name: str) -> 'BaseChatModel':
|
||||
"""Create model instances on demand with API keys from environment."""
|
||||
# Handle chat classes first
|
||||
if name == 'ChatOpenAI':
|
||||
return ChatOpenAI # type: ignore
|
||||
elif name == 'ChatAzureOpenAI':
|
||||
return ChatAzureOpenAI # type: ignore
|
||||
elif name == 'ChatGoogle':
|
||||
return ChatGoogle # type: ignore
|
||||
|
||||
# Handle model instances - these are the main use case
|
||||
try:
|
||||
return get_llm_by_name(name)
|
||||
except ValueError:
|
||||
raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
|
||||
|
||||
|
||||
__all__ = [
|
||||
'ChatOpenAI',
|
||||
'ChatAzureOpenAI',
|
||||
'ChatGoogle',
|
||||
'get_llm_by_name',
|
||||
# OpenAI instances - created on demand
|
||||
'openai_gpt_4o',
|
||||
'openai_gpt_4o_mini',
|
||||
'openai_gpt_4_1_mini',
|
||||
'openai_o1',
|
||||
'openai_o1_mini',
|
||||
'openai_o1_pro',
|
||||
'openai_o3',
|
||||
'openai_o3_mini',
|
||||
'openai_o3_pro',
|
||||
'openai_o4_mini',
|
||||
'openai_gpt_5',
|
||||
'openai_gpt_5_mini',
|
||||
'openai_gpt_5_nano',
|
||||
# Azure instances - created on demand
|
||||
'azure_gpt_4o',
|
||||
'azure_gpt_4o_mini',
|
||||
'azure_gpt_4_1_mini',
|
||||
'azure_o1',
|
||||
'azure_o1_mini',
|
||||
'azure_o1_pro',
|
||||
'azure_o3',
|
||||
'azure_o3_mini',
|
||||
'azure_o3_pro',
|
||||
'azure_gpt_5',
|
||||
'azure_gpt_5_mini',
|
||||
# Google instances - created on demand
|
||||
'google_gemini_2_0_flash',
|
||||
'google_gemini_2_0_pro',
|
||||
'google_gemini_2_5_pro',
|
||||
'google_gemini_2_5_flash',
|
||||
'google_gemini_2_5_flash_lite',
|
||||
]
|
||||
@@ -0,0 +1,97 @@
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, TypeVar, overload
|
||||
|
||||
import httpx
|
||||
from ollama import AsyncClient as OllamaAsyncClient
|
||||
from ollama import Options
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.exceptions import ModelProviderError
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.ollama.serializer import OllamaMessageSerializer
|
||||
from browser_use.llm.views import ChatInvokeCompletion
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatOllama(BaseChatModel):
|
||||
"""
|
||||
A wrapper around Ollama's chat model.
|
||||
"""
|
||||
|
||||
model: str
|
||||
|
||||
# # Model params
|
||||
# TODO (matic): Why is this commented out?
|
||||
# temperature: float | None = None
|
||||
|
||||
# Client initialization parameters
|
||||
host: str | None = None
|
||||
timeout: float | httpx.Timeout | None = None
|
||||
client_params: dict[str, Any] | None = None
|
||||
ollama_options: Mapping[str, Any] | Options | None = None
|
||||
|
||||
# Static
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'ollama'
|
||||
|
||||
def _get_client_params(self) -> dict[str, Any]:
|
||||
"""Prepare client parameters dictionary."""
|
||||
return {
|
||||
'host': self.host,
|
||||
'timeout': self.timeout,
|
||||
'client_params': self.client_params,
|
||||
}
|
||||
|
||||
def get_client(self) -> OllamaAsyncClient:
|
||||
"""
|
||||
Returns an OllamaAsyncClient client.
|
||||
"""
|
||||
return OllamaAsyncClient(host=self.host, timeout=self.timeout, **self.client_params or {})
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return self.model
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
ollama_messages = OllamaMessageSerializer.serialize_messages(messages)
|
||||
|
||||
try:
|
||||
if output_format is None:
|
||||
response = await self.get_client().chat(
|
||||
model=self.model,
|
||||
messages=ollama_messages,
|
||||
options=self.ollama_options,
|
||||
)
|
||||
|
||||
return ChatInvokeCompletion(completion=response.message.content or '', usage=None)
|
||||
else:
|
||||
schema = output_format.model_json_schema()
|
||||
|
||||
response = await self.get_client().chat(
|
||||
model=self.model,
|
||||
messages=ollama_messages,
|
||||
format=schema,
|
||||
options=self.ollama_options,
|
||||
)
|
||||
|
||||
completion = response.message.content or ''
|
||||
if output_format is not None:
|
||||
completion = output_format.model_validate_json(completion)
|
||||
|
||||
return ChatInvokeCompletion(completion=completion, usage=None)
|
||||
|
||||
except Exception as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
@@ -0,0 +1,143 @@
|
||||
import base64
|
||||
import json
|
||||
from typing import Any, overload
|
||||
|
||||
from ollama._types import Image, Message
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
SystemMessage,
|
||||
ToolCall,
|
||||
UserMessage,
|
||||
)
|
||||
|
||||
|
||||
class OllamaMessageSerializer:
|
||||
"""Serializer for converting between custom message types and Ollama message types."""
|
||||
|
||||
@staticmethod
|
||||
def _extract_text_content(content: Any) -> str:
|
||||
"""Extract text content from message content, ignoring images."""
|
||||
if content is None:
|
||||
return ''
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
text_parts: list[str] = []
|
||||
for part in content:
|
||||
if hasattr(part, 'type'):
|
||||
if part.type == 'text':
|
||||
text_parts.append(part.text)
|
||||
elif part.type == 'refusal':
|
||||
text_parts.append(f'[Refusal] {part.refusal}')
|
||||
# Skip image parts as they're handled separately
|
||||
|
||||
return '\n'.join(text_parts)
|
||||
|
||||
@staticmethod
|
||||
def _extract_images(content: Any) -> list[Image]:
|
||||
"""Extract images from message content."""
|
||||
if content is None or isinstance(content, str):
|
||||
return []
|
||||
|
||||
images: list[Image] = []
|
||||
for part in content:
|
||||
if hasattr(part, 'type') and part.type == 'image_url':
|
||||
url = part.image_url.url
|
||||
if url.startswith('data:'):
|
||||
# Handle base64 encoded images
|
||||
# Format: data:image/png;base64,<data>
|
||||
_, data = url.split(',', 1)
|
||||
# Decode base64 to bytes
|
||||
image_bytes = base64.b64decode(data)
|
||||
images.append(Image(value=image_bytes))
|
||||
else:
|
||||
# Handle URL images (Ollama will download them)
|
||||
images.append(Image(value=url))
|
||||
|
||||
return images
|
||||
|
||||
@staticmethod
|
||||
def _serialize_tool_calls(tool_calls: list[ToolCall]) -> list[Message.ToolCall]:
|
||||
"""Convert browser-use ToolCalls to Ollama ToolCalls."""
|
||||
ollama_tool_calls: list[Message.ToolCall] = []
|
||||
|
||||
for tool_call in tool_calls:
|
||||
# Parse arguments from JSON string to dict for Ollama
|
||||
try:
|
||||
arguments_dict = json.loads(tool_call.function.arguments)
|
||||
except json.JSONDecodeError:
|
||||
# If parsing fails, wrap in a dict
|
||||
arguments_dict = {'arguments': tool_call.function.arguments}
|
||||
|
||||
ollama_tool_call = Message.ToolCall(
|
||||
function=Message.ToolCall.Function(name=tool_call.function.name, arguments=arguments_dict)
|
||||
)
|
||||
ollama_tool_calls.append(ollama_tool_call)
|
||||
|
||||
return ollama_tool_calls
|
||||
|
||||
# region - Serialize overloads
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: UserMessage) -> Message: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: SystemMessage) -> Message: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: AssistantMessage) -> Message: ...
|
||||
|
||||
@staticmethod
|
||||
def serialize(message: BaseMessage) -> Message:
|
||||
"""Serialize a custom message to an Ollama Message."""
|
||||
|
||||
if isinstance(message, UserMessage):
|
||||
text_content = OllamaMessageSerializer._extract_text_content(message.content)
|
||||
images = OllamaMessageSerializer._extract_images(message.content)
|
||||
|
||||
ollama_message = Message(
|
||||
role='user',
|
||||
content=text_content if text_content else None,
|
||||
)
|
||||
|
||||
if images:
|
||||
ollama_message.images = images
|
||||
|
||||
return ollama_message
|
||||
|
||||
elif isinstance(message, SystemMessage):
|
||||
text_content = OllamaMessageSerializer._extract_text_content(message.content)
|
||||
|
||||
return Message(
|
||||
role='system',
|
||||
content=text_content if text_content else None,
|
||||
)
|
||||
|
||||
elif isinstance(message, AssistantMessage):
|
||||
# Handle content
|
||||
text_content = None
|
||||
if message.content is not None:
|
||||
text_content = OllamaMessageSerializer._extract_text_content(message.content)
|
||||
|
||||
ollama_message = Message(
|
||||
role='assistant',
|
||||
content=text_content if text_content else None,
|
||||
)
|
||||
|
||||
# Handle tool calls
|
||||
if message.tool_calls:
|
||||
ollama_message.tool_calls = OllamaMessageSerializer._serialize_tool_calls(message.tool_calls)
|
||||
|
||||
return ollama_message
|
||||
|
||||
else:
|
||||
raise ValueError(f'Unknown message type: {type(message)}')
|
||||
|
||||
@staticmethod
|
||||
def serialize_messages(messages: list[BaseMessage]) -> list[Message]:
|
||||
"""Serialize a list of browser_use messages to Ollama Messages."""
|
||||
return [OllamaMessageSerializer.serialize(m) for m in messages]
|
||||
@@ -0,0 +1,273 @@
|
||||
from collections.abc import Iterable, Mapping
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Literal, TypeVar, overload
|
||||
|
||||
import httpx
|
||||
from openai import APIConnectionError, APIStatusError, AsyncOpenAI, RateLimitError
|
||||
from openai.types.chat import ChatCompletionContentPartTextParam
|
||||
from openai.types.chat.chat_completion import ChatCompletion
|
||||
from openai.types.shared.chat_model import ChatModel
|
||||
from openai.types.shared_params.reasoning_effort import ReasoningEffort
|
||||
from openai.types.shared_params.response_format_json_schema import JSONSchema, ResponseFormatJSONSchema
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.exceptions import ModelProviderError
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.openai.serializer import OpenAIMessageSerializer
|
||||
from browser_use.llm.schema import SchemaOptimizer
|
||||
from browser_use.llm.views import ChatInvokeCompletion, ChatInvokeUsage
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatOpenAI(BaseChatModel):
|
||||
"""
|
||||
A wrapper around AsyncOpenAI that implements the BaseLLM protocol.
|
||||
|
||||
This class accepts all AsyncOpenAI parameters while adding model
|
||||
and temperature parameters for the LLM interface (if temperature it not `None`).
|
||||
"""
|
||||
|
||||
# Model configuration
|
||||
model: ChatModel | str
|
||||
|
||||
# Model params
|
||||
temperature: float | None = 0.2
|
||||
frequency_penalty: float | None = 0.3 # this avoids infinite generation of \t for models like 4.1-mini
|
||||
reasoning_effort: ReasoningEffort = 'low'
|
||||
seed: int | None = None
|
||||
service_tier: Literal['auto', 'default', 'flex', 'priority', 'scale'] | None = None
|
||||
top_p: float | None = None
|
||||
add_schema_to_system_prompt: bool = False # Add JSON schema to system prompt instead of using response_format
|
||||
|
||||
# Client initialization parameters
|
||||
api_key: str | None = None
|
||||
organization: str | None = None
|
||||
project: str | None = None
|
||||
base_url: str | httpx.URL | None = None
|
||||
websocket_base_url: str | httpx.URL | None = None
|
||||
timeout: float | httpx.Timeout | None = None
|
||||
max_retries: int = 5 # Increase default retries for automation reliability
|
||||
default_headers: Mapping[str, str] | None = None
|
||||
default_query: Mapping[str, object] | None = None
|
||||
http_client: httpx.AsyncClient | None = None
|
||||
_strict_response_validation: bool = False
|
||||
max_completion_tokens: int | None = 4096
|
||||
reasoning_models: list[ChatModel | str] | None = field(
|
||||
default_factory=lambda: [
|
||||
'o4-mini',
|
||||
'o3',
|
||||
'o3-mini',
|
||||
'o1',
|
||||
'o1-pro',
|
||||
'o3-pro',
|
||||
'gpt-5',
|
||||
'gpt-5-mini',
|
||||
'gpt-5-nano',
|
||||
]
|
||||
)
|
||||
|
||||
# Static
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'openai'
|
||||
|
||||
def _get_client_params(self) -> dict[str, Any]:
|
||||
"""Prepare client parameters dictionary."""
|
||||
# Define base client params
|
||||
base_params = {
|
||||
'api_key': self.api_key,
|
||||
'organization': self.organization,
|
||||
'project': self.project,
|
||||
'base_url': self.base_url,
|
||||
'websocket_base_url': self.websocket_base_url,
|
||||
'timeout': self.timeout,
|
||||
'max_retries': self.max_retries,
|
||||
'default_headers': self.default_headers,
|
||||
'default_query': self.default_query,
|
||||
'_strict_response_validation': self._strict_response_validation,
|
||||
}
|
||||
|
||||
# Create client_params dict with non-None values
|
||||
client_params = {k: v for k, v in base_params.items() if v is not None}
|
||||
|
||||
# Add http_client if provided
|
||||
if self.http_client is not None:
|
||||
client_params['http_client'] = self.http_client
|
||||
|
||||
return client_params
|
||||
|
||||
def get_client(self) -> AsyncOpenAI:
|
||||
"""
|
||||
Returns an AsyncOpenAI client.
|
||||
|
||||
Returns:
|
||||
AsyncOpenAI: An instance of the AsyncOpenAI client.
|
||||
"""
|
||||
client_params = self._get_client_params()
|
||||
return AsyncOpenAI(**client_params)
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return str(self.model)
|
||||
|
||||
def _get_usage(self, response: ChatCompletion) -> ChatInvokeUsage | None:
|
||||
if response.usage is not None:
|
||||
completion_tokens = response.usage.completion_tokens
|
||||
completion_token_details = response.usage.completion_tokens_details
|
||||
if completion_token_details is not None:
|
||||
reasoning_tokens = completion_token_details.reasoning_tokens
|
||||
if reasoning_tokens is not None:
|
||||
completion_tokens += reasoning_tokens
|
||||
|
||||
usage = ChatInvokeUsage(
|
||||
prompt_tokens=response.usage.prompt_tokens,
|
||||
prompt_cached_tokens=response.usage.prompt_tokens_details.cached_tokens
|
||||
if response.usage.prompt_tokens_details is not None
|
||||
else None,
|
||||
prompt_cache_creation_tokens=None,
|
||||
prompt_image_tokens=None,
|
||||
# Completion
|
||||
completion_tokens=completion_tokens,
|
||||
total_tokens=response.usage.total_tokens,
|
||||
)
|
||||
else:
|
||||
usage = None
|
||||
|
||||
return usage
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
"""
|
||||
Invoke the model with the given messages.
|
||||
|
||||
Args:
|
||||
messages: List of chat messages
|
||||
output_format: Optional Pydantic model class for structured output
|
||||
|
||||
Returns:
|
||||
Either a string response or an instance of output_format
|
||||
"""
|
||||
|
||||
openai_messages = OpenAIMessageSerializer.serialize_messages(messages)
|
||||
|
||||
try:
|
||||
model_params: dict[str, Any] = {}
|
||||
|
||||
if self.temperature is not None:
|
||||
model_params['temperature'] = self.temperature
|
||||
|
||||
if self.frequency_penalty is not None:
|
||||
model_params['frequency_penalty'] = self.frequency_penalty
|
||||
|
||||
if self.max_completion_tokens is not None:
|
||||
model_params['max_completion_tokens'] = self.max_completion_tokens
|
||||
|
||||
if self.top_p is not None:
|
||||
model_params['top_p'] = self.top_p
|
||||
|
||||
if self.seed is not None:
|
||||
model_params['seed'] = self.seed
|
||||
|
||||
if self.service_tier is not None:
|
||||
model_params['service_tier'] = self.service_tier
|
||||
|
||||
if self.reasoning_models and any(str(m).lower() in str(self.model).lower() for m in self.reasoning_models):
|
||||
model_params['reasoning_effort'] = self.reasoning_effort
|
||||
del model_params['temperature']
|
||||
del model_params['frequency_penalty']
|
||||
|
||||
if output_format is None:
|
||||
# Return string response
|
||||
response = await self.get_client().chat.completions.create(
|
||||
model=self.model,
|
||||
messages=openai_messages,
|
||||
**model_params,
|
||||
)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
return ChatInvokeCompletion(
|
||||
completion=response.choices[0].message.content or '',
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
else:
|
||||
response_format: JSONSchema = {
|
||||
'name': 'agent_output',
|
||||
'strict': True,
|
||||
'schema': SchemaOptimizer.create_optimized_json_schema(output_format),
|
||||
}
|
||||
|
||||
# Add JSON schema to system prompt if requested
|
||||
if self.add_schema_to_system_prompt and openai_messages and openai_messages[0]['role'] == 'system':
|
||||
schema_text = f'\n<json_schema>\n{response_format}\n</json_schema>'
|
||||
if isinstance(openai_messages[0]['content'], str):
|
||||
openai_messages[0]['content'] += schema_text
|
||||
elif isinstance(openai_messages[0]['content'], Iterable):
|
||||
openai_messages[0]['content'] = list(openai_messages[0]['content']) + [
|
||||
ChatCompletionContentPartTextParam(text=schema_text, type='text')
|
||||
]
|
||||
|
||||
# Return structured response
|
||||
response = await self.get_client().chat.completions.create(
|
||||
model=self.model,
|
||||
messages=openai_messages,
|
||||
response_format=ResponseFormatJSONSchema(json_schema=response_format, type='json_schema'),
|
||||
**model_params,
|
||||
)
|
||||
|
||||
if response.choices[0].message.content is None:
|
||||
raise ModelProviderError(
|
||||
message='Failed to parse structured output from model response',
|
||||
status_code=500,
|
||||
model=self.name,
|
||||
)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
|
||||
parsed = output_format.model_validate_json(response.choices[0].message.content)
|
||||
|
||||
return ChatInvokeCompletion(
|
||||
completion=parsed,
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
except RateLimitError as e:
|
||||
error_message = e.response.json().get('error', {})
|
||||
error_message = (
|
||||
error_message.get('message', 'Unknown model error') if isinstance(error_message, dict) else error_message
|
||||
)
|
||||
raise ModelProviderError(
|
||||
message=error_message,
|
||||
status_code=e.response.status_code,
|
||||
model=self.name,
|
||||
) from e
|
||||
|
||||
except APIConnectionError as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
|
||||
except APIStatusError as e:
|
||||
try:
|
||||
error_message = e.response.json().get('error', {})
|
||||
except Exception:
|
||||
error_message = e.response.text
|
||||
error_message = (
|
||||
error_message.get('message', 'Unknown model error') if isinstance(error_message, dict) else error_message
|
||||
)
|
||||
raise ModelProviderError(
|
||||
message=error_message,
|
||||
status_code=e.response.status_code,
|
||||
model=self.name,
|
||||
) from e
|
||||
|
||||
except Exception as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
@@ -0,0 +1,15 @@
|
||||
from dataclasses import dataclass
|
||||
|
||||
from browser_use.llm.openai.chat import ChatOpenAI
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatOpenAILike(ChatOpenAI):
|
||||
"""
|
||||
A class for to interact with any provider using the OpenAI API schema.
|
||||
|
||||
Args:
|
||||
model (str): The name of the OpenAI model to use.
|
||||
"""
|
||||
|
||||
model: str
|
||||
@@ -0,0 +1,165 @@
|
||||
from typing import overload
|
||||
|
||||
from openai.types.chat import (
|
||||
ChatCompletionAssistantMessageParam,
|
||||
ChatCompletionContentPartImageParam,
|
||||
ChatCompletionContentPartRefusalParam,
|
||||
ChatCompletionContentPartTextParam,
|
||||
ChatCompletionMessageFunctionToolCallParam,
|
||||
ChatCompletionMessageParam,
|
||||
ChatCompletionSystemMessageParam,
|
||||
ChatCompletionUserMessageParam,
|
||||
)
|
||||
from openai.types.chat.chat_completion_content_part_image_param import ImageURL
|
||||
from openai.types.chat.chat_completion_message_function_tool_call_param import Function
|
||||
|
||||
from browser_use.llm.messages import (
|
||||
AssistantMessage,
|
||||
BaseMessage,
|
||||
ContentPartImageParam,
|
||||
ContentPartRefusalParam,
|
||||
ContentPartTextParam,
|
||||
SystemMessage,
|
||||
ToolCall,
|
||||
UserMessage,
|
||||
)
|
||||
|
||||
|
||||
class OpenAIMessageSerializer:
|
||||
"""Serializer for converting between custom message types and OpenAI message param types."""
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_text(part: ContentPartTextParam) -> ChatCompletionContentPartTextParam:
|
||||
return ChatCompletionContentPartTextParam(text=part.text, type='text')
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_image(part: ContentPartImageParam) -> ChatCompletionContentPartImageParam:
|
||||
return ChatCompletionContentPartImageParam(
|
||||
image_url=ImageURL(url=part.image_url.url, detail=part.image_url.detail),
|
||||
type='image_url',
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _serialize_content_part_refusal(part: ContentPartRefusalParam) -> ChatCompletionContentPartRefusalParam:
|
||||
return ChatCompletionContentPartRefusalParam(refusal=part.refusal, type='refusal')
|
||||
|
||||
@staticmethod
|
||||
def _serialize_user_content(
|
||||
content: str | list[ContentPartTextParam | ContentPartImageParam],
|
||||
) -> str | list[ChatCompletionContentPartTextParam | ChatCompletionContentPartImageParam]:
|
||||
"""Serialize content for user messages (text and images allowed)."""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
serialized_parts: list[ChatCompletionContentPartTextParam | ChatCompletionContentPartImageParam] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_parts.append(OpenAIMessageSerializer._serialize_content_part_text(part))
|
||||
elif part.type == 'image_url':
|
||||
serialized_parts.append(OpenAIMessageSerializer._serialize_content_part_image(part))
|
||||
return serialized_parts
|
||||
|
||||
@staticmethod
|
||||
def _serialize_system_content(
|
||||
content: str | list[ContentPartTextParam],
|
||||
) -> str | list[ChatCompletionContentPartTextParam]:
|
||||
"""Serialize content for system messages (text only)."""
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
serialized_parts: list[ChatCompletionContentPartTextParam] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_parts.append(OpenAIMessageSerializer._serialize_content_part_text(part))
|
||||
return serialized_parts
|
||||
|
||||
@staticmethod
|
||||
def _serialize_assistant_content(
|
||||
content: str | list[ContentPartTextParam | ContentPartRefusalParam] | None,
|
||||
) -> str | list[ChatCompletionContentPartTextParam | ChatCompletionContentPartRefusalParam] | None:
|
||||
"""Serialize content for assistant messages (text and refusal allowed)."""
|
||||
if content is None:
|
||||
return None
|
||||
if isinstance(content, str):
|
||||
return content
|
||||
|
||||
serialized_parts: list[ChatCompletionContentPartTextParam | ChatCompletionContentPartRefusalParam] = []
|
||||
for part in content:
|
||||
if part.type == 'text':
|
||||
serialized_parts.append(OpenAIMessageSerializer._serialize_content_part_text(part))
|
||||
elif part.type == 'refusal':
|
||||
serialized_parts.append(OpenAIMessageSerializer._serialize_content_part_refusal(part))
|
||||
return serialized_parts
|
||||
|
||||
@staticmethod
|
||||
def _serialize_tool_call(tool_call: ToolCall) -> ChatCompletionMessageFunctionToolCallParam:
|
||||
return ChatCompletionMessageFunctionToolCallParam(
|
||||
id=tool_call.id,
|
||||
function=Function(name=tool_call.function.name, arguments=tool_call.function.arguments),
|
||||
type='function',
|
||||
)
|
||||
|
||||
# endregion
|
||||
|
||||
# region - Serialize overloads
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: UserMessage) -> ChatCompletionUserMessageParam: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: SystemMessage) -> ChatCompletionSystemMessageParam: ...
|
||||
|
||||
@overload
|
||||
@staticmethod
|
||||
def serialize(message: AssistantMessage) -> ChatCompletionAssistantMessageParam: ...
|
||||
|
||||
@staticmethod
|
||||
def serialize(message: BaseMessage) -> ChatCompletionMessageParam:
|
||||
"""Serialize a custom message to an OpenAI message param."""
|
||||
|
||||
if isinstance(message, UserMessage):
|
||||
user_result: ChatCompletionUserMessageParam = {
|
||||
'role': 'user',
|
||||
'content': OpenAIMessageSerializer._serialize_user_content(message.content),
|
||||
}
|
||||
if message.name is not None:
|
||||
user_result['name'] = message.name
|
||||
return user_result
|
||||
|
||||
elif isinstance(message, SystemMessage):
|
||||
system_result: ChatCompletionSystemMessageParam = {
|
||||
'role': 'system',
|
||||
'content': OpenAIMessageSerializer._serialize_system_content(message.content),
|
||||
}
|
||||
if message.name is not None:
|
||||
system_result['name'] = message.name
|
||||
return system_result
|
||||
|
||||
elif isinstance(message, AssistantMessage):
|
||||
# Handle content serialization
|
||||
content = None
|
||||
if message.content is not None:
|
||||
content = OpenAIMessageSerializer._serialize_assistant_content(message.content)
|
||||
|
||||
assistant_result: ChatCompletionAssistantMessageParam = {'role': 'assistant'}
|
||||
|
||||
# Only add content if it's not None
|
||||
if content is not None:
|
||||
assistant_result['content'] = content
|
||||
|
||||
if message.name is not None:
|
||||
assistant_result['name'] = message.name
|
||||
if message.refusal is not None:
|
||||
assistant_result['refusal'] = message.refusal
|
||||
if message.tool_calls:
|
||||
assistant_result['tool_calls'] = [OpenAIMessageSerializer._serialize_tool_call(tc) for tc in message.tool_calls]
|
||||
|
||||
return assistant_result
|
||||
|
||||
else:
|
||||
raise ValueError(f'Unknown message type: {type(message)}')
|
||||
|
||||
@staticmethod
|
||||
def serialize_messages(messages: list[BaseMessage]) -> list[ChatCompletionMessageParam]:
|
||||
return [OpenAIMessageSerializer.serialize(m) for m in messages]
|
||||
@@ -0,0 +1,208 @@
|
||||
from collections.abc import Mapping
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, TypeVar, overload
|
||||
|
||||
import httpx
|
||||
from openai import APIConnectionError, APIStatusError, AsyncOpenAI, RateLimitError
|
||||
from openai.types.chat.chat_completion import ChatCompletion
|
||||
from openai.types.shared_params.response_format_json_schema import (
|
||||
JSONSchema,
|
||||
ResponseFormatJSONSchema,
|
||||
)
|
||||
from pydantic import BaseModel
|
||||
|
||||
from browser_use.llm.base import BaseChatModel
|
||||
from browser_use.llm.exceptions import ModelProviderError, ModelRateLimitError
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.openrouter.serializer import OpenRouterMessageSerializer
|
||||
from browser_use.llm.schema import SchemaOptimizer
|
||||
from browser_use.llm.views import ChatInvokeCompletion, ChatInvokeUsage
|
||||
|
||||
T = TypeVar('T', bound=BaseModel)
|
||||
|
||||
|
||||
@dataclass
|
||||
class ChatOpenRouter(BaseChatModel):
|
||||
"""
|
||||
A wrapper around OpenRouter's chat API, which provides access to various LLM models
|
||||
through a unified OpenAI-compatible interface.
|
||||
|
||||
This class implements the BaseChatModel protocol for OpenRouter's API.
|
||||
"""
|
||||
|
||||
# Model configuration
|
||||
model: str
|
||||
|
||||
# Model params
|
||||
temperature: float | None = None
|
||||
top_p: float | None = None
|
||||
seed: int | None = None
|
||||
|
||||
# Client initialization parameters
|
||||
api_key: str | None = None
|
||||
http_referer: str | None = None # OpenRouter specific parameter for tracking
|
||||
base_url: str | httpx.URL = 'https://openrouter.ai/api/v1'
|
||||
timeout: float | httpx.Timeout | None = None
|
||||
max_retries: int = 10
|
||||
default_headers: Mapping[str, str] | None = None
|
||||
default_query: Mapping[str, object] | None = None
|
||||
http_client: httpx.AsyncClient | None = None
|
||||
_strict_response_validation: bool = False
|
||||
|
||||
# Static
|
||||
@property
|
||||
def provider(self) -> str:
|
||||
return 'openrouter'
|
||||
|
||||
def _get_client_params(self) -> dict[str, Any]:
|
||||
"""Prepare client parameters dictionary."""
|
||||
# Define base client params
|
||||
base_params = {
|
||||
'api_key': self.api_key,
|
||||
'base_url': self.base_url,
|
||||
'timeout': self.timeout,
|
||||
'max_retries': self.max_retries,
|
||||
'default_headers': self.default_headers,
|
||||
'default_query': self.default_query,
|
||||
'_strict_response_validation': self._strict_response_validation,
|
||||
'top_p': self.top_p,
|
||||
'seed': self.seed,
|
||||
}
|
||||
|
||||
# Create client_params dict with non-None values
|
||||
client_params = {k: v for k, v in base_params.items() if v is not None}
|
||||
|
||||
# Add http_client if provided
|
||||
if self.http_client is not None:
|
||||
client_params['http_client'] = self.http_client
|
||||
|
||||
return client_params
|
||||
|
||||
def get_client(self) -> AsyncOpenAI:
|
||||
"""
|
||||
Returns an AsyncOpenAI client configured for OpenRouter.
|
||||
|
||||
Returns:
|
||||
AsyncOpenAI: An instance of the AsyncOpenAI client with OpenRouter base URL.
|
||||
"""
|
||||
if not hasattr(self, '_client'):
|
||||
client_params = self._get_client_params()
|
||||
self._client = AsyncOpenAI(**client_params)
|
||||
return self._client
|
||||
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return str(self.model)
|
||||
|
||||
def _get_usage(self, response: ChatCompletion) -> ChatInvokeUsage | None:
|
||||
"""Extract usage information from the OpenRouter response."""
|
||||
if response.usage is None:
|
||||
return None
|
||||
|
||||
prompt_details = getattr(response.usage, 'prompt_tokens_details', None)
|
||||
cached_tokens = prompt_details.cached_tokens if prompt_details else None
|
||||
|
||||
return ChatInvokeUsage(
|
||||
prompt_tokens=response.usage.prompt_tokens,
|
||||
prompt_cached_tokens=cached_tokens,
|
||||
prompt_cache_creation_tokens=None,
|
||||
prompt_image_tokens=None,
|
||||
# Completion
|
||||
completion_tokens=response.usage.completion_tokens,
|
||||
total_tokens=response.usage.total_tokens,
|
||||
)
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: None = None) -> ChatInvokeCompletion[str]: ...
|
||||
|
||||
@overload
|
||||
async def ainvoke(self, messages: list[BaseMessage], output_format: type[T]) -> ChatInvokeCompletion[T]: ...
|
||||
|
||||
async def ainvoke(
|
||||
self, messages: list[BaseMessage], output_format: type[T] | None = None
|
||||
) -> ChatInvokeCompletion[T] | ChatInvokeCompletion[str]:
|
||||
"""
|
||||
Invoke the model with the given messages through OpenRouter.
|
||||
|
||||
Args:
|
||||
messages: List of chat messages
|
||||
output_format: Optional Pydantic model class for structured output
|
||||
|
||||
Returns:
|
||||
Either a string response or an instance of output_format
|
||||
"""
|
||||
openrouter_messages = OpenRouterMessageSerializer.serialize_messages(messages)
|
||||
|
||||
# Set up extra headers for OpenRouter
|
||||
extra_headers = {}
|
||||
if self.http_referer:
|
||||
extra_headers['HTTP-Referer'] = self.http_referer
|
||||
|
||||
try:
|
||||
if output_format is None:
|
||||
# Return string response
|
||||
response = await self.get_client().chat.completions.create(
|
||||
model=self.model,
|
||||
messages=openrouter_messages,
|
||||
temperature=self.temperature,
|
||||
top_p=self.top_p,
|
||||
seed=self.seed,
|
||||
extra_headers=extra_headers,
|
||||
)
|
||||
|
||||
usage = self._get_usage(response)
|
||||
return ChatInvokeCompletion(
|
||||
completion=response.choices[0].message.content or '',
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
else:
|
||||
# Create a JSON schema for structured output
|
||||
schema = SchemaOptimizer.create_optimized_json_schema(output_format)
|
||||
|
||||
response_format_schema: JSONSchema = {
|
||||
'name': 'agent_output',
|
||||
'strict': True,
|
||||
'schema': schema,
|
||||
}
|
||||
|
||||
# Return structured response
|
||||
response = await self.get_client().chat.completions.create(
|
||||
model=self.model,
|
||||
messages=openrouter_messages,
|
||||
temperature=self.temperature,
|
||||
top_p=self.top_p,
|
||||
seed=self.seed,
|
||||
response_format=ResponseFormatJSONSchema(
|
||||
json_schema=response_format_schema,
|
||||
type='json_schema',
|
||||
),
|
||||
extra_headers=extra_headers,
|
||||
)
|
||||
|
||||
if response.choices[0].message.content is None:
|
||||
raise ModelProviderError(
|
||||
message='Failed to parse structured output from model response',
|
||||
status_code=500,
|
||||
model=self.name,
|
||||
)
|
||||
usage = self._get_usage(response)
|
||||
|
||||
parsed = output_format.model_validate_json(response.choices[0].message.content)
|
||||
|
||||
return ChatInvokeCompletion(
|
||||
completion=parsed,
|
||||
usage=usage,
|
||||
)
|
||||
|
||||
except RateLimitError as e:
|
||||
raise ModelRateLimitError(message=e.message, model=self.name) from e
|
||||
|
||||
except APIConnectionError as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
|
||||
except APIStatusError as e:
|
||||
raise ModelProviderError(message=e.message, status_code=e.status_code, model=self.name) from e
|
||||
|
||||
except Exception as e:
|
||||
raise ModelProviderError(message=str(e), model=self.name) from e
|
||||
@@ -0,0 +1,26 @@
|
||||
from openai.types.chat import ChatCompletionMessageParam
|
||||
|
||||
from browser_use.llm.messages import BaseMessage
|
||||
from browser_use.llm.openai.serializer import OpenAIMessageSerializer
|
||||
|
||||
|
||||
class OpenRouterMessageSerializer:
|
||||
"""
|
||||
Serializer for converting between custom message types and OpenRouter message formats.
|
||||
|
||||
OpenRouter uses the OpenAI-compatible API, so we can reuse the OpenAI serializer.
|
||||
"""
|
||||
|
||||
@staticmethod
|
||||
def serialize_messages(messages: list[BaseMessage]) -> list[ChatCompletionMessageParam]:
|
||||
"""
|
||||
Serialize a list of browser_use messages to OpenRouter-compatible messages.
|
||||
|
||||
Args:
|
||||
messages: List of browser_use messages
|
||||
|
||||
Returns:
|
||||
List of OpenRouter-compatible messages (identical to OpenAI format)
|
||||
"""
|
||||
# OpenRouter uses the same message format as OpenAI
|
||||
return OpenAIMessageSerializer.serialize_messages(messages)
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user