Observe and act
The observation object, element refs, the action vocabulary, frames, and the read-only tools.
The observation
GET /v1/browsers/{id}/observe returns what the page is, as text, with a stable ref on every interactive element:
[ref=e1] link "More information..." href=https://www.iana.org/domains/example[ref=e2] heading "Example Domain"| Field | Meaning |
|---|---|
frame_id | The observation's identity; every action names it |
tree, truncated | The accessibility-derived tree with [ref=eN] markers; truncated when bounded |
targets | The interactive elements: ref, role, name, and with ?boxes=1 their viewport rectangle |
tabs | Stable tab ids, one active; new marks a tab the site opened |
dialogs, dialog_policy | Dialogs handled since the last observation, and the current policy (dismiss by default) |
downloads | Downloads and their state |
navigation | load_state, pending_requests, is_pdf |
errors | Page errors and refused navigations since the last observation |
scroll, scrollables | Pixels above and below the viewport; scrollable regions, each with the ref of its line when the tree marks it [scrollable] — pass it to scroll to move that region instead of the page |
new_refs, occluded_refs | Elements new since the last observation; elements something covers (no ref) |
pagination | Recognised pagination controls |
challenge | A recognised CAPTCHA / bot-challenge wall (kind, vendor, sitekey), or null |
sizes | What was bounded, and by how much |
occlusion_check | full, partial or skipped |
?boxes=1 adds rectangles; ?attributes=1 adds form attributes to refs. Observe and preview wait up to one second for in-flight requests so the frame after an action shows its result. GET …/preview is the same object plus a JPEG image (≤ 2 MiB); GET …/snapshot is the tree alone.
Actions
POST /v1/browsers/{id}/action with the action, the current frame_id and, for element actions, a ref:
{ "type": "click", "frame_id": "<frame UUID>", "ref": "e3" }{ "type": "type", "frame_id": "<frame UUID>", "ref": "e4", "text": "Search term" }{ "type": "select", "frame_id": "<frame UUID>", "ref": "e5", "values": ["option-value"] }{ "type": "scroll", "frame_id": "<frame UUID>", "delta_y": 550 }{ "type": "press", "frame_id": "<frame UUID>", "key": "Enter" }| Action | Parameters |
|---|---|
click, hover | ref |
type | ref, text (≤ 4000 characters), slowly?, submit? |
select | ref, values |
Typing a saved credential
A browser created with credential_ids carries each credential under a name derived from its vault name (Shop login → shop_login; the observation lists them as secrets). Write the placeholder where the value should go — { "type": "type", "ref": "e4", "text": "{{secret:shop_login}}" } — and the browser substitutes the value only when the page's origin is one of the credential's allowed_origins; anywhere else the action fails, and an unknown name fails the same way. Observations, events and your own reads show the placeholder, never the value.
| press | key, ref? — Enter, Tab, Shift+Tab, Escape, Backspace, Delete, arrows, Home, End, PageUp, PageDown |
| scroll | delta_y (±2000), ref? |
| click_at | x, y — for an element without a ref, from a boxes observation; the element takes focus, as under a pointer |
| insert_text | text — typed at the keyboard focus, for a field without a ref after click_at on it; type with a ref is the better tool when there is one |
| wait | state and/or ms |
| back, forward, reload | — |
| switch_tab, close_tab, new_tab | tab (an id from tabs); close_tab without one closes the active tab; new_tab takes url? |
| dialog_policy | policy: dismiss or accept, prompt_text? |
| find | text, role, name, max — read-only, answers with matches |
| text | format: text or markdown, max_chars (≤ 65536) — read-only, answers with the content |
| upload | Hosted runs only, from a staged Workspace file |
A page-changing action returns the observation after it. A read-only action (find, text) answers with its result and leaves the frame valid. selector and force are not accepted: refs, not CSS.
Frames
Every action needs the frame_id of a current observation. A frame lasts up to 30 seconds; navigation and any page-changing action invalidate earlier frames, and acting on an old one answers stale_frame (409) — observe again. The SDKs hold the last frame, observe when they have none, re-observe once on stale_frame, drop the frame after a page-changing action and keep it after a read-only one.
What input is
Clicks and typing are synthetic DOM activation with occlusion checks — not trusted pointer events (isTrusted is false). Arbitrary pixel or canvas input, dragging and file pickers are not available; click_at activates the element under a point. Actions require the execute scope, and answer conflict while an Agent run owns the browser.
In code
state = browser.observe(boxes=True)button = browser.find(role="button", name="sign in")["matches"]browser.type("e4", "hello", submit=True)browser.press("Escape")browser.scroll(600)browser.switch_tab(state["tabs"][0]["id"])page = browser.text(format="markdown", max_chars=20000)["text"]const state = await browser.observe({ boxes: true });const { matches } = await browser.find({ role: 'button', name: 'sign in' });await browser.type('e4', 'hello', { submit: true });await browser.press('Escape');await browser.scroll(600);await browser.switchTab(state.tabs[0].id);const { text } = await browser.text({ format: 'markdown', max_chars: 20000 });act(name, params) runs any action by its registry name with the same parameter names.