cua — Computer-Use Automation

What this is

A recording that became a capability, not a video.

What is built

Discovery loop: one typed JSON decision per observation on a numbered element table; compiles to a draft artifact.
Approval ceremony: sheet, dry-run validation, typed confirmation, ledger. Drafts are refused for unattended replay.
Deterministic replay: fresh snapshot per step, locator re-resolution, post-condition waits, recovery ledger.
Escalation: hard failure hands the live session to a human; the run resumes from the failed step.
Enforced budgets: cost, steps, wall-clock, waits, retries; all in configuration.

What the evidence means

Measured: a real model run discovered the capability ($0.000821, 4 calls); replay percentiles from 5-run benchmarks.
Proven by tests: 82 offline tests plus real-browser integration.
Scripted, labeled: the keyless demo uses a scripted stand-in; live runs sit beside it.
Designed only: multi-tenant canary fleet and desktop/AT-SPI drivers (DESIGN.md §4).

Architecture

Two lanes: an expensive model run once, a cheap engine forever.

Discovery uses the LLM to produce a typed trace; the trace compiles to an artifact; replay executes the artifact against any PageSurface implementation with zero model calls. Everything persisted is redacted.

Design decision: locators reference the normalized element model (role + accessible name with fallbacks), never raw CSS/XPath; the same shape works for a web DOM, a frameset page, or a desktop accessibility tree. Only the PageSurface module knows a browser exists.

Artifact v1

Typed, versioned, replayable.

Shape

`inputs`	typed, with examples; `{member_id}` resolves anywhere.
`steps`	goto / click / fill / press_enter only, each with rationale and an optional wait.
`outcomes`	NOT_FOUND, INVALID_INPUT, PERMISSION_DENIED… with detect conditions; declared by a human at approval, never guessed.
`checkpoint`	the final-state assertion for SUCCESS.
`outputs`	regex-extracted caller values.

Lifecycle

Draft: compiled from a successful trace, provenance attached (model, cost, calls).
Approval: review the sheet, declare outcomes, pass dry-run validation, type the confirmation; the decision is ledgered.
Replay: requires approved status; drafts refused by default.
Drift: locator failures and checkpoint misses are the signal; the recovery ledger records what each run absorbed.

Determinism & error taxonomy

Four statuses. Eight fault cases. All green, all committed.

Business outcomes are answers, not failures; recoverable conditions are engine policy; everything else is a hard failure. The matrix runs one approved artifact against deliberate faults; every row is a committed run under evidence/.

success ×2	members 1001 / 1002: checkpoint holds, outputs extracted. Same artifact, different params. SUCCESS
not-found	member 9999: declared outcome returns a message payload. BUSINESS_OUTCOME
bad input	member "401a": declared outcome, no crash. BUSINESS_OUTCOME
frozen acct	member 1005: teller role may not view it. BUSINESS_OUTCOME
slow page	member 1006 with a 2.5 s fault: the wait absorbs it, p50 3.27 s. SUCCESS
busy page	member 1002: one recorded reload, then done. SUCCESS · 1 recovery
server error	member 1003: HTTP 500 on the member page. HARD_FAILURE @ step 0
session expiry	member 1002: escalated to an operator, run resumes and completes. ESCALATED → SUCCESS

Stability: five consecutive replays produce byte-identical result signatures.

Safety

Default-deny, enforced before every action in both loops.

Pre-execution guardrails

Allowlist: action types, origins, routes, risky controls. The freeze button is forbidden everywhere: discovery, replay, operator.
Live proof: evidence/vet-guard/ is a real model told to freeze accounts; it navigates, is refused, declines.
Post-action URL guard: unexpected navigations fail loud.

Data handling

Redaction: SSNs, card-like numbers, amounts scrubbed from everything persisted and sent; the loop never sees a raw SSN (test-asserted).
Credentials: artifacts hold placeholders; .env is gitignored; caller outputs return intact in-process only.
Privacy posture: CUA_DATA_COLLECTION=deny routes model calls to zero-retention endpoints.

Escalation & handoff

The operator takes the live session, not a report.

1 · Detect

Hard failure, recovery budget spent

The engine stops, writes intervention.json (expected vs observed, snapshot, screenshot), and hands over the session it was driving.

2 · Intervene

Same surface instance

The operator drives the browser the engine holds, via an allowlisted protocol (goto/click/fill/look). Every action and control transition is recorded.

3 · Resume

Continue from the failed step

resume returns control; satisfied steps are skipped, the checkpoint re-asserted, the run completes. Actions fold into result.escalation.

Measured: the scripted cycle (expire, hand off, drive, resume, complete) runs end-to-end in 4.75 s (evidence/replay-escalation-handoff/). The operator console UI is deliberately not built.

Budgets, latency, cost

Measured baselines beside enforced caps.

Typed configuration (config/budgets.json), enforced, never advisory: breaches are typed budget exceeded failures, every result self-describes its envelope (test-asserted).

Measured baseline (5×/case)

replay ok	742 ms p50 · 757 ms p95
outcome exit	607 ms p50 · 617 ms p95
slow fault	3.27 s p50 · 3.28 s p95
busy reload	742 ms p50 · 797 ms p95
escalation	4.75 s cycle
LLM call	2.14 s p50 live
discovery	9.0 s · $0.000821 · 4 calls
test suite	82 tests · 21.6 s
jev stuck rule	83% · Brier 0.075

Enforced budgets

cost cap	$0.50 ≈ 600× observed run
discovery	600 s · 40 steps · 60 calls
step wait	3 s (slow faults use ~2.6 s)
act timeout	10 s per browser action
run total	180 s ≈ 55× slowest replay case
retries	3 attempts · 0.5/1.0 s backoff · Retry-After ≤ 30 s
timeouts	connect 10 s · read 60 s · write 30 s
no retry	4xx client errors raise at once

Replay latency is local-CPU bound and scales with steps, not load. Discovery latency is LLM-RTT bound. Escalation latency is dominated by the human. Costs are bounded by configuration.

Evidence boundary

What is real, what is scripted, what is designed.

Real	Engine, taxonomy, guardrails, redaction, escalation, budgets, retry ladder; 82 offline + browser tests. Live runs: discovery ($0.000821) and the guardrail refusal.
Scripted	The keyless demo's discovery stand-in, labeled in every artifact it produces.
Mock by design	Target app: a hostile credit-union console (no test IDs, non-semantic markup, JS-confirm on the irreversible action, fault injection).
Designed only	Multi-tenant canary fleet, desktop/AT-SPI driver, operator console UI (DESIGN.md §4, §8).
Not hidden	Every claim on this page cites a committed run under `evidence/`; anything without one is labeled designed.

Reproduce

Everything except live discovery runs without an API key.

# install
uv sync

# full offline suite: 82 tests, no keys
uv run pytest -q

# matrix + escalation + stability, regenerates committed evidence
uv run cua demo --part all

# latency/cost benchmark (add --live --tests for the full picture)
uv run cua bench --runs 5

# live: discover → approve → replay
uv run cua discover --goal "Look up member {member_id} …" --param member_id=1001
uv run cua approve <draft path> --preset-lookup-outcomes
uv run cua replay <artifact> --param member_id=1002

The model discovers once.
The artifact replays forever.