# Why 99% Accurate Browser Agents Still Fail — Derek Meegan, Browserbase

## Executive summary

Browser agents face a critical scaling challenge: while individual steps may be highly accurate (e.g., 99%), the overall success rate over long, multi-step transactions drops drastically (e.g., 36% over 100 steps). To move from demo to reliable production use, the talk argues that success must be measured per transaction, not per run. The solution involves designing a hybrid architecture that minimizes model dependency by encapsulating complex, non-ambiguous steps into deterministic tools, such as OCR verification, dedicated download functions, and standardized authentication modules, thereby improving performance, cost, and maintainability.

## Key takeaways

- Success Measurement: Per-Transaction vs. Per-Run: Reliability should be measured by the per-transaction success rate, allowing for retries, rather than the raw success rate of a single run. The customer cares that the workflow completes reliably, regardless of the number of retries required. (8:42)
- The Cost/Value Disconnect: Cost accumulation (model calls, compute) is continuous across every step, but value realization is terminal (only achieved when the entire task is complete). This 'no partial credit' dynamic requires architectural intervention. (6:12)
- Architectural Determinism: To achieve production reliability, complex operations (like downloading files or authenticating) must be pulled out of the model's decision-making process and implemented as deterministic, reusable tools. (13:01, 14:36)

## Technical details

- Agent Interaction Layers: Agents interact with the browser through several layers: the Document Object Model (DOM)/HTML, screenshots, the accessibility tree (semantic textual representation), and a dynamic code execution runtime (JavaScript/CDP). Industry strategies include creating a hybrid textual representation (HTML + accessibility tree), using screenshots, or dynamic code execution. (2:32)
- Browser Trajectories: Trajectories range from purpose-built (least agentic) to just-in-time automations (most agentic). Production systems should focus on transactional workflows—completing a defined unit of work end-to-end. (3:17)
- Optimizing Agent Systems: A robust system incorporates: 1) A deterministic download tool to retrieve files; 2) An OCR tool for real-time verification against a system of record; 3) Pulling out stable authentication logic; and 4) Defining 'skills' that act as standard operating procedures to guide the agent along the critical path. (13:31, 14:36)

## Practical implications

- When designing automated workflows, prioritize performance and reliability over simply maximizing model capability. Treat cost and maintainability as engineering optimization problems.
- Implement explicit verification steps (e.g., OCR checks) within the agent's workflow to provide deterministic proof of success, rather than relying solely on the agent's final output.
- Modularize the system by extracting stable, non-ambiguous business logic (like authentication) into dedicated, reusable functions to reduce the agent's overall complexity and risk surface.

## Topics

Browser Automation, AI Agents, System Architecture, Reliability Engineering, Web Scraping, Browserbase, Stagehand

Source: https://www.youtube.com/watch?v=5xi_S1f9sDU