# Bringing agents onto the world wide web — Paul Klein IV, Browserbase

## Executive summary

The talk argues that the current bottleneck for building reliable web agents is not model capability (AI), but rather the missing engineering infrastructure—the 'harness' and scalable systems. Reliable browser agents must be multimodal, harness-engineered with memory/skills, and operate on consistent, robust infrastructure. The speaker emphasizes that solving this overhang is an engineering problem that companies can tackle today, accelerating AI diffusion into real-world enterprise use cases (e.g., logistics, banking).

## Key takeaways

- The bottleneck is the harness, not the model.: Model capabilities have advanced significantly; therefore, agents struggle due to missing scaffolding and systems around the LLM (the 'harness'). Building a custom harness optimized for a domain can improve results beyond baseline models. (4:32)
- Reliable browser agents require three components.: They must be multimodal, harness-engineered (using skills and memory), and run on reliable infrastructure that guarantees consistent page rendering across all runs. (7:05)
- The web needs to adapt for agents.: Key improvements required from the web include better accessibility trees, adoption of Chrome's Web MCP, solving agent login/authentication securely, and establishing a trust framework (a 'Verisign moment') for agents. (10:24)
- Infrastructure must be scalable and consistent.: Production environments require infrastructure that can handle thousands of agents, maintain consistency (e.g., preventing mobile/desktop layout shifts), and ideally achieve SOC 2 compliance at scale. (9:09)

## Technical details

- Agent Architecture: The most reliable agents are multimodal, often intercepting network requests and writing code to replay them rather than relying solely on pixel-based driving. They utilize skills and memory for context efficiency. (7:05)
- Harness Engineering: A harness is the scaffolding around an LLM, providing tools like code interpreters, audio/video input capabilities, and subagents. It allows agents to benefit from structured knowledge (e.g., WebMCP) and skills published via platforms like browser.sh. (7:05)
- Web Standards for Agents: The web needs to improve by implementing accessibility trees and adopting Chrome's Web MCP, which allows websites to publish structured components that agents can interact with in a context-efficient manner. (10:24)
- Infrastructure Requirements: Production infrastructure must ensure consistency by rendering pages identically every time, regardless of device or run state. The speaker noted the difficulty in achieving SOC 2 compliance for such setups at scale. (9:09)

## Practical implications

- Focus development efforts on building robust agent harnesses and infrastructure rather than solely waiting for model improvements.
- Implement observability features (e.g., screen recordings, network logs) to create data feedback loops that allow agents to self-improve over time.
- Design agent login/sign-up flows with an 'agent-first' perspective, anticipating automated access and secure authentication methods.
- Build model-agnostic infrastructure to prevent vendor lock-in as AI models evolve.

## Topics

Web Agents, AI Infrastructure, Harness Engineering, Multimodality, System Reliability, Web Standards, Browserbase, Playwright CLI, WebMCP (Chrome)

Source: https://www.youtube.com/watch?v=GqoNrUz8hEU