# The Dark Arts of Web Automation: Teaching Agents to Use Websites Like Humans — Corey Gallon, Rexmore

## Executive summary

This talk details advanced web automation techniques for AI agents to interact with websites that actively resist bots. The core methodology involves treating the agent as a 'meat bag with a mouse,' leveraging the Chrome DevTools Protocol (CDP) through a Command Line Interface (CLI). Automation is structured around a 'Sense, Act, Verify' loop and progresses up a three-rung 'Meatbag Ladder'—moving from simple API calls to simulated human input (jitter, overshoot)—to defeat modern security measures like Cloudflare Turnstile and reCAPTCHA v2.

## Key takeaways

- CLI vs. MCP for Automation: While capability is comparable (both achieved tasks 83% of the time), a CLI approach significantly outperforms an MCP server in speed, reuse, and cost. Specifically, a CLI sequence can be programmed once and reused infinitely without model calls, making it faster and up to 75 times cheaper in token cost than repeated model interaction.
- The Sense-Act-Verify Loop: Effective automation requires a continuous loop: Sensing (via DOM, accessibility tree, or screenshot), Acting (clicking/typing), and Verifying (checking network traffic or screen state). This process must be repeated until the desired action is confirmed.
- The Meatbag Ladder Methodology: Web resistance requires escalating techniques. Rung One uses synthetic JavaScript clicks (API-like). Rung Two employs real CDP input events, which the page recognizes as trusted inputs. Rung Three simulates full human behavior, including jitter and deliberate overshoot, to bypass advanced bot detection.
- Solving reCAPTCHA v2: The final challenge (reCAPTCHA v2) is solved by splitting the work: deterministic code handles the rapid, repeatable actions (trusted clicks, iframe piercing), while a specialized AI agent provides the 'eyes and brain' to interpret visual challenges (e.g., identifying specific objects in a grid).

## Technical details

- Chrome DevTools Protocol (CDP): The CDP is the underlying mechanism used to drive the browser, allowing agents to perform actions that mimic human interaction. The agent tool, Chrome Agent, facilitates writing code to replay these complex CDP interactions.
- Web UI as a Permissionless API: In certain corporate environments (like Office 365), the web user interface itself can function as a 'permissionless API,' allowing automation via simple synthetic clicks and programmatic filling, bypassing the need for formal application registration or admin approval.
- Advanced CAPTCHA Defeat: Defeating modern security measures requires overcoming technical hurdles like shadow roots (Cloudflare Turnstile) and cross-origin iframes. The solution involves calculating the element's screen position and firing a trusted click directly at that coordinate.

## Practical implications

- Build engineers can design more robust automation pipelines by implementing the 'Sense, Act, Verify' loop and escalating techniques based on observed web resistance.
- The preference for CLI over MCP suggests that stateful, deterministic code execution is superior to model-based reasoning for high-speed, repeatable tasks like CI/CD testing or data scraping.
- Understanding CDP allows engineers to programmatically simulate trusted user inputs, bypassing basic JavaScript click detection mechanisms.

## Topics

Web Automation, AI Agents, Chrome DevTools Protocol (CDP), Build Engineering, Security Bypass, CLI Development, Chrome Agent, Captivus GitHub Repository

Source: https://www.youtube.com/watch?v=26RtyAm9y_Q