# The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs

## Executive summary

The video addresses a critical gap in agentic AI: the lack of robust infrastructure for interacting with the open web. The speaker argues that relying solely on large language models (LLMs) or general browser automation is inefficient, expensive, and unreliable. By applying principles from professional web scraping—specifically prioritizing cost control, validating content before processing, and using a browser only when absolutely necessary—the agent's performance can be dramatically improved across all stages of the purchase workflow.

## Key takeaways

- The Core Problem in Agentic AI: Many agents fail or waste resources because they assume an HTTP 200 status code means valid content. Feeding LLMs large HTML blocks containing CAPTCHAs or non-product data leads to wasted tokens (e.g., wasting 70% of tokens when only three out of ten pages are valid).
- Discovery Stage Optimization: Instead of using a browser with a predefined list of retailers, the agent should use a dedicated search API. This approach returns compact JSON (<2,000 tokens), is fast (<700ms), and allows the agent to fan out queries across many popular search engines.
- Decision Stage Optimization: The decision stage can be optimized by replacing browser scraping with a dedicated scraper API. This ensures only valid content is returned, fails loudly (with explicit errors on blocks), supports markdown, and allows parallel requests while billing only for successful results ('No cure, no pay').
- Checkout Stage Enhancement: Since the checkout process requires highly dynamic interaction, a browser is necessary. The solution involves hardening the agent using Playwright MCP combined with an Oxylabs headless browser that provides stealth, residential proxy support, and geolocation capabilities.

## Technical details

- Web Scraping Principles: The guiding principles for building reliable agents are: 1) Cost matters. 2) Use a browser only when absolutely necessary. 3) Validate content (HTTP 200 does not guarantee validity).
- Fast Search API Implementation: Used in the Discovery stage, this tool provides access to multiple search engines via a compact JSON format, enabling agents to formulate and execute fan-out queries efficiently.
- Oxylabs Scraper API (Decision Stage): This lightweight REST API allows for parallel requests and returns only valid content. It supports markdown output and runs a full browser under the hood if the target website is dynamic, while also supporting geolocation localization.
- Playwright MCP & Headless Browser (Checkout): For the final purchase stage, Playwright MCP was used with an enhanced headless browser that incorporates stealth features, residential proxying, and geo-location to ensure localized stock availability during checkout.

## Practical implications

- Build agents with a dedicated, cost-aware infrastructure layer rather than relying solely on general browser automation.
- Implement content validation checks to prevent LLMs from processing invalid or CAPTCHA-filled data, significantly reducing token waste.
- Use specialized APIs (e.g., search APIs) for initial discovery phases to maximize speed and minimize costs before resorting to heavy browser usage.

## Topics

Agentic AI, Web Scraping, LLM Infrastructure, Build Engineering, API Design, Giedrius Šteimantas, Oxylabs

Source: https://www.youtube.com/watch?v=XsvUhpnHepE