The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs
Summary
The video addresses a critical gap in agentic AI: the lack of robust infrastructure for interacting with the open web. The speaker argues that relying solely on large language models (LLMs) or general browser automation is inefficient, expensive, and unreliable. By applying principles from professional web scraping—specifically prioritizing cost control, validating content before processing, and using a browser only when absolutely necessary—the agent's performance can be dramatically improved across all stages of the purchase workflow.
Key takeaways
-
The Core Problem in Agentic AI
13:55
Many agents fail or waste resources because they assume an HTTP 200 status code means valid content. Feeding LLMs large HTML blocks containing CAPTCHAs or non-product data leads to wasted tokens (e.g., wasting 70% of tokens when only three out of ten pages are valid).
-
Discovery Stage Optimization
9:50
Instead of using a browser with a predefined list of retailers, the agent should use a dedicated search API. This approach returns compact JSON (<2,000 tokens), is fast (<700ms), and allows the agent to fan out queries across many popular search engines.
-
Decision Stage Optimization
The decision stage can be optimized by replacing browser scraping with a dedicated scraper API. This ensures only valid content is returned, fails loudly (with explicit errors on blocks), supports markdown, and allows parallel requests while billing only for successful results ('No cure, no pay').
-
Checkout Stage Enhancement
Since the checkout process requires highly dynamic interaction, a browser is necessary. The solution involves hardening the agent using Playwright MCP combined with an Oxylabs headless browser that provides stealth, residential proxy support, and geolocation capabilities.
Technical details
-
Web Scraping Principles
275s
The guiding principles for building reliable agents are: 1) Cost matters. 2) Use a browser only when absolutely necessary. 3) Validate content (HTTP 200 does not guarantee validity).
-
Fast Search API Implementation
650s
Used in the Discovery stage, this tool provides access to multiple search engines via a compact JSON format, enabling agents to formulate and execute fan-out queries efficiently.
-
Oxylabs Scraper API (Decision Stage)
This lightweight REST API allows for parallel requests and returns only valid content. It supports markdown output and runs a full browser under the hood if the target website is dynamic, while also supporting geolocation localization.
-
Playwright MCP & Headless Browser (Checkout)
For the final purchase stage, Playwright MCP was used with an enhanced headless browser that incorporates stealth features, residential proxying, and geo-location to ensure localized stock availability during checkout.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.