Inside Clay's Eval Stack: 300M Agent Runs, One LangSmith Pipeline
Summary
Clay detailed its agentic stack, featuring Claygent (go-to-market research) and Sculptor (go-to-market engineering workflow orchestration). The discussion emphasized that scaling agents to millions of runs requires a robust evaluation philosophy. Key architectural shifts include adopting a data lake for unified first/third-party data access, implementing comprehensive multi-stage evaluations (from local dev to CI), and leveraging large context models (like Fable) to enable complex, self-iterating agent loops.
Key takeaways
-
Agentic Systems are Becoming Interfaces
Clay is evolving into an 'agent interface,' making all parts of the product available via CLI and public API. This creates a flywheel where internal and external agents share the same tools, allowing failures in tool invocation to provide user signal for continuous improvement.
-
Evals are Non-Negotiable for Scale
3:23
With billions of runs, comprehensive evaluation suites are critical. Clay's philosophy mandates different levels of testing: low-lift local dev checks, and staging/CI environments that mirror production harnesses.
-
Data Lake Foundation for Agents
To handle disparate data sources (LangChain traces, Snowflake, Postgres, ClickHouse), Clay is moving to a data lake architecture. This allows agents to run wild on unified first-party and third-party data while enabling safe shadow builds.
Technical details
-
Agent Capabilities
0s
Claygent is a research agent focused on web research and first-party data integration, running over 300 million runs monthly. Sculptor is the engineering agent used for building and orchestrating go-to-market workflows, handling over 100,000 messages weekly.
-
Evaluation Philosophy (The Four Quadrants)
419s
Clay's eval coverage moves beyond simple 'goldens.' They utilize structured checks for complex queries and employ multi-turn evals. The most useful finding was that deterministic, multi-turn evals were more reliable than managing a noisy agent acting as the user.
-
Addressing Evaluation Drift
579s
The biggest challenge is 'eval drift' (production vs. test data). Clay mitigates this by incorporating customer support tickets, human-annotated goldens for judging drift, and using use case classifiers to ensure evals cover real production scenarios.
-
Large Context Breakthrough
The availability of large context windows (e.g., Fable) allows agents to process massive datasets (e.g., 'Look at these 10,000 examples and find trends') in a single pass, moving beyond previous vibe-based analysis.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.