SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe
Summary
Nubank achieved a massive acceleration in deploying AI agents—reporting up to 20x faster shipping—by shifting agent evaluation from relying solely on slow, expensive production data or manual curation to using grounded simulations. The core methodology involves running synthetic, multi-turn conversations against the agent within a specialized framework (Snowglobe), allowing teams to rapidly test models and variants in a continuous 'ship, observe, simulate, repeat' loop.
Key takeaways
-
Simulations Shortcircuit the Eval Bottleneck
10:44
Traditional agent evaluation data is complex (multi-turn, stateful trajectories) and time-consuming to gather via manual authoring or limited production traces. Simulations allow teams to generate thousands of multi-turn conversations quickly, reducing release cycles from weeks to hours.
-
The Simulation Loop: Ship, Observe, Simulate
A robust agent development cycle involves shipping the agent, observing real data, creating strong evaluations (using techniques like automated prompt optimization), and then using simulations to generate synthetic data. This simulated data is piped into the evaluation pipeline for continuous improvement.
-
Closing the Sim-to-Real Gap
To trust simulation results, it is crucial to establish metrics and processes that compare simulated performance against real production data (offline/online human review). The correlation between sim quality via evals and real data was found to be high.
Technical details
-
Agent Evaluation Data Complexity
349s
Unlike single-turn QA, agent evaluation requires multi-turn data points that are full trajectories, including internal tool calls and maintaining consistent state across all interactions.
-
Snowglobe Simulation Process
694s
The Snowglobe SDK wraps the agent without code changes. It requires inputs defining personas, use cases, and data points, generating synthetic conversations with mocked tools (e.g., fake addresses/CCs) that maintain consistency across the simulated execution.
-
Performance Metrics
270s
Evaluation metrics can be built using LLM-as-a-judge classifiers, which are aligned with human judgment and refined through auto-optimization techniques.
Mentioned resources
- KD paper
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.