SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe
Nubank achieved a massive acceleration in deploying AI agents—reporting up to 20x faster shipping—by shifting agent evaluation from relying solely on slow, expensive production data or manual curation to using grounded simulations. The core methodology involves running synthetic, multi-turn conversations against the agent within a specialized framework (Snowglobe), allowing teams to rapidly test models and variants in a continuous 'ship, observe, simulate, repeat' loop.
Key takeaways
-
Simulations Shortcircuit the Eval Bottleneck
10:44
Traditional agent evaluation data is complex (multi-turn, stateful trajectories) and time-consuming to gather via manual authoring or limited production traces. Simulations allow teams to generate thousands of multi-turn conversations quickly, reducing release cycles from weeks to hours.
-
The Simulation Loop: Ship, Observe, Simulate
A robust agent development cycle involves shipping the agent, observing real data, creating strong evaluations (using techniques like automated prompt optimization), and then using simulations to generate synthetic data. This simulated data is piped into the evaluation pipeline for continuous improvement.
-
Closing the Sim-to-Real Gap
To trust simulation results, it is crucial to establish metrics and processes that compare simulated performance against real production data (offline/online human review). The correlation between sim quality via evals and real data was found to be high.