# SimulationMaxxing: How Nubank ships agents 20× faster with simulations — Shreya Rajpal, Snowglobe

## Executive summary

Nubank achieved a massive acceleration in deploying AI agents—reporting up to 20x faster shipping—by shifting agent evaluation from relying solely on slow, expensive production data or manual curation to using grounded simulations. The core methodology involves running synthetic, multi-turn conversations against the agent within a specialized framework (Snowglobe), allowing teams to rapidly test models and variants in a continuous 'ship, observe, simulate, repeat' loop.

## Key takeaways

- Simulations Shortcircuit the Eval Bottleneck: Traditional agent evaluation data is complex (multi-turn, stateful trajectories) and time-consuming to gather via manual authoring or limited production traces. Simulations allow teams to generate thousands of multi-turn conversations quickly, reducing release cycles from weeks to hours.
- The Simulation Loop: Ship, Observe, Simulate: A robust agent development cycle involves shipping the agent, observing real data, creating strong evaluations (using techniques like automated prompt optimization), and then using simulations to generate synthetic data. This simulated data is piped into the evaluation pipeline for continuous improvement.
- Closing the Sim-to-Real Gap: To trust simulation results, it is crucial to establish metrics and processes that compare simulated performance against real production data (offline/online human review). The correlation between sim quality via evals and real data was found to be high.

## Technical details

- Agent Evaluation Data Complexity: Unlike single-turn QA, agent evaluation requires multi-turn data points that are full trajectories, including internal tool calls and maintaining consistent state across all interactions.
- Snowglobe Simulation Process: The Snowglobe SDK wraps the agent without code changes. It requires inputs defining personas, use cases, and data points, generating synthetic conversations with mocked tools (e.g., fake addresses/CCs) that maintain consistency across the simulated execution.
- Performance Metrics: Evaluation metrics can be built using LLM-as-a-judge classifiers, which are aligned with human judgment and refined through auto-optimization techniques.

## Practical implications

- Implement a dedicated simulation layer (like Snowglobe) into the CI/CD pipeline for agent development to drastically reduce testing time.
- Structure evaluation data collection to handle multi-turn, stateful trajectories rather than single Q&A pairs.
- Establish a feedback loop that uses simulated data alongside real production traces to continuously optimize and derisk agent updates before live deployment.

## Topics

AI Agents, Evaluation (Evals), Simulation, Build Engineering, LLM Deployment, KD paper

Source: https://www.youtube.com/watch?v=KMR_RBoCa4M