# Building Turbopuffer: Gergely Orosz (@pragmaticengineer ) × Simon Eskildsen (CEO)

## Executive summary

The discussion provides a deep dive into building highly scalable and resilient infrastructure, focusing heavily on state management challenges in large-scale distributed systems. Key engineering lessons include moving beyond simple benchmarks to model real-world failure modes (e.g., connection layer failures), optimizing for P99 latency when using object storage like S3, and adapting architecture to current cloud constraints, particularly the increasing demand for CPUs driven by AI/RL workloads.

## Key takeaways

- Modeling Failure in CI: To ensure system reliability, it is crucial to simulate low-level failures (like database connection loss) rather than just mocking components. The use of custom proxies or tools like `GDB` allows testing the application's failure handling at the connection layer, uncovering issues that are difficult to reproduce in production.
- The Importance of P99 Latency: When designing large-scale systems, especially those involving multiple round trips (like navigating a tree structure on S3), optimization must focus on the P99 latency, not just the average (P50). This is critical for accurate performance prediction.
- CPU Scarcity in AI Workloads: The demand curve for CPUs is shifting right due to AI and Reinforcement Learning (RL) workloads, which require significant CPU cycles for training and general-purpose agent execution. This scarcity is a major constraint that cloud providers are managing through power allocation.
- Architectural Simplicity Wins: The principle of 'simplicity above everything' was key to the development philosophy, allowing for rapid iteration and focusing on core functionality rather than complex features. This approach helped achieve significant cost reductions (e.g., reducing a client's bill by 95%).

## Technical details

- Database Resilience & Failure Simulation: The discussion highlighted techniques for testing distributed state, including implementing a Layer 7 proxy (like `Toxyroxy`) that sits between the application and the database. This allows simulating failures like data corruption or connection loss without relying on simple mocks.
- Vector Storage & S3 Optimization: For storing vectors (used in AI/search), the architecture can leverage Amazon S3 for durability. Performance optimization involves clustering algorithms and minimizing round trips, as P99 latency is dictated by the cumulative time of multiple requests.
- Cloud Infrastructure Constraints: Scaling in hyperscale cloud environments (AWS, GCP, Azure) requires careful planning for resource allocation. Current constraints include CPU scarcity due to AI/RL demand and the need to manage complex machine types (e.g., C4s on GCP) rather than relying solely on single instance types.

## Practical implications

- When benchmarking infrastructure, do not rely solely on simple metrics; model complex failure scenarios and measure P99 latency.
- Understand the unit economics of cloud services (e.g., comparing S3 costs vs. traditional database hosting) to justify architectural decisions.
- In distributed systems design, always plan for component failures at the connection layer, not just the application layer.

## Topics

Distributed Systems, Database Engineering, Cloud Infrastructure, AI/ML Infra, Scalability, Turbopuffer, Napkin Math Project

Source: https://www.youtube.com/watch?v=jQDXzEVHMSE