Scalability

The deployment path moves from general-purpose, proprietary models (low control) to highly customized, controlled deployments that require explicit management of scale and cost.

Ideal for rapid model iteration and deployment, offering automatic scaling, observability, and native Weights & Biases tracing without requiring infrastructure management.

Provides a controlled, production-ready path for custom models, allowing users to bring their own weights, select GPUs, and use open runtimes while CoreWeave handles operations and cost visibility.

Offers maximum control for ultra-high scale and latency-critical workloads. It is a fully managed service on CoreWeave Kubernetes Service (CKS), supporting runtimes like LLMD and Nvidia Dynamo for self-hosted, distributed inference.

Agents are categorized into 'agents we use' (productivity tools, coding assistants) and 'agents we build' (production-scale systems). The approach for built agents requires careful architectural planning.

A harness is defined by subtraction: take an agent and remove the model component; everything left over is the harness. This includes the infrastructure, skills, and tools.

For production agents, the harness must manage complex concerns like loop management, scaling, payments, identity, runtime, context management, and observability. Attempting to containerize everything together is incorrect for high scale.

Build engineers must avoid 'slop ops' (clicking around a console to deploy resources). Instead, agents must build infrastructure using Infrastructure as Code (IaC) to maintain ownership and control over cloud deployments.

To ensure system reliability, it is crucial to simulate low-level failures (like database connection loss) rather than just mocking components. The use of custom proxies or tools like `GDB` allows testing the application's failure handling at the connection layer, uncovering issues that are difficult to reproduce in production.

When designing large-scale systems, especially those involving multiple round trips (like navigating a tree structure on S3), optimization must focus on the P99 latency, not just the average (P50). This is critical for accurate performance prediction.

The demand curve for CPUs is shifting right due to AI and Reinforcement Learning (RL) workloads, which require significant CPU cycles for training and general-purpose agent execution. This scarcity is a major constraint that cloud providers are managing through power allocation.

The principle of 'simplicity above everything' was key to the development philosophy, allowing for rapid iteration and focusing on core functionality rather than complex features. This approach helped achieve significant cost reductions (e.g., reducing a client's bill by 95%).