# Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio

## Executive summary

This talk details the architecture and trade-offs involved in building an LLM Gateway—a middleware layer managing routing, authentication, fallbacks, and governance between applications and model providers. The core challenge is that availability, latency, guardrails, and cost cannot all be maximized simultaneously during degradation. Key architectural advice includes preferring per-request fallbacks over blind retries or circuit breakers, tracking P99 latency per model/route (not gateway-wide), and treating guardrails as unreliable services requiring fail-open/fail-close decisions.

## Key takeaways

- LLM Gateway Core Trade-offs: An LLM gateway must manage a permanent fight between availability, latency, guardrails, and cost. Degradation forces the system to sacrifice one of these four elements (1:21).
- Fallback Strategy: Instead of blind retries or circuit breakers, implement per-request fallbacks (trying Model A, then B if A fails) or parallel requests for low-latency needs. This prevents multiplying cost and tail latencies (2:33).
- Latency Measurement Best Practices: Do not measure gateway-wide latency; instead, track P99 per model and per route. Setting timeouts on a per-model/per-route basis is critical to preventing silent outages (7:08).
- Guardrails as Services: Treat guardrails (e.g., toxicity filters, PII checks) like any other service that can fail. Teams must decide in advance whether the system should 'fail open' (serve request despite failure) or 'fail close' (block request) (10:40).
- Centralization vs. Governance: While many teams seek a central gateway, the underlying need is often centralized governance (e.g., cost tracking, rate limiting), which can be achieved by decentralizing traffic while centralizing policy enforcement (13:02).

## Technical details

- LLM Gateway Functionality: The gateway acts as middleware providing routing, authentication, fallback logic, rate limits, and governance between applications and model providers (0:49).
- Failure Handling Patterns: Avoid traditional retries or circuit breakers for LLMs due to high cost and latency impact. Prefer per-request fallbacks or parallel requests if extreme low latency is required (2:33).
- Performance Metrics: Focus on tracking P99 per model/per route, especially for mixed workloads involving embedding, classification, and reasoning models. Reasoning models are highly non-deterministic, potentially taking 2 seconds to 60 seconds (8:17).
- Guardrail Placement: Guardrails can be implemented via pre-hooks (safest, adds serial latency), parallel hooks (best for structured output), or post-hooks (best for auditing/monitoring) (10:40).
- System Resilience: Implement load shedding and ensure API keys are segregated per route/use case to prevent noisy tenants from causing service degradation (14:11).

## Practical implications

- Design the LLM gateway to explicitly manage trade-offs between availability, latency, guardrails, and cost.
- Prioritize per-request fallbacks over global retries or circuit breakers for robust failure handling.
- Implement granular timeouts (per model/per route) as a primary defense against silent outages.
- Ensure the fallback provider has higher capacity headroom than the primary provider to serve as a reliable last line of defense.
- Decouple centralized governance (cost tracking, rate limiting) from centralizing all traffic flow.

## Topics

LLM Gateways, System Architecture, Reliability Engineering, Distributed Systems, AI/ML Operations

Source: https://www.youtube.com/watch?v=zrZ1amZBSPw