# Are LLM Performance Benchmarks Reliable? — Ashok Chandrasekar & Jason Kramberger, Google

## Executive summary

This talk addresses the critical issue of unreliable LLM performance benchmarks, arguing that often the failure point is the benchmark harness itself, not the inference server. The speakers introduce `Inference Perf`, a CNCF project, which provides a standardized, high-fidelity solution. This tool solves common pitfalls—such as Python's Global Interpreter Lock (GIL) limitations, client-side latency inflation, and non-reproducible configurations—by offering declarative configuration, robust load generation across multiple processes, and combined client/server telemetry for accurate production-scale measurement.

## Key takeaways

- Benchmark Harness vs. System Under Test: Many performance discrepancies are incorrectly attributed to the server when the fault lies with the benchmark harness. Issues include GIL limitations in Python, client-side latency inflation (e.g., up to 58 seconds), and non-deterministic settings (e.g., model temperature set to zero).
- Need for High Metric Fidelity: Production-scale benchmarking requires observing both the system under test and the client's behavior simultaneously. The proposed solution, `Inference Perf`, provides client-side telemetry alongside server metrics to ensure accurate diagnosis.
- Standardization of Workloads: The `LLMD` project provides a published workload catalog defining state-of-the-art inference tasks (e.g., agentic generation, tree of thought, batch summarization) in a standardized, declarative format, improving reproducibility across different tools.

## Technical details

- Inference Perf Architecture: `Inference Perf` is a CNCF project that uses a main process to schedule requests (Poisson, constant rate, or fixed concurrency) and fans them across multiple worker processes. This architecture solves the GIL problem in Python and ensures accurate load generation up to 5,000 QPS.
- LLM Performance Metrics: Reliable benchmarking requires measuring not just single QPS, but a suite of metrics including time to first token, time per output token, and adherence to Service Level Objectives (SLOs) like P90.
- Workload Configuration: The system supports complex, reproducible configurations, including defining input/output length distributions for multi-turn conversation replay, which is crucial for simulating real-world production workloads.

## Practical implications

- When benchmarking LLM inference stacks, always verify that the benchmark harness itself is not introducing artifacts (e.g., GIL limitations, latency inflation).
- Adopt standardized, declarative tools like `Inference Perf` to ensure reproducible results across different environments and teams.
- Focus on simulating complex, real-world workloads (e.g., multi-turn conversations, agentic generation) rather than simple synthetic QPS tests.

## Topics

LLM Inference, Performance Benchmarking, Distributed Systems, CNCF, High-Throughput Computing, Inference Perf, LLMD, Prism

Source: https://www.youtube.com/watch?v=l1-D89bAuOA