# Stop Guessing Your vLLM Config: Benchmarking LLMs on Runpod - Chris Vela, Runpod

## Executive summary

This talk addresses the common pitfall of LLM deployment: guessing configuration parameters. Chris Vela demonstrates that moving LLMs from R&D to production requires data-driven benchmarking. He introduces BenchPod, a dashboard built on vLLM, which allows engineers to test various configurations (like tuning `max batch tokens`) and measure performance metrics (Time to First Token, Inter-Token Latency) without requiring disruptive restarts, ensuring reliable, scalable deployment.

## Key takeaways

- Benchmarking is Crucial for Production Readiness: Do not rely on default vLLM recipes. Before deployment, systematically test configurations based on the use case (e.g., coding agent vs. summarization) and expected concurrency to ensure optimal performance and cost efficiency.
- BenchPod Streamlines LLM Testing: BenchPod is a centralized benchmarking dashboard built on vLLM that allows rapid testing of configurations and metrics without the time sink of cold starts or image restarts.
- Optimization Knobs Matter for Latency: Tuning parameters like `max batch tokens` can significantly decrease inter-token latency, leading to much faster job completion times compared to base configurations.
- Hardware and Versioning are Critical: The LLM ecosystem moves quickly, requiring attention to specific versions (e.g., using the correct vLLM version for Kimi K3) and hardware capabilities (e.g., running Kimi K3 on a B300).

## Technical details

- vLLM Benchmarking: The speaker proposes BenchPod, a template built on vLLM, which centralizes metrics and allows quick configuration changes, solving the pain point of waiting for cold starts.
- Performance Optimization: Optimizing the `max batch tokens` parameter was demonstrated to decrease inter-token latency, significantly speeding up long-running tasks compared to default vLLM recipes.
- Deployment Workflow: The process moves from R&D to production by: 1) Selecting a GPU pod, 2) Comparing configs, 3) Testing with an agent, and 4) Optimizing metrics (TTF, ITL, TPOT) before scaling out.
- System Resilience: The platform supports self-healing capabilities, ensuring that instances (like Kimi K3) remain operational even after hardware failures.

## Practical implications

- Implement dedicated benchmarking stages in your CI/CD pipeline for LLM services, rather than relying on basic default configurations.
- Focus benchmarking efforts on specific metrics (Time to First Token, Inter-Token Latency) relevant to your application's use case (e.g., low latency for chat agents).
- Utilize dedicated benchmarking tools (like BenchPod) to minimize downtime and cold start overhead when iterating on model configurations.
- When scaling, consider the trade-off between using the most powerful 'frontier' model and a smaller, more efficient model that meets the task requirements.

## Topics

LLM Benchmarking, vLLM, Build Engineering, CI/CD, GPU Optimization, AI Deployment, Runpod, BenchPod

Source: https://www.youtube.com/watch?v=P_0_Th_ZzIg