# What Is LLM Inference? How AI Models Generate Responses

## Executive summary

LLM inference is the process by which a static, trained model (a set of weights) generates real-time responses to new inputs, forming the backbone of modern chatbots and AI applications. This process is managed by an inference engine (e.g., vLLM, Ollama) and involves several critical steps: tokenization, hardware routing, and efficient batching. The core computation is divided into two distinct phases—the compute-bound prefill phase and the memory bandwidth-bound decode phase—each requiring specific optimization to ensure low latency and high throughput.

## Key takeaways

- Inference vs. Training: Inference is the process of running a trained model against new input (happening every request), while training is a one-time process that tunes the static weights. Inference is crucial because it reoccurs with every request and often costs more over time than training.
- The Role of the Inference Engine: An inference engine (like vLLM or Ollama) manages the process by scheduling requests, managing GPU resources, and turning a pile of weights into a service that can handle requests reliably, quickly, and at scale.
- Optimizing Throughput with Batching: To prevent wasting GPU resources, modern engines use continuous batching, where a scheduler slots a new request into the hardware immediately after another request finishes, keeping the GPUs near full utilization.

## Technical details

- Tokenization: This initial step translates raw text into tokens (which can be words, parts of words, or punctuation) and maps them to arbitrary token IDs. The model only processes the stream of IDs, and this step determines the cost per token.
- Prefill Phase (Compute-Bound): When a request enters the model, the prefill phase reads the entire prompt at once to build an understanding, creating a Key-Value (KV) cache. This step is compute-bound, meaning its speed is limited by the raw computational power of the hardware.
- Decode Phase (Memory Bandwidth Bound): The decode phase generates the response one token at a time. It uses the KV cache and runs a forward pass to produce a probability distribution over the vocabulary. This step is memory bandwidth bound, meaning its speed is limited by how fast data can move out of memory.
- Latency Metrics: Two key metrics are used to measure LLM latency: Time to First Token (TTFT), which measures the prefill step duration; and Inter Token Latency (ITL), which measures the time between subsequent tokens during the decode phase.

## Practical implications

- Optimizing the inference stack is critical for reducing operational costs, as inference costs accumulate with every single request.
- Understanding the distinction between compute-bound (Prefill) and memory bandwidth-bound (Decode) phases allows engineers to target specific hardware or software optimizations.
- Using inference engines that expose standard OpenAI compatible endpoints simplifies application development, allowing for easy model swapping without code changes.

## Topics

LLM Inference, AI Model Serving, GPU Optimization, Tokenization, KV Cache, vLLM, Ollama, Llama CPP

Source: https://www.youtube.com/watch?v=615-yWyS1n0