What Is LLM Inference? How AI Models Generate Responses

Summary

LLM inference is the process by which a static, trained model (a set of weights) generates real-time responses to new inputs, forming the backbone of modern chatbots and AI applications. This process is managed by an inference engine (e.g., vLLM, Ollama) and involves several critical steps: tokenization, hardware routing, and efficient batching. The core computation is divided into two distinct phases—the compute-bound prefill phase and the memory bandwidth-bound decode phase—each requiring specific optimization to ensure low latency and high throughput.

Technical details

• Tokenization 215s

  This initial step translates raw text into tokens (which can be words, parts of words, or punctuation) and maps them to arbitrary token IDs. The model only processes the stream of IDs, and this step determines the cost per token.

• Prefill Phase (Compute-Bound) 390s

  When a request enters the model, the prefill phase reads the entire prompt at once to build an understanding, creating a Key-Value (KV) cache. This step is compute-bound, meaning its speed is limited by the raw computational power of the hardware.

• Decode Phase (Memory Bandwidth Bound) 420s

  The decode phase generates the response one token at a time. It uses the KV cache and runs a forward pass to produce a probability distribution over the vocabulary. This step is memory bandwidth bound, meaning its speed is limited by how fast data can move out of memory.

• Latency Metrics 460s

  Two key metrics are used to measure LLM latency: Time to First Token (TTFT), which measures the prefill step duration; and Inter Token Latency (ITL), which measures the time between subsequent tokens during the decode phase.

Mentioned resources

• vLLM (Inference Engine)
• Ollama (Inference Engine)
• Llama CPP (Inference Engine)

Channel & topics

• IBM Technology
• LLM Inference
• AI Model Serving
• GPU Optimization
• Tokenization
• KV Cache
• vLLM
• Ollama
• Llama CPP