# Local AI 201

## Executive summary

The session provides an advanced deep dive into local AI deployment, emphasizing that successful LLM inference is not determined by hardware capacity alone. Instead, it requires selecting a balanced stack comprising the right model, quantization level, and specialized inference engine (e.g., VLLM, llama.cpp) for the specific use case—whether single-user chat or high-concurrency agentic workflows. Key performance metrics like memory bandwidth are shown to be more critical than raw memory capacity when scaling up requests.

## Key takeaways

- Start with the Use Case, Not the Hardware: When designing a local AI solution, always begin by defining the required use case (e.g., single-user chatbot vs. 50-person agentic workflow). The hardware, model, and engine stack must then be selected to support that specific requirement.
- Memory Bandwidth is Critical for Throughput: For serving multiple requests (high throughput), memory bandwidth is often a more critical bottleneck than total memory capacity. For example, the RTX 5090 was shown to achieve significantly higher performance due to its high bandwidth compared to other devices.
- Engine Selection Dictates Performance Under Load: The choice of inference engine (e.g., VLLM vs. llama.cpp) and kernel optimization is paramount. Improperly selecting an engine can severely limit performance, causing a high-bandwidth device to perform worse than a lower-bandwidth machine under load.
- Local AI Offers Superior Privacy and Control: Running LLMs locally provides massive advantages in security, privacy, and control compared to relying on third-party cloud APIs. This allows users to fully tune the stack for long-term stability.

## Technical details

- LLM Inference Phases: Every request has two phases: Prefill (how quickly tokens/information are loaded into the LLM) and Decode (how quickly tokens are generated, which is typically measured as throughput).
- Inference Engine Functionality: A robust inference engine must act as a 'traffic cop,' managing multiple concurrent requests through memory management, kernel dispatching, and scheduling to prevent out-of-memory crashes.
- Local AI Stacks and Tools: Common engines include: llama.cpp (general purpose, friendly for variety), VLLM (optimized for multi-request workloads with continuous batching), and ODS (a full-stack deployment system that simplifies local AI setup).
- Hardware Performance Benchmarks: The demonstration compared multiple hardware devices (RTX Pro 6000, DGX Spark, Stricks Halo, M5 MacBook Pro) running the same model (Quen 3.627B at Q4). The results demonstrated that bandwidth speed directly correlates to hosting capacity.

## Practical implications

- For build engineers designing local AI services, prioritize optimizing the inference engine and kernel selection over simply maximizing hardware memory capacity.
- Utilize full-stack deployment systems like ODS to drastically reduce setup time (from months to minutes) while maintaining professional-grade customization for complex applications.
- When scaling from single-user chat to multi-user agentic workflows, performance bottlenecks will shift toward memory bandwidth and concurrent request handling capabilities.

## Topics

Local AI, LLM Inference Engines, Quantization, System Architecture, High Performance Computing (HPC), ODS, llama.cpp, VLLM, local.ai website

Source: https://www.youtube.com/watch?v=90k2HRT8Ito