Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google
The deployment of AI on edge devices (robotics, mobile phones) is constrained primarily by DRAM cost, necessitating the development of tiny LLMs. The talk details strategies—including aggressive quantization and fine-tuning—to shrink models like Gemma to run efficiently on low-power hardware (e.g., Raspberry Pi, Qualcomm NPUs). Tiny models (<500M parameters) are ideal for wide-scale deployment but often require specialized tasks (ASR, vision, function calling) and robust synthetic data generation for fine-tuning.
Key takeaways
-
DRAM Cost is the Primary Edge Constraint
1:45
The main challenge in edge AI deployment is not compute power, but DRAM cost. Mobile phone manufacturers are reducing on-device RAM, making model size critical. (3:25)
-
Quantization Enables Small Footprints
6:08
Models can be aggressively quantized (e.g., Gemma to 2.9 bits per weight) and optimized using techniques like per-layer embeddings to minimize memory footprint, allowing deployment on devices with limited RAM. (6:08)
-
Tiny Models Require Fine-Tuning
14:20
While small models are easy to use via zero-shot prompting, achieving high reliability for specific tasks like function calling or voice dictation requires fine-tuning using synthetic data sets. (14:28)
-
Voice-to-Function Calling is Key
16:00
Fine-tuning a small model to handle both ASR and function calling allows for robust, offline interaction on low-tier IoT/edge devices, which is crucial where complex UIs are difficult. (14:28)