# Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

## Executive summary

The deployment of AI on edge devices (robotics, mobile phones) is constrained primarily by DRAM cost, necessitating the development of tiny LLMs. The talk details strategies—including aggressive quantization and fine-tuning—to shrink models like Gemma to run efficiently on low-power hardware (e.g., Raspberry Pi, Qualcomm NPUs). Tiny models (<500M parameters) are ideal for wide-scale deployment but often require specialized tasks (ASR, vision, function calling) and robust synthetic data generation for fine-tuning.

## Key takeaways

- DRAM Cost is the Primary Edge Constraint: The main challenge in edge AI deployment is not compute power, but DRAM cost. Mobile phone manufacturers are reducing on-device RAM, making model size critical. (3:25)
- Quantization Enables Small Footprints: Models can be aggressively quantized (e.g., Gemma to 2.9 bits per weight) and optimized using techniques like per-layer embeddings to minimize memory footprint, allowing deployment on devices with limited RAM. (6:08)
- Tiny Models Require Fine-Tuning: While small models are easy to use via zero-shot prompting, achieving high reliability for specific tasks like function calling or voice dictation requires fine-tuning using synthetic data sets. (14:28)
- Voice-to-Function Calling is Key: Fine-tuning a small model to handle both ASR and function calling allows for robust, offline interaction on low-tier IoT/edge devices, which is crucial where complex UIs are difficult. (14:28)

## Technical details

- Model Scaling & Constraints: The shift from large LLMs to tiny models (<500M parameters) is driven by the need for deployment on older laptops and consumer edge devices, overcoming DRAM cost limitations. (12:07)
- Quantization Techniques: Optimizing memory footprint involves using mixed quantization (e.g., two-bit, four-bit, eight-bit) to reduce the required storage for weights and enable deployment on low-RAM devices. (6:08)
- Performance Benchmarks: A 2B parameter model running on a Raspberry Pi achieves approximately 7.6 tokens/second decode (without MTP). On a Qualcomm NPU, it can achieve ~31 tokens/second pre-fill and fast real-time inference for vision tasks. (9:30)
- Function Calling Reliability: Fine-tuning models like Function Gemma allows conversion of arbitrary free text into specific function calls with high reliability (>86%), enabling complex agentic behavior on the edge. (14:28)

## Practical implications

- The constraint on edge AI is shifting from compute to memory (DRAM).
- For wide-scale deployment, fine-tuning small models using synthetic data is the most effective playbook.
- Offline voice dictation and visual intelligence are achievable features for consumer devices without cloud subscriptions.

## Topics

Edge AI, Tiny LLMs, Quantization, Function Calling, Robotics, AI Edge Gallery, Google AI Edge Team GitHub, Mobile Actions Dataset

Source: https://www.youtube.com/watch?v=hacEQHHhu2Q