The Video Signal

AI Engineer

Why Large? Tiny LMs & Agents on Edge/Robotics — Cormac Brick, Google

Published 2026-07-25 · Duration 21:45

Summary

The deployment of AI on edge devices (robotics, mobile phones) is constrained primarily by DRAM cost, necessitating the development of tiny LLMs. The talk details strategies—including aggressive quantization and fine-tuning—to shrink models like Gemma to run efficiently on low-power hardware (e.g., Raspberry Pi, Qualcomm NPUs). Tiny models (<500M parameters) are ideal for wide-scale deployment but often require specialized tasks (ASR, vision, function calling) and robust synthetic data generation for fine-tuning.

Download summary

Key takeaways

  1. DRAM Cost is the Primary Edge Constraint 1:45

    The main challenge in edge AI deployment is not compute power, but DRAM cost. Mobile phone manufacturers are reducing on-device RAM, making model size critical. (3:25)

  2. Quantization Enables Small Footprints 6:08

    Models can be aggressively quantized (e.g., Gemma to 2.9 bits per weight) and optimized using techniques like per-layer embeddings to minimize memory footprint, allowing deployment on devices with limited RAM. (6:08)

  3. Tiny Models Require Fine-Tuning 14:20

    While small models are easy to use via zero-shot prompting, achieving high reliability for specific tasks like function calling or voice dictation requires fine-tuning using synthetic data sets. (14:28)

  4. Voice-to-Function Calling is Key 16:00

    Fine-tuning a small model to handle both ASR and function calling allows for robust, offline interaction on low-tier IoT/edge devices, which is crucial where complex UIs are difficult. (14:28)

Technical details

  • Model Scaling & Constraints 727s

    The shift from large LLMs to tiny models (<500M parameters) is driven by the need for deployment on older laptops and consumer edge devices, overcoming DRAM cost limitations. (12:07)

  • Quantization Techniques 368s

    Optimizing memory footprint involves using mixed quantization (e.g., two-bit, four-bit, eight-bit) to reduce the required storage for weights and enable deployment on low-RAM devices. (6:08)

  • Performance Benchmarks 458s

    A 2B parameter model running on a Raspberry Pi achieves approximately 7.6 tokens/second decode (without MTP). On a Qualcomm NPU, it can achieve ~31 tokens/second pre-fill and fast real-time inference for vision tasks. (9:30)

  • Function Calling Reliability 903s

    Fine-tuning models like Function Gemma allows conversion of arbitrary free text into specific function calls with high reliability (>86%), enabling complex agentic behavior on the edge. (14:28)

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.