# Understand the Gemma 4 model family

## Executive summary

The Gemma 4 family is a set of multimodal, open-source large language models available in five sizes across four architectures. The models range from efficient, dense, on-device options (E2B, E4B) utilizing Per-Layer Embeddings (PLE), to advanced architectures like the encoder-free 12B model, the Mixture-of-Experts (MoE) 26B model, and the highly capable 31B dense model. Each size is optimized for specific use cases, including dedicated vision and audio processing.

## Key takeaways

- E2B and E4B Models: These smaller, dense models are optimized for on-device usage and utilize Per-Layer Embeddings (PLE), which are lookup tables for processing queries. They process audio and images using dedicated encoders.
- 12B Model Architecture: This model is suitable for high-end laptops and employs an encoder-free method, removing dedicated encoders (e.g., audio encoder) and directly projecting audio to the LLM.
- 26B Model (MoE): This Mixture-of-Experts (MoE) model uses 26 billion parameters but only activates four billion at any given time ('A' for active). It features a larger vision encoder, making it ideal for difficult vision tasks.
- 31B Model: This is the most capable dense model in the Gemma 4 family. It utilizes a larger vision encoder and is presented as the top-tier model.

## Technical details

- Model Architecture & Efficiency: The Gemma 4 family includes five sizes across four architectures. Smaller models (E2B, E4B) are dense and use Per-Layer Embeddings (PLE). The 12B model uses an encoder-free method, projecting modalities directly to the LLM.
- Mixture-of-Experts (MoE): The 26B model is an MoE architecture, meaning it activates only a subset of its parameters (4B) despite having 26B total parameters.

## Practical implications

- The choice of model (E2B, 12B, 26B, 31B) should be dictated by the required performance level and deployment environment (e.g., on-device vs. high-end laptop).
- The 26B MoE model is specifically recommended for complex vision tasks due to its larger vision encoder and efficient parameter activation.

## Topics

Large Language Models, Multimodal AI, Model Architecture, Mixture-of-Experts (MoE), On-Device AI, Gemma 4 Resources

Source: https://www.youtube.com/watch?v=4oxA9o_OmWo