Understand the Gemma 4 model family
The Gemma 4 family is a set of multimodal, open-source large language models available in five sizes across four architectures. The models range from efficient, dense, on-device options (E2B, E4B) utilizing Per-Layer Embeddings (PLE), to advanced architectures like the encoder-free 12B model, the Mixture-of-Experts (MoE) 26B model, and the highly capable 31B dense model. Each size is optimized for specific use cases, including dedicated vision and audio processing.
Key takeaways
-
E2B and E4B Models
These smaller, dense models are optimized for on-device usage and utilize Per-Layer Embeddings (PLE), which are lookup tables for processing queries. They process audio and images using dedicated encoders.
-
12B Model Architecture
This model is suitable for high-end laptops and employs an encoder-free method, removing dedicated encoders (e.g., audio encoder) and directly projecting audio to the LLM.
-
26B Model (MoE)
This Mixture-of-Experts (MoE) model uses 26 billion parameters but only activates four billion at any given time ('A' for active). It features a larger vision encoder, making it ideal for difficult vision tasks.
-
31B Model
This is the most capable dense model in the Gemma 4 family. It utilizes a larger vision encoder and is presented as the top-tier model.