# Per-Layer Embeddings (PLE) in Gemma 4 explained

## Executive summary

This video explains Per-Layer Embeddings (PLE), a technique used in Google's Gemma 4 E2B and E4B models. PLE enhances token representation by providing a unique, layer-specific embedding for every token, allowing the model to capture more diverse information. Crucially, this boosts model performance and representational power without requiring an increase in trainable parameters during computation.

## Key takeaways

- Function of Per-Layer Embeddings (PLE): PLE gives each token a unique, layer-specific embedding, meaning the token 'hi' has a different representation on layer one versus layer six, leading to more diverse representation.
- Efficiency and Parameters: While the full embedding table is large, only a small portion is needed during inference. The architecture and encoders are considered 'effective parameters' because they are the only parameters actively used for computation.
- Performance Improvement: PLE increases model performance and representational power by adding layer-specific embeddings without actually increasing the total number of parameters.

## Technical details

- Gemma 4 Architecture: The models discussed are Gemma 4 E2B and E4B. The 'E' in E2B references that the model learns from the token embedding layers.
- Token Embedding vs. PLE: A standard token embedding layer provides one representation for a token (e.g., 'hi'). PLE expands this by adding layer-specific embeddings, providing a more powerful, multi-dimensional representation.
- Effective Parameters: The computation relies on 'effective parameters' (the architecture and encoders) because the large Per-Layer Embeddings and token embedding table are stored on flash storage and are not needed during the active computation phase.

## Practical implications

- Enables high-performance language models (Gemma 4) with enhanced representational power.
- Optimizes model deployment by boosting performance without increasing the computational parameter count.
- Suggests efficient memory management by storing large embedding tables on flash storage rather than requiring them during inference.

## Topics

Gemma 4, Per-Layer Embeddings (PLE), Token Embedding, Large Language Models (LLMs), Model Optimization, Effective Parameters, Further PLE Resources

Source: https://www.youtube.com/watch?v=s82Ho5HdltE