# Between the Layers– Interpreting Large Language Models - Michelle Frost - NDC Toronto 2026

## Executive summary

This talk provides a deep dive into AI Interpretability, arguing that for Large Language Models (LLMs) to achieve reliability and trust, we must move beyond external explanations (explainability) and understand the internal mechanisms (interpretability). The speaker outlines advanced research—particularly Anthropic's work on mechanistic interpretability—that focuses on decomposing model activations into sparse, single-meaning features. This shift allows engineers to treat LLMs not as black boxes, but as auditable systems that can be tested, monitored, and debugged at the circuit level.

## Key takeaways

- Interpretability vs. Explainability: Interpretability is the 'inside view' (understanding which internal parts wrote the answer), while explainability is the 'outside view' (a human-readable reason for the output). Mechanistic evidence comes from ablating suspected units and observing predictable behavior changes, whereas a salency map or local surrogate provides an account.
- The Need for Operational Trust: To build trust in LLMs, we must make concepts like truthfulness and calibration operational. This requires understanding the model's internal mechanisms to ensure consistency (robustness) and prevent unsafe answers or agreement bias (syncopancy).
- The Interpretability Litmus Test: A system should achieve 'understanding' if it can be able to: Name the units, Test their causal importance, Predict counterfactual failures (e.g., misinformation), Instrument them in production, and allow another engineer to Reproduce findings.
- Modern Mechanistic Approaches: Advanced research uses tools like Sparse Autoencoders (SAEs) and overcomplete dictionaries to decompose model activations into 'features'—internal signals that track a single concept (monosemanticity). This allows for the mapping of complex behaviors to specific, nameable circuits.
- The Evolution of AI Safety: Interpretability is presented as 'ethical infrastructure.' By understanding internal mechanisms (e.g., default circuits that trigger refusals), we can move from simply measuring outputs to actively debugging and governing the system's behavior.

## Technical details

- Model Architecture Components: Transformers process text by mapping tokens into vectors using an embedding matrix. Within a transformer block, each token creates three versions of itself: the Query (what it looks for), the Key (what it offers), and the Value (its content). Information is passed through attention mechanisms and Multi-Layer Perceptrons (MLPs) into a residual stream.
- Mechanistic Interpretability Tools: Key tools include: **Sparse Autoencoders (SAEs)**, which are used to unmix overlapped signals; **Attribution Graphs**, which map features influencing tokens across layers; and **Default Circuits**, which represent inherent safety behaviors or refusals.
- Causal Testing: The core method involves generating a mechanistic story (a hypothesis) and then using probes or steering ablations to test if the predicted behavior change occurs when internal units are manipulated. This provides causal evidence.
- Fairness Assessment: Counterfactual edits—the smallest realistic changes that flip a model's decision (e.g., flipping gender pronouns)—are crucial for proving bias, as demonstrated by the Compass dataset analysis.

## Practical implications

- Implement a systematic approach to AI governance by treating interpretability as 'ethical infrastructure.'
- Shift testing focus from merely evaluating outputs (accuracy) to performing internal audits and causal tests on model mechanisms.
- When developing systems, always document the specific units/features that contribute to an output, rather than relying solely on high-level rationales.
- In build engineering, incorporate 'mechanism level monitors' into CI/CD pipelines to track internal behavior changes across model versions.

## Topics

AI Interpretability, Large Language Models (LLMs), Mechanistic AI, Causal Inference, Model Governance and Ethics, Anthropic Research Papers, Neuronipedia

Source: https://www.youtube.com/watch?v=17QQKoQFRJU