# How to Cut Your LLM Classification Costs by 90%

## Executive summary

This session details the Model Cascade architecture, a technique to drastically reduce LLM classification costs (by over 90%) while maintaining high accuracy (e.g., 95%+). Instead of running an expensive 'Oracle' model on every record, the cascade uses a cheap proxy model for initial predictions and only escalates to the Oracle when the proxy's confidence score is low. The optimal decision threshold ($\tau$) is found offline by simulating performance across samples.

## Key takeaways

- Model Cascade Architecture: The pipeline first sends every record (e.g., support ticket) to a cheap LLM API (the proxy/nano model). It generates a prediction $P(x)$ and a confidence score $C(x)$. A decision point then determines if the score $C(x)$ is high or low, routing only low-confidence records to the expensive 'Oracle' model.
- Cost Optimization Principle: The goal is to find a threshold $\tau$ that minimizes cost while maintaining a target accuracy (e.g., 95%) relative to the Oracle model. Lowering $\tau$ increases cost savings but risks dropping below the required accuracy.
- Threshold Determination: The optimal threshold $\tau$ is found offline by simulating the cascade using a labeled sample of tickets, calculating the resulting accuracy and cost for various potential thresholds $C(x_i)$.

## Technical details

- Model Cascade Algorithm Steps: 1. Sample Selection: Run the proxy model on a representative sample of records (e.g., 200-500 tickets) to obtain $P(x_i)$ and $C(x_i)$. 2. Labeling: Obtain ground truth labels $O(x_i)$ from the Oracle for the sample. 3. Threshold Simulation: Simulate the cascade using various thresholds $\tau$ on the sample data, recording resulting accuracy and cost pairs. 4. Optimization: Select the lowest threshold $\tau$ that meets or exceeds the target accuracy while minimizing total cost.
- Cascade Inference: For all tickets $X$: If $C(x) \ge \tau$, accept $P(x)$. Otherwise, run $O(x)$ (the Oracle).
- Statistical Guarantees: The procedure is a best estimate based on samples. For statistical guarantees, concentration inequalities can be used instead of naive sample accuracy calculation.

## Practical implications

- The technique is highly effective for classification tasks, such as routing support tickets or model routing in coding agents.
- It provides a data-driven way to estimate the required confidence threshold ($\tau$), unlike black-box model routers which lack transparency regarding accuracy guarantees.

## Topics

LLM Cost Reduction, Model Cascade, Classification, Confidence Scoring, AI Evals, AI Evals September 2026 cohort, Hamel Husain Website

Source: https://www.youtube.com/watch?v=FWFjLF_VoVI