The Video Signal

AI Engineer

How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI

Published 2026-09-09 · Duration 22:26

Summary

The capacity of frontier LLMs to follow complex instructions has increased by an estimated tenfold in the last year, moving the instruction ceiling from 200-300 rules to 2,000 to 5,000 rules. This shift fundamentally changes prompt engineering from a 'compression problem' (fitting rules into a small budget) to a 'verification problem' (confirming the model actually obeyed the rules). Developers must now focus on robust output validation (evals) rather than worrying about the length of the skills file.

Download summary

Key takeaways

  1. Instruction Ceiling Increased by 10x 10:12

    A year ago, models struggled with 200 to 300 instructions. Current frontier models can handle up to 2,000 instructions, with the best models reaching 5,000 rules (6:12).

  2. Failure Mode Shift: From Forgetting to Refusing 20:00

    Models no longer fail by simply forgetting instructions. Failure modes are now diverse: Deepseek V4 Pro forgets; Claude Opus 4.7 refuses at the API level due to safety classifiers; Gemini 3.1 Pro runs out of thinking tokens; and GPT 5.5 writes a partial report and politely states the request is 'stupid' (12:00).

  3. Engineering Focus Shifts to Verification

    The challenge is no longer fitting rules into a small prompt (compression); it is verifying that the model actually followed all instructions, which requires external output checking (an eval) (14:00).

Technical details

  • IFScale Benchmark 158s

    The benchmark requires the model to write a business report while accurately including a list of specific, exact words (keywords). This measures 'accuracy' (percentage of rules followed) versus 'density' (number of rules) (2:58).

  • Model Performance (Current Frontier) 612s

    The test showed that current models (e.g., GPT 5.5, Gemini 3.1 Pro) scored 100% immediately on the original test, requiring the benchmark to be raised to 10,000 words to find the true ceiling (6:12).

  • Failure Mode Comparison 1000s

    Deepseek V4 Pro is 'traditional,' simply forgetting instructions. Claude Opus 4.7 fails due to a sensitive safety classifier triggering on random words (e.g., 'anthrax' and 'cyanide') (10:00). Gemini 3.1 Pro fails by exhausting its thinking tokens, resulting in no output (11:00).

Mentioned resources

  • IFScale (Benchmark/Test)
  • GPT 5.5 (LLM Model)
  • Claude Opus 4.7 (LLM Model)
  • Gemini 3.1 Pro (LLM Model)
  • Deepseek V4 Pro (LLM Model)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.