# How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI

## Executive summary

The capacity of frontier LLMs to follow complex instructions has increased by an estimated tenfold in the last year, moving the instruction ceiling from 200-300 rules to 2,000 to 5,000 rules. This shift fundamentally changes prompt engineering from a 'compression problem' (fitting rules into a small budget) to a 'verification problem' (confirming the model actually obeyed the rules). Developers must now focus on robust output validation (evals) rather than worrying about the length of the skills file.

## Key takeaways

- Instruction Ceiling Increased by 10x: A year ago, models struggled with 200 to 300 instructions. Current frontier models can handle up to 2,000 instructions, with the best models reaching 5,000 rules (6:12).
- Failure Mode Shift: From Forgetting to Refusing: Models no longer fail by simply forgetting instructions. Failure modes are now diverse: Deepseek V4 Pro forgets; Claude Opus 4.7 refuses at the API level due to safety classifiers; Gemini 3.1 Pro runs out of thinking tokens; and GPT 5.5 writes a partial report and politely states the request is 'stupid' (12:00).
- Engineering Focus Shifts to Verification: The challenge is no longer fitting rules into a small prompt (compression); it is verifying that the model actually followed all instructions, which requires external output checking (an eval) (14:00).

## Technical details

- IFScale Benchmark: The benchmark requires the model to write a business report while accurately including a list of specific, exact words (keywords). This measures 'accuracy' (percentage of rules followed) versus 'density' (number of rules) (2:58).
- Model Performance (Current Frontier): The test showed that current models (e.g., GPT 5.5, Gemini 3.1 Pro) scored 100% immediately on the original test, requiring the benchmark to be raised to 10,000 words to find the true ceiling (6:12).
- Failure Mode Comparison: Deepseek V4 Pro is 'traditional,' simply forgetting instructions. Claude Opus 4.7 fails due to a sensitive safety classifier triggering on random words (e.g., 'anthrax' and 'cyanide') (10:00). Gemini 3.1 Pro fails by exhausting its thinking tokens, resulting in no output (11:00).

## Practical implications

- Developers no longer need to keep skills files under 200 instructions; they can be significantly longer.
- If a use case requires 100 or 300 specific rules, they can be placed directly in the prompt rather than being sharded across multiple specialized agents.
- The primary engineering focus must shift from prompt length management to implementing robust output validation (evals) to confirm model compliance.

## Topics

AI, LLM Prompt Engineering, LLM Benchmarking, Build Systems, AI Reliability, IFScale, GPT 5.5, Claude Opus 4.7, Gemini 3.1 Pro, Deepseek V4 Pro

Source: https://www.youtube.com/watch?v=XzJD1bvXKjs