How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI
The capacity of frontier LLMs to follow complex instructions has increased by an estimated tenfold in the last year, moving the instruction ceiling from 200-300 rules to 2,000 to 5,000 rules. This shift fundamentally changes prompt engineering from a 'compression problem' (fitting rules into a small budget) to a 'verification problem' (confirming the model actually obeyed the rules). Developers must now focus on robust output validation (evals) rather than worrying about the length of the skills file.
Key takeaways
-
Instruction Ceiling Increased by 10x
10:12
A year ago, models struggled with 200 to 300 instructions. Current frontier models can handle up to 2,000 instructions, with the best models reaching 5,000 rules (6:12).
-
Failure Mode Shift: From Forgetting to Refusing
20:00
Models no longer fail by simply forgetting instructions. Failure modes are now diverse: Deepseek V4 Pro forgets; Claude Opus 4.7 refuses at the API level due to safety classifiers; Gemini 3.1 Pro runs out of thinking tokens; and GPT 5.5 writes a partial report and politely states the request is 'stupid' (12:00).
-
Engineering Focus Shifts to Verification
The challenge is no longer fitting rules into a small prompt (compression); it is verifying that the model actually followed all instructions, which requires external output checking (an eval) (14:00).