Agentic Engineering Benchmarks: How I RANK Astra, Fable 5.1, and Open-Weights
Summary
The video argues that relying on generalized AI benchmarks (like the Artificial Analysis Index) is misleading because model performance is a complex, three-dimensional problem involving performance, cost, and speed. The speaker advocates for selecting a curated set of five specialized benchmarks—Terminal-Bench, APEX Agents, AutomationBench, AA-Omniscience, and DeepSWE—to accurately assess model capabilities for specific, real-world agentic engineering tasks, rather than relying on single aggregate scores.
Key takeaways
-
Model Selection is a 3D Problem
0:40
Choosing a model requires balancing performance, cost, and speed simultaneously. A model that performs best may be prohibitively expensive or slow, making the trade-off critical for production systems.
-
Focus on Variance, Not Indices
1:50
The most valuable benchmarks show significant variance in model performance, indicating 'alpha' or information gain. Flat-lining benchmarks (like some long context retrieval tests) are considered saturated and uninformative.
-
The Importance of Guardrails (Alignment)
10:30
AutomationBench is highlighted for its focus on guardrail adherence. A model must not only complete the objective but also avoid triggering guardrail violations, which is crucial for reliable, production-grade agents.
-
The Value of Honesty (AA-Omniscience)
17:30
The AA-Omniscience benchmark measures hallucination rate and the cost of honesty. It is critical that an agent can 'opt to not answer' when lacking information, as this prevents cascading errors in long-running agent pipelines.
-
Long-Horizon Work is Key
27:30
DeepSWE v1.1 is emphasized for testing long-horizon software engineering tasks. The goal is to build systems that operate autonomously with minimal human oversight, requiring models that maintain consistency over extended periods.
Technical details
-
Terminal-Bench v4.0
130s
A pure agentic coding benchmark where an agent runs in a prepared container, executes a harness loop, and a verifier checks the final state. It is used to assess the trade-off between performance, cost, and speed. Astra is noted as a strong winner on cost efficiency.
-
APEX Agents
400s
A proxy for knowledge worker domains, testing agent performance across three expert-authored professions: investment banking analysis, management consulting, and corporate law. This moves the focus beyond pure software engineering.
-
AutomationBench
630s
Tests classical software automation across six business domains (Finance, HR, Marketing, etc.). Its key metric is the ability to complete objectives without triggering guardrail violations, making it a measure of alignment.
-
AA-Omniscience
1050s
The hallucination benchmark, which assesses the cost of honesty. It grades answers as correct, incorrect, partial, or 'not attempted,' rewarding agents that refuse to answer when uncertain.
-
DeepSWE v1.1
1650s
A benchmark designed for long-horizon software engineering tasks, simulating complex, sustained work. The speaker notes that short, high-level prompts should allow agents to accomplish large amounts of work, indicating high potential.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.