# Agentic Engineering Benchmarks: How I RANK Astra, Fable 5.1, and Open-Weights

## Executive summary

The video argues that relying on generalized AI benchmarks (like the Artificial Analysis Index) is misleading because model performance is a complex, three-dimensional problem involving performance, cost, and speed. The speaker advocates for selecting a curated set of five specialized benchmarks—Terminal-Bench, APEX Agents, AutomationBench, AA-Omniscience, and DeepSWE—to accurately assess model capabilities for specific, real-world agentic engineering tasks, rather than relying on single aggregate scores.

## Key takeaways

- Model Selection is a 3D Problem: Choosing a model requires balancing performance, cost, and speed simultaneously. A model that performs best may be prohibitively expensive or slow, making the trade-off critical for production systems.
- Focus on Variance, Not Indices: The most valuable benchmarks show significant variance in model performance, indicating 'alpha' or information gain. Flat-lining benchmarks (like some long context retrieval tests) are considered saturated and uninformative.
- The Importance of Guardrails (Alignment): AutomationBench is highlighted for its focus on guardrail adherence. A model must not only complete the objective but also avoid triggering guardrail violations, which is crucial for reliable, production-grade agents.
- The Value of Honesty (AA-Omniscience): The AA-Omniscience benchmark measures hallucination rate and the cost of honesty. It is critical that an agent can 'opt to not answer' when lacking information, as this prevents cascading errors in long-running agent pipelines.
- Long-Horizon Work is Key: DeepSWE v1.1 is emphasized for testing long-horizon software engineering tasks. The goal is to build systems that operate autonomously with minimal human oversight, requiring models that maintain consistency over extended periods.

## Technical details

- Terminal-Bench v4.0: A pure agentic coding benchmark where an agent runs in a prepared container, executes a harness loop, and a verifier checks the final state. It is used to assess the trade-off between performance, cost, and speed. Astra is noted as a strong winner on cost efficiency.
- APEX Agents: A proxy for knowledge worker domains, testing agent performance across three expert-authored professions: investment banking analysis, management consulting, and corporate law. This moves the focus beyond pure software engineering.
- AutomationBench: Tests classical software automation across six business domains (Finance, HR, Marketing, etc.). Its key metric is the ability to complete objectives without triggering guardrail violations, making it a measure of alignment.
- AA-Omniscience: The hallucination benchmark, which assesses the cost of honesty. It grades answers as correct, incorrect, partial, or 'not attempted,' rewarding agents that refuse to answer when uncertain.
- DeepSWE v1.1: A benchmark designed for long-horizon software engineering tasks, simulating complex, sustained work. The speaker notes that short, high-level prompts should allow agents to accomplish large amounts of work, indicating high potential.

## Practical implications

- Do not rely on single index scores; instead, select a diverse set of benchmarks that align with the specific domain (e.g., coding, finance, compliance) of your intended agent workflow.
- Prioritize benchmarks that test guardrail adherence (AutomationBench) and the ability to gracefully fail/refuse to answer (AA-Omniscience).
- Adopt a 'model stack' approach rather than selecting a single model, combining compute from multiple models to optimize for cost and performance across different tasks.
- Focus on building systems that operate autonomously (long-horizon work) to achieve true agentic engineering, moving beyond human-in-the-loop processes.

## Topics

Agentic Engineering, LLM Benchmarking, Model Stacking, Guardrails, Cost Optimization, AI Workflow, Claude Fable & Mythos 5.1, GPT-6 Astra, Terminal-Bench v4.0, APEX Agents Leaderboard, AutomationBench, AA-Omniscience, DeepSWE v1.1

Source: https://www.youtube.com/watch?v=9weiIHy9T_0