Channel
Digests from Hamel Husain — Hamel Husain videos on AI engineering, evaluation, and practical machine learning workflows.
Learn the 7 most common mistakes in AI evaluation (Evals). From using generic metrics to failing to validate LLM judges, this guide shows how to build Evals you can actually trust.
Read article
For complex AI agents, traditional input-output testing fails. The solution is 'simulation': replaying real historical production traces against changes to detect side effects and ensure the agent's behavior remains consistent.
Read article
An in-depth comparison of Opus 5 vs. Opus 5.5 reveals that while the newer model improves writing flow, both still struggle with 'slop' and require heavy human editing for technical content.
Read article
Stop using generic metrics and off-the-shelf LLM judges. The key to AI evaluation is always looking at your data first, using custom annotation tools, and validating judges with human labels.
Read article
Deep dive into evaluating OCR and AI agents on complex documents. Learn how to perform root cause analysis when an agent fails, tracing the error back to upstream data issues like OCR inaccuracies or missing watermarks.
Read article
Standard PDFs often fail to provide structural metadata needed for screen readers or LLMs. Learn how Data Lab's API solves this by transforming complex documents into highly structured, machine-readable formats.
Read article
Watch how an AI agent attempts to purchase a course using a Mercury agent card. The demo highlights the security benefits of scoped permissions but reveals significant friction points in the payment process.
Read article
Learn how to build better AI evaluations using eight Claude Code skills. Focus on Error Discovery and Eval Audit to move beyond writing evals before analyzing real product failures.
Read article
The speaker argues that for LLMs, context matters more than the specific wording of the prompt. Overly elaborate 'mannered' prompts are often imprecise, suggesting that simple, literal context is the most powerful guide.
Read article
In the age of AI, data science skills—like handling noisy, non-deterministic signals—are more valuable than ever. Human judgment is critical for measuring AI output and designing robust RAG systems.
Read article
Building trustworthy AI agents requires more than just an answer. The product must be designed to show its work, providing full data provenance, SQL queries, and intermediate calculations for expert verification.
Read article
The speaker critiques AI 'slop' and argues that accurate model evaluation requires building small, realistic datasets and testing against specific failure modes, rather than relying on leaderboards or vendor demos.
Read article
A deep dive into Claude's new `plugin eval` tool. While promising for automated skill testing, the speakers highlight that the process is currently bottlenecked by manual data collection and lacks an 'in-the-loop' annotation interface.
Read article
This presentation covers three aspects of search agents: evaluation, synthetic data generation using ORBIT, and trajectory analysis via tools like Hawkeye. It emphasizes that efficiency is as critical as accuracy.
Read article
Continuous discovery emphasizes starting with a desired outcome, mapping unmet needs via Opportunity Solution Trees (OST), and using structured interviews to gather behavioral stories. AI accelerates synthesis and coaching, but rigor is key.
Read article
Stop relying on MTEB scores! Learn how to optimize embedding models by mastering quantization, managing vector dimensions, and building cheap first-phase/expensive reranker search architectures.
Read article
The hard part of background agents is always the environment. Focus on environmental engineering, building robust dev boxes instead of optimizing the agent itself. This requires separating control and data planes for stability.
Read article
Discover how Multi-Vector Retrieval solves the limitations of single vector embeddings for AI agents. Learn about sparse multi-vector encoding, scaling to billion documents, and achieving sub-50ms P99 latency.
Read article
Before fine-tuning an LLM, focus on optimizing your evaluation environment. The three core parts of any eval are the Task Set, the Harness (the execution loop), and the Scoring Function. Advanced training involves Reinforcement Learning (RL).
Read article
This session details the Model Cascade architecture, a technique to drastically reduce LLM classification costs (by over 90%) while maintaining high accuracy (e.g., 95%+). Instead of running an expensive 'Oracle' model on every record, the cascade uses a cheap proxy model for initial predictions and only escalates to the Oracle when the proxy's confidence score is low. The optimal decision threshold ($\tau$) is found offline by simulating performance across samples.
Read article