LLM — The Video Signal

Sep 22, 2026 · 55:19

Ask the Experts: Evaluating Agent Skills | Nemotron Labs

This session details the evaluation of autonomous AI agent skills using NVIDIA's open-source framework, SkillEvaluator. The system provides a comprehensive, multi-tier pipeline to ensure that skills are safe, reliable, and genuinely useful before deployment. The evaluation process covers static security scanning (detecting risks like prompt injection and data exfiltration), deduplication checks, and live agent testing in a sandbox environment. This framework is critical for establishing quality gates in AI-driven workflows.

Key takeaways

Three-Tier Evaluation Framework 2:00
Skill evaluation utilizes a three-tier framework: 1) Static/Security checks (using SkillSpector), 2) Deduplication checks against an internal registry, and 3) Live Agent Testing, which assesses if the skill improves task performance in a sandbox environment.
Security Scanning for Agent Risks 4:00
SkillSpector is a standalone security scanner that checks for agent-native risks, including prompt injection, data exfiltration, trigger abuse, and tool poisoning. It combines static analysis with optional LLM semantic analysis.
Measuring Skill Utility (Skill Lift) 7:30
The evaluation measures 'Skill Lift' across five dimensions: efficiency, correctness, effectiveness, discoverability, and security. This determines if the skill genuinely helps the agent perform the task better compared to a baseline.
Live Agent Testing Methodology 9:00
Live agent evaluation runs the agent in a secure sandbox, performing an A/B test by comparing the agent's performance when the skill is available versus when it is restricted. This confirms if the skill is functionally necessary for the task.

Watch on YouTube Full article

Sep 17, 2026 · 9:38

What Is MLflow? Tracing AI Agents & LLM Workflows

This video details the necessity of advanced observability for multi-agent and LLM workflows, arguing that traditional monitoring (HTTP status, response time) is insufficient for diagnosing complex AI failures. MLflow is presented as an OpenTelemetry-compatible platform that captures detailed traces, allowing engineers to monitor not just request success, but the internal logic, tool usage, and quality of the AI's decision path. Key techniques include using LLM judges for non-deterministic evaluation and integrating quality gates into the CI pipeline.

Key takeaways

Beyond Request-Level Monitoring
Multi-agent systems require visibility into internal failures like silent tool failures, cascading latency, context overflow, and non-determinism, which standard monitoring dashboards cannot detect. (0:00)
MLflow's Core Primitive: The Trace 0:47
MLflow uses 'traces'—complete records of one request—composed of 'spans.' Each span details a single action (e.g., LLM call, DB query), capturing inputs, outputs, latency, and token counts. (0:47)
Advanced Evaluation with LLM Judges 2:32
For non-deterministic AI, MLflow allows using a second model (LLM Judge) to grade agent output against criteria like tool call correctness, relevance, and safety. (2:32)
Production Deployment Best Practices 4:10
For production, the tracking server must use a real database (e.g., Postgres/MySQL), asynchronous trace logging should be enabled, and sampling should be used to manage overhead. (4:10)

Watch on YouTube Full article

Sep 4, 2026 · 34:48

Hugging Face Journal Club: AI Research Preference Models

This discussion summarizes Meta's research on Research Preference Models (RPMs), which utilize Large Language Models (LLMs) to predict the success of machine learning experiments. Given that ML evaluations are computationally expensive (potentially taking days or weeks on GPUs), RPMs guide autonomous agents by selecting the most promising candidates for evaluation, drastically reducing required compute time while maintaining high performance. The process involves complex tree search mechanisms and can be enhanced through ensembling multiple LLM judges.

Key takeaways

RPM Goal: Reducing Compute Budget 2:35
The primary goal is to avoid evaluating all possible ML candidates, which consumes excessive compute resources. RPMs select the most promising experiments (mutations) to evaluate next, reducing required time from potentially days down to hours while achieving performance comparable to an 'Oracle' [0:02:35].
RPM Types and Functionality 2:58
Two main types are discussed: the Inference-only RPM (using a frozen LLM to reason over plans/code) and the Agentic RPM, which can run small-scale pilot experiments to further refine predictions [0:02:58].
The Search Process (Tree Traversal) 3:45
The process is modeled as a tree search, starting from a root node (initial experiment). Candidates are generated as children nodes; the RPM scores these candidates, and the agent selects the best one to explore next. This mechanism resembles Monte Carlo Tree Search (MCTS) [0:03:45].
Ensembling for Robustness 8:13
To improve reliability, the research suggests evaluating candidates using ensembles of multiple frontier models (e.g., GPT-5 Opus). Techniques include majority vote and an LLM arbiter ensemble to mitigate issues like reward hacking [0:08:23].

Watch on YouTube Full article

Aug 27, 2026 · 15:01

LLM & AI Agent Benchmarks vs Reality: Why AI Applications Break

The performance of Large Language Models (LLMs) in real-world AI applications often deviates significantly from high benchmark scores. Building reliable AI systems requires balancing three critical factors—accuracy, latency/performance, and cost. Evaluation must therefore encompass both 'model evaluation' (assessing intelligence and accuracy) and 'system evaluation' (measuring scalability, throughput, and cost). For complex agents, this process extends to evaluating every step in the decision chain.

Key takeaways

Benchmark vs. Reality Gap
A high score on a leaderboard does not guarantee real-world performance; production environments test for latency, accuracy, and cost simultaneously.
The Three Pillars of AI Design 2:05
AI applications must balance Accuracy (correctness), Performance (response time/latency), and Cost. Optimizing for two often compromises the third.
Agent Evaluation is Multi-Layered 11:20
Evaluating agents requires checking every link in the decision chain, including system performance, formatting, safety/bias, factual accuracy, and domain-specific checks.

Watch on YouTube Full article