The Video Signal

Topic

Security Scanning

All digests tagged Security Scanning

Ask the Experts: Evaluating Agent Skills | Nemotron Labs thumbnail

· 55:19

Ask the Experts: Evaluating Agent Skills | Nemotron Labs

This session details the evaluation of autonomous AI agent skills using NVIDIA's open-source framework, SkillEvaluator. The system provides a comprehensive, multi-tier pipeline to ensure that skills are safe, reliable, and genuinely useful before deployment. The evaluation process covers static security scanning (detecting risks like prompt injection and data exfiltration), deduplication checks, and live agent testing in a sandbox environment. This framework is critical for establishing quality gates in AI-driven workflows.

Key takeaways

  1. Three-Tier Evaluation Framework 2:00

    Skill evaluation utilizes a three-tier framework: 1) Static/Security checks (using SkillSpector), 2) Deduplication checks against an internal registry, and 3) Live Agent Testing, which assesses if the skill improves task performance in a sandbox environment.

  2. Security Scanning for Agent Risks 4:00

    SkillSpector is a standalone security scanner that checks for agent-native risks, including prompt injection, data exfiltration, trigger abuse, and tool poisoning. It combines static analysis with optional LLM semantic analysis.

  3. Measuring Skill Utility (Skill Lift) 7:30

    The evaluation measures 'Skill Lift' across five dimensions: efficiency, correctness, effectiveness, discoverability, and security. This determines if the skill genuinely helps the agent perform the task better compared to a baseline.

  4. Live Agent Testing Methodology 9:00

    Live agent evaluation runs the agent in a secure sandbox, performing an A/B test by comparing the agent's performance when the skill is available versus when it is restricted. This confirms if the skill is functionally necessary for the task.

Watch on YouTube Full article

We Scored a Real Snyk Skill Against Anthropic's Rules thumbnail

· 15:19

We Scored a Real Snyk Skill Against Anthropic's Rules

This video details a live review process where a Snyk skill (`SKILL.md`) was evaluated using Tessl's `tessl review run` against Anthropic's best practices. The initial score of 87% was successfully improved to 90% by applying fixes, demonstrating how automated tools can enhance skill quality and security. Key focus areas include implementing progressive disclosure to prevent context bloat, improving skill conciseness, and using Snyk's Agent Scan tool to detect vulnerabilities like prompt injection in both first-party and third-party skills.

Key takeaways

  1. Skill Quality Improvement via Automated Review

    The review process successfully increased the skill score from 87% to 90% by applying fixes, demonstrating that automated tools can significantly improve adherence to best practices (e.g., Anthropic's guidelines).

  2. Importance of Progressive Disclosure 10:08

    Skills should not be overly dense or verbose. Implementing progressive disclosure—breaking large skills into smaller, referenced sub-files—prevents context bloat and ensures the agent only loads necessary information.

  3. Security Scanning with Agent Scan

    Snyk's dedicated tool, Agent Scan (available on GitHub), can scan skills for security vulnerabilities, including prompt injection, which is crucial when integrating third-party or user-written skills.

Watch on YouTube Full article