The Video Signal

Topic

Quality Assurance

All digests tagged Quality Assurance

Trying the new Claude Eval tool thumbnail

· 58:39

Trying the new Claude Eval tool

The video provides a technical deep dive into the new Claude Evals tool, designed to evaluate the performance of AI plugins and skills. While the tool automates test case generation and scoring (comparing runs with and without the plugin), the speakers highlight significant limitations. The process is currently highly dependent on manual, pre-collected data (e.g., defining failure modes, providing positive/negative examples) and lacks an integrated, 'in-the-loop' annotation interface, making its real-world application challenging for robust build pipelines.

Key takeaways

  1. Claude Plugin Eval Functionality 2:05

    The `Claude plugin eval` tool allows users to evaluate plugins by generating test cases, running the plugin against them, scoring the results, and comparing the scores with and without the plugin's intervention. This process generates a terminal output and an HTML report.

  2. Evaluation Input Requirements 10:10

    The tool requires users to define quality standards by providing examples of 'good' and 'bad' outputs, identifying specific failure modes (e.g., overediting, abrupt transitions, jargon), and supplying real prompts or past outputs.

  3. Process Limitations 31:40

    The current chat-based interface is criticized for being 'out of the loop.' Effective evaluation requires continuous, in-situ feedback and annotation, which the current tool does not adequately support.

Watch on YouTube Full article

.NET Testing Techniques You Didn’t Know You Needed - Dante De Ruwe - NDC Copenhagen 2026 thumbnail

· 58:44

.NET Testing Techniques You Didn’t Know You Needed - Dante De Ruwe - NDC Copenhagen 2026

This talk explores advanced .NET testing techniques designed to improve test effectiveness and build confidence in code quality beyond basic unit tests. Key methods include Property-Based Testing (PBT) for generating vast input variations, Snapshot Testing for tracking complex output changes over time, and Mutation Testing for verifying if existing tests are actually capable of catching bugs. The speaker emphasizes that the goal is not just to write many tests, but to write fewer, highly effective tests.

Key takeaways

  1. Test Reliability vs. Code Trust 2:00

    Tests should be viewed as a 'parachute'—they must have no holes. A failing test when requirements haven't changed indicates a bug; passing when the code changes suggests incomplete coverage or potential issues.

  2. Randomized and Property-Based Testing (PBT) 7:10

    Instead of relying solely on example cases, PBT defines mathematical properties that the code must always satisfy (e.g., a + b = b + a). This is crucial for complex structures or when edge case enumeration is impossible.

  3. Snapshot Testing 13:20

    This technique captures and saves the output of a function (the 'snapshot'). Subsequent test runs fail if the output changes unexpectedly, making it ideal for verifying API contracts (like OpenAPI specs) or complex data structures without writing explicit assertions for every field.

  4. Mutation Testing 26:20

    This advanced technique introduces small bugs (mutations) into the code and checks if the existing test suite fails. A high mutation score indicates that tests are effective at catching subtle errors, debunking false confidence from high line coverage metrics.

Watch on YouTube Full article