# Trying the new Claude Eval tool

## Executive summary

The video provides a technical deep dive into the new Claude Evals tool, designed to evaluate the performance of AI plugins and skills. While the tool automates test case generation and scoring (comparing runs with and without the plugin), the speakers highlight significant limitations. The process is currently highly dependent on manual, pre-collected data (e.g., defining failure modes, providing positive/negative examples) and lacks an integrated, 'in-the-loop' annotation interface, making its real-world application challenging for robust build pipelines.

## Key takeaways

- Claude Plugin Eval Functionality: The `Claude plugin eval` tool allows users to evaluate plugins by generating test cases, running the plugin against them, scoring the results, and comparing the scores with and without the plugin's intervention. This process generates a terminal output and an HTML report.
- Evaluation Input Requirements: The tool requires users to define quality standards by providing examples of 'good' and 'bad' outputs, identifying specific failure modes (e.g., overediting, abrupt transitions, jargon), and supplying real prompts or past outputs.
- Process Limitations: The current chat-based interface is criticized for being 'out of the loop.' Effective evaluation requires continuous, in-situ feedback and annotation, which the current tool does not adequately support.

## Technical details

- Claude Plugin Eval Command: The evaluation process is initiated by running the command `claude plugin eval` from the root directory of the plugins folder.
- Evaluation Methodology: The tool performs an ablation study, comparing the performance score of a skill/plugin (e.g., 'plain writing skill') when it is active versus when it is disabled (baseline).
- Skill/Plugin Scope: The tool is designed to evaluate plugins, though the speakers questioned if it could evaluate standalone skills, noting that the plugin directory is the required execution context.
- Data Requirements: Effective use requires a comprehensive dataset of positive and negative test cases, detailed failure modes, and specific prompts to guide the evaluation process.

## Practical implications

- For build engineers, this represents a potential shift toward automated, AI-driven quality gates for complex components (skills/plugins).
- The process emphasizes the need for robust data collection pipelines (annotation apps) to feed the evaluation tool, suggesting that the data preparation phase is as critical as the execution phase.
- The discussion highlights the need for 'in-the-loop' development tools that allow developers to provide real-time feedback during the skill's execution, rather than relying solely on post-hoc testing.

## Topics

AI Development, Plugin Architecture, Automated Testing, Quality Assurance, LLM Evaluation, Claude plugin eval, plain writing skill, Claude Devs (Twitter)

Source: https://www.youtube.com/watch?v=K85b9r-4RvI