# Score Every Production Trace with an LLM Judge, from Your Terminal (LangSmith CLI)

## Executive summary

This walkthrough demonstrates how to implement automated, large-scale evaluation of AI agents using an LLM-as-a-judge within LangSmith. By setting up an online evaluation, developers can programmatically score incoming chat traces (e.g., checking for user frustration) without manually reviewing thousands of interactions. The process involves using a coding agent to install LangSmith skills and configure the evaluation, followed by managing the sampling rate to control costs.

## Key takeaways

- Online LLM-as-a-Judge Setup: Online evaluations allow an LLM to automatically score incoming agent traces based on a defined prompt and rubric (e.g., detecting user frustration). The output is a score and reasoning attached directly to the trace.
- Using LangSmith CLI for Automation: A coding agent can be instructed to use the LangSmith CLI to create and manage the LLM judge evaluator, automating the setup process.
- Cost Management via Sampling Rate: The sampling rate can be updated (e.g., from 100% to 50%) to reduce evaluation costs while maintaining continuous monitoring.

## Technical details

- LLM-as-a-Judge Structure: The evaluation uses an LLM judge, which requires a prompt and a rubric to determine the metric (e.g., user frustration). The result is a score (binary: 1 for frustrated, 0 for not frustrated) and detailed reasoning.
- LangSmith Skills Integration: The coding agent must first be equipped with the latest LangSmith skills by running the `latest skills` command to ensure up-to-date interaction knowledge.
- Evaluation Configuration: The evaluation is configured to run on all incoming traces, checking for negative user experience and assigning a binary score.

## Practical implications

- Automating quality assurance for AI agents by programmatically scoring user interactions.
- Enabling continuous monitoring of agent performance and user experience at scale.
- Providing cost control mechanisms by adjusting the evaluation sampling rate.

## Topics

LLM Evaluation, LangSmith, AI Agents, CI/CD, Natural Language Processing, LangSmith CLI, Online evaluations (LLM-as-a-judge)

Source: https://www.youtube.com/watch?v=XE5-hb-H-OY