# Can One Model Run Your App And Evals?

## Executive summary

The discussion addresses the common concern regarding model bias when using the same Large Language Model (LLM) for both the primary application task (Agent) and the evaluation/judging process (Judge). The consensus is that using the same model family is generally acceptable because the judge performs a distinct task from the main pipeline. The critical factor for successful evaluation is achieving strong alignment with human labels, which should guide model selection over worrying about potential model bias.

## Key takeaways

- Model Reuse for Judging: Using the same model for the LLM judge and the main agent is usually acceptable because the judge is performing a different task than the main LLM pipeline.
- Focus on Alignment, Not Separation: The primary focus should be on achieving good alignment with human labels, rather than strictly separating the judge model from the agent model.
- Optimal Judge Tasks: Recommended judge tasks are scoped binary classification tasks. Success depends on achieving good precision and recall on a held-out test set.

## Technical details

- LLM Evaluation Strategy: When selecting judge models, it is recommended to start with the most capable model available to establish strong alignment with human labels. Cost optimization can be addressed later.
- Model Provider Complexity: Onboarding a different model provider (e.g., moving from OpenAI to Anthropic) can be non-trivial and complex, suggesting that the added complexity may not justify the benefit if alignment with human labels is achievable with the current model.

## Practical implications

- Prioritize achieving strong alignment with human labels (e.g., through iterative alignment) over worrying about potential model bias.
- When building an AI product, establish reliable evaluation criteria using the most capable model first, and then optimize for cost.
- If evaluation struggles, consider the complexity of switching model providers before assuming a different model is necessary.

## Topics

LLM Evaluation (Evals), Model Bias, Binary Classification, Precision and Recall, Model Architecture, Free Evals Guide

Source: https://www.youtube.com/watch?v=kkoCM-yn2kA