# How to Ace AI Eval Interview Questions (for Engineers & PMs)

## Executive summary

This guide outlines a structured approach for tackling AI evaluation (e-vals) assessments in job interviews. Candidates are advised to structure their artifact by focusing heavily on data analysis (50-70% of effort), detailing experiments run, summarizing results, and discussing key learnings and limitations. The core emphasis is on demonstrating resourcefulness, skepticism regarding provided data, and strong product judgment by analyzing trade-offs (e.g., complexity vs. performance) rather than simply maximizing scores.

## Key takeaways

- Structure the Evaluation Artifact: The recommended structure is: Executive Summary (brief), Data Analysis (most critical section), Experiments Run, Summary of Results, Key Learnings/Limitations, and Next Steps/Recommendations.
- Prioritize Data Skepticism: Never assume the provided dataset is correct. Look for missing, inconsistent, or insufficient labels, and identify data noise. Solving for missing negative examples (e.g., guardrails) is often the primary challenge.
- Showcase Resourcefulness: When data is limited, demonstrate resourcefulness by leveraging external datasets (e.g., Hugging Face) or generating synthetic data (e.g., for negative examples) to augment the provided data set.
- Frame Experiments and Trade-offs: Present experiments in a table format, comparing different approaches (e.g., 'GPT-6 Luna with prompt only' vs. 'GPT-6 Luna with external database lookup'). Discussing trade-offs (cost, latency, complexity) and identifying overfitting (e.g., high validation score, low test score) is crucial for demonstrating judgment.
- Address Ambiguous Metrics: If the interviewer provides a questionable metric (e.g., combining helpfulness and tone into a composite score), compute the score but also explain why you would refactor or improve the metric, thereby teaching the interviewer and avoiding a trap.

## Technical details

- Data Quality Issues: Identifying inconsistent labels (e.g., species name mismatch) or insufficient label data is critical. The goal is to solve for missing examples, such as positive/negative cases for guardrails.
- Overfitting Detection: When reviewing results, look for discrepancies between validation set scores and test set scores. A significant drop indicates overfitting, which must be discussed.
- Evaluation Metrics: Candidates should be prepared to discuss various metrics (e.g., TPR, TNR, F1 score) and be ready to argue why a provided metric might be suboptimal.
- Agent Improvement Techniques: Methods to improve an agent include using external databases for lookup, automated prompt tuning (e.g., Jeppa), or varying the underlying LLM (e.g., comparing different versions of GPT-6 Luna).

## Practical implications

- Systematically analyze the data set for flaws (inconsistent labels, missing examples) before building any solution.
- Structure the final presentation to emphasize data analysis and trade-off discussions over simply reporting the highest possible score.
- When faced with ambiguous or flawed instructions, propose a technical refactoring or alternative metric to demonstrate advanced judgment.
- Always consider the cost and latency trade-offs alongside performance metrics.

## Topics

AI Evaluation (E-vals), Data Engineering, Machine Learning Model Evaluation, Product Management (PM) for AI, Experiment Design, AI Evals Guide

Source: https://www.youtube.com/watch?v=6RTgIr08LMs