# Building a Harness with Jev

## Executive summary

This talk introduces Jev, a new System 1 model from TypeSafe AI, designed for fast, structured decision-making. Unlike traditional LLMs (System 2), Jev does not generate text but instead evaluates a state and questions to return typed answers and probabilities, making it significantly faster (up to 200x) and cheaper (up to 400x) for classification-style tasks. Jev can be integrated into agent harnesses via LangChain's TypeSafe integration to enhance model routing, implement auto-mode for risk assessment, and function as a highly efficient judge for online evaluations.

## Key takeaways

- Jev as a System 1 Model: Jev is a System 1 model that evaluates a state and questions to return typed answers and probabilities, rather than generating text. This makes it ideal for specialized, structured decision tasks.
- Performance Advantage: Jev is claimed to be 20 to 200 times faster and 40 to 400 times cheaper than LLMs for classification-style tasks.
- Three Question Types: Jev can answer three types of questions: Choice (multiple choice), Score (on a scale), and Boolean (yes/no). It can process multiple questions from a single state in parallel.
- Use Case: Model Routing: Jev can assess a given prompt against criteria to help decide whether a fast/cheap model or a more powerful/expensive model should be used, optimizing agent performance.
- Use Case: Jev as a Judge: Jev can score an agent's answer against a provided rubric (e.g., correctness, grounding) for online evaluations, offering a cheaper, faster, and more consistent alternative to LLM-as-a-judge methods.

## Technical details

- Agent Architecture: The standard agent loop involves an LLM taking action by calling tools, receiving structured results, and continuing the process until completion. This loop is enhanced by structured outputs and tool calling.
- Structured Outputs: This primitive allows binding an output type to a model, ensuring the final result adheres to a given JSON schema rather than raw text.
- Jev Integration: Jev can be used in LangChain via the newly released LangChain TypeSafe integration. Users must acquire a TypeSafe API key to invoke the classifier.
- Auto Mode: Jev can analyze whether given tool calls are risky, allowing runtime blocking of potentially dangerous actions (e.g., deleting databases).

## Practical implications

- Build engineers can optimize agent performance and cost by offloading classification and decision tasks from general-purpose LLMs to specialized models like Jev.
- The ability to use Jev for model routing and risk assessment enhances the reliability and safety of complex, code-driven applications.
- Jev provides a scalable, cost-effective alternative to human oversight or expensive LLM-based evaluation during large-scale agent testing (online evals).
- The integration of Jev into LangChain allows for immediate adoption of System 1 decision-making capabilities in existing agent harnesses.

## Topics

AI Models, Agent Development, Structured Outputs, System 1 AI, LangChain, Model Evaluation, Building a Harness with Jev, Jev-as-a-Judge for Agent Evals, LangSmith, LangChain Academy

Source: https://www.youtube.com/watch?v=VE5dsWll06M