Building a Harness with Jev
This talk introduces Jev, a new System 1 model from TypeSafe AI, designed for fast, structured decision-making. Unlike traditional LLMs (System 2), Jev does not generate text but instead evaluates a state and questions to return typed answers and probabilities, making it significantly faster (up to 200x) and cheaper (up to 400x) for classification-style tasks. Jev can be integrated into agent harnesses via LangChain's TypeSafe integration to enhance model routing, implement auto-mode for risk assessment, and function as a highly efficient judge for online evaluations.
Key takeaways
-
Jev as a System 1 Model
2:18
Jev is a System 1 model that evaluates a state and questions to return typed answers and probabilities, rather than generating text. This makes it ideal for specialized, structured decision tasks.
-
Performance Advantage
2:45
Jev is claimed to be 20 to 200 times faster and 40 to 400 times cheaper than LLMs for classification-style tasks.
-
Three Question Types
6:30
Jev can answer three types of questions: Choice (multiple choice), Score (on a scale), and Boolean (yes/no). It can process multiple questions from a single state in parallel.
-
Use Case: Model Routing
8:00
Jev can assess a given prompt against criteria to help decide whether a fast/cheap model or a more powerful/expensive model should be used, optimizing agent performance.
-
Use Case: Jev as a Judge
Jev can score an agent's answer against a provided rubric (e.g., correctness, grounding) for online evaluations, offering a cheaper, faster, and more consistent alternative to LLM-as-a-judge methods.