Building a Harness with Jev
Summary
This talk introduces Jev, a new System 1 model from TypeSafe AI, designed for fast, structured decision-making. Unlike traditional LLMs (System 2), Jev does not generate text but instead evaluates a state and questions to return typed answers and probabilities, making it significantly faster (up to 200x) and cheaper (up to 400x) for classification-style tasks. Jev can be integrated into agent harnesses via LangChain's TypeSafe integration to enhance model routing, implement auto-mode for risk assessment, and function as a highly efficient judge for online evaluations.
Key takeaways
-
Jev as a System 1 Model
2:18
Jev is a System 1 model that evaluates a state and questions to return typed answers and probabilities, rather than generating text. This makes it ideal for specialized, structured decision tasks.
-
Performance Advantage
2:45
Jev is claimed to be 20 to 200 times faster and 40 to 400 times cheaper than LLMs for classification-style tasks.
-
Three Question Types
6:30
Jev can answer three types of questions: Choice (multiple choice), Score (on a scale), and Boolean (yes/no). It can process multiple questions from a single state in parallel.
-
Use Case: Model Routing
8:00
Jev can assess a given prompt against criteria to help decide whether a fast/cheap model or a more powerful/expensive model should be used, optimizing agent performance.
-
Use Case: Jev as a Judge
Jev can score an agent's answer against a provided rubric (e.g., correctness, grounding) for online evaluations, offering a cheaper, faster, and more consistent alternative to LLM-as-a-judge methods.
Technical details
-
Agent Architecture
60s
The standard agent loop involves an LLM taking action by calling tools, receiving structured results, and continuing the process until completion. This loop is enhanced by structured outputs and tool calling.
-
Structured Outputs
110s
This primitive allows binding an output type to a model, ensuring the final result adheres to a given JSON schema rather than raw text.
-
Jev Integration
430s
Jev can be used in LangChain via the newly released LangChain TypeSafe integration. Users must acquire a TypeSafe API key to invoke the classifier.
-
Auto Mode
520s
Jev can analyze whether given tool calls are risky, allowing runtime blocking of potentially dangerous actions (e.g., deleting databases).
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.