JevEval logopeterindia.net
LLM Evaluation

JevEval

Jev as a judge for LLM evaluation: bounded questions, calibrated probabilities and repeatable scores, without an LLM judge.

33 topics•6 categories•Updated November 2026

What JevEval Is

6 entries

The metric, the model behind it, and its variants.

01JevEval metric (DeepEval docs)A custom DeepEval metric where you write bounded questions and Jev answers them; no evaluation LLM is involved. 02Introducing JevEval: Jev-as-a-JudgeDeepEval's announcement of Jev as a judge, with the cost and latency claims. 03Introducing Jev for EvalsJev as a language-model-free judge, and the hybrid and system_one evaluation modes. 04Jev and System One (TypeSafe AI)Jev is a decision model, not a text generator: it takes state and a bounded question and returns typed decisions as probabilities. 05ConversationalJevEvalThe multi-turn variant, which judges conversations turn by turn. 06Trajectory modeSingle-turn JevEval without evaluation parameters judges an agent's span tree; it needs a trace.

The Three Question Types

3 entries

Every JevEval metric is built from these bounded questions.

07Noul: yes/no propositionA yes/no statement; the value is the probability that it is true. 08Score: ordered levelsOrdered levels from worst to best; the value is the expected level position, scaled from 0 to 1. 09Choice: weighted optionsUnordered options with credits from 0 to 1, or none when a question does not apply.

How Scoring Works

4 entries

What comes out and how to read it.

10Weighted mean of answersThe score is the weighted mean of the applicable question values, so the same input gives the same score. 11Threshold and strict modeThe default threshold is 0.5; strict mode returns 1 only if every applicable question passes. 12score_breakdown and reasonPer-question probabilities, values and confidence, and a reason built from them rather than generated. 13Weights and prioritiesPer-question weights state which criteria matter most, directly in code.

Jev vs LLM-as-a-Judge

7 entries

Where a decision model fits, and where a generative judge is still the better tool.

14G-Eval (LLM-as-a-judge)DeepEval says to use G-Eval when you want an LLM to draft the evaluation logic from a short criterion. 15DeepEval: LLM-as-a-Judge in 2026Techniques and best practices for LLM judges. 16Jev vs LLM-as-a-Judge (OpenRouter)OpenRouter's tutorial comparing the two approaches. 17Can Jev replace your LLM judge? (MLflow)MLflow's look at quality, cost and latency, and using Jev through make_judge. 18Jev vs. LLM-as-a-Judge (Comet Opik)Comet's comparison, with Jev available in Opik. 19JEV vs. LLMs as Rubric Judges (arXiv)A preprint comparing Jev with LLM judges on rubric scoring. 20LLM-as-a-Judge SolutionsThis site's directory of LLM-as-a-judge platforms, frameworks and research.

Getting Started

5 entries

Install, examples and related tooling.

21JevEval examplesWorked examples of JevEval metrics. 22DeepEval and TypeSafe AI integrationHow to connect DeepEval to Jev with a TypeSafe API key. 23DeepEval classifiersLabel-style classifiers built on Choice questions, such as refusal detection. 24TypeSafe AI documentationReference for Jev and its primitives. 25DeepEval metrics introductionOverview of DeepEval's metrics.

Related Pages

8 entries

Related pages on this site.

26LLM-as-a-Judge SolutionsJudge platforms and research. 27LLM Evaluation MethodsMethods for evaluating LLMs. 28LLMsDirectory of LLMs. 29LLM HubHub of LLM guides. 30LLMOpsLLM operations and tooling. 31AI Agent Observability OverviewObserving AI agents. 32AI Agent GuardrailsGuardrails for agents. 33Evaluation Metrics for Deep LearningMeasuring model quality.

Frequently Asked Questions

What is JevEval?

A custom metric in DeepEval that scores an LLM interaction using Jev. You write bounded questions about the input, output or other fields, Jev answers them with calibrated probabilities, and the score is a fixed weighted mean.

What is Jev?

A model from TypeSafe AI described as a System One decision model. It does not generate text. It takes state and a bounded question and returns typed decisions as probabilities.

Does JevEval use an LLM judge?

No. According to DeepEval's documentation, nothing is generated at any stage, including the reason, so no evaluation LLM or LLM provider is needed for JevEval itself.

Can it judge custom LLMs?

Yes. It judges the outputs, tool calls and conversations of any LLM application you can express as a DeepEval test case, including fine-tuned and self-hosted models. It is a judge of those systems, not a judge model you configure.

What are Noul, Score and Choice?

The three question types. Noul is a yes/no proposition, Score is an ordered scale from worst to best, and Choice is a set of options with credits.

How is the score calculated?

As the weighted mean of the values of the applicable questions. Strict mode returns 1 only when every applicable question passes.

How does it differ from G-Eval?

G-Eval has an LLM draft and apply the evaluation logic from a criterion. JevEval is for when you already know what to decide and want it decided the same way on every run.

Where is a generative judge still better?

Third-party tests report that Jev can match an LLM judge on closed rubrics with the evidence supplied, while LLM judges may track expert ratings better on open-ended tasks. Test both on your data.

What does it cost and how fast is it?

DeepEval's launch post states a per-input-token price and roughly 100 ms per query. Those are the vendor's figures, so check current pricing and measure latency yourself.