LLM Evaluation Methods (2026): BLEU, G-Eval, LLM-as-Judge & Agents

Why Evaluation Needs Several Methods

4 entries

The source post groups 11 methods by what they evaluate: reference-based metrics when ground truth exists, judge-based methods when it does not, human and deterministic checks, agent-specific methods, and a safety gate. No single method is enough, because overlap metrics miss meaning, judges can be biased, and humans do not scale, so production teams usually combine them.

0111 LLM evaluation methods (Daily Dose of Data Science, LinkedIn)The post this page follows, listing the 11 methods and presenting Comet Opik as a way to build the evaluation layer. 02LLM-as-a-Judge SolutionsTools and platforms for judge-based evaluation, on this site. 03LLMOps, Observability and ToolsLifecycle, tracing, evaluation and monitoring tools, on this site. 04Evaluation Metrics for Deep LearningClassic metrics for classification, regression and detection, on this site.

Reference-Based Metrics (1-3)

6 entries

Use these when a reference answer exists, such as translation, summarisation or extraction with labelled outputs. They are cheap and deterministic but compare wording or embeddings rather than judging whether an answer is correct.

051. BLEUScores n-gram precision: how much of the generated text overlaps with the reference. Original paper: BLEU, a method for automatic evaluation of machine translation (ACL 2002). 06SacreBLEUReference implementation for reproducible BLEU scores. 072. ROUGEFocuses on recall: how much of the reference content appears in the output. Original paper: ROUGE, a package for automatic evaluation of summaries (ACL 2004). 08ROUGE implementation (Google Research)Python implementation of the ROUGE metric. 093. BERTScoreCompares contextual embeddings rather than exact words, so semantically similar answers can still score well. Paper: BERTScore, evaluating text generation with BERT. 10BERTScore implementationOfficial code for BERTScore. GitHub lists 0.3.13 (February 2023) as its latest release.

Judge-Based Methods (4-6)

3 entries

Use these when ground truth is unavailable or incomplete. An LLM scores outputs against criteria, which scales better than humans but inherits the judge's biases, so validate judges against human labels.

114. G-EvalUses an LLM, with chain-of-thought and a form-filling paradigm, to score an output against defined criteria. Paper: G-Eval, NLG evaluation using GPT-4 with better human alignment. 125. LLM-as-a-JudgeGives a model a rubric and asks it to score or compare outputs. Paper: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, which also studies judge biases. 136. LLM JuriesRuns several independent judges and aggregates their verdicts to reduce reliance on one evaluator. Paper: Replacing Judges with Juries, evaluating LLM generations with a panel of diverse models.

Human and Deterministic Evaluation (7-8)

2 entries

Human review anchors every automated method, while deterministic logic gives repeatable checks for rules that can be written down.

147. Human EvaluationPeople score outputs on dimensions such as correctness, relevance and helpfulness. It is the reference point for validating automated judges, but it is slow and costly to scale. Chatbot Arena shows a large-scale pairwise version. 158. DAG-based EvaluationEncodes evaluation as deterministic decision logic, routing outputs through explicit checks to a final verdict. DeepEval documents a DAG metric that combines LLM judgements with decision-tree branching.

Agent-Specific Methods (9-10)

4 entries

Agents act over several steps and turns, so scoring only the final answer misses most failures.

169. Trajectory AccuracyEvaluates the sequence of actions an agent took, not just its final answer. The AgentEvals library provides trajectory matching and LLM-judge evaluators for LangChain-style agents. 17DeepEval task completionDocumentation for scoring whether an agent completed its task from its execution trace. 1810. Multi-turn EvaluationScores behaviour across a whole conversation, including consistency, memory and coherence. DeepEval documents conversational G-Eval for this. 19AI Agent Observability OverviewTracing and monitoring agents in production, on this site.

Safety Evaluation (11)

2 entries

The post treats safety as a gate: outputs are checked for toxicity, bias, sensitive-data leakage and policy violations before they are accepted.

2011. Safety EvaluationChecks outputs against safety categories before acceptance. Llama Guard is an example of an LLM-based input-output safeguard classifier. 21PurpleLlama (Meta)Meta's umbrella project for trust and safety tools, including Llama Guard and cybersecurity evaluations.

Evaluation Tools and Frameworks

11 entries

Open-source frameworks that implement these methods. Release facts are from each project's GitHub page.

22Opik (Comet)Open-source stack for tracing, debugging, test suites and agent evaluations; the post presents it as a way to build the evaluation layer. 23Opik documentationComet's documentation for Opik. 24DeepEvalOpen-source LLM evaluation framework with G-Eval, DAG, conversational and agent metrics. 25RagasEvaluation toolkit for RAG and LLM applications; GitHub lists v0.4.3 (January 2026) as the latest release. 26promptfooCLI and library for testing prompts, models and agents with assertions and LLM graders. 27Inspect (UK AISI)Framework for LLM evaluations, including agent evaluations, from the UK AI Security Institute. 28lm-evaluation-harness (EleutherAI)Few-shot benchmark harness for language models; GitHub lists v0.4.13 (August 2026) as the latest release. 29LightEval (Hugging Face)Hugging Face's evaluation toolkit; GitHub lists v0.13.0 (November 2025) as the latest release. 30Arize PhoenixOpen-source tracing and evaluation for LLM applications. 31LangfuseOpen-source LLM engineering platform with tracing, datasets and evaluations. 32OpenAI EvalsFramework and registry of evals; its last GitHub release is from May 2024, so check activity before adopting it.

Frequently Asked Questions

What are the main LLM evaluation methods?

The source post lists 11: BLEU, ROUGE and BERTScore (reference-based), G-Eval, LLM-as-Judge and LLM juries (judge-based), human evaluation and DAG-based evaluation, trajectory accuracy and multi-turn evaluation for agents, and safety evaluation.

When should I use BLEU, ROUGE or BERTScore?

When you have reference answers. BLEU measures n-gram precision, ROUGE measures recall of reference content and BERTScore compares contextual embeddings, so it tolerates paraphrase. None judges factual correctness.

What is the difference between G-Eval and LLM-as-a-Judge?

G-Eval scores an output against defined criteria using an LLM, while LLM-as-a-Judge is the broader pattern of giving a model a rubric to score or compare outputs. Both depend on the judge model and should be checked against human labels.

Why use an LLM jury?

A jury aggregates verdicts from several independent judges, which reduces reliance on one evaluator and its biases, at the cost of more calls.

How do you evaluate an AI agent?

Score the trajectory, meaning the actions taken, as well as the final answer, and score behaviour across whole conversations for consistency, memory and coherence.

How current is this page?

Links were checked on 30 October 2026. The method list follows a post by Daily Dose of Data Science on LinkedIn; paper and tool facts come from the papers and project pages. Tool versions change often, so check each repository.