LLM-as-a-Judge Solutions (2026) | DeepEval, Ragas, Langfuse, Phoenix, Braintrust, Prometheus, Judge Arena & Bias Research

What Is LLM-as-a-Judge?

11 entries

LLM-as-a-judge uses a capable language model to score or compare outputs from an AI system against criteria such as correctness, helpfulness, faithfulness to retrieved context or safety, as a scalable alternative to human review. It works best with clear rubrics, pairwise comparisons or reference answers, and judges that are themselves checked against human labels, because judges show position, verbosity and self-preference biases.

01Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaThe paper that established the approach and measured agreement with humans. 02G-Eval: NLG Evaluation using GPT-4 with Better Human AlignmentChain-of-thought rubrics and form-filling for scoring. 03A Survey on LLM-as-a-JudgeSurvey of methods, reliability and applications. 04LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation MethodsSurvey of evaluation approaches. 05Using LLM-as-a-Judge for evaluation (Hamel Husain)Practical guide to building judges that align with human judgement. 06Patterns for LLM Evaluators (Eugene Yan)Survey of patterns and pitfalls. 07LLM-as-a-judge: a complete guide (Evidently AI)Introduction with prompts and examples. 08LLM-as-a-Judge (Cameron Wolfe)Deep dive into techniques and findings. 09Using LLM-as-a-Judge for an automated evaluation (Hugging Face Cookbook)Step-by-step notebook. 10Define success criteria and build evaluations (Anthropic)Guidance on grading methods, including model-based grading. 11Demystifying evals for AI agents (Anthropic)How to design evaluations for agents.

Evaluation & Observability Platforms with Judges

12 entries

Platforms that run judges on traces and datasets.

12Langfuse: LLM-as-a-JudgeManaged judges over production traces and datasets. 13LangSmith evaluationOffline and online evaluators, including LLM judges. 14Arize PhoenixOpen-source observability and evaluation with LLM-based evaluators. 15BraintrustEvals, logging and prompt iteration with scorers. 16Opik (Comet)Open-source LLM evaluation and tracing. 17W&B WeaveTracing and evaluation from Weights & Biases (docs now on CoreWeave). 18GalileoEvaluation and observability for AI agents. 19Patronus AIEvaluators and judge models for LLM applications. 20Confident AIEvaluation platform built on DeepEval. 21HoneyHiveEvaluation and monitoring for AI agents. 22MLflow GenAI evaluationScorers and LLM judges in MLflow. 23Evidently AIOpen-source evaluation and monitoring with LLM judges.

Open-Source Evaluation Frameworks

13 entries

Libraries you run in your own pipelines.

24DeepEvalPytest-style LLM evaluation with G-Eval and many LLM-judged metrics. 25DeepEval custom LLM metricsG-Eval, DAG and custom judge metrics. 26RagasEvaluation for RAG and agents with LLM-based metrics. 27TruLensFeedback functions for evaluating and tracking LLM apps. 28PromptfooTest prompts and models with LLM-graded assertions and red-teaming. 29Inspect AIUK AI Security Institute framework with model-graded scorers. 30OpenAI EvalsFramework and registry of evaluations, including model-graded evals. 31OpenAI Evals guideHosted evals with model graders. 32lm-evaluation-harnessStandard harness for benchmarking language models. 33LightevalHugging Face's evaluation toolkit. 34GiskardTesting and red-teaming for LLM applications. 35UpTrainOpen-source LLM evaluation. 36LangfuseEvaluation overview.

Cloud Evaluation Services

4 entries

Judge-based evaluation from the major clouds.

37Google Gen AI evaluation serviceModel-based metrics and rubrics on Google Cloud. 38Microsoft Foundry evaluatorsGeneral-purpose and RAG evaluators, formerly Azure AI Foundry. 39Amazon Bedrock: LLM-as-a-judge evaluationEvaluate models and RAG with judge models. 40Amazon Bedrock evaluationAutomatic and human model evaluation.

Judge Models, Arenas & Meta-Benchmarks

12 entries

Open judge models and tests of the judges themselves.

41Prometheus-EvalOpen-source judge models and tooling for rubric-based evaluation. 42Prometheus models on Hugging FaceOpen evaluator language models. 43Prometheus: Inducing Fine-grained Evaluation CapabilityPaper on an open evaluator model. 44Prometheus 2Open model for evaluating other language models. 45Judge ArenaCompare LLM judges through human votes. 46Patronus GLIDERSmall judge model with explainable ratings. 47RewardBenchBenchmark of reward models and preference judgments. 48RewardBench paperEvaluating reward models for language modeling. 49JudgeBenchBenchmark of LLM judges on challenging responses. 50MT-Bench and FastChat LLM judgeOriginal judge code and data. 51AlpacaEvalAutomatic evaluator of instruction-following models. 52ArenaCrowd-sourced model rankings (formerly LMArena / Chatbot Arena).

Reliability & Bias Research

3 entries

Known weaknesses of LLM judges.

53Large Language Models are not Fair EvaluatorsShows position bias in LLM judges. 54LLM Evaluators Recognize and Favor Their Own GenerationsSelf-preference bias. 55Justice or Prejudice? Quantifying Biases in LLM-as-a-JudgeCatalogue of judge biases.

11 entries

Neighbouring pages on this site.

56LLMOpsOperating LLM applications. 57Agentic AI Observability ToolsTracing and monitoring agents. 58AI Agent Observability OverviewObservability for agents. 59Guardrails for Agentic AIGuardrails for agents. 60AI Agent Governance SolutionsGovernance for agents. 61AI Code Reviewing ToolsAI code review tools. 62Open-Source Benchmarks for AI Coding AgentsBenchmarks for coding agents. 63RAG ArchitecturesRAG architectures. 64AgenticOpsOperating agents. 65Agentic AIAgentic AI hub. 66Large Language ModelsLLM hub.

Frequently Asked Questions

What is LLM-as-a-judge?

Using a language model to grade or compare the outputs of another model or AI system against a rubric, as a scalable complement to human review.

How reliable are LLM judges?

They can match human preferences well on clear criteria, but show position, verbosity and self-preference biases. Use pairwise comparisons, swapped order, reference answers and calibration against human-labelled samples.

Which tools should I start with?

DeepEval, Ragas or Promptfoo for code-first testing, and Langfuse, LangSmith, Arize Phoenix or Braintrust when you also need tracing, datasets and online evaluation.

Can small open models act as judges?

Yes. Prometheus-Eval and GLIDER are open judge models, and benchmarks such as RewardBench and JudgeBench test how good any judge is.

How current is this list?

Links were checked in October 2026. Sites that block automated checks are kept and may need a manual click; renamed products are labelled.

LLM-as-a-Judge Solutions: Evaluation Platforms, Frameworks, Judge Models & Research