Evaluating & Improving Agentic Applications (2026): Frameworks, Benchmarks, Tools

1. Foundations: What to Evaluate in an Agent

11 entries

Guidance on measuring agents: outcomes, trajectories, tool use and cost. Evaluate the final result and the path the agent took to reach it.

01Anthropic: Demystifying Evals for AI AgentsEngineering guide to building evaluations for agents, graders and tasks. 02Anthropic: Building Effective AgentsWorkflows versus agents, with advice on testing in sandboxes and adding guardrails. 03OpenAI: Evaluate Agent WorkflowsTrace grading and datasets for assessing agent workflows. 04OpenAI: Working with EvalsCreate evals, datasets and graders on the OpenAI platform. 05Google Cloud: Evaluate Gen AI Agents (Vertex AI)Agent evaluation on Vertex AI, including trajectory and response metrics. 06Google ADK: Evaluate AgentsEvaluate tool-use trajectories and final responses in the Agent Development Kit. 07Microsoft Foundry: Agent EvaluatorsBuilt-in evaluators for intent resolution, tool-call accuracy and task adherence. 08AWS: Amazon Bedrock AgentCore EvaluationsManaged evaluation of agent quality on AWS. 09LangChain: Evaluating Deep AgentsLessons on evaluating long-running agents. 10Hamel Husain: Your AI Product Needs EvalsPractitioner guide to error analysis and evaluation loops. 11Eugene Yan: Patterns for Building LLM-based SystemsEvals, guardrails and feedback loops for LLM products.

2. Open-Source Evaluation Frameworks

19 entries

Libraries you run yourself to score agents and LLM applications. Pick one that fits your language, tracing and CI setup.

12DeepEvalPytest-style framework with agent metrics such as task completion and tool correctness. 13DeepEval on GitHubSource for the DeepEval evaluation framework. 14RagasMetrics for RAG and agents, including tool-call accuracy and goal accuracy. 15Ragas on GitHubSource for Ragas. 16promptfooTest and red-team prompts, agents and RAG from the command line and CI. 17promptfoo on GitHubSource for promptfoo. 18OpenAI EvalsFramework and registry of evals for language models. 19UK AISI InspectFramework from the UK AI Security Institute for evaluating models and agents. 20Inspect EvalsCommunity collection of ready-made evaluations for Inspect. 21EleutherAI lm-evaluation-harnessStandard harness for benchmarking language models. 22Arize PhoenixOpen-source tracing and evaluation for LLM applications and agents. 23Opik by CometOpen-source tracing, evaluation and agent optimisation. 24TruLensEvaluate and track LLM apps and agents with feedback functions. 25GiskardTesting and red-teaming for LLM agents and RAG. 26MLflow GenAI EvaluationEvaluate and monitor GenAI apps and agents with MLflow. 27AgentaOpen-source prompt management, evaluation and observability. 28UpTrainOpen-source toolkit for evaluating and improving LLM applications. 29OpenEvalsReady-made evaluators from LangChain, including LLM-as-judge and agent trajectory checks. 30AgentEvalsEvaluators for agent trajectories.

3. Evaluation and Observability Platforms

14 entries

Hosted or self-hostable platforms that combine tracing, datasets, evals and monitoring. Most support OpenTelemetry-style traces and CI gating.

31LangSmithTracing, datasets and evaluation for LLM applications and agents. 32LangfuseOpen-source tracing, prompt management and evaluation. 33BraintrustEvals, tracing and experiments for AI products. 34Weights & Biases WeaveTrace, evaluate and iterate on LLM applications. 35GalileoEvaluation and observability for agents, with agent-specific metrics. 36Arize AXObservability and evaluation platform for AI agents. 37Confident AIPlatform built around DeepEval for testing and monitoring. 38Patronus AIEvaluation and monitoring models for generative AI. 39Maxim AISimulation, evaluation and observability for AI agents. 40AgentOpsMonitoring, cost tracking and debugging for agents. 41HeliconeOpen-source LLM observability and gateway. 42Datadog LLM ObservabilityTrace agents and run evaluations within Datadog. 43Azure AI Foundry ObservabilityEvaluate, trace and monitor generative AI apps on Azure. 44Vertex AI Gen AI Evaluation ServiceGoogle Cloud's managed evaluation service.

4. Agent Benchmarks

12 entries

Public benchmarks for measuring agent capability across tool use, web, code and computer tasks. Use them to compare models and agents, then build your own for your tasks.

45SWE-benchResolving real GitHub issues in software repositories. 46τ-bench (tau-bench)Tool-agent-user interaction in retail and airline domains. 47GAIAGeneral AI assistant benchmark of real-world questions needing tools. 48AgentBenchEvaluating LLMs as agents across eight environments. 49WebArenaRealistic web environments for autonomous agents. 50OSWorldComputer-use tasks across real operating systems. 51Terminal-BenchAgents completing tasks in a terminal. 52BrowseCompOpenAI benchmark for browsing agents finding hard-to-locate information. 53ToolBenchLarge-scale tool-use dataset and evaluation. 54Berkeley Function Calling LeaderboardRanks models on function and tool calling. 55MLE-benchMachine learning engineering tasks drawn from Kaggle competitions. 56Open-Source Benchmarks for AI Coding AgentsCoding agent benchmarks on this site.

5. Evaluation Methods: Graders, Trajectories and LLM-as-Judge

8 entries

How agent outputs and trajectories are scored. Combine code-based checks, model graders and human review.

57Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaFoundational study of using strong models as judges, with known biases. 58G-EvalCriteria-based scoring with chain-of-thought and form filling. 59Agent-as-a-JudgeUsing agentic systems to evaluate other agentic systems. 60A Survey on LLM-as-a-JudgeSurvey of reliability, bias and methods for model judges. 61Survey on Evaluation of LLM-based AgentsMaps benchmarks and frameworks for agent evaluation. 62Why Do Multi-Agent LLM Systems Fail?Failure taxonomy useful for error analysis. 63LLM Evaluation MethodsBLEU, G-Eval, LLM-as-judge and agent evaluation on this site. 64Evaluation Metrics for Deep LearningClassic metrics for deep learning models.

6. Improving Agents: Feedback, Optimisation and Self-Improvement

10 entries

Techniques and tools that use evaluation results to make agents better. Close the loop: evaluate, find the failure, change the prompt, tools or model, then re-test.

65ReflexionAgents improve by reflecting on feedback and storing it in memory. 66Self-RefineIterative refinement using the model's own feedback. 67DSPyProgrammatic prompt and pipeline optimisation against a metric. 68GEPA: Reflective Prompt EvolutionPrompt optimisation using reflection on trajectories. 69TextGradAutomatic differentiation via text feedback. 70OPRO: Large Language Models as OptimizersUsing an LLM to optimise prompts against a score. 71Agent LightningTrain agents with reinforcement learning from their traces. 72OpenAI: Reinforcement Fine-TuningFine-tune models against graders. 73Anthropic: Writing Effective Tools for AgentsImprove tools through evaluation-driven iteration. 74Anthropic: Effective Context Engineering for AI AgentsManaging context to improve agent reliability.

7. Related Pages on This Site

8 entries

Neighbouring directories for operating and securing agents. Evaluation sits beside observability, guardrails and operations.

75AI Agent Observability: MELT Framework OverviewMetrics, events, logs and traces for agents. 76AgenticOps (AgentOps)Operating autonomous AI agents. 77LLMOpsOperating LLM applications in production. 78Guardrails for an AI AgentFive checkpoints and tools. 79Top Agentic AI FrameworksFrameworks for building agents. 80AI Agent FrameworksAgent frameworks compared. 81Multi-Agent System PatternsDesign, orchestration and decomposition patterns. 82Observability ToolsObservability platforms and tools.

Frequently Asked Questions

How do you evaluate an agentic application?

Define tasks with clear success criteria, run the agent in a controlled environment, and score both the final outcome and the trajectory: the tools called, their arguments, the order of steps, cost and latency. Combine code-based checks, model graders and human review, and run the suite on every change.

How is evaluating an agent different from evaluating an LLM?

A model answers once, while an agent takes many steps, calls tools and changes state. You must grade the end state and the path, handle non-determinism by running tasks several times, and check side effects such as files or records changed.

What are the main evaluation frameworks for agents?

Open-source options include DeepEval, Ragas, promptfoo, UK AISI Inspect, OpenAI Evals, OpenEvals and AgentEvals, Arize Phoenix, Opik and TruLens. Platforms that add tracing, datasets and monitoring include LangSmith, Langfuse, Braintrust, Galileo and Weights & Biases Weave.

Which metrics matter for agents?

Common ones are task completion, tool-call accuracy, argument correctness, step efficiency, goal accuracy, faithfulness to retrieved context, safety violations, cost and latency. Pick a few that match how users judge success rather than tracking everything.

Can an LLM judge an agent's work?

Yes, LLM-as-judge and agent-as-judge methods are widely used, but judges show biases such as favouring longer answers. Calibrate them against human labels, use clear rubrics, and prefer deterministic checks where a result can be verified by code.

Which benchmarks test agents?

Public benchmarks include SWE-bench for code, tau-bench for tool-agent-user dialogue, GAIA for assistant tasks, WebArena for web tasks, OSWorld for computer use, Terminal-Bench for command line work and the Berkeley Function Calling Leaderboard. They compare models, but your own tasks should decide what ships.

How do you improve an agent after evaluating it?

Read failed traces, group the errors, then change the prompt, tool descriptions, context, model or workflow and re-run the suite. Techniques such as DSPy, GEPA, Reflexion and reinforcement fine-tuning automate parts of this loop against a metric.

How current is this list?

Links were checked in November 2026. A few sites block automated checks and may need a manual click. This page is a resource list, not engineering advice, and tool descriptions reflect each project's own positioning.