Evaluation Metrics for Deep Learning (2026) | Classification, Regression, Vision, NLP, LLM Benchmarks & Fairness

Choosing and Reading Metrics

4 entries

A deep learning model is only as good as the measurement used to judge it, and no single number fits every task. Classification uses accuracy, precision, recall, F1 and the area under ROC and precision-recall curves; regression and forecasting use error measures such as MAE, RMSE and MASE; vision uses intersection-over-union, mean average precision and generative-image scores such as FID; language tasks use BLEU, ROUGE, BERTScore, perplexity and increasingly benchmark suites and model-graded evaluation for large language models; retrieval uses ranking metrics such as nDCG; and trustworthy systems also need calibration, fairness, robustness and efficiency measures. This page collects definitions, papers, benchmarks and libraries for each area. Which metric to choose depends on the cost of different errors, class balance and what the model is used for, and the Statistics for Deep Learning page covers how to tell whether a difference between two scores is real.

01Statistics for Deep LearningEvaluation, significance testing, calibration and experiment design. 02Probability Theory for Deep LearningWeblog on likelihood, cross-entropy, KL divergence and perplexity. 03Mathematics for Deep LearningHub for the mathematics used in deep learning. 04Open-Source Benchmarks for AI Coding AgentsBenchmarks used to evaluate coding agents.

Classification Metrics

9 entries

Measures for predicting classes, from accuracy to ranking quality.

05scikit-learn: Classification metricsReference implementations of accuracy, precision, recall, F-scores, ROC AUC and more. 06Confusion matrixThe table of true and predicted classes from which most metrics are computed. 07Precision and recallDefinitions of precision, recall and the F-measure. 08Receiver operating characteristicROC curves and the area under the curve. 09The Relationship Between Precision-Recall and ROC CurvesDavis and Goadrich on when precision-recall curves are more informative on imbalanced data. 10Matthews correlation coefficientA balanced single-number summary of a binary confusion matrix. 11Cohen's kappaAgreement between predictions and labels corrected for chance. 12Brier scoreMean squared error of predicted probabilities. 13Cross-entropy (log loss)The loss used for training and for scoring predicted probabilities.

Regression & Forecasting Metrics

8 entries

Error measures for numeric and time-series predictions.

14scikit-learn: Regression metricsMAE, MSE, R-squared, MAPE, pinball loss and other regression scores. 15Mean squared errorAverage squared prediction error, which penalises large errors. 16Coefficient of determination (R-squared)Share of variance explained by the model. 17Mean absolute percentage errorA scale-free error measure with known weaknesses near zero. 18Another look at measures of forecast accuracy (MASE)Hyndman and Koehler propose the mean absolute scaled error. 19Quantile regression (pinball loss)Loss for evaluating predicted quantiles and intervals. 20Huber lossA loss that is less sensitive to outliers than squared error. 21Continuous ranked probability scoreA proper score for probabilistic forecasts.

Computer Vision & Image Generation Metrics

11 entries

Detection, segmentation and image quality measures.

22Jaccard index (intersection over union)IoU, the overlap measure behind detection and segmentation scores. 23Sørensen–Dice coefficientOverlap measure widely used for segmentation. 24COCO detection evaluationHow mean average precision is computed for COCO object detection. 25Panoptic SegmentationIntroduces the panoptic quality metric. 26Structural similarity index (SSIM)Perceptual image similarity measure. 27Peak signal-to-noise ratioReconstruction quality measure for images and video. 28GANs Trained by a Two Time-Scale Update Rule (FID)Introduces the Fréchet Inception Distance. 29Improved Techniques for Training GANs (Inception Score)Introduces the Inception Score. 30Demystifying MMD GANs (KID)Introduces the Kernel Inception Distance. 31The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (LPIPS)Learned perceptual image patch similarity. 32CLIPScoreReference-free evaluation of image captions using CLIP.

Language, Translation & Speech Metrics

11 entries

Text overlap, embedding-based and recognition metrics.

33BLEU: a Method for Automatic Evaluation of Machine TranslationThe original BLEU paper. 34A Call for Clarity in Reporting BLEU ScoresArgues for standardised scoring and introduces sacreBLEU. 35ROUGE: A Package for Automatic Evaluation of SummariesThe original ROUGE paper. 36METEORTranslation metric that adds stemming, synonyms and word order. 37chrF: character n-gram F-scoreCharacter-level metric for machine translation. 38BERTScoreText generation evaluation using contextual embeddings. 39BLEURTLearned evaluation metric based on BERT. 40COMETNeural framework for machine translation evaluation. 41Perplexity of fixed-length models (Hugging Face)Practical guide to computing language-model perplexity. 42SQuAD: 100,000+ Questions for Machine Comprehension of TextIntroduces exact-match and F1 scoring for question answering. 43Word error rateThe standard metric for speech recognition.

LLM Evaluation & Benchmarks

15 entries

Benchmarks, harnesses and model-graded evaluation for large language models.

44Measuring Massive Multitask Language Understanding (MMLU)Multiple-choice knowledge benchmark across 57 subjects. 45Holistic Evaluation of Language Models (HELM)Framework evaluating models across many scenarios and metrics. 46Beyond the Imitation Game (BIG-bench)Collaborative benchmark of hundreds of tasks. 47Evaluating Large Language Models Trained on Code (HumanEval, pass@k)Introduces HumanEval and the pass@k metric. 48SWE-benchBenchmark of real GitHub issues for software engineering agents. 49TruthfulQABenchmark of whether models imitate human falsehoods. 50GPQAGraduate-level science questions designed to resist search. 51Instruction-Following Evaluation (IFEval)Verifiable instruction-following benchmark. 52Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaStudies using strong LLMs to grade other models. 53Chatbot ArenaCrowdsourced pairwise preference platform for ranking models. 54RagasReference-free evaluation of retrieval-augmented generation. 55Adding Error Bars to EvalsStatistical approach to reporting uncertainty in LLM evaluations. 56EleutherAI lm-evaluation-harnessOpen framework for running language models on many benchmarks. GitHub lists v0.4.13 as the latest release, from May 2026. 57Inspect AIOpen evaluation framework from the UK AI Security Institute. 58OpenAI EvalsFramework and registry of evaluations for language models.

Retrieval, Ranking & Recommendation Metrics

5 entries

Scoring ordered lists and embeddings.

59Evaluation measures (information retrieval)Precision at k, recall, average precision, nDCG and related measures. 60Discounted cumulative gainDCG and normalised DCG for graded relevance. 61Mean reciprocal rankAverage of the reciprocal rank of the first relevant result. 62BEIRHeterogeneous benchmark for zero-shot retrieval. 63MTEB: Massive Text Embedding BenchmarkBenchmark of text embedding models across tasks.

Calibration, Fairness, Robustness & Efficiency

8 entries

Metrics beyond accuracy for trustworthy and practical models.

64On Calibration of Modern Neural NetworksIntroduces expected calibration error for deep networks. 65Fairness and Machine Learning (Barocas, Hardt, Narayanan)Free book on fairness definitions and their trade-offs. 66FairlearnToolkit for assessing and improving fairness. 67AI Fairness 360Open-source toolkit of fairness metrics and mitigation algorithms. GitHub lists v0.6.1 as the latest release, from February 2024, so check its activity before relying on it. 68RobustBenchStandardised benchmark for adversarial robustness. 69Benchmarking Neural Network Robustness to Common Corruptions (ImageNet-C)Corruption and perturbation benchmarks for image classifiers. 70MLPerf Inference BenchmarkDescribes the MLPerf inference benchmark suite for latency and throughput. 71CodeCarbonEstimates the carbon emissions of computing workloads.

Training Diagnostics & Experiment Tracking

4 entries

Watching metrics during training and across runs.

72scikit-learn: Learning curvesDiagnosing bias and variance from training and validation scores. 73TensorBoardVisualises loss, metrics and other training signals. 74Weights & BiasesExperiment tracking and visualisation platform. 75MLflowOpen-source platform for tracking experiments and evaluating models.

Metric Libraries

6 entries

Software that implements the metrics above.

76scikit-learn metricsStandard metrics for classification, regression and clustering. 77TorchMetricsPyTorch metrics with distributed support. 78Keras metricsBuilt-in metrics for Keras models. 79Hugging Face EvaluateLibrary of metrics and comparisons for models and datasets. 80sacreBLEUReproducible BLEU, chrF and TER scoring. GitHub lists v2.6.0 as the latest release, from January 2025, so check its activity before relying on it. 81pycocotoolsOfficial COCO API for detection and segmentation evaluation.

10 entries

Neighbouring pages on this site.

82Deep LearningFrameworks, algorithms and datasets. 83Machine LearningMachine learning hub. 84LLMOpsOperating and monitoring language-model applications. 85ML Model Monitoring ToolsTools for monitoring models in production. 86StatisticsFree textbooks, open data, software and journals. 87Top Deep Learning AlgorithmsCore deep learning algorithms. 88Hyperparameter Optimisation ToolsTools for tuning models. 89Vector DatabasesThe retrieval layer behind RAG. 90RAG Architectures ElucidatedRetrieval-augmented generation patterns. 91Deployment Strategies for Deep Learning ModelsServing modes, rollout strategies, model servers, optimisation, edge and LLM deployment.

Frequently Asked Questions

Why does deep learning need statistics?

Training objectives such as cross-entropy and squared error are maximum likelihood estimates under assumed noise models, and questions about overfitting, evaluation, uncertainty and data shift are statistical questions. Without them it is hard to tell a real improvement from noise.

What statistics topics should I learn first for deep learning?

Start with random variables and distributions, expectation and variance, maximum likelihood and Bayes' theorem, then the bias-variance tradeoff, cross-validation and the bootstrap. After that, calibration, hypothesis testing for model comparison and distribution shift matter most in practice.

How do I know whether one model is really better than another?

Evaluate on held-out data, report uncertainty (for example confidence intervals from the bootstrap or variation across random seeds), and use an appropriate test when comparing models. The evaluation section lists papers on comparing classifiers and on error bars for language-model evaluations.

What is calibration and why does it matter?

A model is calibrated when its stated probabilities match observed frequencies. Guo et al. found that modern neural networks are often overconfident, and the uncertainty section lists methods such as temperature scaling, deep ensembles and conformal prediction.

Does the bias-variance tradeoff still apply to large neural networks?

The classical U-shaped curve describes many models, but work on double descent shows that test error can fall again as models grow past the point of fitting the training data. The generalisation section lists the main papers.

How is this page different from Probability Theory for AI?

That page covers probability, Bayesian inference and generative models. This page focuses on statistical practice for deep learning: estimation and regularisation, generalisation, evaluation and significance, uncertainty, distribution shift and experiment design.