Statistics for Deep Learning logopeterindia.net
Knowledge Hub

Statistics for Deep Learning

The statistics behind deep learning — estimation and regularisation, generalisation, honest evaluation, uncertainty and calibration, distribution shift and experiment design.

90 topics•10 categories•Updated October 2026

Why Statistics Matters for Deep Learning

3 entries

Deep learning is statistical estimation at scale. Training by minimising cross-entropy or squared error is maximum likelihood estimation under an assumed noise model; weight decay corresponds to a prior in maximum a posteriori estimation; and the gap between training and test performance is the generalisation question that statistical learning theory studies. Statistics also decides whether an evaluation can be trusted: how large a test set must be, whether a difference between two models is more than noise, how results vary across random seeds, whether stated confidence is calibrated, and whether the data seen in deployment resembles the training data. This page collects free books, courses, papers and tools for each of these areas. For the probability side, see the Probability Theory for AI page.

01Maximum likelihood estimationChoosing parameters that make the observed data most probable, the principle behind standard training losses. 02Bias–variance tradeoffHow model complexity trades systematic error against sensitivity to the training sample. 03Empirical risk minimizationMinimising average loss on training data as a proxy for expected loss on new data.

Foundations & Learning Resources

11 entries

Textbooks and courses for the statistics behind machine learning.

04An Introduction to Statistical LearningFree textbook with applications in R and Python, a gentle entry to statistical learning. 05The Elements of Statistical LearningFree graduate-level text by Hastie, Tibshirani and Friedman. 06Probabilistic Machine Learning (Murphy)Kevin Murphy's books on machine learning from a probabilistic and statistical view. 07Understanding Deep Learning (Prince)Free book that derives loss functions from maximum likelihood and covers generalisation. 08Deep Learning book: Machine Learning BasicsChapter 5: estimators, bias and variance, maximum likelihood and Bayesian statistics. 09MIT 18.650 Statistics for ApplicationsOpen course on estimation, hypothesis testing and regression. 10All of Statistics (Wasserman)Concise text on statistical inference for readers with a computing background. 11Think StatsFree book teaching statistics through Python and real data. 12Dive into Deep Learning: Probability and StatisticsInteractive chapter on probability and statistics for deep learning. 13StatQuestShort video explanations of statistics and machine learning concepts. 14Seeing TheoryVisual introduction to probability and statistics from Brown University.

Estimation, Loss Functions & Regularisation

8 entries

How training objectives and regularisers have statistical meanings.

15Cross-entropyThe loss that equals negative log-likelihood for classification. 16Maximum a posteriori estimationEstimation with a prior; L2 weight decay corresponds to a Gaussian prior. 17Decoupled Weight Decay Regularization (AdamW)Shows why weight decay and L2 regularisation differ under adaptive optimisers. 18Dropout: A Simple Way to Prevent OverfittingThe JMLR paper introducing dropout as a regulariser. 19Batch NormalizationNormalising layer inputs using mini-batch statistics. 20Rethinking the Inception Architecture (label smoothing)Introduces label smoothing as a regulariser for classification. 21Early stoppingStopping training when validation error stops improving, a form of regularisation. 22Bootstrap aggregating (bagging)Averaging models trained on resampled data, the statistical idea behind ensembles.

Generalisation & Statistical Learning Theory

8 entries

Why networks that fit their training data can still generalise.

23Understanding deep learning requires rethinking generalizationShows networks can fit random labels, challenging classical explanations. 24Reconciling modern machine-learning practice and the bias–variance trade-offDescribes the double-descent risk curve. 25Deep Double DescentReports double descent across model size, training time and data size. 26Scaling Laws for Neural Language ModelsEmpirical power-law relationships between loss, model size, data and compute. 27Training Compute-Optimal Large Language ModelsThe Chinchilla study of how to balance model size and training tokens. 28Vapnik–Chervonenkis theoryClassical theory of capacity and generalisation bounds. 29Probably approximately correct learningFramework for how many examples a learner needs. 30Rademacher complexityA data-dependent measure of function-class richness.

Evaluation, Metrics & Significance Testing

10 entries

Measuring models honestly and knowing when a difference is real.

31scikit-learn: Metrics and scoringReference for classification, regression and ranking metrics. 32scikit-learn: Cross-validationResampling schemes for estimating generalisation performance. 33Bootstrapping (statistics)Resampling to estimate uncertainty without distributional assumptions. 34Statistical Comparisons of Classifiers over Multiple Data SetsDemšar's recommended tests for comparing learners across datasets. 35Adding Error Bars to EvalsStatistical approach to reporting uncertainty in language-model evaluations. 36Deep Reinforcement Learning that MattersShows how seeds and implementation details change reported results. 37Troubling Trends in Machine Learning ScholarshipCritique of evaluation and reporting practices. 38The Relationship Between Precision-Recall and ROC CurvesDavis and Goadrich on when each curve is more informative. 39Statistical hypothesis testBackground on p-values, significance and power. 40Multiple comparisons problemWhy testing many hypotheses or model variants inflates false positives.

Uncertainty, Calibration & Bayesian Deep Learning

8 entries

Knowing what a model does not know.

41On Calibration of Modern Neural NetworksFinds modern networks are often overconfident and evaluates temperature scaling. 42Deep EnsemblesSimple, scalable predictive uncertainty from ensembles of networks. 43Dropout as a Bayesian ApproximationInterprets dropout at test time as approximate Bayesian inference. 44Bayesian Deep Learning and a Probabilistic Perspective of GeneralizationArgues for marginalisation over parameters in deep networks. 45A Gentle Introduction to Conformal PredictionDistribution-free prediction sets with coverage guarantees. 46Laplace ReduxEffortless Bayesian deep learning with Laplace approximations. 47Uncertainty BaselinesHigh-quality implementations of baseline uncertainty methods. 48Calibration (statistics)Background on calibrated probabilities and reliability diagrams.

Data, Sampling & Distribution Shift

10 entries

When the data does not match the assumptions.

49WILDSBenchmark of real-world distribution shifts. 50Do ImageNet Classifiers Generalize to ImageNet?Measures accuracy drops on a newly collected test set. 51Shortcut Learning in Deep Neural NetworksDescribes models relying on spurious features. 52Underspecification Presents Challenges for Credibility in Modern Machine LearningShows equivalent validation performance can hide different deployed behaviour. 53Datasheets for DatasetsDocumentation practice for dataset provenance and limits. 54Confident LearningEstimating label noise in datasets. 55cleanlabTools for finding label errors and data issues. 56imbalanced-learnPython library for resampling imbalanced datasets. 57Dataset shiftCovariate, prior and concept shift explained. 58Data leakage (machine learning)How information from outside the training set contaminates evaluation.

Experiments, Tuning & Reproducibility

8 entries

Designing experiments that give trustworthy answers.

59Random Search for Hyper-Parameter OptimizationWhy random search often beats grid search. 60OptunaHyperparameter optimisation framework. 61Deep Learning Tuning PlaybookGoogle Research guide to systematically tuning deep learning models. 62A Recipe for Training Neural NetworksAndrej Karpathy's practical checklist for training and debugging. 63Rules of Machine LearningGoogle guide to practical machine learning engineering. 64Trustworthy Online Controlled ExperimentsBook and site on A/B testing methodology. 65Design of experimentsFoundations of planning comparisons and ablations. 66Power of a testDetermining how much data a comparison needs to detect an effect.

Libraries & Tools

10 entries

Software for statistical analysis and probabilistic modelling.

67SciPy statsDistributions, tests and descriptive statistics for Python. 68statsmodelsStatistical models, tests and data exploration in Python. 69scikit-learnMachine learning library with evaluation and model-selection tools. 70PyMCProbabilistic programming library for Bayesian modelling. 71StanPlatform for statistical modelling and Bayesian inference. 72ArviZDiagnostics and visualisation for Bayesian models. 73PyroProbabilistic programming on PyTorch. 74TensorFlow ProbabilityProbabilistic reasoning and statistical analysis in TensorFlow. 75torch.distributionsProbability distributions and sampling in PyTorch. 76TorchMetricsMetrics library for PyTorch models.

Related Pages

14 entries

Neighbouring pages on this site.

77Probability Theory for AIBayesian inference, uncertainty and generative models. 78Linear Algebra for AIVectors, matrices, SVD and tensors. 79Linear Algebra for Deep LearningWeblog on the matrix maths inside deep learning. 80StatisticsFree textbooks, open data, software and journals. 81Deep LearningFrameworks, algorithms and datasets. 82Machine LearningMachine learning hub. 83MathematicsMathematics resources. 84Hyperparameter Optimisation ToolsTools for tuning models. 85Data ScienceData science resources. 86Causal AICausal inference and models. 87Python Libraries for Data ScientistsPython data science libraries. 88Probability Theory for Deep LearningA worked explainer: likelihood and loss functions, softmax, entropy, priors, VAEs, diffusion and sampling. 89Mathematics for Deep LearningHub for the mathematics used in deep learning: linear algebra, calculus, optimisation, probability and numerical computing. 90Evaluation Metrics for Deep LearningClassification, regression, vision, language and LLM metrics, benchmarks and libraries.

Frequently Asked Questions

Why does deep learning need statistics?

Training objectives such as cross-entropy and squared error are maximum likelihood estimates under assumed noise models, and questions about overfitting, evaluation, uncertainty and data shift are statistical questions. Without them it is hard to tell a real improvement from noise.

What statistics topics should I learn first for deep learning?

Start with random variables and distributions, expectation and variance, maximum likelihood and Bayes' theorem, then the bias-variance tradeoff, cross-validation and the bootstrap. After that, calibration, hypothesis testing for model comparison and distribution shift matter most in practice.

How do I know whether one model is really better than another?

Evaluate on held-out data, report uncertainty (for example confidence intervals from the bootstrap or variation across random seeds), and use an appropriate test when comparing models. The evaluation section lists papers on comparing classifiers and on error bars for language-model evaluations.

What is calibration and why does it matter?

A model is calibrated when its stated probabilities match observed frequencies. Guo et al. found that modern neural networks are often overconfident, and the uncertainty section lists methods such as temperature scaling, deep ensembles and conformal prediction.

Does the bias-variance tradeoff still apply to large neural networks?

The classical U-shaped curve describes many models, but work on double descent shows that test error can fall again as models grow past the point of fitting the training data. The generalisation section lists the main papers.

How is this page different from Probability Theory for AI?

That page covers probability, Bayesian inference and generative models. This page focuses on statistical practice for deep learning: estimation and regularisation, generalisation, evaluation and significance, uncertainty, distribution shift and experiment design.