Ethical Hacking Tools

Training objectives such as cross-entropy and squared error are maximum likelihood estimates under assumed noise models, and questions about overfitting, evaluation, uncertainty and data shift are statistical questions. Without them it is hard to tell a real improvement from noise.

Start with random variables and distributions, expectation and variance, maximum likelihood and Bayes' theorem, then the bias-variance tradeoff, cross-validation and the bootstrap. After that, calibration, hypothesis testing for model comparison and distribution shift matter most in practice.

Evaluate on held-out data, report uncertainty (for example confidence intervals from the bootstrap or variation across random seeds), and use an appropriate test when comparing models. The evaluation section lists papers on comparing classifiers and on error bars for language-model evaluations.

A model is calibrated when its stated probabilities match observed frequencies. Guo et al. found that modern neural networks are often overconfident, and the uncertainty section lists methods such as temperature scaling, deep ensembles and conformal prediction.

The classical U-shaped curve describes many models, but work on double descent shows that test error can fall again as models grow past the point of fitting the training data. The generalisation section lists the main papers.

That page covers probability, Bayesian inference and generative models. This page focuses on statistical practice for deep learning: estimation and regularisation, generalisation, evaluation and significance, uncertainty, distribution shift and experiment design.