Probability gives deep learning its objectives (maximum likelihood), its output layers (softmax and friends), its regularisers (priors), its generative models (VAEs, diffusion, autoregressive language models) and its vocabulary for uncertainty. For a curated list of books, courses and tools, see the Probability Theory for AI directory on this site; this article is the explanation, and its companion on linear algebra covers the matrix side.
01Networks as Conditional Distributions
A supervised model is best understood as defining a conditional distribution p(y | x; θ): given an input x and parameters θ, how likely is each possible output y? A classifier gives a categorical distribution over classes, a regression network gives the parameters of a distribution over a number (usually the mean of a Gaussian), and a language model gives a categorical distribution over the next token. Training means choosing θ so that the distribution assigns high probability to the data we actually observed.
02Random Variables, Expectation and Variance
A random variable takes values according to a distribution. Two summaries are used constantly. The expectation E[X] is the probability-weighted average, and the variance Var[X] = E[(X − E[X])²] measures spread. Almost every training objective is an expectation, namely the average loss over the data distribution, and almost every training run estimates it by averaging over a finite sample. The distributions that appear most are the Bernoulli (one binary outcome), the categorical (one of K outcomes) and the Gaussian (a continuous value with a mean and a variance).
03Likelihood: Where Loss Functions Come From
The likelihood of the training data is the probability the model assigns to it. For independent examples it is a product, so the standard move is to take the logarithm, which turns the product into a sum and avoids numerical underflow. Maximising likelihood is then the same as minimising the negative log-likelihood (NLL):
The familiar losses are special cases, depending on the distribution you assume for the output:
- Categorical output gives cross-entropy loss. If the model gives the true class probability 0.665, that example's loss is -ln 0.665 ≈ 0.408.
- Bernoulli output (a sigmoid unit) gives binary cross-entropy.
- Gaussian output with fixed variance gives mean squared error: -log N(y; μ, σ²) = (y - μ)² / (2σ²) plus a constant, so minimising it with respect to μ is minimising squared error.
- Laplace output gives absolute error (L1 loss).
Choosing a loss is therefore choosing a noise model. That is why squared error is a poor fit for heavy-tailed noise and why classification uses cross-entropy rather than squared error on probabilities.
04Softmax and Temperature
The network's final layer produces real-valued scores called logits. Softmax converts them into a categorical distribution, and a temperature T rescales the logits first:
For logits [2, 1, 0] and T = 1, the exponentials are 7.389, 2.718 and 1, which sum to 11.107, so the probabilities are 0.665, 0.245 and 0.090. A lower temperature sharpens the distribution toward the largest logit; a higher one flattens it toward uniform. Softmax is unchanged if the same constant is subtracted from every logit, and implementations use this to avoid overflow, computing with the log-sum-exp trick. Frameworks fuse softmax and the log in the loss for the same numerical reason, which is why classification losses usually take logits rather than probabilities.
05Entropy, Cross-Entropy and KL Divergence
Entropy measures the average surprise of a distribution p, and cross-entropy measures the average surprise when events from p are scored using a model q. KL divergence is the gap between them:
Because the data distribution's entropy H(p) does not depend on the model, minimising cross-entropy is the same as minimising KL(p ‖ q), pushing the model distribution toward the data distribution. The KL divergence is not symmetric, so KL(p ‖ q) and KL(q ‖ p) behave differently; the direction used in a method has consequences for what it does.
For language models, perplexity is the exponential of the average cross-entropy per token (in nats). A model that spreads probability uniformly over a 50,000-token vocabulary has a loss of ln 50,000 ≈ 10.82 nats and a perplexity of 50,000; a model that is on average as unsure as a fair choice between 20 tokens has a perplexity of 20. Perplexity values are only comparable between models that use the same tokeniser and test text.
06Priors, MAP and Regularisation
Bayes' theorem says the posterior over parameters is proportional to likelihood times prior: p(θ | D) ∝ p(D | θ) p(θ). Maximum a posteriori (MAP) estimation maximises this, so the loss gains a term, -log p(θ). With a zero-mean Gaussian prior on the weights, that term is proportional to the squared weight norm, which is L2 regularisation; with a Laplace prior it is proportional to the L1 norm. Regularisation can therefore be read as a belief that weights should be small.
A fully Bayesian treatment keeps the whole posterior rather than a single point estimate and averages predictions over it, but that is expensive for large networks. Practical approximations include deep ensembles (train several networks and average), Monte Carlo dropout (Gal and Ghahramani interpret dropout at test time as approximate Bayesian inference) and Laplace approximations.
07Randomness in Training
Training is stochastic by design. A minibatch gradient is an unbiased estimate of the full-data gradient: its expectation over random batches equals the true gradient, and its variance shrinks as the batch grows. Weights are initialised randomly, data is shuffled, and dropout multiplies activations by random Bernoulli masks. This explains why results differ between runs with different seeds, which is why a single run is weak evidence when comparing methods.
08Generative Models
Generative models learn a distribution over data itself. Three families dominate:
- Autoregressive models factor the joint probability with the chain rule, p(x) = Π p(xₜ | x₍<t₎), and train each factor with cross-entropy. This is how language models work.
- Variational autoencoders (Kingma and Welling) cannot compute the likelihood directly, so they maximise a lower bound, the ELBO: log p(x) ≥ E_q[log p(x | z)] - KL(q(z | x) ‖ p(z)). The reparameterisation trick writes a sample as z = μ + σ ⊙ ε with ε drawn from a standard normal, which lets gradients flow through the sampling step.
- Diffusion models (Ho et al.) define a fixed process that gradually adds Gaussian noise to data and train a network to reverse it, so generation is a sequence of denoising steps.
At generation time, a distribution has to be turned into an output. Greedy decoding takes the most likely token; sampling draws from the distribution, optionally after changing the temperature or truncating it with top-k or nucleus (top-p) sampling. Holtzman et al. introduced nucleus sampling after finding that maximising likelihood tends to produce dull, repetitive text, whereas sampling the unreliable tail of the distribution produces incoherent text.
09Uncertainty and Calibration
Kendall and Gal distinguish two kinds of uncertainty. Aleatoric uncertainty comes from noise in the data and does not go away with more data; epistemic uncertainty comes from limited knowledge about the model and can shrink with more data. A model is calibrated when its stated probabilities match observed frequencies, so that predictions made with 80% confidence are right about 80% of the time. Guo et al. found that modern deep networks are often overconfident, and showed that a simple post-hoc fix, temperature scaling, can help considerably. A high softmax probability is therefore not automatically a trustworthy confidence.
Softmax is not a probability of being right
Softmax outputs sum to one by construction, even for inputs unlike anything seen in training. Always evaluate calibration on held-out data, especially for inputs that differ from the training distribution.
10Cheat Sheet: Idea to Where It Appears
| Idea | Where it appears | Key fact |
|---|---|---|
| Maximum likelihood | Almost every training loss | Minimise negative log-likelihood |
| Categorical + softmax | Classifiers, next-token prediction | Loss is -log p(true class) |
| Gaussian noise model | Regression | Fixed variance gives mean squared error |
| Cross-entropy and KL | Training and distillation | H(p, q) = H(p) + KL(p ‖ q) |
| Perplexity | Language-model evaluation | exp(average cross-entropy per token) |
| Prior and MAP | Weight decay, L1 and L2 | Gaussian prior gives L2 |
| ELBO, reparameterisation | VAEs | Reconstruction term minus KL term |
| Temperature, top-k, top-p | Sampling from language models | Reshape or truncate the output distribution |
| Calibration | Confidence reporting | Confidence should match accuracy |
11How to Study It
Derive the cross-entropy loss and the mean squared error loss from their noise models on paper once, work through a softmax by hand, and compute the KL divergence between two small distributions. Then read about the ELBO. These exercises make most deep learning papers easier to follow. The sources below include the probability chapter of the Deep Learning book, two free textbooks and the original papers.
Sources and further reading:
- Deep Learning book, chapter 3: Probability and Information Theory (Goodfellow, Bengio, Courville).
- Deep Learning book, chapter 5: Machine Learning Basics — maximum likelihood and Bayesian statistics.
- Probabilistic Machine Learning (Kevin Murphy) — free books.
- Understanding Deep Learning (Simon Prince) — derives loss functions from maximum likelihood.
- Auto-Encoding Variational Bayes — the VAE and the reparameterisation trick.
- Denoising Diffusion Probabilistic Models.
- The Curious Case of Neural Text Degeneration — nucleus sampling.
- Dropout as a Bayesian Approximation.
- What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? — aleatoric and epistemic uncertainty.
- On Calibration of Modern Neural Networks.
- torch.distributions (PyTorch documentation).
- On this site: Probability Theory for AI, Statistics for Deep Learning, Linear Algebra for Deep Learning, Deep Learning, Generative AI.