The Complete Guide to Reinforcement Learning

Core Concepts

Reinforcement learning (RL) is the branch of machine learning concerned with how an agent ought to take actions in an environment to maximize cumulative reward. Unlike supervised learning, there are no labeled examples — only a scalar feedback signal, delayed consequences, and the agent's own experience.

Policy, Value, and Return

A policy π maps states to actions (deterministic) or to a probability distribution over actions (stochastic). The agent's goal is to find the policy that maximizes expected return — the discounted sum of future rewards.

Two quantities anchor almost every RL algorithm: the state-value function V^π(s), the expected return from state s under policy π, and the action-value function Q^π(s,a), the expected return from taking action a in state s and then following π.

Markov Decision Processes & the Bellman Equations

Almost all of RL is formalized as a Markov Decision Process (MDP) — a tuple ⟨S, A, P, R, γ⟩ of states, actions, transition dynamics, reward function, and discount factor. The Markov property assumes the future depends only on the current state, not the full history.

Exploration vs. Exploitation

An agent that only exploits its current knowledge may get stuck in a suboptimal routine; one that only explores never capitalizes on what it has learned. Balancing the two is a defining challenge of RL.

Value-Based Methods

Value-based methods learn V(s) or Q(s,a) and derive a policy implicitly (e.g. act greedily with respect to Q). They form the historical and conceptual backbone of RL.

Dynamic Programming

Policy Iteration alternates policy evaluation (compute V^π) and policy improvement (act greedily w.r.t. V^π) until convergence. Value Iteration merges both steps by directly applying the Bellman optimality backup. Both require a known model and are typically limited to small, discrete state spaces.

Monte Carlo Methods

Estimate value functions by averaging complete episode returns, with no bootstrapping and no model required. Simple and unbiased, but high-variance and only applicable to episodic tasks.

Temporal-Difference (TD) Learning

TD methods combine the best of both worlds: like Monte Carlo, they learn directly from raw experience without a model; like dynamic programming, they bootstrap — updating estimates based on other learned estimates rather than waiting for a final outcome.

Q-Learning & SARSA

The two workhorse TD control algorithms differ in one crucial way: Q-learning is off-policy (it learns the optimal Q* regardless of the behavior policy), while SARSA is on-policy (it learns the value of the policy actually being followed, including its exploration).

Deep Q-Networks & Value-Function Approximation

Tabular Q-learning breaks down once state spaces become continuous or astronomically large (e.g. raw pixels). Deep Q-Networks (DQN), introduced by DeepMind in 2013–2015, replaced the Q-table with a neural network Q(s,a;θ) and proved it could learn to play Atari games directly from pixels.

# sample a minibatch of transitions from replay buffer
s, a, r, s_next, done = replay_buffer.sample(batch_size)

# target uses a separate, periodically-synced target network
target = r + (1 - done) * gamma * max(target_net(s_next), dim=-1)

# minimize temporal-difference error
q_values = policy_net(s).gather(a)
loss = huber_loss(q_values, target.detach())
loss.backward()
optimizer.step()

Policy Gradient & Actor-Critic Methods

Instead of learning values and deriving a policy indirectly, policy-gradient methods parameterize the policy π_θ(a|s) directly and optimize it via gradient ascent on expected return. This handles continuous action spaces naturally and can learn genuinely stochastic policies.

Model-Based Reinforcement Learning

Model-based methods learn (or are given) a model of the environment's dynamics p(s′,r|s,a), then use it to plan — simulating rollouts internally instead of relying solely on real interaction. This can dramatically improve sample efficiency.

The trade-off is model bias: an imperfect learned model can compound errors over long planning horizons, so most production systems either use short planning horizons or fall back to model-free fine-tuning.

Multi-Agent Reinforcement Learning

Many real settings involve multiple learning agents interacting — cooperatively, competitively, or both. This breaks the single-agent MDP assumption, since the environment becomes non-stationary from any one agent's perspective as others also learn.

Key Dichotomies

Three axes classify almost every RL algorithm discussed above. Knowing where a method sits on each tells you a lot about its sample efficiency, stability, and applicability.

Milestones in Reinforcement Learning

A field shaped by decades of theory before an explosive decade of deep-learning-powered breakthroughs.

Real-World Applications

RL has moved well beyond games and simulated benchmarks into domains where sequential decisions under uncertainty carry real consequences.

Tools & Frameworks

A modern RL stack typically layers a simulation environment, an algorithm library, and an experiment-tracking layer.

Glossary of Key Terms

Quick definitions for terms used throughout this guide.

Agent

The decision-making learner that interacts with an environment.

Policy (π)

A mapping from states to actions or action probabilities.

Value Function

Expected return from a state (V) or state-action pair (Q).

Reward Shaping

Modifying the reward signal to make learning faster without changing the optimal policy.

Discount Factor (γ)

Weight applied to future rewards, between 0 (myopic) and 1 (far-sighted).

Episode

One complete sequence of interaction from an initial state to a terminal state.

Bootstrapping

Updating an estimate using other learned estimates rather than only real outcomes.

Advantage Function

A(s,a) = Q(s,a) − V(s); how much better an action is than the state's average.

Credit Assignment

Determining which past actions were responsible for a later reward.

Sample Efficiency

How much environment interaction an algorithm needs to reach a given performance level.

Partial Observability

When the agent cannot directly observe the true underlying state (formalized as a POMDP).

Reward Hacking

An agent exploiting unintended loopholes in a reward function rather than solving the intended task.

Curriculum Learning

Training on a sequence of increasingly difficult tasks or environments.

Sim-to-Real Transfer

Applying a policy trained in simulation to a physical, real-world system.

Regret

The cumulative gap between an agent's rewards and those of an optimal strategy.

RLHF

Reinforcement Learning from Human Feedback — using a learned reward model built from human preferences.