What is Reinforcement Learning?
RL is fundamentally different from supervised and unsupervised learning. There is no labelled dataset. Instead, an agent learns by interacting with an environment, taking actions and receiving feedback in the form of rewards. The goal: learn a policy — a strategy for choosing actions — that maximises cumulative reward over time.
Real-world examples: training a robot to walk (each step = action, not falling = reward), teaching a chess engine to play (each move = action, winning = reward), tuning a recommendation system (each recommendation = action, click = reward), training ChatGPT with RLHF (each response = action, human preference = reward signal).
The Five Core Components
AgentThe learner and decision-maker. It observes states, takes actions, and receives rewards. Could be a robot, a game-playing program, or a trading algorithm.
EnvironmentEverything outside the agent. It receives actions and returns the next state and reward. Could be a physical world, a game simulator, or a financial market.
State $s$A complete description of the environment at a given moment. The agent uses the current state to decide its next action. Example: board position in chess, sensor readings of a robot.
Action $a$A choice the agent can make from the action space $\mathcal{A}$. Discrete (move left/right, attack/defend) or continuous (steering angle, throttle).
Reward $r$A scalar signal from the environment after each action. Encodes the goal: positive for good outcomes, negative for bad. The agent maximises total future reward, not just immediate reward.
The RL Interaction Loop
Agent–Environment Loop$$s_0 \xrightarrow{a_0} (r_1, s_1) \xrightarrow{a_1} (r_2, s_2) \xrightarrow{a_2} \cdots$$At each timestep $t$: agent observes $s_t$, selects action $a_t \sim \pi(a|s_t)$, environment returns reward $r_{t+1}$ and next state $s_{t+1}$.
Python
import gymnasium as gym # pip install gymnasium
# Classic CartPole environment: balance a pole on a cart
env = gym.make("CartPole-v1", render_mode=None)
# State: [cart_pos, cart_velocity, pole_angle, pole_angular_velocity]
# Action: 0 = push left, 1 = push right
# Reward: +1 for every timestep the pole stays upright
# Episode ends: pole falls (angle > 12°) or cart moves off screen or 500 steps
obs, info = env.reset(seed=42)
print(f"Initial state: {obs}") # 4-dim vector
print(f"Action space: {env.action_space}") # Discrete(2)
print(f"Obs space: {env.observation_space}") # Box(4,)
# ── Random agent (baseline — no learning)
total_reward = 0
obs, _ = env.reset()
for step in range(500):
action = env.action_space.sample() # random action
obs, reward, terminated, truncated, info = env.step(action)
total_reward += reward
if terminated or truncated:
break
print(f"Random agent survived {step+1} steps, total reward: {total_reward}")Policy and Value Functions
The policy $\pi(a|s)$ is the agent's strategy — a probability distribution over actions given the current state. The agent's goal is to find the optimal policy $\pi^*$ that maximises expected return.
Discounted Return$$G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1} = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots$$$\gamma \in [0,1)$: discount factor. Future rewards are worth less than immediate ones. $\gamma \approx 0$: myopic. $\gamma \approx 1$: far-sighted.
State-Value and Action-Value Functions$$V^\pi(s) = \mathbb{E}_\pi[G_t \mid s_t = s] \qquad Q^\pi(s,a) = \mathbb{E}_\pi[G_t \mid s_t = s, a_t = a]$$$V^\pi(s)$: expected return from state $s$ following policy $\pi$. $Q^\pi(s,a)$: expected return from taking action $a$ in state $s$ then following $\pi$. The optimal Q-function $Q^*(s,a)$ gives the best possible return.
The Bellman Equation
The cornerstone of RL — a recursive relationship expressing $V^\pi(s)$ in terms of immediate reward and successor state values:
Bellman Optimality Equation$$Q^*(s,a) = \mathbb{E}\left[R_{t+1} + \gamma \max_{a'} Q^*(s_{t+1}, a') \mid s_t=s, a_t=a\right]$$The optimal Q-value of $(s,a)$ equals the immediate reward plus the discounted best Q-value achievable from the next state. This is the update target in Q-Learning.
Exploration vs. Exploitation
The central dilemma of RL: exploit what you know (choose the action with highest estimated reward) vs. explore new actions (discover potentially better ones). The $\epsilon$-greedy strategy balances both:
$\epsilon$-Greedy Policy$$a_t = \begin{cases} \arg\max_a Q(s_t,a) & \text{with probability } 1-\epsilon \\ \text{random action} & \text{with probability } \epsilon \end{cases}$$
Python
import numpy as np
class EpsilonGreedy:
def __init__(self, n_actions, eps_start=1.0, eps_end=0.01, eps_decay=0.995):
self.n_actions = n_actions
self.eps = eps_start
self.eps_end = eps_end
self.eps_decay = eps_decay
def select_action(self, q_values):
if np.random.random() < self.eps:
return np.random.randint(self.n_actions) # explore
return np.argmax(q_values) # exploit
def decay(self):
self.eps = max(self.eps_end, self.eps * self.eps_decay)
# Epsilon decays from 1.0 → 0.01 over ~1000 steps
eg = EpsilonGreedy(n_actions=2)
for i in range(10):
eg.decay()
if i % 2 == 0:
print(f" Step {i*100+100:4d}: eps = {eg.eps:.4f}")RL Algorithm Taxonomy
Model-FreeAgent doesn't build a model of the environment. Learns directly from experience. Q-Learning, DQN, PPO, SAC fall here. Most practical algorithms.
Model-BasedAgent builds an internal world model to simulate future states. More sample-efficient but harder to train correctly. AlphaZero, Dreamer.
Value-BasedLearn Q-values; derive policy by taking the action with highest Q. Q-Learning, DQN.
Policy-BasedDirectly optimise the policy $\pi_\theta$ via gradient ascent on expected return. REINFORCE, PPO, SAC.
Actor-CriticCombine both: actor (policy) + critic (value function). Reduces variance of policy gradient estimates. A2C, PPO, SAC.
Summary
- RL = agent + environment. Agent learns by trial-and-error, not from labelled data.
- Goal: maximise discounted return $G_t = \sum \gamma^k R_{t+k+1}$.
- $Q^*(s,a)$: optimal action-value. Bellman equation expresses it recursively.
- $\epsilon$-greedy: explore randomly with prob $\epsilon$, exploit best known action otherwise. Decay $\epsilon$ over time.