Probability and random variables

Let $\Omega$ be a set, whose elements $\omega$ are the possible outcomes of an experiment; $\Omega$ is the sample space.

A collection $\mathcal{F}$ of subsets of $\Omega$ is a $\sigma$-field (measure theorists say $\sigma$-algebra) if it contains $\emptyset$, is closed under complementation, and is closed under countable unions. Its members are the events; to say that $A$ is an event is exactly to say $A \in \mathcal{F}$. Closure under countable intersections follows by De Morgan.

A probability measure on $(\Omega, \mathcal{F})$ is a function $\mathbb{P} : \mathcal{F} \to [0,1]$ with

1. $\mathbb{P}(\emptyset) = 0$ and $\mathbb{P}(\Omega) = 1$;
2. countable additivity: whenever $A_1, A_2, \dots$ are members of $\mathcal{F}$ that are pairwise disjoint, meaning $A_i \cap A_j = \emptyset$ for $i \neq j$, $$\mathbb{P}\Bigl( \bigcup_{i=1}^{\infty} A_i \Bigr) \; = \; \sum_{i=1}^{\infty} \mathbb{P}(A_i) .$$

The triple $(\Omega, \mathcal{F}, \mathbb{P})$ is a probability space. It is a measure space (see Sigma-algebras and measures) whose total mass is $1$.

An event $A$ with $\mathbb{P}(A) = 0$ is called null; if $\mathbb{P}(A) = 1$ we say $A$ occurs almost surely, abbreviated a.s. A null event is not the same as the impossible event $\emptyset$: a dart thrown at a target hits some particular point, and the event that it hits that point had probability zero.

The elementary case is a finite $\Omega$ with all outcomes equally likely, $\mathcal{F}$ the collection of all subsets, and $\mathbb{P}(A) = \lvert A \rvert / \lvert \Omega \rvert$. Then probability is counting, and the whole difficulty is combinatorial.

Let $(\Omega, \mathcal{F}, \mathbb{P})$ be a probability space.

Conditional probability. If $B$ is an event with $\mathbb{P}(B) > 0$, the probability of $A$ given $B$ is

For fixed $B$ the map $A \mapsto \mathbb{P}(A \mid B)$ is itself a probability measure on $(\Omega, \mathcal{F})$: it is non-negative, assigns $1$ to $\Omega$, and inherits countable additivity from $\mathbb{P}$ because $(\bigcup_i A_i) \cap B = \bigcup_i (A_i \cap B)$ preserves disjointness. So every result proved for probability measures holds verbatim for conditional probabilities.

Rearranged, the definition is the chain rule $\mathbb{P}(A \cap B) = \mathbb{P}(B)\,\mathbb{P}(A \mid B)$, which iterates: provided $\mathbb{P}(A_1 \cap \cdots \cap A_{n-1}) > 0$,

Independence. Events $A$ and $B$ are independent if

When $\mathbb{P}(B) > 0$ this says exactly $\mathbb{P}(A \mid B) = \mathbb{P}(A)$, which is the meaning; the product form is taken as the definition because it is symmetric in $A$ and $B$ and remains meaningful when $\mathbb{P}(B) = 0$.

A family $\{A_i : i \in I\}$ of events is independent if

and pairwise independent if this holds for every $J$ of size $2$. Pairwise independence is strictly weaker; see the remark below.

Finally, for an event $C$ with $\mathbb{P}(C) > 0$, events $A$ and $B$ are conditionally independent given $C$ if $\mathbb{P}(A \cap B \mid C) = \mathbb{P}(A \mid C)\,\mathbb{P}(B \mid C)$, that is, if they are independent under the conditional measure.

Partition. A finite or countable family $\{B_i\}$ of events is a partition of $\Omega$ if the $B_i$ are pairwise disjoint and their union is $\Omega$. Every outcome then lies in exactly one $B_i$.

Let $(\Omega, \mathcal{F}, \mathbb{P})$ be a probability space as in Probability spaces.

A random variable is a function $X : \Omega \to \mathbb{R}$ such that

such an $X$ is called $\mathcal{F}$-measurable. The condition is exactly what is needed for the probabilities below to be defined; it is automatic when $\mathcal{F}$ consists of all subsets of $\Omega$, as it does for a finite or countable sample space.

Following the usual abuse, the event $\{\omega : X(\omega) \le x\}$ is written $\{X \le x\}$ and its probability $\mathbb{P}(X \le x)$.

The distribution function of $X$ is

written $F_X$ when the variable must be named. Measure-theoretically, $X$ pushes $\mathbb{P}$ forward to a Borel probability measure on the line,

called the distribution of $X$; and $F$ determines $\mathbb{P}_X$ completely. Two random variables with the same distribution function are identically distributed, whether or not they are defined on the same space, and no purely distributional statement can tell them apart.

The two tails of $X$ are $T_1(x) = \mathbb{P}(X > x) = 1 - F(x)$ and $T_2(x) = \mathbb{P}(X \le -x) = F(-x)$ for large positive $x$; how fast they decay governs which moments exist.

$X$ is discrete when its range lies inside a countable subset $\{x_1, x_2, \dots\}$ of the line. Its (probability) mass function is

which vanishes off the countable set and satisfies $\sum_i f(x_i) = 1$. Then $F(x) = \sum_{x_i \le x} f(x_i)$ is a step function, constant between the $x_i$ and jumping by $f(x_i)$ at $x_i$; such a distribution is called atomic, and the points $x_i$ with $f(x_i) > 0$ are its atoms. Riley, Hobson and Bence call $f$ the probability function and $F$ the cumulative probability function.

$X$ is continuous when some integrable $f : \mathbb{R} \to [0,\infty)$, called a (probability) density function for $X$, reproduces $F$ by integration:

Then $F$ is continuous, $\mathbb{P}(X = x) = 0$ for every $x$, and

The density is not unique — changing $f$ at finitely many points changes no integral — but wherever $F$ is differentiable one takes $f(x) = F'(x)$. Physicists usually define the density the other way round, by $\mathbb{P}(x < X \le x + dx) = f(x)\,dx$, which is the same statement in infinitesimal form.

Two cautions. First, the word "continuous" describes $F$, not the function $X$ on $\Omega$. Second, these two classes are not exhaustive: there exist singular distribution functions, continuous but with no density, and there exist mixtures of all three kinds. A convenient device for handling discrete and continuous variables in one formalism, used throughout physics, is to allow densities containing Dirac deltas, writing $f(x) = \sum_i p_i\,\delta(x - x_i)$ for a discrete variable; see The Dirac delta.