Stochastic processes

Markov chains, Poisson processes, Brownian motion and stochastic differential equations.

In this chapter 11

Tick a unit to add it to your PDF basket.

Markov processes
Diffusions and stochastic calculus

Markov processes

Definition

Discrete-time Markov chains

Adapted

Add to PDF

Sets up a chain on a countable state space through its transition matrix, derives the Chapman-Kolmogorov equations that make the n-step probabilities matrix powers, classifies states by recurrence, mean recurrence time and period, and characterises the stationary distribution as the left eigenvector of the transition matrix with eigenvalue one.

Prerequisites: Conditional expectation, Generating functions, Eigenvalues and eigenvectors

Note(What a Markov chain buys you)Original

A random process is in general an unmanageable object: the joint law of $X_0, X_1, \dots, X_n$ needs a number for every path. A Markov chain is the case in which that joint law collapses into a single matrix of one-step probabilities, because the chain's memory of its past is entirely contained in where it is now. Everything in this unit is a consequence of that collapse: $n$ -step probabilities become matrix powers, long-run behaviour becomes a statement about eigenvalues, and the equilibrium of the chain becomes a left eigenvector.

The limit theorem that says the chain actually converges to that equilibrium is the subject of Convergence of Markov chains; the walk on $\mathbb{Z}^d$ , the canonical example with an infinite state space, is treated in Random walks. Here we fix the objects and the bookkeeping.

Definition(The Markov property, transition matrices and

n

-step probabilities)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.1, Definitions (6.1.1), (6.1.4) and (6.1.6), and Theorem (6.1.5), pp. 239-241

Let $S$ be a countable set, the state space, whose elements we may and do take to be integers. A sequence $X = \{X_n : n \ge 0\}$ of random variables taking values in $S$ , defined on a probability space $(\Omega, \mathcal{F}, \mathbb{P})$ , is a Markov chain if

\mathbb{P}\bigl(X_n = s \mid X_0 = x_0, \ X_1 = x_1, \ \dots, \ X_{n-1} = x_{n-1}\bigr) \ = \ \mathbb{P}\bigl(X_n = s \mid X_{n-1} = x_{n-1}\bigr)

for every $n \ge 1$ and all states $s, x_1, \dots, x_{n-1}$ for which the conditioning event has positive probability. Grimmett and Stirzaker record two equivalent forms: one may condition on the values at any finite increasing set of past epochs $n_1 < n_2 < \dots < n_k \le n$ and only the last one survives, and one may ask for $X_{m+n}$ given $X_0, \dots, X_m$ and only $X_m$ survives. The conditional probabilities here are those of Conditional expectation, taken with respect to the discrete conditioning events.

The chain is homogeneous if $\mathbb{P}(X_{n+1} = j \mid X_n = i)$ does not depend on $n$ . Its transition matrix is then $P = (p_{ij})_{i,j \in S}$ with

p_{ij} = \mathbb{P}(X_{n+1} = j \mid X_n = i) .

Every chain in this unit is homogeneous. The matrix $P$ is stochastic: its entries are non-negative and each of its rows sums to $1$ , since from state $i$ the chain must go somewhere. Conversely every stochastic matrix is the transition matrix of some chain.

The $n$ -step transition probabilities are $p_{ij}(n) = \mathbb{P}(X_{m+n} = j \mid X_m = i)$ , which by homogeneity and the Chapman-Kolmogorov statement below do not depend on $m$ ; we write $P_n$ for the matrix $(p_{ij}(n))$ and set $p_{ij}(0) = \delta_{ij}$ . Finally, write $\mu^{(n)}_i = \mathbb{P}(X_n = i)$ and let $\mu^{(n)}$ be the row vector of these numbers.

A warning on conventions: $p_{ij}$ is the probability of a jump from $i$ to $j$ , so that probability mass is propagated by multiplying a row vector on the right by $P$ . Some older texts, and much of the physics literature on master equations, use the transposed convention in which the transition matrix acts on column vectors from the left. Grimmett and Stirzaker note that the transposed convention was formerly in use and warn the reader about it. We use the row convention throughout the handbook.

Definition(Chapman-Kolmogorov:

n

-step probabilities are matrix powers)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.1, Theorem (6.1.7) and Lemma (6.1.8), pp. 241-242

Let $X$ be a homogeneous Markov chain with transition matrix $P$ . Then for all states $i, j$ and all integers $n, r \ge 0$ ,

p_{ij}(n + r) \ = \ \sum_{k \in S} p_{ik}(n)\, p_{kj}(r) ,

the Chapman-Kolmogorov equations. In matrix form $P_{n+r} = P_n P_r$ , and therefore

P_n = P^{\,n} , \qquad n \ge 0 .

Moreover the distribution of the chain evolves by $\mu^{(m+n)} = \mu^{(m)} P^{\,n}$ , and in particular

\mu^{(n)} = \mu^{(0)} P^{\,n} .

Thus the law of the whole process is determined by the pair $(\mu^{(0)}, P)$ .

Definition(Recurrence, mean recurrence time, period, irreducibility)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.2, Definitions (6.2.1), (6.2.7), (6.2.8), (6.2.10), (6.2.11) and the first-passage probabilities (6.2.2), pp. 246-249; Grimmett & Stirzaker (2020), §6.3, Definition (6.3.1) and Theorem (6.3.2), pp. 250-251

Write $\mathbb{P}_i$ for $\mathbb{P}(\,\cdot \mid X_0 = i)$ and $\mathbb{E}_i$ for the corresponding expectation, and let

T_j = \min\{n \ge 1 : X_n = j\}

be the first passage time to $j$ , with $T_j = \infty$ if the chain never visits $j$ after time $0$ . Put $f_{ij}(n) = \mathbb{P}_i(T_j = n)$ and $f_{ij} = \sum_{n \ge 1} f_{ij}(n) = \mathbb{P}_i(T_j < \infty)$ .

State $i$ is recurrent (Grimmett and Stirzaker also write persistent) if $f_{ii} = 1$ , that is, if return to $i$ from $i$ is certain; otherwise $f_{ii} < 1$ and $i$ is transient.
The mean recurrence time of $i$ is $\mu_i = \mathbb{E}_i(T_i) = \sum_n n f_{ii}(n)$ when $i$ is recurrent, and $\mu_i = \infty$ when $i$ is transient. A recurrent state is positive (or non-null) if $\mu_i < \infty$ and null if $\mu_i = \infty$ ; a recurrent state can be null, so certainty of return does not imply a finite expected return time.
The period of $i$ is $d(i) = \gcd\{n \ge 1 : p_{ii}(n) > 0\}$ . The state is aperiodic if $d(i) = 1$ and periodic otherwise. A state that is recurrent, positive and aperiodic is called ergodic.

Say $i$ leads to $j$ , written $i \to j$ , if $p_{ij}(m) > 0$ for some $m \ge 0$ , and that $i$ and $j$ intercommunicate, $i \leftrightarrow j$ , if each leads to the other. Intercommunication is an equivalence relation, and intercommunicating states share their period, their transience or recurrence, and their nullity or positivity. The chain is irreducible if $S$ is a single equivalence class.

Grimmett and Stirzaker record the following for a finite state space $S$ , with no irreducibility assumed: some state of the chain must be recurrent, and nullity is impossible, every recurrent state being positive. Combining this with the class properties, every state of a finite irreducible chain is positive recurrent.

Definition(When a state is recurrent, and the stationary distribution)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.2, Theorem (6.2.3) and Corollary (6.2.4), pp. 247-248; Grimmett & Stirzaker (2020), §6.4, Definition (6.4.1) and Theorem (6.4.3) with equation (6.4.11), pp. 254-257; Grimmett & Stirzaker (2020), §6.6, Theorem (6.6.1), p. 269

(a) A summability criterion. Define the generating functions $P_{ij}(s) = \sum_{n \ge 0} s^n p_{ij}(n)$ and $F_{ij}(s) = \sum_{n \ge 0} s^n f_{ij}(n)$ for $\lvert s \rvert < 1$ , with $f_{ij}(0) = 0$ . Then

P_{ii}(s) = 1 + F_{ii}(s) P_{ii}(s) , \qquad P_{ij}(s) = F_{ij}(s) P_{jj}(s) \ \ (i \ne j) .

Consequently state $j$ is recurrent if and only if $\sum_n p_{jj}(n) = \infty$ . If $j$ is recurrent then $\sum_n p_{ij}(n) = \infty$ for every $i$ with $f_{ij} > 0$ ; if $j$ is transient then $\sum_n p_{ij}(n) < \infty$ , and hence $p_{ij}(n) \to 0$ , for every $i$ .

(b) Stationary distributions. A row vector $\pi = (\pi_j : j \in S)$ is a stationary distribution of the chain if $\pi_j \ge 0$ , $\sum_j \pi_j = 1$ , and

\pi = \pi P , \qquad \text{that is} \qquad \pi_j = \sum_{i \in S} \pi_i p_{ij} \ \text{ for all } j .

Iterating gives $\pi P^{\,n} = \pi$ for all $n$ , so a chain started with $X_0 \sim \pi$ has $X_n \sim \pi$ for every $n$ . Note that $\pi$ is a left eigenvector of $P$ with eigenvalue $1$ , not a right one.

(c) Existence and uniqueness. A stationary distribution exists for an irreducible chain exactly when every one of its states is positive recurrent. In that case $\pi$ is unique and

\pi_i = \frac{1}{\mu_i} \qquad \text{for every } i \in S ,

where $\mu_i$ is the mean recurrence time. No assumption of aperiodicity is needed for this.

(d) The finite case. If $S$ is finite with $N = \lvert S \rvert$ and the chain is irreducible with period $d$ , the Perron-Frobenius theorem says that $\lambda_1 = 1$ is an eigenvalue of $P$ ; that the $d$ complex $d$ -th roots of unity are eigenvalues; that the remaining eigenvalues $\lambda_{d+1}, \dots, \lambda_N$ satisfy $\lvert \lambda_r \rvert < 1$ ; and that there is a unique $\pi$ with $\pi P = \pi$ , all of whose entries are strictly positive.

Idea of the proofOriginal

Two ideas carry the whole unit.

The first is that conditioning on where the chain is at an intermediate time decomposes a long journey into two independent-looking short ones. Summing over the intermediate state is exactly a matrix product, which is why $P_n = P^n$ and why the long-run questions turn into linear algebra.

The second is that a renewal happens at every return to a fixed state $k$ : by the Markov property the chain starts afresh there, so the excursions between consecutive visits to $k$ are independent and identically distributed. The time between visits has mean $\mu_k$ , so in a long run of $n$ steps the chain makes about $n/\mu_k$ visits to $k$ , and the fraction of time spent at $k$ is about $1/\mu_k$ . That heuristic is the content of $\pi_k = 1/\mu_k$ , and it also explains the summability criterion: $\sum_n p_{kk}(n)$ is the expected number of visits to $k$ , which is infinite exactly when return is certain and the visits never stop.

Proof(Chapman-Kolmogorov, the evolution of

\mu^{(n)}

, and the summability criterion)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.1, proofs of Theorem (6.1.7) and Lemma (6.1.8), pp. 241-242; Grimmett & Stirzaker (2020), §6.2, proofs of Theorem (6.2.3) and Corollary (6.2.4), pp. 247-248

Chapman-Kolmogorov. Fix $i, j$ and $m, n, r \ge 0$ . Partition the event $\{X_{m+n+r} = j\}$ according to the state occupied at the intermediate epoch $m+n$ :

\mathbb{P}(X_{m+n+r} = j \mid X_m = i) \ = \ \sum_{k \in S} \mathbb{P}(X_{m+n+r} = j, \ X_{m+n} = k \mid X_m = i) ,

which is legitimate because $S$ is countable and the events are disjoint. Each summand factorises as

\mathbb{P}(X_{m+n+r} = j \mid X_{m+n} = k,\, X_m = i)\ \mathbb{P}(X_{m+n} = k \mid X_m = i) ,

by the elementary identity $\mathbb{P}(A \cap B \mid C) = \mathbb{P}(A \mid B \cap C)\,\mathbb{P}(B \mid C)$ . The Markov property, in the form that allows conditioning on any finite set of past epochs, lets us drop the condition $X_m = i$ from the first factor; homogeneity then identifies the two factors as $p_{kj}(r)$ and $p_{ik}(n)$ . This is the displayed equation, and in matrix language $P_{n+r} = P_n P_r$ . Taking $r = 1$ and inducting on $n$ gives $P_n = P^{\,n}$ , and in particular the right-hand side does not depend on the starting epoch $m$ .

Evolution of the distribution. Condition on $X_m$ :

\mu^{(m+n)}_j = \mathbb{P}(X_{m+n} = j) = \sum_{i \in S} \mathbb{P}(X_{m+n} = j \mid X_m = i)\, \mathbb{P}(X_m = i) = \sum_i \mu^{(m)}_i p_{ij}(n) ,

which is the $j$ -th entry of the row vector $\mu^{(m)} P^{\,n}$ . Setting $m = 0$ gives $\mu^{(n)} = \mu^{(0)} P^{\,n}$ . $\square$

The summability criterion. Fix $i$ and $j$ . Decompose a visit to $j$ at time $m \ge 1$ according to the time $r$ of the first visit to $j$ . These $m$ possibilities are disjoint and exhaust the event, so by the Markov property applied at the first-passage time,

p_{ij}(m) \ = \ \sum_{r=1}^{m} f_{ij}(r)\, p_{jj}(m - r) , \qquad m \ge 1 .

The right-hand side is a convolution. Multiplying by $s^m$ and summing over $m \ge 1$ , and recalling $p_{ij}(0) = \delta_{ij}$ , turns it into the product of generating functions (see Generating functions):

P_{ij}(s) - \delta_{ij} = F_{ij}(s) P_{jj}(s) ,

which is both displayed identities at once. Taking $i = j$ and solving,

P_{jj}(s) = \frac{1}{1 - F_{jj}(s)} \qquad (\lvert s \rvert < 1) .

Now let $s \uparrow 1$ . Both $P_{jj}$ and $F_{jj}$ have non-negative coefficients, so Abel's theorem lets us pass to the limit termwise: $F_{jj}(s) \to f_{jj}$ and $P_{jj}(s) \to \sum_n p_{jj}(n)$ , the two limits being finite or infinite together with the sums. Hence $\sum_n p_{jj}(n) = \infty$ precisely when $f_{jj} = 1$ , that is, precisely when $j$ is recurrent. The statements for $i \ne j$ follow from $P_{ij}(s) = F_{ij}(s) P_{jj}(s)$ by letting $s \uparrow 1$ again: the left side blows up iff $P_{jj}$ does and $f_{ij} = F_{ij}(1) > 0$ . $\square$

Remark(Where each hypothesis is used)Original

Countability of $S$ . It enters at the first line of the Chapman-Kolmogorov proof, where the event is split into a countable disjoint union and summed. For an uncountable state space the sum must be replaced by an integral against a transition kernel, and the result is the same equation in integral form.

Homogeneity. Used only to identify $\mathbb{P}(X_{m+n+r} = j \mid X_{m+n} = k)$ with $p_{kj}(r)$ . Without it the Chapman-Kolmogorov equations still hold in the form $P(m, m+n+r) = P(m, m+n) P(m+n, m+n+r)$ , but the matrices depend on the epoch and are no longer powers of a single $P$ , so the eigenvalue analysis is lost.

The Markov property. Used exactly once in each proof: to delete the condition $X_m = i$ in Chapman-Kolmogorov, and to restart the chain at the first passage time in the convolution identity. If it fails, the convolution identity fails and the generating-function criterion with it.

Irreducibility in part (c). Without it the conclusion is false in both directions. The two-state chain with $p_{11} = p_{22} = 1$ is positive recurrent in every state yet has infinitely many stationary distributions, namely every $\pi = (a, 1-a)$ ; so uniqueness genuinely needs irreducibility. And a chain with an absorbing state $0$ reachable from a transient part has a stationary distribution concentrated on $0$ while other states are transient, so "has a stationary distribution" does not by itself make every state positive recurrent.

Positive recurrence in part (c). Null recurrence is not an exotic pathology: the symmetric simple random walk on $\mathbb{Z}$ is irreducible and recurrent, every $\mu_i$ is infinite, and it has no stationary distribution at all, only the stationary measure $x_i \equiv 1$ , which cannot be normalised (see Random walks). This is where the finiteness of $S$ does real work: by the lemma above, a finite irreducible chain cannot be null recurrent, so it always has exactly one $\pi$ .

Aperiodicity. Deliberately not assumed anywhere in this unit. The chain on $\{1,2\}$ with $p_{12} = p_{21} = 1$ is irreducible and positive recurrent, has the unique stationary distribution $\pi = (\tfrac12, \tfrac12)$ , and yet $p_{11}(n)$ alternates between $1$ and $0$ and converges to nothing. Stationarity is an algebraic statement; convergence to $\pi$ is a separate analytic statement that does need aperiodicity, and it is proved in Convergence of Markov chains.

Method(Analysing a chain given its transition matrix)Original

Write $P$ down and check it is stochastic. Every row must sum to $1$ . An arithmetic slip here invalidates everything after it.
Draw the directed graph with an arrow $i \to j$ whenever $p_{ij} > 0$ , and find its communicating classes. The chain is irreducible exactly when the graph is strongly connected.
Classify. For a finite irreducible chain, stop: all states are positive recurrent (lemma above). Otherwise identify the closed recurrent classes and the transient states; from a transient state the chain eventually enters one of the closed classes and stays there, so analyse each closed class as a chain in its own right.
Find the period of one state in each class; all states in a class share it. In practice: if $p_{ii} > 0$ for some $i$ in the class, the class is aperiodic. Otherwise take the gcd of the lengths of the cycles through $i$ in the graph.
Solve $\pi P = \pi$ with $\sum_j \pi_j = 1$ . Because $P$ is stochastic, the all-ones column vector satisfies $P\mathbf{1} = \mathbf{1}$ , so $P - I$ is singular and the $N$ equations $\pi(P - I) = 0$ are linearly dependent. Discard any one of them and append the normalisation $\sum_j \pi_j = 1$ in its place.
Read off the mean recurrence times as $\mu_i = 1/\pi_i$ .
For $n$ -step probabilities, diagonalise $P$ if you can: with $P = B^{-1} \Lambda B$ one has $P^{\,n} = B^{-1} \Lambda^n B$ , and the eigenvalues other than $1$ control how fast $p_{ij}(n)$ settles down. If $P$ is not diagonalisable, use its Jordan form (Jordan normal form); the extra polynomial factors $n^{k}$ do not change which terms decay.

Example(A three-state chain: classification, stationary distribution, and a three-step probability)Original

A turbulence monitor samples a flow once per minute and records which of three regimes it is in: $1$ = laminar, $2$ = intermittent, $3$ = fully turbulent. The record is modelled as a homogeneous Markov chain with

P \ = \ \begin{pmatrix} \tfrac12 & \tfrac12 & 0 \\[2pt] \tfrac13 & \tfrac13 & \tfrac13 \\[2pt] 0 & \tfrac14 & \tfrac34 \end{pmatrix} .

Step 1: $P$ is stochastic. The row sums are $\tfrac12 + \tfrac12 = 1$ , $\tfrac13 \cdot 3 = 1$ and $\tfrac14 + \tfrac34 = 1$ .

Step 2: irreducibility. The arrows are $1 \to 2$ , $2 \to 1$ , $2 \to 3$ , $3 \to 2$ , together with the loops at $1$ , $2$ and $3$ . So $1 \leftrightarrow 2 \leftrightarrow 3$ and the chain is irreducible. The state space is finite, so every state is positive recurrent.

Step 3: aperiodicity. $p_{11} = \tfrac12 > 0$ , so $d(1) = 1$ ; by the class property all three states are aperiodic, hence all three are ergodic.

Step 4: the stationary distribution. Solve $\pi P = \pi$ , that is

\begin{aligned} \tfrac12 \pi_1 + \tfrac13 \pi_2 &= \pi_1 , \\ \tfrac12 \pi_1 + \tfrac13 \pi_2 + \tfrac14 \pi_3 &= \pi_2 , \\ \tfrac13 \pi_2 + \tfrac34 \pi_3 &= \pi_3 . \end{aligned}

The first equation gives $\tfrac13 \pi_2 = \tfrac12 \pi_1$ , so $\pi_2 = \tfrac32 \pi_1$ . The third gives $\tfrac13 \pi_2 = \tfrac14 \pi_3$ , so $\pi_3 = \tfrac43 \pi_2 = 2 \pi_1$ . The second equation is then automatically satisfied, as it must be. Normalising, $\pi_1(1 + \tfrac32 + 2) = 1$ , so $\pi_1 = \tfrac29$ and

\pi \ = \ \Bigl( \tfrac29, \ \tfrac13, \ \tfrac49 \Bigr) .

A check: $\pi P$ has first entry $\tfrac29 \cdot \tfrac12 + \tfrac13 \cdot \tfrac13 = \tfrac19 + \tfrac19 = \tfrac29$ . Correct.

Step 5: mean recurrence times. By part (c) of the statement, $\mu_i = 1/\pi_i$ , so

\mu_1 = \tfrac92 = 4.5 , \qquad \mu_2 = 3 , \qquad \mu_3 = \tfrac94 = 2.25 \ \text{minutes}.

The monitor returns to full turbulence on average every $2.25$ minutes, and to laminar flow every $4.5$ minutes.

Step 6: a three-step probability by Chapman-Kolmogorov. First square $P$ . Its top row is

\begin{aligned} p_{11}(2) &= p_{11}p_{11} + p_{12}p_{21} = \tfrac12 \cdot \tfrac12 + \tfrac12 \cdot \tfrac13 = \tfrac14 + \tfrac16 = \tfrac{5}{12}, \\ p_{12}(2) &= p_{11}p_{12} + p_{12}p_{22} = \tfrac12 \cdot \tfrac12 + \tfrac12 \cdot \tfrac13 = \tfrac{5}{12}, \\ p_{13}(2) &= p_{12}p_{23} = \tfrac12 \cdot \tfrac13 = \tfrac16 . \end{aligned}

These sum to $1$ , as they must. Now apply Chapman-Kolmogorov with $n = 2$ , $r = 1$ :

p_{13}(3) = \sum_{k=1}^{3} p_{1k}(2)\, p_{k3} = \tfrac{5}{12}\cdot 0 + \tfrac{5}{12}\cdot \tfrac13 + \tfrac16 \cdot \tfrac34 = \tfrac{5}{36} + \tfrac18 = \tfrac{10}{72} + \tfrac{9}{72} = \boxed{\tfrac{19}{72}} \approx 0.2639 .

So a flow observed laminar now is fully turbulent three minutes later with probability $19/72$ . For comparison, the stationary value is $\pi_3 = 4/9 \approx 0.4444$ : three steps is not yet long enough to forget the laminar start. The eigenvalues of $P$ are $1$ and $\tfrac{7 \pm \sqrt{73}}{24}$ , numerically $0.64767$ and $-0.06433$ ; the largest of these in modulus is $0.64767$ , and since $0.64767^{\,n}$ falls below $0.01$ only at $n = 11$ , the chain needs of order ten minutes to settle. Direct computation confirms this: $p_{13}(8) = 0.42387$ and $p_{13}(20) = 0.44433$ , against $\pi_3 = 0.44444$ .

Application(Kinetic Monte Carlo and Markov state models)Original

Discrete-time chains are the standard coarse-graining of a complicated continuous dynamics onto a manageable set of labels.

In Markov state models of molecular dynamics one partitions the configuration space of a protein into a few hundred metastable sets, runs many short molecular-dynamics trajectories, and estimates $p_{ij}$ as the observed fraction of trajectories that are in set $j$ a lag time $\tau$ after being in set $i$ . The payoff is exactly the content of this unit: the stationary vector $\pi$ gives the equilibrium populations of the conformations (and hence their free-energy differences, $\Delta F_{ij} = -k_{\mathrm{B}} T \log(\pi_j/\pi_i)$ ), the mean recurrence times $1/\pi_i$ give residence statistics, and the eigenvalues of $P$ other than $1$ give the relaxation timescales $-\tau/\log \lvert \lambda_r \rvert$ of the slow conformational transitions, all from short simulations that individually never see a full folding event.

In lattice models of surface growth and radiation damage, kinetic Monte Carlo evolves an adatom configuration by hopping moves; recording the configuration after each sweep gives a chain whose stationary distribution is the equilibrium ensemble. Whether such a chain actually relaxes to $\pi$ , and how fast, is the subject of Convergence of Markov chains; the design principle that makes $\pi$ come out right in the first place is detailed balance, treated there.

The same bookkeeping, with $S$ the set of web pages and $P$ the hyperlink matrix smoothed by a small uniform term, is the PageRank computation: the ranking vector is the left eigenvector $\pi$ of $P$ with eigenvalue $1$ , computed by the power iteration $\mu^{(n)} = \mu^{(0)} P^{\,n}$ of the evolution lemma.

Theorem

Convergence of Markov chains

Adapted

Add to PDF

Proves by coupling that the n-step probabilities of an irreducible aperiodic chain converge to the reciprocal of the mean recurrence time, shows that a distribution in detailed balance with the transition matrix is automatically stationary and makes the chain reversible, and gives a geometric bound on the remaining total-variation distance in terms of the second-largest eigenvalue modulus.

Prerequisites: Discrete-time Markov chains, Convergence of random variables, Spectral theorem for Hermitian matrices

Note(What is being proved, and what is not)Original

Discrete-time Markov chains produced a stationary distribution $\pi$ as a solution of the algebraic equation $\pi = \pi P$ . Nothing there said that a chain started anywhere else ever approaches $\pi$ . This unit supplies the missing analytic statement, identifies the extra hypothesis it needs (aperiodicity), and then asks the practical question that follows: how long must one wait.

Three things are proved. First, the limit theorem: $p_{ij}(n) \to 1/\mu_j$ for an irreducible aperiodic chain, with no positivity assumption, so that the transient and null recurrent cases are included as the statement $p_{ij}(n) \to 0$ . Second, a sufficient condition, detailed balance, which both identifies $\pi$ cheaply and makes the chain reversible. Third, a geometric bound on the error, controlled by the second largest eigenvalue modulus of $P$ .

Definition(Total variation distance, reversibility, detailed balance)AdaptedAdapted from Grimmett & Stirzaker (2020), §4.12, equations (4.12.7) and (4.12.8) with the footnote on normalisation, p. 145; Grimmett & Stirzaker (2020), §6.5, Definition (6.5.2) and the equations (6.5.3), pp. 265-266

Let $X$ be an irreducible homogeneous Markov chain on a countable state space $S$ with transition matrix $P = (p_{ij})$ , and write $p_{i\cdot}(n)$ for the mass function $k \mapsto p_{ik}(n)$ .

The total variation distance between two mass functions $f$ and $g$ on $S$ is

d_{\mathrm{TV}}(f, g) \ = \ \sum_{k \in S} \lvert f_k - g_k \rvert \ = \ 2 \sup_{A \subseteq S} \bigl\lvert \textstyle\sum_{k \in A} f_k - \sum_{k \in A} g_k \bigr\rvert .

Convention. Grimmett and Stirzaker define $d_{\mathrm{TV}}$ without a factor of one half, and note in a footnote that some authors halve it; much of the mixing-time literature uses the halved version, for which the supremum form has no factor $2$ and the distance never exceeds $1$ . We follow Grimmett and Stirzaker throughout this unit, so our $d_{\mathrm{TV}}$ lies in $[0,2]$ . Halve every displayed bound to convert.

Suppose $X$ has stationary distribution $\pi$ and $X_0 \sim \pi$ , so that $X_n \sim \pi$ for every $n$ . For a finite horizon $N$ , the time-reversal of $X$ is $Y_n = X_{N-n}$ , $0 \le n \le N$ . The chain is reversible if $Y$ has the same transition matrix as $X$ , which (by the time-reversal statement below) is the same as requiring the detailed balance equations

\pi_i\, p_{ij} \ = \ \pi_j\, p_{ji} \qquad \text{for all } i, j \in S .

More generally, a transition matrix $P$ and a distribution $\lambda$ are said to be in detailed balance if $\lambda_i p_{ij} = \lambda_j p_{ji}$ for all $i, j$ ; an irreducible chain whose $P$ is in detailed balance with its stationary distribution is called reversible in equilibrium.

Theorem(The Markov chain limit theorem)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.4(B), Example (6.4.19) and Theorem (6.4.20) with remarks (a)-(d), pp. 259-260; Grimmett & Stirzaker (2020), §6.4(A), Theorem (6.4.3), p. 254

Let $X$ be a Markov chain on a countable state space $S$ which is irreducible and aperiodic, with transition matrix $P$ and mean recurrence times $\mu_j = \mathbb{E}_j(T_j)$ . Then

p_{ij}(n) \ \longrightarrow \ \frac{1}{\mu_j} \qquad \text{as } n \to \infty, \ \text{ for every } i, j \in S ,

with the convention $1/\infty = 0$ . Four consequences are worth separating.

When the chain is transient, or recurrent with every $\mu_j$ infinite, the limit is $p_{ij}(n) \to 0$ for all $i, j$ : a limit exists, but it is not a probability distribution.
If the chain is positive recurrent then it has a unique stationary distribution $\pi$ , with $\pi_j = 1/\mu_j$ , and $p_{ij}(n) \to \pi_j$ .
The limit does not depend on the starting state $i$ . Hence for any initial distribution $\mu^{(0)}$ , $\mathbb{P}(X_n = j) = \sum_i \mu^{(0)}_i p_{ij}(n) \ \longrightarrow \ \frac{1}{\mu_j} ,$ so the chain forgets where it started.
Aperiodicity cannot be dropped, but it can be worked around. If $X$ is irreducible with period $d$ , then $\{X_{nd} : n \ge 0\}$ is aperiodic and $p_{jj}(nd) \to d/\mu_j$ .

Theorem(Detailed balance, reversibility, and a cheap route to

\pi

)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.5, Theorem (6.5.1), Definition (6.5.2) and Theorem (6.5.4), pp. 265-266

Let $X$ be irreducible and positive recurrent with stationary distribution $\pi$ , and let $X_0 \sim \pi$ .

(a) The reversed chain is a chain. $Y_n = X_{N-n}$ is a Markov chain, with transition probabilities

\mathbb{P}(Y_{n+1} = j \mid Y_n = i) \ = \ \frac{\pi_j}{\pi_i}\, p_{ji} .

In particular $X$ is reversible exactly when $\pi_i p_{ij} = \pi_j p_{ji}$ for all $i, j$ .

(b) Detailed balance implies stationarity. Suppose an irreducible chain with transition matrix $P$ admits some distribution $\pi$ for which the balance relation $\pi_i p_{ij} = \pi_j p_{ji}$ holds at every pair of states $i, j \in S$ . That $\pi$ is then stationary, and the chain is reversible in equilibrium.

This is the practical content: detailed balance is a system of two-term equations, one for each edge of the transition graph, whereas $\pi = \pi P$ couples every state to every other. When a guess satisfies detailed balance one has found $\pi$ with no linear algebra at all. The converse fails: a stationary distribution need not be in detailed balance with $P$ , and a chain that is not reversible still has its $\pi$ .

Theorem(Geometric rate of convergence for a finite reversible chain)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.6, Theorem (6.6.1) and the bound following it, pp. 269-270; Grimmett & Stirzaker (2020), §6.14, Theorem (6.14.9) and the expansion (6.14.11), pp. 333-334

Let $S$ be finite with $\lvert S \rvert = N$ and let $X$ be irreducible and aperiodic with transition matrix $P$ and stationary distribution $\pi$ . By the Perron-Frobenius theorem the eigenvalues of $P$ may be listed as $\lambda_1 = 1, \lambda_2, \dots, \lambda_N$ with $\lvert \lambda_r \rvert < 1$ for $r \ge 2$ ; order them so that $\lambda_2$ has the largest modulus among these. The quantity $\lvert \lambda_2 \rvert$ is called the second largest eigenvalue modulus, or SLEM. If the eigenvalues are distinct, so that $P = B^{-1}\Lambda B$ is diagonalisable, then $P^{\,n} = B^{-1}\Lambda^n B$ , every row of $P^{\,n}$ tends to $\pi$ , and there is a constant $c < \infty$ with

\lvert p_{ij}(n) - \pi_j \rvert \ \le \ c \,\lvert \lambda_2 \rvert^{\,n} \qquad \text{for all } i, j \text{ and all } n .

Grimmett and Stirzaker state the bound in this diagonalisable setting; more generally, if $\lambda_2$ has multiplicity $m$ one must allow a polynomial factor, and the entries of $P^{\,n}$ approach their limits with an error of order $n^{m-1}\lvert \lambda_2\rvert^{\,n}$ .

If in addition $X$ is reversible in equilibrium, the constant can be made explicit. Equip $\mathbb{R}^N$ with the inner product $\langle x, y \rangle = \sum_{i \in S} \pi_i x_i y_i$ ; reversibility makes $P$ self-adjoint for this inner product, so its eigenvalues are real and its right eigenvectors $v_1, \dots, v_N$ may be taken to form an orthonormal basis, with $v_1 \equiv \mathbf{1}$ . Writing $v_r(i)$ for the $i$ -th entry of $v_r$ ,

\sum_{k \in S} \lvert p_{ik}(n) - \pi_k \rvert \ \le \ N \,\lvert \lambda_2 \rvert^{\,n} \, \sup\bigl\{ \lvert v_r(i) \rvert : r \in S \bigr\} , \qquad i \in S, \ n \ge 1 .

The left-hand side is the total variation distance between $p_{i\cdot}(n)$ and $\pi$ . So convergence is geometric at rate $\lvert \lambda_2 \rvert$ , and the number of steps needed to bring the distance below a tolerance $\varepsilon$ grows like $\log(1/\varepsilon) / \log(1/\lvert\lambda_2\rvert)$ . A chain with $\lvert \lambda_2 \rvert$ close to $1$ is said to be slowly mixing.

Idea of the proofOriginal

The limit theorem is proved by coupling, which replaces an analytic estimate by a construction. Run two copies of the chain side by side and independently, one started at $i$ and one started at $j$ . If the two copies ever occupy the same state at the same time, then from that moment on they have the same conditional law, because a chain's future depends on nothing but its present position and the matrix $P$ . So the difference $p_{ik}(n) - p_{jk}(n)$ can only come from the event that they have not yet met, and it is enough to show that they meet almost surely. The pair is itself a Markov chain on $S \times S$ , and it is irreducible exactly because the original chain is aperiodic: if $X$ had period $2$ , the two copies started at even and odd distance could never be at the same place at the same time. That one sentence is where aperiodicity is used, and it is the only place.

Once the limit is known to be independent of the start, averaging over the start against $\pi$ identifies it as $\pi$ itself, because averaging a constant sequence changes nothing.

Detailed balance is a different kind of statement and needs no limit at all. It says that in equilibrium the probability flux from $i$ to $j$ matches the flux back, edge by edge. Stationarity only demands that the total flux into each node matches the total flux out. Local balance obviously implies global balance, and that one-line implication is the whole proof.

Proof(The limit theorem, by coupling)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.4(B), proof of Theorem (6.4.20) including the coupling inequality (6.4.21) and the bounded convergence step (6.4.22), pp. 260-262

Assume first that $X$ is positive recurrent; the transient and null recurrent cases are dealt with at the end.

Step 1: the coupled chain. Let $X = \{X_n\}$ and $Y = \{Y_n\}$ be independent chains, each with state space $S$ and transition matrix $P$ , and set $Z_n = (X_n, Y_n)$ . Then $Z$ is a Markov chain on $S \times S$ with transition probabilities

\mathbb{P}\bigl(Z_{n+1} = (k,l) \mid Z_n = (i,j)\bigr) = p_{ik}\,p_{jl} ,

the product arising from the independence of the two copies.

Step 2: $Z$ is irreducible — this is where aperiodicity enters. For an irreducible aperiodic chain and any pair of states $a, b$ there is an $N(a,b)$ with $p_{ab}(n) > 0$ for all $n \ge N(a,b)$ : the set $\{n : p_{aa}(n) > 0\}$ is closed under addition and has greatest common divisor $1$ , so it contains every sufficiently large integer, and composing with one path from $a$ to $b$ shifts this to $b$ . Hence for any $(i,j)$ and $(k,l)$ and all $n$ beyond the maximum of the two thresholds, $p_{ik}(n)\,p_{jl}(n) > 0$ , so $Z$ can get from $(i,j)$ to $(k,l)$ . Without aperiodicity this fails: for the chain on $\{1,2\}$ with $p_{12} = p_{21} = 1$ , the pair started at $(1,2)$ visits only $(1,2)$ and $(2,1)$ .

Step 3: $Z$ is positive recurrent. $X$ has a unique stationary distribution $\pi$ . The product $\nu_{ij} = \pi_i \pi_j$ is a distribution on $S \times S$ , and it is stationary for $Z$ because

\sum_{i,j} \nu_{ij}\, p_{ik}p_{jl} = \Bigl(\sum_i \pi_i p_{ik}\Bigr)\Bigl(\sum_j \pi_j p_{jl}\Bigr) = \pi_k \pi_l = \nu_{kl} .

Since $Z$ is irreducible and possesses a stationary distribution, it is positive recurrent.

Step 4: the two copies meet. Fix any $s \in S$ , start the pair at $Z_0 = (i,j)$ , and let

T = \min\{n \ge 1 : Z_n = (s,s)\} .

By recurrence of the irreducible chain $Z$ , the state $(s,s)$ is reached from $(i,j)$ with probability one, so $\mathbb{P}(T < \infty) = 1$ .

Step 5: the coupling inequality. Suppose $T \le n$ . At time $T$ the two copies are both at $s$ ; from then on, their conditional distributions coincide, since each depends only on $P$ and on the common current position. Therefore, conditionally on $\{T \le n\}$ , $X_n$ and $Y_n$ have the same law, and

p_{ik}(n) = \mathbb{P}(X_n = k, T \le n) + \mathbb{P}(X_n = k, T > n) = \mathbb{P}(Y_n = k, T \le n) + \mathbb{P}(X_n = k, T > n) .

Bounding the first term by $\mathbb{P}(Y_n = k) = p_{jk}(n)$ and the second by $\mathbb{P}(T > n)$ gives $p_{ik}(n) \le p_{jk}(n) + \mathbb{P}(T > n)$ , and the same argument with $i$ and $j$ exchanged gives

\bigl\lvert p_{ik}(n) - p_{jk}(n) \bigr\rvert \ \le \ \mathbb{P}(T > n) \ \longrightarrow \ 0 ,

since $T$ is almost surely finite. So if the limit of $p_{ik}(n)$ exists it cannot depend on $i$ .

Step 6: identifying the limit. Average the last display against $\pi$ in the variable $i$ . Since $\sum_i \pi_i p_{ik}(n) = \pi_k$ by stationarity,

\lvert \pi_k - p_{jk}(n) \rvert \ = \ \Bigl\lvert \sum_i \pi_i \bigl( p_{ik}(n) - p_{jk}(n) \bigr) \Bigr\rvert \ \le \ \sum_i \pi_i \bigl\lvert p_{ik}(n) - p_{jk}(n) \bigr\rvert .

The summands are bounded by $1$ and tend to $0$ . To pass to the limit on an infinite $S$ , split the sum at a finite set $F$ : the part over $F$ is a finite sum of terms tending to zero, and the part outside is at most $2\sum_{i \notin F}\pi_i$ , which enlarging $F$ forces below any prescribed tolerance, because $\pi$ is a probability distribution. Hence $p_{jk}(n) \to \pi_k = 1/\mu_k$ .

Step 7: the remaining cases. If $X$ is transient then $\sum_n p_{ij}(n) < \infty$ for all $i, j$ , so $p_{ij}(n) \to 0 = 1/\mu_j$ . If $X$ is null recurrent the coupled chain $Z$ cannot be positive recurrent: starting from $(i,i)$ , the return time of $Z$ to $(i,i)$ is at least the return time of $X$ to $i$ , whose mean is infinite. If $Z$ is transient then $p_{ij}(n)^2 = \mathbb{P}(Z_n = (j,j) \mid Z_0 = (i,i)) \to 0$ and we are done; if $Z$ is null recurrent, Step 5 still applies, and Grimmett and Stirzaker complete the argument by a diagonal-selection device which shows that a non-zero subsequential limit would produce a stationary distribution and so contradict null recurrence. $\square$

Proof(Reversal and detailed balance)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.5, proofs of Theorem (6.5.1) and Theorem (6.5.4), pp. 265-266

(a) The reversed chain. Write a trajectory of $Y$ as a trajectory of $X$ read backwards. For states $i_0, \dots, i_{n+1}$ ,

\mathbb{P}\bigl(Y_{n+1} = i_{n+1} \mid Y_n = i_n, \dots, Y_0 = i_0\bigr) = \frac{\mathbb{P}(X_{N-n-1} = i_{n+1},\, X_{N-n} = i_n, \dots, X_N = i_0)}{\mathbb{P}(X_{N-n} = i_n, \dots, X_N = i_0)} .

Because $X_0 \sim \pi$ and $\pi$ is stationary, every $X_m$ has distribution $\pi$ , and the numerator is the product $\pi_{i_{n+1}} p_{i_{n+1} i_n} p_{i_n i_{n-1}} \cdots p_{i_1 i_0}$ while the denominator is $\pi_{i_n} p_{i_n i_{n-1}} \cdots p_{i_1 i_0}$ . All the factors $p_{i_m i_{m-1}}$ with $m \le n$ cancel, leaving $\pi_{i_{n+1}} p_{i_{n+1} i_n}/\pi_{i_n}$ . This depends on $i_n$ and $i_{n+1}$ only, which proves both that $Y$ is Markov and that its transition probabilities are $(\pi_j/\pi_i)p_{ji}$ . Equality of these with $p_{ij}$ is exactly $\pi_i p_{ij} = \pi_j p_{ji}$ .

(b) Detailed balance implies stationarity. Suppose $\pi$ is a distribution with $\pi_i p_{ij} = \pi_j p_{ji}$ for all $i, j$ . Sum this over $i$ , holding $j$ fixed:

\sum_{i \in S} \pi_i p_{ij} \ = \ \sum_{i \in S} \pi_j p_{ji} \ = \ \pi_j \sum_{i \in S} p_{ji} \ = \ \pi_j ,

the last step because $P$ is stochastic, so row $j$ sums to one. The left side is the $j$ -th entry of $\pi P$ ; hence $\pi P = \pi$ and $\pi$ is stationary. Reversibility then follows from part (a). $\square$

Remark(Where the hypotheses are used, and what fails without them)Original

Aperiodicity is used once, in Step 2, to make the coupled chain irreducible. Its necessity is visible in the simplest example: on $S = \{1,2\}$ with $p_{12} = p_{21} = 1$ , the chain is irreducible and positive recurrent, $\pi = (\tfrac12, \tfrac12)$ is the unique stationary distribution, and yet $p_{11}(n)$ equals $1$ for even $n$ and $0$ for odd $n$ . The Cesaro averages still converge to $\pi$ , and part 4 of the limit theorem recovers the full statement along multiples of the period. A standard repair is to make the chain lazy: replace $P$ by $L = aP + (1-a)I$ with $0 < a < 1$ . Then $L$ is irreducible and aperiodic, has the same stationary distribution, and is reversible whenever $P$ is.

Irreducibility is used in Step 4 (to reach $(s,s)$ ) and in Step 3 (uniqueness of $\pi$ ). Without it, different starting states can have genuinely different limits: a chain with two absorbing states converges to whichever one it is started in.

Positivity of $\pi$ in Step 6 is where the chain's being a probability distribution matters. For a null recurrent chain there is still a stationary measure, but it has infinite total mass and cannot be used to average; that is why Step 7 needs a separate argument.

Independence of the two copies in Step 1 is a choice, not a necessity. The coupling only has to be a joint construction of two processes with the right marginals which meet quickly; cleverer, dependent couplings are how sharp mixing-time bounds are actually obtained.

Reversibility is not needed for the limit theorem at all; it is needed only for the explicit constant in the spectral bound, because it is what makes $P$ self-adjoint for the inner product $\langle x,y\rangle = \sum_i \pi_i x_i y_i$ and hence gives real eigenvalues and an orthonormal eigenbasis (see The spectral theorem for Hermitian operators). An irreversible $P$ may have complex eigenvalues and may fail to be diagonalisable; the bound $c\lvert\lambda_2\rvert^n$ survives, with a polynomial correction $n^{m-1}$ when $\lambda_2$ has multiplicity $m$ in a Jordan block, but the constant $c$ is no longer explicit.

Finiteness of $S$ is needed for the spectral statements only. The limit theorem itself holds on any countable $S$ , and there the rate can be anything at all: an infinite chain may converge sub-geometrically.

Method(Getting a chain to converge to a prescribed

\pi

, and estimating how long it takes)Original

Given a target distribution $\pi$ on a finite set $S$ :

Choose a proposal matrix $H = (h_{ij})$ that is irreducible and easy to sample from; symmetry ( $h_{ij} = h_{ji}$ ) is convenient but not required.
Accept with the Hastings probability. From state $i$ , draw a proposal $j$ with probability $h_{ij}$ and accept it with probability $a_{ij} = \min\left\{1, \ \frac{\pi_j h_{ji}}{\pi_i h_{ij}}\right\} ;$ on rejection stay at $i$ . The resulting matrix is $p_{ij} = h_{ij}a_{ij}$ for $i \ne j$ , with $p_{ii}$ whatever is left over. Only ratios $\pi_j/\pi_i$ are needed, so an unknown normalising constant in $\pi$ is irrelevant. When $H$ is symmetric this reduces to the Metropolis rule $a_{ij} = \min\{1, \pi_j/\pi_i\}$ .
Verify detailed balance, which is immediate from the form of $a_{ij}$ : for $i \neq j$ , $\pi_i h_{ij} a_{ij} = \min\{\pi_i h_{ij}, \pi_j h_{ji}\}$ , an expression symmetric in $i$ and $j$ . By the statement above, $\pi$ is then stationary.
Check aperiodicity. If any proposal is ever rejected with positive probability then some $p_{ii} > 0$ and the chain is aperiodic. If not, make the chain lazy.
Estimate the rate. Compute, or bound, the SLEM $\lvert \lambda_2 \rvert$ . The number of steps to reach total variation distance $\varepsilon$ is of order $\log(1/\varepsilon)/\log(1/\lvert\lambda_2\rvert)$ , and the reversible bound above turns this into an inequality with explicit constants.
Discard the burn-in. Estimate $\sum_\theta g(\theta)\pi_\theta$ by $n^{-1}\sum_{r=M}^{M+n-1} g(X_r)$ , with $M$ chosen large enough that the bound in step 5 is below the accuracy wanted.

Example(A lazy Ehrenfest chain: exact convergence against the spectral bound)Original

A partition with a small hole separates two chambers, $A$ and $B$ , which hold $m = 2$ gas molecules between them. A fair coin is tossed at each tick of a clock; heads leaves the configuration alone, while tails sends one molecule, chosen uniformly from the two, through the hole. Take $X_n$ to be chamber $A$ 's occupancy at time $n$ , so that $S = \{0,1,2\}$ . From state $i$ the chosen molecule is in $A$ with probability $i/2$ , so

p_{i,i-1} = \tfrac12 \cdot \tfrac{i}{2}, \qquad p_{i,i+1} = \tfrac12\Bigl(1 - \tfrac{i}{2}\Bigr), \qquad p_{ii} = \tfrac12 ,

that is

P \ = \ \begin{pmatrix} \tfrac12 & \tfrac12 & 0 \\[2pt] \tfrac14 & \tfrac12 & \tfrac14 \\[2pt] 0 & \tfrac12 & \tfrac12 \end{pmatrix} .

Stationary distribution by detailed balance. Rather than solve $\pi = \pi P$ , look for $\pi$ with $\pi_i p_{i,i+1} = \pi_{i+1}p_{i+1,i}$ ; for a chain that only moves to neighbours these are the only non-trivial detailed balance equations. They read $\pi_0 \cdot \tfrac12 = \pi_1 \cdot \tfrac14$ and $\pi_1 \cdot \tfrac14 = \pi_2 \cdot \tfrac12$ , giving $\pi_1 = 2\pi_0$ and $\pi_2 = \pi_0$ . Normalising,

\pi = \Bigl( \tfrac14, \ \tfrac12, \ \tfrac14 \Bigr) ,

the binomial distribution $\binom{2}{i}2^{-2}$ , as physical intuition demands. By the detailed balance theorem $\pi$ is stationary and the chain is reversible in equilibrium. The chain is irreducible and $p_{00} = \tfrac12 > 0$ , so it is aperiodic, and the limit theorem applies: the mean recurrence times are $\mu_0 = \mu_2 = 4$ and $\mu_1 = 2$ ticks.

Eigenvalues and eigenvectors. One checks directly that

P\begin{pmatrix}1\\1\\1\end{pmatrix} = 1 \cdot \begin{pmatrix}1\\1\\1\end{pmatrix}, \qquad P\begin{pmatrix}1\\0\\-1\end{pmatrix} = \tfrac12\begin{pmatrix}1\\0\\-1\end{pmatrix}, \qquad P\begin{pmatrix}1\\-1\\1\end{pmatrix} = 0 .

For instance the middle one: the first row gives $\tfrac12 \cdot 1 + \tfrac12 \cdot 0 = \tfrac12$ , the second $\tfrac14 \cdot 1 + \tfrac14 \cdot (-1) = 0$ , the third $\tfrac12 \cdot 0 + \tfrac12 \cdot (-1) = -\tfrac12$ . So the spectrum is $\{1, \tfrac12, 0\}$ and the SLEM is $\lvert \lambda_2 \rvert = \tfrac12$ . Normalising in the inner product $\langle x,y\rangle = \sum_i \pi_i x_i y_i$ : the first vector has $\langle v,v\rangle = \tfrac14 + \tfrac12 + \tfrac14 = 1$ ; the second has $\tfrac14 \cdot 1 + \tfrac12 \cdot 0 + \tfrac14 \cdot 1 = \tfrac12$ , so $v_2 = \sqrt{2}\,(1,0,-1)$ ; the third has $\tfrac14 + \tfrac12 + \tfrac14 = 1$ already. They are mutually orthogonal, as reversibility guarantees.

Exact $n$ -step probabilities. Expanding $p_{ik}(n) = \sum_r v_r(k)\pi_k \lambda_r^{\,n} v_r(i)$ from state $i = 0$ :

p_{00}(n) = \tfrac14\bigl(1 + 2 \cdot 2^{-n}\bigr) = \tfrac14 + 2^{-(n+1)} , \qquad p_{01}(n) = \tfrac12 , \qquad p_{02}(n) = \tfrac14 - 2^{-(n+1)} ,

valid for $n \ge 1$ . (Check at $n = 1$ : $\tfrac14 + \tfrac14 = \tfrac12$ , $\tfrac12$ , $0$ — the first row of $P$ .) Hence the exact total variation distance from the stationary distribution, started from the empty chamber, is

d_{\mathrm{TV}}\bigl(p_{0\cdot}(n), \pi\bigr) = 2^{-(n+1)} + 0 + 2^{-(n+1)} = 2^{-n} .

At $n = 5$ this equals $\tfrac{1}{32} = 0.03125$ .

Comparison with the bound. The reversible bound gives, with $N = 3$ , $\lvert\lambda_2\rvert = \tfrac12$ and $\sup_r \lvert v_r(0)\rvert = \max\{1, \sqrt2, 1\} = \sqrt2$ ,

d_{\mathrm{TV}}\bigl(p_{0\cdot}(n), \pi\bigr) \ \le \ 3\sqrt{2}\, \cdot 2^{-n} ,

which at $n = 5$ is $3\sqrt2/32 = 0.1326$ . The bound holds, and overestimates the true distance by the factor $3\sqrt2 \approx 4.24$ — it gets the geometric rate exactly right and only the prefactor is loose. To force $d_{\mathrm{TV}} \le 10^{-3}$ the bound demands $n \ge \log_2(3\sqrt2 \times 10^{3}) = 12.05$ , so $n = 13$ ; the truth is $n = 10$ .

Application(Markov chain Monte Carlo and detailed balance in statistical physics)Original

Every Monte Carlo simulation of a thermal system is an instance of this unit. One wants averages $\langle g \rangle = \sum_\theta g(\theta)\pi(\theta)$ over the Boltzmann distribution $\pi(\theta) = Z^{-1}\exp(-\beta E(\theta))$ on a configuration space far too large to enumerate, and the partition function $Z$ is unknown. The Metropolis recipe in the method above builds a chain whose stationary distribution is exactly $\pi$ , using only the energy differences $E(\theta') - E(\theta)$ , because the acceptance probability $\min\{1, e^{-\beta \Delta E}\}$ depends on $\pi$ only through ratios and $Z$ cancels. Detailed balance is the condition physicists impose on the update rule precisely because it guarantees, by the theorem above, that the Boltzmann weight is left invariant; it is a sufficient condition, imposed for convenience, not a law of nature, and schemes that are stationary without being reversible are also in use.

The limit theorem is then the statement that the simulation gives the right answer at all, and aperiodicity is the fine print: a deterministic sweep through the lattice in a model with no self-transitions can be periodic, which is one reason random site selection or a laziness parameter is standard.

The rate statement is where the physics gets hard. Near a continuous phase transition the SLEM of a local-update chain for the Ising model approaches $1$ as the system size grows, the relaxation time diverges as $L^{z}$ with a dynamical exponent $z \approx 2$ for Metropolis dynamics, and the simulation suffers critical slowing down. Cluster algorithms are designed to keep detailed balance while drastically reducing $\lvert \lambda_2 \rvert$ . The eigenvalue $\lambda_2$ and its eigenvector also carry physical meaning away from criticality: they identify the slowest collective mode of the dynamics, which in a chain built from molecular dynamics (see Discrete-time Markov chains) is the rate-limiting conformational change.

Finally, the same bound tells an experimentalist when an equilibration protocol is long enough. For the Ehrenfest model of the example, generalised to $m$ molecules, the SLEM of the lazy chain is $1 - 1/m$ , so equilibration takes of order $m \log m$ ticks: the characteristic time grows only a little faster than the number of molecules, which is why gases equilibrate fast.

Theorem

Random walks

Adapted

Add to PDF

Solves the gambler's ruin problem for a simple random walk between two absorbing barriers, and proves Polya's trichotomy: the symmetric nearest-neighbour walk returns to its starting point with probability one on the line and on the plane, but not in space, where the return probability is about 0.34.

Prerequisites: Discrete-time Markov chains, Generating functions, Characteristic functions

Definition(Simple random walk, on the line and on the lattice)AdaptedAdapted from Grimmett & Stirzaker (2020), §3.9, equations (3.9.1)-(3.9.2) and Lemmas (3.9.3)-(3.9.5), pp. 77-79; Grimmett & Stirzaker (2020), Problem (6.15.9), p. 340

Let $X_1, X_2, \dots$ be independent, all carrying the same two-point law, which puts mass $p$ on the value $+1$ and mass $q = 1 - p$ on the value $-1$ , with $0 < p < 1$ . The simple random walk started at $a \in \mathbb{Z}$ is the sequence

S_0 = a , \qquad S_n = a + \sum_{i=1}^{n} X_i \quad (n \ge 1) .

The walk is symmetric if $p = q = \tfrac12$ . It is spatially homogeneous ( $\mathbb{P}(S_n = j \mid S_0 = a)$ depends only on $j - a$ ), temporally homogeneous, and has the Markov property, so it is a Markov chain on $\mathbb{Z}$ in the sense of Discrete-time Markov chains, with transition probabilities $p_{i,i+1} = p$ , $p_{i,i-1} = q$ and all others zero.

Two variants are needed below.

Absorbing barriers. Fix an integer $N \ge 1$ and restrict to the state space $\{0, 1, \dots, N\}$ , declaring $0$ and $N$ to be absorbing: $p_{00} = p_{NN} = 1$ . In gambling language, a player holding $k$ units stakes one unit on each of a sequence of independent bets won with probability $p$ , and stops on reaching $0$ (ruin) or $N$ (the target).

The lattice walk in $d$ dimensions. Let $\mathbf{S}_0 = \mathbf{0}$ and $\mathbf{S}_n = \sum_{i \le n}\boldsymbol{\epsilon}_i$ , where the steps $\boldsymbol{\epsilon}_i$ are independent and uniform on the $2d$ unit vectors $\pm \mathbf{e}_1, \dots, \pm \mathbf{e}_d$ of $\mathbb{Z}^d$ , each with probability $1/(2d)$ . For $d = 1$ this is the symmetric walk above.

Occupation and first-return probabilities. Whichever of these unrestricted walks is under discussion — the walk on $\mathbb{Z}$ with parameter $p$ , started at $a = 0$ , or the symmetric lattice walk in $d$ dimensions — we write

p_0(n) = \mathbb{P}(\text{walk is at the origin at time } n) , \qquad f_0(n) = \mathbb{P}(\text{the first return to the origin occurs at time } n) ,

with $p_0(0) = 1$ and $f_0(0) = 0$ . In every case a single step changes the parity of the sum of the coordinates, so $p_0(n) = 0$ for odd $n$ and every state has period $2$ .

The walk is called recurrent if it returns to its starting point with probability one and transient otherwise; by spatial homogeneity this does not depend on the starting point, and it is the recurrence of the origin as a state of the chain.

Theorem(Gambler's ruin)AdaptedAdapted from Grimmett & Stirzaker (2020), §3.9, Example (3.9.6), the difference equations (3.9.7) and (3.9.8) and their solutions, including (3.9.9), pp. 79-80; Grimmett & Stirzaker (2020), Exercise (3.9.1), p. 80

Take the walk of the definition, confined as described to the states $0, 1, \dots, N$ and absorbed at whichever end it reaches first, and start it from $k$ , where $0 \le k \le N$ . Write $\mathbb{P}_k$ and $\mathbb{E}_k$ for probabilities and expectations of the walk so started, write $p_k$ for the chance that it finishes at $0$ rather than at $N$ (the ruin probability, indexed by the starting state; it has nothing to do with the occupation probabilities $p_0(n)$ of the definition, which are indexed by time, and the clash of the two at $p_0 = p_0(0) = 1$ is a coincidence of values), write $D_k$ for the mean number of steps it makes before stopping, and put $\rho = q/p$ . Then

p_k \ = \ \begin{cases} \dfrac{\rho^{\,k} - \rho^{\,N}}{1 - \rho^{\,N}} , & p \ne \tfrac12 , \\[10pt] 1 - \dfrac{k}{N} , & p = \tfrac12 , \end{cases} \qquad\qquad D_k \ = \ \begin{cases} \dfrac{1}{q - p}\left( k - N\,\dfrac{1 - \rho^{\,k}}{1 - \rho^{\,N}} \right) , & p \ne \tfrac12 , \\[10pt] k\,(N - k) , & p = \tfrac12 . \end{cases}

Absorption is certain, and the time $T$ it takes has a geometric tail: writing $\delta = \min(p, q)^N > 0$ ,

\mathbb{P}_k(T > mN) \ \le \ (1 - \delta)^m \qquad (m = 0, 1, 2, \dots) .

In particular $\mathbb{P}_k(T < \infty) = 1$ and every moment of $T$ is finite, so $D_k = \mathbb{E}_k(T)$ is the finite number displayed above, and $1 - p_k$ is the probability of reaching the upper barrier $N$ .

Theorem(Polya's trichotomy: recurrence in one and two dimensions, transience in three)AdaptedAdapted from Grimmett & Stirzaker (2020), §5.3, Theorem (5.3.1) and Corollary (5.3.4), pp. 183-185; Grimmett & Stirzaker (2020), §6.2, Corollary (6.2.4) and Example (6.2.12), pp. 248-249; Grimmett & Stirzaker (2020), Problems (5.12.5), (5.12.6), p. 231, and Problem (6.15.9), p. 340

(a) The criterion, specialised to the walk. By the summability criterion of Discrete-time Markov chains, a state $j$ is recurrent exactly when $\sum_n p_{jj}(n) = \infty$ . Here $p_0(n) = 0$ at odd $n$ , so the lattice walk is recurrent exactly when

\sum_{n \ge 0} p_0(2n) = \infty ,

and this sum is the expected number of visits the walk makes to the origin.

(b) One dimension, general $p$ . For the simple random walk on $\mathbb{Z}$ , the generating functions $P_0(s) = \sum_n s^n p_0(n)$ and $F_0(s) = \sum_n s^n f_0(n)$ of the occupation and first-return probabilities satisfy

P_0(s) = 1 + P_0(s)F_0(s), \qquad P_0(s) = (1 - 4pqs^2)^{-1/2}, \qquad F_0(s) = 1 - (1 - 4pqs^2)^{1/2} ,

so that the chance of a return to the origin ever occurring is

F_0(1) = 1 - \lvert p - q \rvert .

Recurrence therefore holds precisely at $p = \tfrac12$ , and there the mean time until the walk first comes back diverges: the symmetric walk on $\mathbb{Z}$ is null recurrent, and every state has period $2$ . For $p \neq \tfrac12$ every state is transient. Equivalently, $p_0(2n) \sim (4pq)^n/\sqrt{\pi n}$ , which is summable precisely when $4pq < 1$ , that is when $p \neq \tfrac12$ .

(c) Two dimensions. For the symmetric walk on $\mathbb{Z}^2$ ,

p_0(2n) \ = \ \left[ \binom{2n}{n} 4^{-n} \right]^2 \ \sim \ \frac{1}{\pi n} ,

so $\sum_n p_0(2n) = \infty$ and the walk is recurrent: it returns to the origin with probability one.

(d) Three dimensions. For the symmetric walk on $\mathbb{Z}^3$ ,

p_0(2n) \ = \ \binom{2n}{n} 2^{-2n} \sum_{i+j+k=n} \left( \frac{n!}{3^{\,n}\, i!\,j!\,k!} \right)^{2} \ \le \ \frac{C}{n^{3/2}}

for a finite constant $C$ , so $\sum_n p_0(2n) < \infty$ and the walk is transient.

Idea of the proofOriginal

Both halves of the unit come from the Markov property, used in two different ways.

For gambler's ruin, condition on the first bet. The walk started at $k$ is, after one step, a walk started at $k+1$ or at $k-1$ , with no memory of how it got there. So the unknown $p_k$ satisfies a linear second-order difference equation in $k$ — the discrete analogue of Laplace's equation — with boundary values $1$ at $0$ and $0$ at $N$ . Its characteristic equation $p\theta^2 - \theta + q = 0$ has the roots $1$ and $q/p$ . When $p \ne q$ the two roots are distinct and the powers $1^k$ and $(q/p)^k$ span the solutions; in the symmetric case the root $1$ is repeated, the second solution is $k$ rather than a second power, and the general solution reads $A_1 + A_2 k$ . The expected duration satisfies the same equation with an inhomogeneous term $1$ , because one step has been used up.

For recurrence, condition instead on the time of first return. That turns the sequence $p_0(n)$ into the convolution of itself with the first-return sequence, and convolution is multiplication of generating functions, which inverts to $P_0 = 1/(1 - F_0)$ . Letting $s \uparrow 1$ converts the question "is return certain?" into the question "is $\sum_n p_0(n)$ infinite?" — a question about the size of the walk, not its combinatorics. And the size is set by the central limit scaling: after $n$ steps the walk is spread over a region of diameter $\sqrt{n}$ in each direction, so the chance of being at any one named site, the origin included, is of order $n^{-d/2}$ . The series $\sum n^{-d/2}$ diverges for $d = 1, 2$ and converges for $d \ge 3$ . Everything below is this one-line dimension count made exact.

Proof(Gambler's ruin)AdaptedAdapted from Grimmett & Stirzaker (2020), §3.9, Example (3.9.6) and the solution of (3.9.7) and (3.9.8), pp. 79-80

Absorption is certain and $D_k$ is finite. This has to come first, because the difference equations below are solved in the real numbers, which presupposes that their solutions are finite; the recursions themselves are also satisfied by $D \equiv \infty$ . Let $T$ be the absorption time and $\delta = \min(p,q)^N$ . Whatever the walk has done so far, if it is not yet absorbed it sits at some $i$ with $0 < i < N$ , and if its next $N$ steps are all $+1$ it reaches $N$ after $N - i < N$ of them and is absorbed. That block of steps has probability $p^N \ge \delta$ , and the steps are independent of the past, so from any position not yet absorbed the chance of absorption within the next $N$ steps is at least $\delta$ . Iterating over $m$ disjoint blocks of $N$ steps gives $\mathbb{P}_k(T > mN) \le (1-\delta)^m$ . Hence $\mathbb{P}_k(T = \infty) = 0$ , and summing the tail,

D_k = \mathbb{E}_k(T) = \sum_{n \ge 0}\mathbb{P}_k(T > n) \ \le \ N\sum_{m \ge 0}(1-\delta)^m = \frac{N}{\delta} \ < \ \infty .

(The same bound makes every moment of $T$ finite, which is the content of Exercise (3.9.1) of the source.) Since absorption is certain, $p_k$ and $1 - p_k$ are the probabilities of the two alternatives, finishing at $0$ and finishing at $N$ .

The difference equation. Let $1 \le k \le N-1$ and condition on the first step. With probability $p$ the walk moves to $k+1$ , and by the Markov property and temporal homogeneity it is then absorbed at $0$ with probability $p_{k+1}$ ; with probability $q$ it moves to $k-1$ and is absorbed at $0$ with probability $p_{k-1}$ . Hence

p_k = p\,p_{k+1} + q\,p_{k-1} , \qquad 1 \le k \le N-1 ,

with $p_0 = 1$ and $p_N = 0$ because those states are absorbing.

Solving it. Substituting the trial solution $p_k = \theta^{\,k}$ and dividing by $\theta^{\,k-1}$ gives the characteristic equation $p\theta^2 - \theta + q = 0$ , whose roots are $\theta = 1$ and $\theta = q/p = \rho$ , since $p + q = 1$ .

Case $p \ne \tfrac12$ . The roots are distinct, so the general solution is $p_k = A_1 + A_2\rho^{\,k}$ . The boundary conditions give $A_1 + A_2 = 1$ and $A_1 + A_2\rho^{\,N} = 0$ , so $A_2 = 1/(1 - \rho^{\,N})$ and $A_1 = -\rho^{\,N}/(1-\rho^{\,N})$ , which is the stated formula.

Case $p = \tfrac12$ . The root $\theta = 1$ is repeated, so the general solution is $p_k = A_1 + A_2 k$ , and the boundary conditions give $A_1 = 1$ , $A_2 = -1/N$ .

The duration. Condition on the first step again, this time keeping track of the step taken. From $k$ , one step is spent and the walk then needs $D_{k\pm1}$ further steps on average, so

D_k = p\,(1 + D_{k+1}) + q\,(1 + D_{k-1}) = 1 + p D_{k+1} + q D_{k-1} , \qquad 1 \le k \le N-1 ,

with $D_0 = D_N = 0$ . This is the same homogeneous equation with the constant forcing term $1$ , so the general solution is a particular solution plus $A_1 + A_2\rho^{\,k}$ (or $A_1 + A_2 k$ when $p = \tfrac12$ ).

Case $p \ne \tfrac12$ . Try the particular solution $D_k = ck$ . Then $ck = 1 + cp(k+1) + cq(k-1) = 1 + ck + c(p - q)$ , so $c = 1/(q-p)$ . Imposing $D_0 = D_N = 0$ on $D_k = k/(q-p) + A_1 + A_2\rho^{\,k}$ gives $A_1 + A_2 = 0$ and $N/(q-p) + A_1 + A_2\rho^{\,N} = 0$ , so $A_2 = -A_1$ and $A_1 (1 - \rho^{\,N}) = -N/(q-p)$ . Substituting back,

D_k = \frac{1}{q-p}\left( k - N\,\frac{1 - \rho^{\,k}}{1 - \rho^{\,N}} \right) .

Case $p = \tfrac12$ . Now $p - q = 0$ , so a linear particular solution cannot work; try $D_k = ck^2$ , giving $ck^2 = 1 + \tfrac{c}{2}[(k+1)^2 + (k-1)^2] = 1 + ck^2 + c$ , so $c = -1$ . With $D_k = -k^2 + A_1 + A_2 k$ and $D_0 = D_N = 0$ we get $A_1 = 0$ and $A_2 = N$ , that is $D_k = k(N-k)$ .

Both solutions are legitimate because the tail bound at the head of the proof has already made $p_k$ and $D_k$ finite. $\square$

Proof(Recurrence in one and two dimensions, transience in three)AdaptedAdapted from Grimmett & Stirzaker (2020), §5.3, Theorem (5.3.1) and Corollary (5.3.4), pp. 183-185; Grimmett & Stirzaker (2020), §6.2, Corollary (6.2.4), p. 248; Grimmett & Stirzaker (2020), Problems (5.12.5), (5.12.6), p. 231, and Problem (6.15.9), p. 340

The criterion. This part is taken over from the prerequisite rather than reproved: the renewal decomposition and its generating-function consequence are established for a general chain in Discrete-time Markov chains, and we record only what they say for the walk. Splitting a visit to the origin at time $n \ge 1$ according to the time $r$ of the first return gives $n$ disjoint possibilities that exhaust the event. On each of them the walk starts afresh at the origin at time $r$ — note that $\{\text{first return at } r\}$ is not an event about the state at a single time but a condition on the whole path up to $r$ , so the Markov property is applied at the fixed time $r$ to each such path separately and the contributions are then summed — and therefore

p_0(n) = \sum_{r=1}^{n} f_0(r)\, p_0(n-r) .

Multiplying by $s^n$ , summing over $n \ge 1$ and using $p_0(0) = 1$ turns this convolution into $P_0(s) = 1 + F_0(s)P_0(s)$ , that is $P_0(s) = 1/(1 - F_0(s))$ for $\lvert s \rvert < 1$ ; letting $s \uparrow 1$ termwise, which Abel's theorem permits because all coefficients are non-negative,

\sum_{n \ge 0} p_0(n) = \frac{1}{1 - f} , \qquad f := \sum_{n \ge 1} f_0(n) = \mathbb{P}(\text{ever return}) ,

read with the convention $1/0 = \infty$ : if $f = 1$ then $F_0(s) \uparrow 1$ and $P_0(s) = 1/(1 - F_0(s)) \uparrow \infty$ , so the left-hand side, a series of non-negative terms, diverges. So the sum is infinite exactly when $f = 1$ , which is part (a); the left-hand side is also the expected number of visits to the origin. What is left to do, and what the rest of the proof does, is to compute $p_0(n)$ in each dimension.

(b) One dimension. The walk is at the origin at time $n$ only if $n = 2m$ is even and exactly $m$ of the steps were $+1$ , so

p_0(2m) = \binom{2m}{m}(pq)^m .

The generating function of $\binom{2m}{m}x^m$ is $(1-4x)^{-1/2}$ , so $P_0(s) = (1 - 4pqs^2)^{-1/2}$ and therefore $F_0(s) = 1 - 1/P_0(s) = 1 - (1-4pqs^2)^{1/2}$ . At $s = 1$ ,

f = F_0(1) = 1 - \sqrt{1 - 4pq} = 1 - \sqrt{(p+q)^2 - 4pq} = 1 - \lvert p - q\rvert ,

using $p + q = 1$ . Hence $f = 1$ if and only if $p = q = \tfrac12$ . When $p = \tfrac12$ , $F_0(s) = 1 - (1-s^2)^{1/2}$ and $F_0'(s) = s(1-s^2)^{-1/2} \to \infty$ as $s \uparrow 1$ , so the mean return time is infinite and the walk is null recurrent. The same conclusion follows from the criterion: Stirling's formula $m! \sim m^{m+1/2}e^{-m}\sqrt{2\pi}$ gives $\binom{2m}{m} \sim 4^m/\sqrt{\pi m}$ and hence

p_0(2m) \sim \frac{(4pq)^m}{\sqrt{\pi m}} ,

which sums to $\infty$ iff $4pq = 1$ , that is iff $p = \tfrac12$ .

(c) Two dimensions. The walk is at the origin at time $2n$ exactly when, for some $k$ , it made $k$ steps east, $k$ west, $n-k$ north and $n-k$ south. The number of orderings is the multinomial coefficient, and each of the $4^{2n}$ step sequences is equally likely, so

p_0(2n) = 4^{-2n}\sum_{k=0}^{n}\frac{(2n)!}{k!\,k!\,(n-k)!\,(n-k)!} .

Factor the summand as $\binom{2n}{n}\binom{n}{k}^2$ , which is an identity between factorials:

\binom{2n}{n}\binom{n}{k}^2 = \frac{(2n)!}{n!\,n!}\cdot\left(\frac{n!}{k!\,(n-k)!}\right)^2 = \frac{(2n)!}{k!^2\,(n-k)!^2} .

By Vandermonde's identity $\sum_{k=0}^n \binom{n}{k}^2 = \binom{2n}{n}$ , so the sum collapses and

p_0(2n) = 4^{-2n}\binom{2n}{n}^{2} = \left[\binom{2n}{n}4^{-n}\right]^{2} \sim \left(\frac{1}{\sqrt{\pi n}}\right)^{2} = \frac{1}{\pi n} ,

again by Stirling. Since $\sum_n 1/(\pi n)$ diverges, the criterion gives recurrence, and the return probability is $1$ .

(d) Three dimensions. Now the walk is at the origin at time $2n$ exactly when, for some $i, j, k \ge 0$ with $i + j + k = n$ , it made $i$ steps along each of $\pm\mathbf{e}_1$ , $j$ along each of $\pm\mathbf{e}_2$ and $k$ along each of $\pm\mathbf{e}_3$ . There are $6^{2n}$ equally likely step sequences, so

p_0(2n) = 6^{-2n}\sum_{i+j+k=n}\frac{(2n)!}{(i!\,j!\,k!)^{2}} = \binom{2n}{n}2^{-2n}\sum_{i+j+k=n}\left(\frac{n!}{3^{\,n}\,i!\,j!\,k!}\right)^{2} ,

where the second form follows by writing $6^{-2n} = 4^{-n}9^{-n}$ and $(2n)!/(i!j!k!)^2 = \binom{2n}{n}\,(n!/(i!j!k!))^2$ . The rewriting is the point: the numbers $w_{ijk} = n!/(3^n i!\,j!\,k!)$ are non-negative and sum to $1$ , because $\sum_{i+j+k=n} n!/(i!j!k!) = 3^n$ by the multinomial theorem. Therefore

\sum_{i+j+k=n} w_{ijk}^{2} \ \le \ \Bigl(\max_{i+j+k=n} w_{ijk}\Bigr)\sum_{i+j+k=n} w_{ijk} \ = \ \max_{i+j+k=n} w_{ijk} .

The multinomial coefficient $n!/(i!j!k!)$ is largest when the three parts are as equal as possible, since replacing a pair $(i, j)$ with $i \ge j+2$ by $(i-1, j+1)$ multiplies it by $i/(j+1) > 1$ . So the maximum is attained at the nearest-to-equal split, whose parts differ from $n/3$ by less than $1$ . For $n$ divisible by $3$ that split is $(n/3, n/3, n/3)$ , and Stirling applied to $n!/((n/3)!)^3$ gives

\max_{i+j+k=n} \frac{n!}{i!\,j!\,k!} \ \sim \ \frac{3^{\,n+3/2}}{2\pi n} , \qquad\text{so}\qquad \max_{i+j+k=n} w_{ijk} \ \sim \ \frac{3\sqrt3}{2\pi n} .

For the other two residues each part is $n/3 + O(1)$ , which changes the Stirling estimate by a factor $1 + O(1/n)$ only, so the same asymptotic holds for every $n$ ; numerically the maximum is in fact slightly smaller off the divisible case, $n\max_{i+j+k=n} w_{ijk}$ being $0.8158$ at $n = 121$ and $0.8224$ at $n = 120$ , both below the limiting value $3\sqrt3/(2\pi) = 0.8270$ .

Combining this with $\binom{2n}{n}2^{-2n} \sim 1/\sqrt{\pi n}$ gives

p_0(2n) \ \le \ \frac{C}{n^{3/2}}\bigl(1 + o(1)\bigr), \qquad C = \frac{3\sqrt3}{2\pi\sqrt{\pi}} = 0.4666 .

The series $\sum_n n^{-3/2}$ converges, so $\sum_n p_0(2n) < \infty$ and the origin is transient. By spatial homogeneity every state is transient. $\square$

Remark(Where the hypotheses are used, and how sharp the conclusions are)Original

Symmetry in parts (c) and (d). Only the symmetric walk is treated there, and for good reason: a lattice walk with any non-zero mean drift is transient in every dimension, because the strong law of large numbers of Laws of large numbers already pushes $\mathbf{S}_n/n$ away from the origin. Recurrence in $d = 1, 2$ is a statement about the borderline case of zero drift.

Independence of the steps. Used in the counting: every sequence of $2n$ steps has the same probability. For a self-avoiding walk, or any walk with memory, none of the formulas survive, and even the two-dimensional case becomes a hard open problem.

Finiteness of $N$ in gambler's ruin. Letting $N \to \infty$ at fixed $k$ in the ruin formula recovers the one-barrier answer: for $p > \tfrac12$ we have $\rho < 1$ , so $p_k \to \rho^{\,k} < 1$ and the player survives forever with positive probability, while for $p \le \tfrac12$ we get $p_k \to 1$ and ruin is certain. The mean duration does not follow that pattern, and the three cases must be kept apart. For $p < \tfrac12$ we have $\rho > 1$ , so in $D_k$ the ratio $N(1-\rho^{\,k})/(1-\rho^{\,N})$ has a numerator linear in $N$ and a denominator geometric in $N$ and therefore tends to $0$ , leaving

D_k \ \longrightarrow \ \frac{k}{q-p} \qquad (p < \tfrac12) :

ruin is certain and quick (at $p = \tfrac25$ , $k = 5$ the limit is $25$ steps, and the finite- $N$ values $19.18$ , $24.80$ , $25.00$ at $N = 10, 20, 50$ climb to it). At $p = \tfrac12$ , by contrast, $D_k = k(N-k) \to \infty$ : ruin is certain while the expected time to it is infinite, so there "ruin is certain" and "ruin happens quickly" come apart. That single case is the null recurrence of part (b) seen from another side; at $p < \tfrac12$ the walk is transient, not null recurrent, and the analogy does not apply to it. (For $p > \tfrac12$ also $D_k \to \infty$ , but for the opposite reason: in the limit absorption at $0$ is not certain at all.)

Higher dimensions. The cited treatment stops at $d = 3$ , and what follows is our own extension. The central limit scaling of the idea block can be made local: after $2n$ steps of the symmetric walk on $\mathbb{Z}^d$ each coordinate has variance $2n/d$ , the coordinates are uncorrelated, and the Gaussian density at the origin is $(4\pi n/d)^{-d/2}$ , which has to be shared between the two parity classes of sites, giving

p_0(2n) \ \sim \ 2\left(\frac{d}{4\pi n}\right)^{d/2} .

For $d = 1, 2, 3$ this returns $1/\sqrt{\pi n}$ , $1/(\pi n)$ and $0.2333\,n^{-3/2}$ , the values computed above, and for every $d \ge 3$ it makes $\sum_n p_0(2n)$ converge: all the higher-dimensional walks are transient too, with the exponent $d/2$ in place of $3/2$ .

The exponent versus the constant. The estimate in (d) is an upper bound, not an asymptotic. Its constant $C = 0.4666$ is exactly twice the true asymptotic constant $3\sqrt3/(4\pi\sqrt\pi) = 0.2333$ , because bounding one factor $w_{ijk}$ by its maximum throws away a factor $2$ in the limit: a direct computation of the exact multinomial sum gives $p_0(80) = 9.136\times10^{-4}$ against the bound $C\cdot 40^{-3/2} = 1.844\times10^{-3}$ . Transience needs only the exponent $3/2$ , so the slack costs nothing.

How transient is three-dimensional space? The criterion gives the return probability exactly: $f = 1 - 1/G$ with $G = \sum_{n\ge0}p_0(n)$ the expected number of visits. By the Fourier inversion formula for a lattice-valued sum (see Characteristic functions, and Riley, Hobson and Bence (Riley, Hobson & Bence (2006), §30.6, eq. (30.64), p. 1155) for the same representation of the displacement of a general multi-step walk), $p_0(n) = (2\pi)^{-3}\int_{[-\pi,\pi]^3}\varphi(\mathbf{k})^n \, d^3k$ with single-step characteristic function $\varphi(\mathbf{k}) = \tfrac13(\cos k_1 + \cos k_2 + \cos k_3)$ , so summing the geometric series gives

G = \frac{1}{(2\pi)^3}\int_{[-\pi,\pi]^3}\frac{d^3 k}{1 - \tfrac13(\cos k_1 + \cos k_2 + \cos k_3)} .

Numerical integration gives $G = 1.51639$ and hence $f = 0.34054$ : a walker on the cubic lattice comes back to where it started about one time in three, and otherwise never. The partial sums of the exact series agree: $\sum_{n=1}^{40} p_0(2n) = 0.44330$ , and estimating the remainder from the true asymptotic constant $C' = 3\sqrt3/(4\pi\sqrt\pi) = 0.2333$ of the previous paragraph,

\sum_{n > 40} p_0(2n) \ \approx \ \int_{40}^{\infty} \frac{C'}{n^{3/2}}\,dn = \frac{2C'}{\sqrt{40}} = 0.0738 ,

which brings the total to $0.5171$ against the exact $G - 1 = 0.51639$ .

Periodicity. Every state of the lattice walk has period $2$ , so the limit theorem of Convergence of Markov chains applies only along even times — and in any case its conclusion here is the uninformative $p_0(2n) \to 0$ , since the walk is null recurrent or transient and has no stationary distribution.

Method(First-passage and absorption problems by conditioning on the first step)Original

The gambler's ruin calculation is a template. For a Markov chain on a finite or semi-infinite set of integers with nearest-neighbour steps:

Name the unknown as a function of the starting state, $h_k$ , and make sure it is a quantity to which the Markov property applies: an absorption probability, an expected hitting time, an expected reward collected before absorption.
Condition on the first step. Each possible first step leads to a state from which the chain starts afresh, so $h_k = \sum_j p_{kj}\,h_j$ for an absorption probability, and $h_k = 1 + \sum_j p_{kj}\,h_j$ for an expected time (the $1$ being the step just taken).
Write the boundary conditions at the absorbing states: $h = 1$ or $0$ for a probability, $h = 0$ for a hitting time.
Solve the difference equation. Substitute $h_k = \theta^{\,k}$ into the homogeneous part to get the characteristic polynomial; distinct roots $\theta_1 \ne \theta_2$ give $A_1\theta_1^{\,k} + A_2\theta_2^{\,k}$ , a repeated root $\theta$ gives $(A_1 + A_2 k)\theta^{\,k}$ . For an inhomogeneous equation add a particular solution, raising the degree of the polynomial trial by one each time it is annihilated by the homogeneous operator.
Fix the constants from the boundary conditions, then sanity-check the limits: the symmetric case, the $N \to \infty$ case, and the cases $k = 0$ and $k = N$ should all give the obvious answers.

Example(A bounded betting run at the roulette wheel)Original

A player arrives at a European roulette table with $20$ chips and decides to bet one chip on red on each spin, stopping when he reaches $50$ chips or loses everything. A European wheel has $37$ pockets, of which $18$ are red, so

p = \tfrac{18}{37}, \qquad q = \tfrac{19}{37}, \qquad \rho = \frac{q}{p} = \frac{19}{18} = 1.0\overline{5} .

Here $k = 20$ and $N = 50$ .

Ruin probability. We need $\rho^{20}$ and $\rho^{50}$ . From $\log \rho = \log(19/18) = 0.0540672$ , we get $20\log\rho = 1.0813444$ and $50\log\rho = 2.7033611$ , so

\rho^{20} = e^{1.0813444} = 2.948641 , \qquad \rho^{50} = e^{2.7033611} = 14.929828 .

Hence

p_{20} = \frac{\rho^{20} - \rho^{50}}{1 - \rho^{50}} = \frac{2.948641 - 14.929828}{1 - 14.929828} = \frac{-11.981187}{-13.929828} = 0.860110 .

So the player is ruined with probability $0.8601$ , and reaches $50$ chips with probability $0.1399$ .

Expected length of the session.

D_{20} = \frac{1}{q-p}\left(20 - 50\,\frac{1 - \rho^{20}}{1-\rho^{50}}\right) .

Now $q - p = 1/37$ , so $1/(q-p) = 37$ , and

\frac{1-\rho^{20}}{1-\rho^{50}} = \frac{1 - 2.948641}{1 - 14.929828} = \frac{-1.948641}{-13.929828} = 0.139890 .

Therefore

D_{20} = 37\,\bigl(20 - 50 \times 0.139890\bigr) = 37 \times \bigl(20 - 6.99450\bigr) = 37 \times 13.00550 = 481.204 .

(The exact rational value is $481.2038$ , so the six-figure intermediate quantities have cost nothing.) The session lasts about $481$ spins on average.

The fair game for comparison. Had the wheel had no zero, $p = q = \tfrac12$ would give

p_{20} = 1 - \tfrac{20}{50} = 0.6 , \qquad D_{20} = 20 \times 30 = 600 .

The single green pocket, a house edge of $1/37 = 2.7\%$ per spin, raises the chance of ruin from $0.600$ to $0.860$ and shortens the expected session from $600$ spins to $481$ . A useful way to see where the money goes: the expected loss is $q - p = 37^{-1}$ chips per spin, so over $481.204$ spins the player expects to lose $481.204/37 = 13.0055$ chips. Independently, the expected final holding is $0 \times 0.860110 + 50 \times 0.139890 = 6.9945$ chips, which is $20 - 13.0055$ . The two calculations agree to five figures, which checks the ruin probability and the mean duration against each other.

Application(Diffusion, polymers, and why chemistry is different in two dimensions)Original

Dimension decides whether diffusing particles find each other. Polya's trichotomy is the reason a diffusion-limited reaction behaves differently on a surface and in a bulk fluid. Two reactants performing independent lattice walks meet exactly when their difference walk, itself a random walk, visits the origin. In three dimensions the difference walk is transient, so a pair that fails to react on a first encounter is likely never to meet again, and the reaction rate is governed by fresh encounters: this is the classical Smoluchowski regime, with a rate constant that settles to a finite value. In two dimensions — a lipid membrane, an adsorbed layer, a thin film — the difference walk is recurrent, so a pair that has once come close will come close again and again, the same partners re-collide many times, and the effective rate constant acquires a logarithmic dependence on time and on system size. Membrane biochemistry and surface catalysis are quantitatively different from bulk chemistry for exactly this reason, and it is a statement about $\sum n^{-d/2}$ .

The same trichotomy controls exciton trapping and defect capture in crystals: a randomly walking excitation in a three-dimensional lattice has a positive chance of escaping a trap forever, while in a one- or two-dimensional conductor capture is certain and only the time matters.

Polymer physics uses the walk as the model of an ideal chain: the positions of the monomers of a flexible polymer in a theta solvent are a random walk with $n$ the degree of polymerisation, giving the end-to-end distance $\sqrt{\langle \lvert \mathbf{S}_n\rvert^2\rangle} = \sqrt{n}\,\ell$ for step length $\ell$ . The characteristic-function representation of the displacement after $n$ independent steps of arbitrary length, used above for the lattice Green function and worked out for a general step density by Riley, Hobson and Bence (Riley, Hobson & Bence (2006), §30.6, eq. (30.64), p. 1155), is precisely the tool used in the polymer literature to get the distribution of the end-to-end vector for a general segment distribution. What the excluded-volume interaction acts on is not recurrence but the number of self-contacts, and the two have different thresholds. The ideal chain in space is transient by part (d) — it comes back to its first monomer only with probability $0.3405$ — and yet it touches itself without limit, because there are many pairs of monomers and each pair has its own chance of coinciding. The expected number of coincident pairs is $\sum_{m \ge 1}(n + 1 - 2m)\,p_0(2m)$ , there being $n+1-2m$ pairs of monomers $2m$ apart along the chain, and since $\sum_m p_0(2m) = G - 1$ converges this grows linearly in $n$ with slope $G - 1 = 0.5164$ : it is $5.80$ at $n = 20$ , $39.9$ at $n = 100$ and $476$ at $n = 1000$ , that is $0.29$ , $0.40$ and $0.48$ contacts per monomer, climbing towards $0.5164$ . Whether forbidding those contacts changes the exponent is settled by a different count, Flory's: the order of $n^2$ monomer pairs each interact with strength $v$ while confined to the volume $R^d = n^{d/2}$ that the ideal chain occupies, so the dimensionless interaction parameter is

z \ \sim \ \frac{v\,n^2}{R^{\,d}} \ = \ v\,n^{(4-d)/2} ,

which grows with $n$ for $d < 4$ , is marginal at $d = 4$ and dies away above. So the swollen exponent $\nu$ in $\lvert\mathbf{S}_n\rvert \sim n^{\nu}$ replaces the ideal $\tfrac12$ for $d < 4$ ( $\nu = \tfrac34$ in two dimensions, $\nu \approx 0.588$ in three), while at $d = 4$ the ideal value survives up to logarithmic corrections. Both questions are governed by the same family of sums $\sum n^{-d/2}$ , but they weight it differently and their thresholds differ: $d = 2$ for return to the origin, $d = 4$ for excluded volume.

Gambler's ruin is the first-passage problem for an absorbing boundary, and reappears whenever a fluctuating quantity is watched against two thresholds: the survival of a small population against extinction and a carrying capacity, the sequential probability ratio test in statistics, the voltage of an integrate-and-fire neuron between reset and threshold, and the risk of ruin of an insurer. The formulas above are the discrete, two-barrier ancestors of the exit-time problems for diffusions solved in The Fokker-Planck equation, and the scaling limit that connects them is Brownian motion.

Definition

The Poisson process

Adapted

Add to PDF

Gives the infinitesimal and interarrival definitions of the Poisson process and shows they describe the same process, derives the Poisson law of the counts and the exponential law of the gaps, proves the conditional property that fixes the number of points and leaves their positions an independent sample, and proves that superposing independent Poisson streams and independently thinning one both return Poisson processes with the obvious intensities.

Prerequisites: Random variables and distributions, Expectation, variance and covariance, Joint distributions and transformations, Conditional expectation, Generating functions, Laws of large numbers

Note(Why this process and no other)Original

A Poisson process is the mathematics of events that happen at a steady average rate and otherwise take no notice of one another: clicks on a Geiger counter, photons at a faint detector, calls at an exchange, raindrops on a tile. Three demands pin it down completely. Events are counted, never un-counted. In a short interval the chance of one event is proportional to the length of the interval, and the chance of two or more is smaller than that by an order. And what happens after a given time is unaffected by what happened before it.

Those three demands are stated below as the definition. Everything else in this unit is a consequence: the number of events in a window has a Poisson distribution, the gaps between events are independent and exponential, two independent streams merged give a third stream of the same kind, and a stream from which events are deleted by independent coin tosses is again of the same kind. The last two facts are what make the process so useful in practice, because real detectors miss particles and real signals are mixtures of sources.

Definition(The Poisson process, infinitesimally)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.8, Definition (6.8.1), p. 277

Let $\lambda > 0$ . A Poisson process with intensity (or rate) $\lambda$ is a family $N = \{N(t) : t \ge 0\}$ of random variables taking values in $\{0, 1, 2, \dots\}$ such that:

$N(0) = 0$ , and $N(s) \le N(t)$ whenever $s < t$ ;
for every $n \ge 0$ , as $h \downarrow 0$ , $\mathbb{P}\bigl(N(t+h) = n+m \,\big|\, N(t) = n\bigr) = \begin{cases} \lambda h + o(h) , & m = 1 , \\[2pt] o(h) , & m > 1 , \\[2pt] 1 - \lambda h + o(h) , & m = 0 ; \end{cases}$ with the $o(h)$ terms uniform in the base time $t$ — which is what time-homogeneity is meant to encode, and what lets the proof below re-read the relation at a shifted base point rather than only pointwise in $t$ .
whenever $s < t$ , the count $N(t) - N(s)$ of arrivals falling in $(s,t]$ is independent of everything the process did on $[0,s]$ , the instants of those earlier arrivals included.

The three clauses are, in order: the process counts; it counts one at a time, at a rate that does not vary; and it has no memory. We call $N(t)$ the number of arrivals (or events, or counts) by time $t$ . The arrival times and interarrival times are

T_0 = 0 , \qquad T_n = \inf\{t : N(t) = n\} , \qquad U_n = T_{n+1} - T_n \quad (n \ge 0) ,

so that $T_n = U_0 + U_1 + \cdots + U_{n-1}$ and, in the other direction, $N(t) = \max\{n : T_n \le t\}$ . Each of the two descriptions therefore determines the other: to know the whole path of $N$ is to know the whole sequence $U_0, U_1, \dots$ , and conversely. The second description is the one used in simulation, and it is the subject of part 2 of the statement below.

Definition(Poisson processes on a region)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.13, Definitions (6.13.1) and (6.13.4) and the remark at (6.13.2), pp. 319-320

The time axis plays no special role, and for superposition and thinning it is cleaner to count points in sets. Let $d \ge 1$ and let $\lambda : \mathbb{R}^d \to \mathbb{R}$ be a non-negative measurable function whose integral

\Lambda(A) \;=\; \int_A \lambda(x)\,dx , \qquad A \in \mathcal{B}^d ,

over the Borel $\sigma$ -field $\mathcal{B}^d$ of $\mathbb{R}^d$ is finite for every bounded $A$ . A random countable subset $\Pi \subseteq \mathbb{R}^d$ is a non-homogeneous Poisson process with intensity function $\lambda$ if the counts $N(A) = \lvert \Pi \cap A\rvert$ satisfy:

$N(A)$ has the Poisson distribution with parameter $\Lambda(A)$ , for every $A \in \mathcal{B}^d$ , where a Poisson distribution of parameter $+\infty$ is to be read as the statement $\mathbb{P}(N(A) = \infty) = 1$ . That convention is the one Grimmett and Stirzaker print on p. 319 for their constant-intensity definition; we carry it over to a varying intensity function, where they do not repeat it. It is needed because the hypothesis on $\lambda$ makes $\Lambda$ finite on bounded sets only: an unbounded $A$ may well have $\Lambda(A) = \infty$ , and then $A$ holds infinitely many points almost surely;
$N(A_1), \dots, N(A_n)$ are independent whenever $A_1, \dots, A_n \in \mathcal{B}^d$ are disjoint.

$\Lambda$ is the mean measure of $\Pi$ . The constant-intensity case $\lambda(x) \equiv \lambda$ , in which $\Lambda(A) = \lambda\lvert A\rvert$ is $\lambda$ times the volume of $A$ , is the homogeneous Poisson process. Grimmett and Stirzaker record that in the constant-intensity case with $d = 1$ the present definition and the previous one describe the same object: if $\Pi$ is a Poisson process on $\mathbb{R}$ with constant intensity $\lambda$ then $M(t) = N([0,t])$ satisfies the infinitesimal definition, and conversely every process satisfying the infinitesimal definition arises this way from some $\Pi$ . For a genuinely varying intensity function no such comparison is available, since clause 2 of the infinitesimal definition fixes one rate $\lambda$ for all times.

Definition(The counts and the gaps)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.8, Theorem (6.8.2), p. 277, Theorem (6.8.10), p. 279, the converse remark and the calculation at the top of p. 280, and Theorem (6.8.11), p. 280

Let $N$ be a Poisson process with intensity $\lambda$ , with arrival times $T_n$ and interarrival times $U_n$ as above.

The counts are Poisson. For each $t \ge 0$ , $\mathbb{P}\bigl(N(t) = j\bigr) = \frac{(\lambda t)^{j}}{j!}\,e^{-\lambda t} , \qquad j = 0, 1, 2, \dots ,$ so $\mathbb{E}N(t) = \operatorname{var}N(t) = \lambda t$ .
The gaps are independent and exponential. The random variables $U_0, U_1, U_2, \dots$ are independent, each with the exponential distribution of parameter $\lambda$ .
The converse. If $U_0, U_1, \dots$ are independent, each exponential with parameter $\lambda$ , and $N$ is built from them by $T_n = \sum_{i<n} U_i$ and $N(t) = \max\{n : T_n \le t\}$ , then $N$ is a Poisson process with intensity $\lambda$ . Consequently $T_n$ has the gamma distribution $\Gamma(\lambda, n)$ , with density $\lambda(\lambda x)^{n-1}e^{-\lambda x}/(n-1)!$ on $x > 0$ , mean $n/\lambda$ and variance $n/\lambda^{2}$ .
Conditional uniformity. Conditional on $\{N(t) = n\}$ , the vector $(T_1, \dots, T_n)$ is distributed as the order statistics $V_{(1)} \le \cdots \le V_{(n)}$ of $n$ independent random variables $V_1, \dots, V_n$ , each uniform on $[0, t]$ .

Statements 1-3 say that the infinitesimal description and the exponential-gaps description are two presentations of one process; statement 4 says that, once the number of arrivals in a window is fixed, their positions carry no further structure.

Definition(The conditional property)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.13, Theorem (6.13.11) and the remark following it, p. 323

Take $\Pi$ as in the region-based definition, with intensity function $\lambda$ and mean measure $\Lambda$ , and fix a Borel set $A$ with $0 < \Lambda(A) < \infty$ . Write

Q(B) \;=\; \frac{\Lambda(B)}{\Lambda(A)} , \qquad B \subseteq A ,

for the mean measure restricted to $A$ and normalized to total mass $1$ ; its density at a point $x$ of $A$ is $\lambda(x)/\Lambda(A)$ . Given that $N(A) = n$ , the configuration $\Pi \cap A$ is then an independent sample of size $n$ from $Q$ . For a constant intensity $Q$ is uniform on $A$ , so in that case one conditions on the number of points and gets $n$ positions thrown independently and uniformly into $A$ .

Part 4 of the previous statement is the case $d = 1$ , $A = [0,t]$ , $\lambda$ constant, rewritten in terms of the arrival times; it is proved separately below, directly from parts 1 and 2, so that the one-dimensional theory does not lean on the region-based definition. The general form stated here is what the thinning proof needs, because there the intensity is allowed to vary with position.

Definition(Superposition and thinning)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.13, Superposition theorem (6.13.5) and Colouring theorem (6.13.14), pp. 320 and 324, and §6.8, Exercises (6.8.1) and (6.8.2), p. 287

Superposition. Take two independent Poisson processes $\Pi'$ and $\Pi''$ on $\mathbb{R}^d$ , with intensity functions $\lambda'$ and $\lambda''$ respectively. Their union $\Pi = \Pi' \cup \Pi''$ is then itself Poisson, with intensity function $\lambda' + \lambda''$ . The same argument covers countably many independent processes with mean measures $\Lambda_i$ , provided $\sum_i \Lambda_i(A) < \infty$ for every bounded $A$ .

Thinning (the colouring theorem). Now take a single Poisson process $\Pi$ on $\mathbb{R}^d$ with intensity function $\lambda$ . Give the point sitting at $x$ the colour green, with probability $\gamma(x)$ , or else scarlet, with probability $\sigma(x) = 1-\gamma(x)$ , and make that choice separately for each point, independently of every other choice. Writing $\Gamma$ for the green points and $\Sigma$ for the scarlet ones, the two sets $\Gamma$ and $\Sigma$ are independent Poisson processes, with intensity functions $\gamma(x)\lambda(x)$ and $\sigma(x)\lambda(x)$ .

On the time axis these read: two independent Poisson streams of rates $\lambda$ and $\mu$ merge into a Poisson stream of rate $\lambda + \mu$ ; and if each arrival of a rate- $\lambda$ stream is retained with probability $p$ , independently of the others, the retained arrivals form a Poisson stream of rate $\lambda p$ , independent of the stream of discarded ones. Grimmett and Stirzaker set these two one-dimensional statements as exercises (flies and wasps landing on a plate; green insects in the soup) and prove the general versions as theorems.

Idea of the proofAdaptedAdapted from Grimmett & Stirzaker (2020), §6.8, the two solution methods for Theorem (6.8.2) and the proof of Theorem (6.8.10), pp. 277-279, and §6.13, the route of the proofs of (6.13.5) and (6.13.14), pp. 320 and 324; Riley, Hobson & Bence (2006), §30.8.2, the binomial limit on p. 1175

Why the counts are Poisson. Condition on the state a short time earlier. Clause 2 of the definition says that in $(t, t+h]$ the process either stays put or moves up one, with probabilities $1 - \lambda h$ and $\lambda h$ to first order, whatever count it has reached by time $t$ — the clause is stated conditionally on that count, which is all this first argument needs of the hypotheses. That turns $p_j(t) = \mathbb{P}(N(t)=j)$ into a chain of differential equations, each coupling $p_j$ to $p_{j-1}$ only. Packing them into the generating function $G(s,t) = \mathbb{E}(s^{N(t)})$ collapses the chain to one first-order equation in $t$ , whose solution $e^{\lambda t(s-1)}$ is recognisably the generating function of a Poisson variable with mean $\lambda t$ . Riley, Hobson and Bence reach the same formula from the other direction, by cutting $[0,t]$ into $n$ slices each carrying one event with probability $p = \lambda t/n$ and letting $n \to \infty$ ; the binomial limit and the differential-equation route are two views of the same first-order bookkeeping.

Why the gaps are exponential. No arrival in $[0, t]$ is exactly the event $\{N(t) = 0\}$ , whose probability is $e^{-\lambda t}$ by statement 1; so the first gap is exponential. For the later gaps, the point is that the event $\{U_0 = t_0\}$ is decided by what happens up to time $t_0$ while $\{U_1 > t\}$ is decided by what happens after it, and clause 3 makes these independent. Induction then peels off one gap at a time.

Why superposition and thinning work. Both are statements that a certain count is Poisson and that counts in disjoint sets are independent, so in both cases there are exactly two things to check. For superposition, the sum of two independent Poisson variables is Poisson with the summed parameters, and independence across disjoint sets is inherited from the two ingredients. For thinning, condition on the total count in a set: by the conditional property the points are then an independent sample from the normalized mean measure, so their colours are independent coin tosses and the counts of the two colours are jointly binomial; multiplying the binomial probability by the Poisson probability of the total makes the joint distribution factorise, and a product of two Poisson factors is exactly what independence of the two colours means.

Proof(1: the counts are Poisson)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.8, proof of Theorem (6.8.2), Method B, pp. 277-278; Riley, Hobson & Bence (2006), §30.8.2, eqs. (30.101)-(30.102), p. 1176

Write $p_j(t) = \mathbb{P}(N(t) = j)$ .

Step 1: a differential-difference system. Split the event $\{N(t+h) = j\}$ according to the value of $N(t)$ , which produces the conditional probabilities $\mathbb{P}(N(t+h) = j \mid N(t) = i)$ . Clause 2 of the definition is stated in exactly that conditional form, so it evaluates them directly, with no appeal to clause 3: $1 - \lambda h + o(h)$ for $j = i$ , $\lambda h + o(h)$ for $j = i+1$ , and $o(h)$ for $j > i+1$ . Since $N$ is non-decreasing, no term with $i > j$ contributes. Hence, for $j \ge 1$ ,

p_j(t+h) = \lambda h\, p_{j-1}(t) + (1 - \lambda h)\,p_j(t) + o(h) ,

and for $j = 0$ the first term is absent, so $p_0(t+h) = (1 - \lambda h)p_0(t) + o(h)$ . Subtracting $p_j(t)$ , dividing by $h$ and letting $h \downarrow 0$ gives

p_j'(t) = \lambda p_{j-1}(t) - \lambda p_j(t) \ (j \ge 1) , \qquad p_0'(t) = -\lambda p_0(t) ,

with $p_j(0) = \delta_{j0}$ because $N(0) = 0$ .

The limit just taken is one-sided, so what it delivers is the right-hand derivative of $p_j$ , whereas step 2 differentiates in $t$ and solves an ordinary differential equation, which asks for the two-sided one. The gap closes because the error terms in clause 2 depend on the length $h$ of the window and not on where the window sits — the rate $\lambda$ is the same at all times, and this is the same reading of the clause that step 2 of the next proof uses. Granting that, the displayed relation holds with $t$ replaced by $t - h$ for any $t > 0$ and small $h > 0$ :

p_j(t) = \lambda h\, p_{j-1}(t-h) + (1-\lambda h)\,p_j(t-h) + o(h) .

Read as it stands it gives $p_j(t) - p_j(t-h) \to 0$ , so each $p_j$ is continuous from the left as well as from the right; divided by $h$ it then gives $\lambda p_{j-1}(t) - \lambda p_j(t)$ for the left-hand derivative too. So $p_j$ is differentiable on $(0,\infty)$ and right-differentiable at $0$ , which is what step 2 uses.

Step 2: collapse the system with a generating function. Put $G(s,t) = \sum_{j \ge 0} p_j(t)s^{j} = \mathbb{E}\bigl(s^{N(t)}\bigr)$ , which converges for $\lvert s\rvert \le 1$ . Fix $r < 1$ and work on the disc $\lvert s\rvert \le r$ . Multiply the $j$ th equation of step 1 by $s^{j}$ and add over $j \ge 0$ . Summing the left-hand sides gives $\partial G/\partial t$ , and this is the one step that needs an argument, since it differentiates a power series term by term: the equations themselves bound the derivatives, $\lvert p_j'(t)\rvert \le \lambda\bigl(p_{j-1}(t) + p_j(t)\bigr) \le 2\lambda$ for every $j$ and every $t$ , so on $\lvert s\rvert \le r$ the differentiated series is dominated term by term by the convergent numerical series $2\lambda\sum_j r^{j}$ , with a bound independent of $t$ . The differentiated series therefore converges uniformly in $t$ , which is exactly the hypothesis that licenses $\sum_j p_j'(t)s^{j} = \partial G/\partial t$ . On the right-hand sides, $\sum_{j\ge 1}\lambda p_{j-1}(t)s^{j}$ equals $\lambda s\,G(s,t)$ and $\sum_{j \ge 0}\lambda p_j(t)s^j$ equals $\lambda G(s,t)$ , so

\frac{\partial G}{\partial t} = \lambda(s-1)\,G , \qquad G(s,0) = 1 ,

the boundary value being $1$ because $N(0) = 0$ almost surely. Since $r < 1$ was arbitrary, this holds for every $\lvert s \rvert < 1$ .

Step 3: solve and read off the distribution. For fixed $s$ this is a linear first-order equation in $t$ with constant coefficient $\lambda(s-1)$ , so $G(s,t) = e^{\lambda t (s-1)}$ . Expanding the exponential,

G(s,t) = e^{-\lambda t}\sum_{j=0}^{\infty}\frac{(\lambda t)^{j}}{j!}\,s^{j} ,

and matching coefficients of $s^j$ against the definition of $G$ gives $p_j(t) = (\lambda t)^{j}e^{-\lambda t}/j!$ . The mean and variance of a Poisson variable with parameter $\lambda t$ are both $\lambda t$ (see Generating functions). $\square$

Grimmett and Stirzaker also record the slower route: solve $p_0' = -\lambda p_0$ to get $p_0(t) = e^{-\lambda t}$ , feed it into the equation for $p_1$ , and induct. Riley, Hobson and Bence derive the same system (their eq. (30.101)) from the switchboard picture and iterate it in that way.

Proof(2 and 3: the gaps, and the converse)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.8, proof of Theorem (6.8.10), p. 279, and the derivation at the top of p. 280; Riley, Hobson & Bence (2006), §30.9.3, eqs. (30.116) and (30.118), pp. 1190-1191

Step 1: the first gap. The first arrival occurs after time $t$ exactly when there is no arrival in $[0,t]$ , so by part 1,

\mathbb{P}(U_0 > t) = \mathbb{P}\bigl(N(t) = 0\bigr) = e^{-\lambda t} , \qquad t \ge 0 ,

which is the exponential law with parameter $\lambda$ .

Step 2: the second gap, given the first. Condition on $U_0 = t_0$ . By step 1 the conditioning event has probability zero, so what follows is a statement about conditional densities and the conditional probabilities computed from them, in the sense of Conditional expectation: we evaluate $\mathbb{P}(U_1 > t \mid U_0 = t_0)$ as a function of $t_0$ , and finding that it does not depend on $t_0$ is what independence of $U_0$ and $U_1$ means. The event $\{U_1 > t\}$ says that no arrival falls in $(t_0, t_0+t]$ . The conditioning event is decided by the arrival pattern in $[0, t_0]$ , and clause 3 of the definition makes the number of arrivals in $(t_0, t_0+t]$ independent of that pattern. Hence

\mathbb{P}(U_1 > t \mid U_0 = t_0) = \mathbb{P}\bigl(\text{no arrival in } (t_0, t_0+t]\bigr) = e^{-\lambda t} ,

the last equality because the right-hand probability does not depend on where the window sits: clause 2 refers only to the length $h$ of the window, so repeating step 1 on $(t_0, t_0+t]$ gives $e^{-\lambda t}$ . The answer does not involve $t_0$ , so $U_1$ is independent of $U_0$ and has the same law.

Step 3: induction. Suppose $U_0 = t_0, \dots, U_{n-1} = t_{n-1}$ — again a null event, and again read through the conditional densities of Conditional expectation — and set $T = t_0 + \cdots + t_{n-1}$ . Then $\{U_n > t\}$ is the event of no arrival in $(T, T+t]$ , which by the same two reasons has probability $e^{-\lambda t}$ regardless of the conditioning values. So $U_n$ is exponential with parameter $\lambda$ and independent of $U_0, \dots, U_{n-1}$ , which completes the induction. $\square$

Step 4: the converse. Suppose instead that we are given independent exponential gaps and define $N$ by $N(t) = \max\{n : T_n \le t\}$ with $T_n = \sum_{i<n}U_i$ . Then $T_n$ , a sum of $n$ independent exponential variables of parameter $\lambda$ , has the gamma distribution $\Gamma(\lambda, n)$ ; this is the law Riley, Hobson and Bence obtain as their eq. (30.118) by asking for $r-1$ events in $[0,x]$ and one more in $[x, x+dx]$ . The construction gives the identity

\{N(t) \ge j\} = \{T_j \le t\} ,

since having at least $j$ arrivals by time $t$ is precisely the $j$ th arrival occurring by then, whence

\mathbb{P}\bigl(N(t) = j\bigr) = \mathbb{P}(T_j \le t) - \mathbb{P}(T_{j+1}\le t) = \frac{(\lambda t)^{j}}{j!}e^{-\lambda t} ,

by integrating the two gamma densities by parts.

Step 5: the converse, clause 1. $T_1 = U_0 > 0$ almost surely, so no arrival has occurred at time $0$ and $N(0) = 0$ ; and $\{n : T_n \le t\}$ grows with $t$ , so $N$ is non-decreasing. That $N(t)$ is finite needs $T_n \to \infty$ : by the strong law of large numbers (Laws of large numbers) $T_n/n \to 1/\lambda$ almost surely, and $1/\lambda > 0$ , so $T_n \to \infty$ and only finitely many arrivals fall in $[0,t]$ .

Step 6: the converse, the restart at a fixed time. The one fact about the exponential law we need is that it forgets: for $u, v \ge 0$ ,

\mathbb{P}(U > u+v \mid U > u) = \frac{e^{-\lambda(u+v)}}{e^{-\lambda u}} = e^{-\lambda v} ,

so the unexpired part of a gap that has already lasted $u$ is again exponential with parameter $\lambda$ . Now fix $t \ge 0$ and condition on the entire arrival pattern on $[0,t]$ : on $N(t) = n$ and on $T_1 \in dt_1, \dots, T_n \in dt_n$ with $t_n \le t$ . As at steps 2 and 3, this conditioning event has probability zero, and the $dt_k$ are the usual shorthand: the statement being proved is one about the conditional law of the gaps that follow time $t$ given the vector $(T_1, \dots, T_n)$ and the event $\{N(t) = n\}$ , read through the conditional densities of Conditional expectation, and its content is that this conditional law does not depend on $n$ or on the values $t_1, \dots, t_n$ . Written in terms of the gaps, the conditioning event is

\{U_0 \in dt_1\} \cap \{U_1 \in d(t_2-t_1)\} \cap \cdots \cap \{U_{n-1} \in d(t_n - t_{n-1})\} \cap \{U_n > t - t_n\} .

It pins down $U_0, \dots, U_{n-1}$ , constrains $U_n$ only through the one-sided event $\{U_n > t-t_n\}$ , and says nothing whatever about $U_{n+1}, U_{n+2}, \dots$ . Since the gaps are independent, two things follow: $U_{n+1}, U_{n+2}, \dots$ are untouched by the conditioning and remain independent exponentials of parameter $\lambda$ ; and, by the lack-of-memory calculation just made, the residual

R = T_{n+1} - t = U_n - (t - t_n)

is exponential with parameter $\lambda$ and independent of them. So, conditionally on the pattern on $[0,t]$ , the sequence $R, U_{n+1}, U_{n+2}, \dots$ is a sequence of independent exponential variables of parameter $\lambda$ .

Step 7: the converse, clauses 3 and 2. The arrivals after time $t$ are at $t + R$ , $t + R + U_{n+1}$ , and so on, so the shifted process $s \mapsto N(t+s) - N(t)$ is built from the sequence $R, U_{n+1}, \dots$ by exactly the recipe that built $N$ from $U_0, U_1, \dots$ . By step 6 that sequence has the same law whatever the conditioning pattern was, so the shifted process has the law of $N$ and is independent of the arrival pattern on $[0,t]$ . That is clause 3, and it also shows that the law of an increment depends only on the length of its window. Clause 2 follows: averaging over the patterns compatible with $N(t) = n$ ,

\mathbb{P}\bigl(N(t+h) = n+m \mid N(t) = n\bigr) = \mathbb{P}\bigl(N(h) = m\bigr) = \frac{(\lambda h)^{m}}{m!}e^{-\lambda h} ,

by step 4. Expanding the exponential gives $1 - \lambda h + o(h)$ for $m = 0$ and $\lambda h + o(h)$ for $m = 1$ , while the whole tail $m \ge 2$ carries probability $1 - e^{-\lambda h} - \lambda h e^{-\lambda h} = \tfrac12(\lambda h)^{2} + O(h^{3}) = o(h)$ . All three lines of clause 2 hold, so $N$ is a Poisson process with intensity $\lambda$ . $\square$

Proof(4: conditional uniformity, and the conditional property on a region)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.8, proof of Theorem (6.8.11), pp. 280-281, and §6.13, proof of Theorem (6.13.11), the partition calculation (6.13.13), pp. 323-324

Part 4: the arrival times, given their number. Fix $n \ge 1$ and $t > 0$ .

Step 1: the joint density of the first $n$ arrival times. By part 2 the gaps $U_0, \dots, U_{n-1}$ are independent exponentials of parameter $\lambda$ , so their joint density is $\lambda^{n}e^{-\lambda(u_0+\cdots+u_{n-1})}$ on $(0,\infty)^{n}$ . Now apply the multivariate density-transformation rule of Joint distributions and transformations: under a smooth bijective change of variables, the new density is the old one composed with the inverse map and divided by the modulus of the Jacobian determinant. The map $(u_0, \dots, u_{n-1}) \mapsto (t_1, \dots, t_n)$ given by $t_k = u_0 + \cdots + u_{k-1}$ is linear and triangular with $1$ s on the diagonal, so its Jacobian determinant is $1$ ; it carries $(0,\infty)^n$ bijectively onto the ordered region $\{0 < t_1 < \cdots < t_n\}$ , and it turns the sum in the exponent into $t_n$ . Hence $T = (T_1, \dots, T_n)$ has density

f_T(t_1, \dots, t_n) = \lambda^{n}e^{-\lambda t_n} \qquad \text{on } \{0 < t_1 < \cdots < t_n\} .

Step 2: impose the condition $N(t) = n$ . Let $C$ be a Borel subset of $\{0 < t_1 < \cdots < t_n \le t\}$ . Given the values of $T$ , the event $\{N(t) = n\}$ says precisely that the next gap outlasts the time remaining, $U_n > t - t_n$ ; and $U_n$ is independent of $U_0, \dots, U_{n-1}$ , hence of $T$ , so that event has conditional probability $e^{-\lambda(t-t_n)}$ given $T = (t_1, \dots, t_n)$ . Integrating that conditional probability against the density of $T$ , which is how a probability is recovered from a conditional probability given a continuous vector (Conditional expectation),

\mathbb{P}\bigl(N(t) = n, \ T \in C\bigr) = \int_C e^{-\lambda(t - t_n)}\,\lambda^{n}e^{-\lambda t_n}\,dt_1\cdots dt_n = \lambda^{n}e^{-\lambda t}\,\lvert C\rvert ,

the two exponentials combining into one that no longer depends on the integration variables; $\lvert C\rvert$ is the volume of $C$ in $\mathbb{R}^{n}$ .

Step 3: divide by the probability of the condition. By part 1, $\mathbb{P}(N(t) = n) = (\lambda t)^{n}e^{-\lambda t}/n!$ , so

\mathbb{P}\bigl(T \in C \mid N(t) = n\bigr) = \frac{\lambda^{n}e^{-\lambda t}\lvert C\rvert}{(\lambda t)^{n}e^{-\lambda t}/n!} = \int_C \frac{n!}{t^{n}}\,dt_1 \cdots dt_n .

The conditional density of $T$ is the constant $n!/t^{n}$ on the ordered region, and $0$ off it. Note that $\lambda$ has disappeared.

Step 4: recognize the answer. Let $V_1, \dots, V_n$ be independent and uniform on $[0,t]$ ; their joint density is $t^{-n}$ on $[0,t]^{n}$ . The ordered vector $(V_{(1)}, \dots, V_{(n)})$ lives on the ordered region, and each point of that region receives the mass of the $n!$ rearrangements of its coordinates, each contributing $t^{-n}$ ; so the order statistics have joint density $n!/t^{n}$ there. That is the density found in step 3, which proves part 4. $\square$

The conditional property on a region. Let $0 < \Lambda(A) < \infty$ .

Step 1: count over a partition. Let $A_1, \dots, A_k$ be disjoint Borel sets with union $A$ , and let $n_1 + \cdots + n_k = n$ . The counts $N(A_i)$ are independent, each Poisson with parameter $\Lambda(A_i)$ , and their sum is $N(A)$ , which is Poisson with parameter $\Lambda(A) = \sum_i \Lambda(A_i)$ . Dividing the joint mass function by that of the sum,

\mathbb{P}\bigl(N(A_1) = n_1, \dots, N(A_k) = n_k \,\big|\, N(A) = n\bigr) = \frac{\prod_i \Lambda(A_i)^{n_i}e^{-\Lambda(A_i)}/n_i!}{\Lambda(A)^{n}e^{-\Lambda(A)}/n!} = \frac{n!}{n_1!\cdots n_k!}\prod_{i=1}^{k} Q(A_i)^{n_i} ,

the exponential factors cancelling because the $\Lambda(A_i)$ add up to $\Lambda(A)$ .

Step 2: read the right-hand side. It is the multinomial probability that $n$ points thrown into $A$ independently, each with law $Q$ , land $n_1$ in $A_1$ , ..., $n_k$ in $A_k$ .

Step 3: a classical input. What remains is to pass from the partition probabilities of step 1 to the law of the whole configuration, and generating a $\sigma$ -field is by itself not enough for that: two probability measures can agree on a generating family and differ on the $\sigma$ -field it generates. The standard repair is to work with a family closed under finite intersections.

Quoted classical input: Dynkin's $\pi$ - $\lambda$ theorem. Call a family $\mathcal{C}$ of events a $\pi$ -system if $E, F \in \mathcal{C}$ implies $E \cap F \in \mathcal{C}$ . Then (i) two probability measures that agree on a $\pi$ -system agree on the $\sigma$ -field it generates, and (ii) if $\mathcal{C}_1, \dots, \mathcal{C}_k$ are $\pi$ -systems, each containing $\Omega$ , with $\mathbb{P}(E_1 \cap \cdots \cap E_k) = \prod_i \mathbb{P}(E_i)$ for every choice of $E_i \in \mathcal{C}_i$ , then the $\sigma$ -fields $\sigma(\mathcal{C}_1), \dots, \sigma(\mathcal{C}_k)$ are independent. The condition $\Omega \in \mathcal{C}_i$ is not decoration. Without it the single full-intersection identity does not give independence once $k \ge 3$ : on $\Omega = \{1,\dots,8\}$ with the uniform measure, the one-element families $\mathcal{C}_1 = \{A\}$ , $\mathcal{C}_2 = \{B\}$ , $\mathcal{C}_3 = \{C\}$ with $A = \{1,2,3,4\}$ , $B = \{1,2,3,5\}$ , $C = \{1,5,6,7\}$ are $\pi$ -systems satisfying $\mathbb{P}(A \cap B \cap C) = \tfrac18 = \mathbb{P}(A)\mathbb{P}(B)\mathbb{P}(C)$ , yet $\mathbb{P}(A \cap B) = \tfrac38 \neq \tfrac14$ , so the three $\sigma$ -fields are not independent. With $\Omega$ available, the identity for any subcollection follows from the full one by padding the missing slots with $\Omega$ , and that is what the proof of independence needs. This theorem is proved in no unit of this handbook and is not in the sources followed here; it is used, in both parts, as a quoted classical result, in the way the $\sigma$ -field machinery of Sigma-algebras and measures is used generally.

Step 4: a $\pi$ -system of count events. For a Borel set $A$ write

\mathcal{C}(A) \;=\; \Bigl\{\, \{N(B_1) = m_1\} \cap \cdots \cap \{N(B_r) = m_r\} \;:\; r \ge 1,\ B_1,\dots,B_r \subseteq A \text{ Borel},\ m_j \ge 0 \,\Bigr\} .

(a) It is a $\pi$ -system, by construction: an intersection of two finite intersections of events $\{N(B) = m\}$ is again a finite intersection of such events, and so is again in $\mathcal{C}(A)$ — including when it is empty, as for $\{N(B)=0\}\cap\{N(B)=1\}$ . It also contains $\Omega$ , as the case $r = 1$ , $B_1 = \emptyset$ , $m_1 = 0$ , since $N(\emptyset) = 0$ identically; this is the hypothesis part (ii) above needs.

Arbitrary Borel sets are used here, and not the cells of one partition, for a definite reason. The cylinders of step 1 — events $\{N(B_1)=n_1,\dots,N(B_r)=n_r\}$ in which $B_1,\dots,B_r$ are the cells of a single finite partition of $A$ — are the natural family to try, and it is tempting to argue that they are closed under intersection because two partitions have a common refinement. That argument is wrong. Take two partitions $\{B_1,B_2\}$ and $\{B_1',B_2'\}$ of $A$ in general position, so that all four refinement cells $B_j \cap B_k'$ carry positive mean measure. Then

\{N(B_1)=1,\,N(B_2)=1\} \;\cap\; \{N(B_1')=1,\,N(B_2')=1\}

is the union of exactly two cylinders over the refinement $\{B_j\cap B_k'\}$ — one point in $B_1\cap B_1'$ and one in $B_2\cap B_2'$ , or one point in $B_1\cap B_2'$ and one in $B_2\cap B_1'$ — and both have positive probability, so the intersection is a union of two distinct refinement cylinders and is therefore not one of them. Passing to a common refinement makes the two generated $\sigma$ -fields sit inside a third; it does not turn an intersection of cylinders into a cylinder. Rather than try to repair that route we avoid it: $\mathcal{C}(A)$ is closed under intersection by its very definition, and by (b) it generates the same $\sigma$ -field.

(b) It generates the $\sigma$ -field of the configuration. That $\sigma$ -field is $\sigma\bigl(N(B) : B \subseteq A \text{ Borel}\bigr)$ , and each $N(B)$ takes values in $\{0,1,2,\dots\}\cup\{\infty\}$ , so it is generated by the events $\{N(B) = m\}$ alone, every one of which lies in $\mathcal{C}(A)$ .

(c) The two laws agree on $\mathcal{C}(A)$ . Fix $B_1,\dots,B_r \subseteq A$ and $m_1,\dots,m_r$ , and let $\mathcal{P}$ be the finite Borel partition of $A$ that they generate: the nonempty sets $B_1^{\pm} \cap \cdots \cap B_r^{\pm}$ , where $B_j^{+} = B_j$ and $B_j^{-} = A \setminus B_j$ . Each $B_j$ is the union of the cells of $\mathcal{P}$ it contains, so $N(B_j) = \sum_{D \subseteq B_j} N(D)$ . Under either law all $n$ points lie in $A$ , so

\bigcap_{j=1}^{r}\{N(B_j) = m_j\} \;=\; \biguplus\;\bigl\{ N(D) = n_D \text{ for all } D \in \mathcal{P} \bigr\} ,

a disjoint union over the count vectors $(n_D)_{D\in\mathcal{P}}$ with $\sum_D n_D = n$ and $\sum_{D \subseteq B_j} n_D = m_j$ for each $j$ . Every event in that union is of the step-1 form, and step 1 gives it the same probability under the conditioned process as under the independent $Q$ -sample; summing the finitely many terms gives the same probability for the left-hand side.

By (a), (b), (c) and part (i) of the quoted theorem, the conditioned process and the independent $Q$ -sample have the same law. $\square$

In the constant-intensity case $Q(B) = \lvert B\rvert/\lvert A\rvert$ is the uniform distribution on $A$ , which for $d=1$ and $A = [0,t]$ returns part 4 once the sample is written in increasing order.

Proof(Superposition and thinning)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.13, proof of the Superposition theorem (6.13.5), p. 320, and proof of the Colouring theorem (6.13.14), p. 324; Riley, Hobson & Bence (2006), p. 1179, additivity of independent Poisson variables

Superposition. Fix $A \in \mathcal{B}^d$ and put $N'(A) = \lvert\Pi'\cap A\rvert$ , $N''(A) = \lvert\Pi''\cap A\rvert$ . These are independent, and Poisson with parameters $\Lambda'(A)$ and $\Lambda''(A)$ .

Step 1: the count in one set. The sum of two independent Poisson variables is Poisson with the sum of the parameters — the generating functions multiply, $e^{\Lambda'(A)(s-1)}e^{\Lambda''(A)(s-1)} = e^{(\Lambda'(A)+\Lambda''(A))(s-1)}$ — so $S(A) = N'(A) + N''(A)$ is Poisson with parameter $\Lambda'(A)+\Lambda''(A) = \int_A(\lambda'+\lambda'')$ . This is the fact Riley, Hobson and Bence state for sums of independent Poisson variables.

Step 2: independence across disjoint sets. If $A_1, A_2, \dots$ are disjoint then the families $\{N'(A_i)\}$ and $\{N''(A_i)\}$ each consist of independent variables, and the two families are independent of each other; so the sums $S(A_i)$ are independent.

Step 3: the union really has $S(A)$ points. $\lvert\Pi\cap A\rvert$ equals $S(A)$ unless some point of $\Pi'$ coincides with a point of $\Pi''$ , in which case the union loses a point. Grimmett and Stirzaker show that almost surely no such coincidence occurs, by covering a bounded $A$ with the $O(2^{nd})$ dyadic boxes $B$ of side $2^{-n}$ that meet it and bounding the chance that one box holds a point of each. Since $\Pi'$ and $\Pi''$ are independent and $\mathbb{P}(N'(B)\ge1) \le \Lambda'(B)$ , that chance is at most $\Lambda'(B)\Lambda''(B)$ . It is the sum over the boxes that has to go to zero, and it does, because the sum factorises against its largest factor:

\sum_{B} \Lambda'(B)\Lambda''(B) \;\le\; \Bigl(\max_{B}\Lambda''(B)\Bigr)\sum_{B}\Lambda'(B) \;=\; \Bigl(\max_{B}\Lambda''(B)\Bigr)\,\Lambda'(\widetilde A) ,

where $\widetilde A$ is the union of the boxes. The second factor is a fixed finite number, while the first tends to $0$ as $n \to \infty$ by absolute continuity of the integral $\Lambda''(B) = \int_B \lambda''$ over sets of volume $2^{-nd} \to 0$ . So the chance of a coincidence anywhere in $A$ is zero. They describe this step as technical and set it aside on a first reading. $\square$

Thinning. Fix $A$ with $\Lambda(A) < \infty$ , and let $N_g$ , $N_s$ count the green and scarlet points of $\Pi$ in $A$ . If $\Lambda(A) = 0$ then $N(A)$ is Poisson with parameter $0$ , so $N(A) = 0$ and $N_g = N_s = 0$ almost surely and the conclusion below holds trivially, with both colour parameters equal to $0$ ; so assume $0 < \Lambda(A) < \infty$ , which is the range in which the conditional property was proved.

Step 1: condition on the total. Given $\lvert\Pi\cap A\rvert = n$ , the conditional property proved above says these $n$ points are distributed as $n$ independent points of $A$ , each drawn from the probability measure $Q(B) = \Lambda(B)/\Lambda(A)$ . This is the general form of the property, with the intensity allowed to vary over $A$ ; the uniform special case of part 4 would not be enough here.

Step 2: the colours are then independent coin tosses. Because the points are independent and each is coloured independently of the others, the conditional probability that a given one is green is the position-averaged colouring probability

\bar\gamma \;=\; \int_A \gamma(x)\,dQ \;=\; \frac{1}{\Lambda(A)}\int_A \gamma(x)\lambda(x)\,dx ,

a single number, to be kept apart from the function $\gamma(\cdot)$ of the statement; write $\bar\sigma = \int_A \sigma(x)\,dQ = 1 - \bar\gamma$ for its scarlet counterpart. The colours of the $n$ points are independent, so, conditionally on the total being $n$ ,

\mathbb{P}\bigl(N_g = g,\ N_s = s \,\big|\, N(A) = n\bigr) = \frac{n!}{g!\,s!}\,\bar\gamma^{\,g}\bar\sigma^{\,s} , \qquad g + s = n .

Step 3: multiply by the Poisson weight and watch it factorise. With $n = g+s$ ,

\mathbb{P}(N_g = g,\ N_s = s) = \frac{(g+s)!}{g!\,s!}\bar\gamma^{\,g}\bar\sigma^{\,s}\cdot\frac{\Lambda(A)^{g+s}e^{-\Lambda(A)}}{(g+s)!} = \frac{\bigl(\bar\gamma\Lambda(A)\bigr)^{g}e^{-\bar\gamma\Lambda(A)}}{g!}\cdot\frac{\bigl(\bar\sigma\Lambda(A)\bigr)^{s}e^{-\bar\sigma\Lambda(A)}}{s!} ,

using $\bar\gamma + \bar\sigma = 1$ to split $e^{-\Lambda(A)}$ . The joint mass function is a product of two Poisson mass functions, so $N_g$ and $N_s$ are independent and Poisson, with parameters

\bar\gamma\Lambda(A) = \int_A \gamma(x)\lambda(x)\,dx , \qquad \bar\sigma\Lambda(A) = \int_A \sigma(x)\lambda(x)\,dx ,

which are the mean measures of the intensity functions $\gamma(x)\lambda(x)$ and $\sigma(x)\lambda(x)$ claimed in the statement.

Step 4: disjoint sets. Let $A_1, \dots, A_k$ be disjoint, each of finite mean measure. The first thing to establish is that the whole configurations $\Pi \cap A_1, \dots, \Pi \cap A_k$ , and not merely the counts $N(A_1), \dots, N(A_k)$ , are independent across $i$ ; clause 2 of the region-based definition says only the latter. The passage from the one to the other is the $\pi$ -system argument set out at steps 3 and 4 of the proof of the conditional property above, and it is worth doing in full, because the obvious family of events is the wrong one.

Take for each $i$ the family $\mathcal{C}(A_i)$ of finite intersections $\{N(B_1) = m_1\} \cap \cdots \cap \{N(B_r) = m_r\}$ with $B_1, \dots, B_r$ arbitrary Borel subsets of $A_i$ . By (a) and (b) of step 4 of the conditional-property proof, with $A_i$ in place of $A$ , it is a $\pi$ -system and it generates $\sigma(\Pi \cap A_i)$ . The cylinders over the cells of one fixed finite partition of $A_i$ are not the family to use: as the two-partition example given there shows, an intersection of two of them is in general a union of refinement cylinders rather than a cylinder, so the common-refinement argument does not supply the $\pi$ -system hypothesis and the quoted theorem would be applied without it.

Now fix $E_i \in \mathcal{C}(A_i)$ for each $i$ , say $E_i = \{N(B_{i1}) = m_{i1}\} \cap \cdots \cap \{N(B_{ir_i}) = m_{ir_i}\}$ , and let $\mathcal{P}_i$ be the finite Borel partition of the union $B_{i1} \cup \cdots \cup B_{ir_i}$ generated by those sets: the nonempty cells $B_{i1}^{\pm} \cap \cdots \cap B_{ir_i}^{\pm}$ , where $B^{+} = B$ and $B^{-} = A_i \setminus B$ , with the all-minus cell discarded. It is the partition of the union of the $B_{ij}$ , and not of $A_i$ , that is wanted here. In (c) of the conditional-property proof the partition was taken over the whole of $A$ , and there the conditioning $\sum_D n_D = n$ held the leftover cell's count down; in the present step the total count over $A_i$ is not conditioned, so the leftover cell $A_i \setminus (B_{i1} \cup \cdots \cup B_{ir_i})$ would carry a count on which $E_i$ imposes no constraint whatever, and the decomposition below would run over infinitely many terms. Dropping that cell removes the difficulty. Each $B_{ij}$ is a union of cells of $\mathcal{P}_i$ , so $N(B_{ij}) = \sum_{D \in \mathcal{P}_i,\, D \subseteq B_{ij}} N(D)$ , and splitting $E_i$ over the values of the cell counts gives

E_i \;=\; \biguplus_{c} c , \qquad c \;=\; \bigl\{ N(D) = n_D \text{ for all } D \in \mathcal{P}_i \bigr\} ,

a disjoint union over the vectors $(n_D)_{D \in \mathcal{P}_i}$ of non-negative integers satisfying $\sum_{D \subseteq B_{ij}} n_D = m_{ij}$ for every $j \le r_i$ . Every cell of $\mathcal{P}_i$ lies in at least one $B_{ij}$ , by construction, so each $n_D$ is bounded by $\max_j m_{ij}$ and only finitely many vectors qualify: this union is genuinely finite. The cells of $\mathcal{P}_1, \dots, \mathcal{P}_k$ taken together are pairwise disjoint, because the $A_i$ are, so clause 2 makes the whole collection of counts $\{N(D) : D \in \mathcal{P}_1 \cup \cdots \cup \mathcal{P}_k\}$ independent. Hence the probability of an intersection $c_1 \cap \cdots \cap c_k$ of one cylinder from each $E_i$ is $\prod_i \mathbb{P}(c_i)$ , and expanding each $E_i$ and multiplying out,

\mathbb{P}\Bigl(\bigcap_{i=1}^{k} E_i\Bigr) \;=\; \sum_{c_1,\dots,c_k} \prod_{i=1}^{k}\mathbb{P}(c_i) \;=\; \prod_{i=1}^{k}\ \sum_{c_i}\mathbb{P}(c_i) \;=\; \prod_{i=1}^{k}\mathbb{P}(E_i) ,

where $c_i$ runs over the cylinders making up $E_i$ ; the middle step is the distributive law, legitimate because each $E_i$ is a union of finitely many cylinders, and the outer ones use that the cylinders making up a given $E_i$ are disjoint. So the $k$ $\pi$ -systems are independent, and part (ii) of the quoted $\pi$ - $\lambda$ theorem makes the $\sigma$ -fields $\sigma(\Pi \cap A_1), \dots, \sigma(\Pi \cap A_k)$ independent. That is the independence of the configurations.

Now the colour of each point is decided by a toss made independently of every other toss and of all the positions. The pair $\bigl(N_g(A_i), N_s(A_i)\bigr)$ is a function of the configuration $\Pi \cap A_i$ and of the tosses attached to its points, so the pairs are independent of one another. That is more than independence of the green counts among themselves and of the scarlet counts among themselves, and it is what the statement needs: combined with step 3, which makes $N_g(A_i)$ and $N_s(A_i)$ independent within each pair, it says that the whole family $\{N_g(A_i)\} \cup \{N_s(A_i)\}$ is independent.

Step 5: sets of infinite mean measure. Clause 1 of the region-based definition quantifies over every Borel set, while steps 1-4 kept the mean measure finite on the sets in play; the sets with $\Lambda(A) = \infty$ , on which the green or the scarlet mean measure may be finite or infinite, are reached from the bounded ones by exhaustion. Let $A$ be an arbitrary Borel set and put $A_m = A \cap \{x : \lvert x\rvert \le m\}$ , so $\Lambda(A_m) < \infty$ because $\Lambda$ is finite on bounded sets, and $A_m \uparrow A$ . Write $\Lambda_\gamma(B) = \int_B \gamma\lambda$ . Then $N_g(A_m) \uparrow N_g(A)$ pointwise, and $\Lambda_\gamma(A_m) \uparrow \Lambda_\gamma(A)$ by monotone convergence. If $\Lambda_\gamma(A) < \infty$ , the Poisson mass functions of parameter $\Lambda_\gamma(A_m)$ converge term by term to those of parameter $\Lambda_\gamma(A)$ , so the laws of the $N_g(A_m)$ converge to the Poisson law of that parameter; an almost-sure limit has the limiting law, so $N_g(A)$ is Poisson with parameter $\Lambda_\gamma(A)$ . If $\Lambda_\gamma(A) = \infty$ , then for each fixed $k$ ,

\mathbb{P}\bigl(N_g(A) \le k\bigr) \le \mathbb{P}\bigl(N_g(A_m) \le k\bigr) \longrightarrow 0 \qquad (m \to \infty) ,

so $N_g(A) = \infty$ almost surely, which is clause 1 read with the infinite-parameter convention. The same argument serves for $N_s$ . The independence statements of steps 3 and 4 survive the limit as well: disjoint Borel sets $A_1, \dots, A_k$ have disjoint bounded pieces $A_1 \cap \{\lvert x\rvert \le m\}, \dots, A_k \cap \{\lvert x\rvert \le m\}$ , for which those steps apply, and independence is preserved under almost-sure limits. Hence $\Gamma$ and $\Sigma$ each satisfy both clauses of the region-based definition, with the intensity functions found in step 3, and the two processes are independent of each other. $\square$

Remark(Where the hypotheses are used, and what fails without them)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.8, Definition (6.8.15) and Theorem (6.8.17), pp. 281-282, and Example (6.8.28), p. 286, and §6.13, Definition (6.13.1) with the convention for an infinite mean measure, p. 319, Definition (6.13.4) and the Superposition theorem (6.13.5), p. 320, and the Colouring theorem (6.13.14), p. 324

Clause 3 (independence of the future from the past) is what makes the gaps independent. It is used twice, both times in the proof of part 2: at step 2 and at step 3, where in each case the conditioning event is a statement about the past and the event being evaluated is a statement about the future. Part 1 does not need it at all — clause 2 is itself stated conditionally on the present count, so it evaluates the conditional probabilities thrown up by splitting $\{N(t+h) = j\}$ over the value of $N(t)$ without any further hypothesis. In the converse it is not available and has to be recovered, which is the work of steps 6 and 7. Its region-based counterpart is clause 2 of the definition on a region, independence of the counts in disjoint sets, and that is what carries the independence through the superposition and thinning proofs.

What fails without it is step 2 of the proof of part 2, and with it the conclusion. Clauses 1 and 2 speak only about the next instant given the present count; neither says anything about how the increment after time $t$ depends on when the earlier arrivals occurred. The counting processes with unit jumps and no simultaneous arrivals are a far larger family than the Poisson processes — every renewal process whose gap distribution is continuous and not exponential belongs to it — and what picks the Poisson process out of that family is exactly the demand that the past be forgotten. Such a renewal process violates clause 2 as well, since its instantaneous arrival rate depends on how long the current gap has already run; that is the lack-of-memory property of step 6 read backwards, as a characterization of the exponential law. So it is not a counterexample in the strict sense, and we offer none: clauses 1 and 2 speak only about the next instant conditional on the current count, never about the arrival times behind that count, and whether they can be met while clause 3 fails is a question this unit does not settle. What the renewal family does establish is the weaker point the proofs need — the conclusions above are not consequences of the unit-jump structure of clause 1 alone, and clause 3 is genuinely used at the two places just listed.

The constancy of $\lambda$ in clause 2 is what makes the gaps identically distributed. If the rate is allowed to depend on the number of arrivals so far, $\lambda \mapsto \lambda_n$ , the definition becomes that of a birth process, and the gaps are still independent but exponential with different parameters $\lambda_n$ . Then the counts need not be Poisson, and worse can happen: Grimmett and Stirzaker's Theorem (6.8.17) says that such a process, with all its birth rates strictly positive, stays finite for all time if and only if $\sum_n \lambda_n^{-1} = \infty$ , so for rapidly growing rates infinitely many arrivals occur in a bounded time. This is the point at which the Poisson process stops being the whole story and Continuous-time Markov chains take over.

The $o(h)$ in the second line of clause 2 forbids simultaneous arrivals, and it is what makes $N$ increase by one at a time. Consider a process that jumps by $2$ whenever a Poisson process of rate $\lambda$ has an arrival. It satisfies clause 1 and clause 3, and it satisfies the $m = 0$ line of clause 2, since the chance of no jump in $(t, t+h]$ is $e^{-\lambda h} = 1 - \lambda h + o(h)$ . It fails the other two lines, and in opposite directions: the chance of exactly one further arrival is $0$ , not $\lambda h + o(h)$ , and the chance of two is $\lambda h + o(h)$ , not $o(h)$ . The second failure is the one at issue here: the $o(h)$ line is by itself enough to rule the process out, whether or not the $m = 1$ line is also consulted. How much the clause buys can be read off Grimmett and Stirzaker's (Grimmett & Stirzaker (2020), §6.8, Example (6.8.28), p. 286) Example (6.8.28). Suppose all that is known of a process $M$ is this: it starts at the origin; its paths never decrease and are right-continuous; every value it takes is a whole number; the increments it makes over disjoint time intervals are independent of one another, and the law of an increment depends on the interval only through its length; and it never moves up by more than one at a time. Those demands already force $M$ to be a Poisson process. Unit jumps together with stationary independent increments is therefore a characterization, not merely a convenient hypothesis.

For the superposition theorem, independence of the two processes is essential, and the coincidence step is not decoration. Superposing a Poisson process with a translated copy of itself gives a point set whose counts in disjoint sets are strongly dependent. The coincidence step is the one place where the fact that the mean measures have no atoms — guaranteed here by $\Lambda(A) = \int_A \lambda$ , an integral of a function — is used; a "Poisson process" allowed to put positive mass on a single point would overlap its independent twin with positive probability, and the union would then have fewer points than the sum of the counts.

For thinning, the colours must be chosen independently of one another and of the positions, apart from the dependence on position allowed through $\gamma(x)$ . Step 2 fails otherwise. A detector whose dead time makes it more likely to miss a particle shortly after recording one does not produce a Poisson stream of recorded particles, however nearly Poisson the incident stream is; the recorded counts are under-dispersed relative to the Poisson law, which is why $\operatorname{var}/\mathbb{E}$ is the standard diagnostic.

Finiteness of $\Lambda$ on bounded sets keeps the counts in bounded regions finite, and that is all it does. It does not keep $\Lambda(A)$ finite for unbounded $A$ , which is why clause 1 of the region-based definition carries the convention Grimmett and Stirzaker state on p. 319 for a constant intensity: a Poisson distribution of infinite parameter means $\mathbb{P}(N(A) = \infty) = 1$ . The process is still defined on such an $A$ ; it simply has infinitely many points there. Every result above that conditions on $N(A)$ , the conditional property and the thinning proof included, assumes $\Lambda(A) < \infty$ explicitly for just this reason; the thinning conclusion is then carried to the sets of infinite mean measure by exhausting them with bounded ones, which is step 5 of that proof.

Method(Modelling a stream of events as a Poisson process)Original

Check the three clauses against the physics, not against the data. Events must be countable and non-simultaneous; the instantaneous rate must not depend on the past (no dead time, no refractory period, no queueing feedback); and the rate must be constant in time. If the rate varies with time but nothing else fails, keep going with the non-homogeneous version, replacing $\lambda t$ by $\Lambda(A) = \int_A \lambda(x)\,dx$ everywhere below.
Fix the intensity $\lambda$ in units of events per unit time. If several independent sources contribute, add their rates: that is superposition.
Apply any thinning. If each event is recorded with probability $p$ , independently of the others, replace $\lambda$ by $\lambda p$ ; the recorded stream is again Poisson, and is independent of the missed stream.
Answer count questions with the Poisson law. The number of events in a window of length $t$ is Poisson with parameter $\lambda t$ ; for a non-homogeneous process, with parameter $\Lambda$ of that window.
Answer waiting-time questions with the gamma law. The time to the $n$ th event is $\Gamma(\lambda, n)$ , with mean $n/\lambda$ and variance $n/\lambda^{2}$ ; for $n = 1$ this is the exponential law. Convert between the two languages with $\{N(t)\ge n\} = \{T_n \le t\}$ , which turns any statement about waiting times into one about counts.
Answer "where did they land" questions with conditional uniformity. Given the number of events in a window, their positions are an independent uniform sample of that window.
Test the model with the index of dispersion. For Poisson counts, $\operatorname{var}N(t) = \mathbb{E}N(t)$ . A sample variance well above the sample mean points to clustering (the rate itself fluctuates); well below it points to a dead time or some other inhibition.

Example(A two-source counting experiment with an imperfect detector)Original

Two weak radioactive sources sit in front of one detector. Source $A$ emits particles towards the detector at a mean rate of $1.2\ \mathrm{s}^{-1}$ , source $B$ at $0.8\ \mathrm{s}^{-1}$ , and the two emission streams are independent Poisson processes. The detector registers each incident particle with probability $p = 0.6$ , independently of everything else. Let $R$ be the process of registered counts.

Step 1: superpose. The incident particles form the union of two independent Poisson processes, so by the superposition theorem they form a Poisson process of rate

\lambda_{\text{inc}} = 1.2 + 0.8 = 2.0\ \mathrm{s}^{-1} .

Step 2: thin. Colour an incident particle "registered" with probability $0.6$ and "missed" with probability $0.4$ , independently of the others. By the colouring theorem $R$ is a Poisson process of rate

\lambda = p\,\lambda_{\text{inc}} = 0.6 \times 2.0 = 1.2\ \mathrm{s}^{-1} ,

independent of the process of missed particles.

Step 3: a count. The number of registered counts in a $3\ \mathrm{s}$ window is Poisson with parameter $\lambda t = 1.2 \times 3 = 3.6$ , so

\mathbb{P}\bigl(R(3) = 2\bigr) = \frac{3.6^{2}}{2!}\,e^{-3.6} = 6.48\,e^{-3.6} = 0.17706 .

Step 4: a waiting time. The time $T_5$ to the fifth registered count is $\Gamma(1.2, 5)$ , with

\mathbb{E}T_5 = \frac{5}{1.2} = 4.1667\ \mathrm{s} , \qquad \operatorname{sd}(T_5) = \frac{\sqrt5}{1.2} = 1.8634\ \mathrm{s} .

Step 5: a waiting time answered as a count. What is the chance that we have to wait more than $4\ \mathrm{s}$ for the third registered count? Use $\{T_3 > 4\} = \{R(4) \le 2\}$ . Here $R(4)$ is Poisson with parameter $1.2\times 4 = 4.8$ , so

\mathbb{P}(T_3 > 4) = e^{-4.8}\Bigl(1 + 4.8 + \tfrac{4.8^{2}}{2}\Bigr) = 17.32\,e^{-4.8} = 0.142539 .

That is the number to check: $\mathbb{P}(T_3 > 4\ \mathrm{s}) = 0.1425$ to four decimal places.

Step 6: a consistency check on the model. The missed particles form an independent Poisson process of rate $0.4 \times 2.0 = 0.8\ \mathrm{s}^{-1}$ , and superposing the registered and missed streams must return the incident stream: $1.2 + 0.8 = 2.0\ \mathrm{s}^{-1}$ , as it does. Also, had we thinned first and superposed afterwards — thinning each source separately to rates $0.72$ and $0.48$ — we would have reached the same $\lambda = 1.2\ \mathrm{s}^{-1}$ . The two operations commute, as they must, since both are read off from the intensity function alone.

Application(Shot noise, dead time, and counting statistics)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.13, the shot-noise sum over the points of a Poisson process, p. 326

Every counting instrument in the physical sciences is calibrated against this process.

Photon and particle counting. A photomultiplier viewing a steady faint source, a Geiger counter beside a long-lived isotope, a cosmic-ray telescope: in each case the arrival stream is Poisson to good accuracy, and the characteristic consequence is that a measurement of $n$ counts carries an uncertainty of about $\sqrt n$ , because $\operatorname{var}N(t) = \mathbb{E}N(t) = \lambda t$ . The familiar rule that one must quadruple the integration time to halve the fractional error, $\sqrt{\lambda t}/(\lambda t) = (\lambda t)^{-1/2}$ , is this identity and nothing more.

Detector efficiency is thinning. A detector of quantum efficiency $p$ turns an incident Poisson stream of rate $\lambda$ into a recorded Poisson stream of rate $p\lambda$ . This is why photon-counting statistics are not degraded in form by an inefficient detector — only the rate falls, and the recorded counts remain Poisson. It is also why departures from Poisson statistics in recorded data are informative: they cannot have been manufactured by a memoryless loss mechanism, so they come either from the source (bunching, clustering, a varying rate) or from a detector with memory.

Dead time is the standard failure of the model. After each count, a real counter is insensitive for a short interval $\tau$ . The retained events are then not an independent thinning, since whether a particle is recorded depends on how recently the last one was; the recorded process is a renewal process whose gaps are $\tau$ plus an exponential variable. Those gaps are less variable than exponential ones of the same mean, and the counts they produce are under-dispersed: $\operatorname{var}R(t)$ falls below $\mathbb{E}R(t)$ . Measuring an index of dispersion below $1$ is the usual way to detect and then correct for dead time.

Shot noise in a circuit. If each arrival at rate $\lambda$ contributes a pulse $r(t - T_i)$ of fixed shape to a current, the total current is $I(t) = \sum_i r(t - T_i)$ , a sum over the points of a Poisson process; Grimmett and Stirzaker call the one-dimensional version of such sums shot noise. The superposition theorem says that two independent sources of such pulses give a current of the same type with the rates added, which is why shot noise from independent channels adds in power rather than in amplitude.

Spatial versions. The region-based definition is used directly for the positions of impurities in a crystal, of stars in a survey field, or of raindrops on a surface. There the thinning theorem is the statement that a magnitude-limited sample from a Poisson field of stars is still a Poisson field, with intensity reduced by the detection probability at each position — the usual justification for modelling a flux-limited catalogue as a Poisson process with a position-dependent intensity.

Definition

Continuous-time Markov chains

Adapted

Add to PDF

Defines a continuous-time Markov chain through its transition semigroup and its generator, shows that the chain is an exponential holding time in each state followed by a jump governed by a discrete chain, derives the Kolmogorov forward and backward equations and the solution by a matrix exponential, and gives the stationary condition and the birth-death criterion.

Prerequisites: Discrete-time Markov chains, The Poisson process, Linear systems and the matrix exponential, Functions of matrices and the matrix exponential

Note(What changes when time becomes continuous)Original

A discrete-time chain is run by one matrix: the transition matrix $P$ , with $P^{n}$ giving the $n$ -step probabilities (Discrete-time Markov chains). In continuous time there is no smallest step, so there is no single matrix to iterate. There is instead a whole family $\{P_t : t \ge 0\}$ , one matrix per elapsed time, and the family multiplies: $P_{s+t} = P_sP_t$ . A family of matrices that multiplies like that is governed by its derivative at $t = 0$ , and that derivative is the generator $G$ . Everything in this unit is an unpacking of the slogan $P_t = e^{tG}$ and of the places where it needs care.

The generator also has a direct probabilistic reading, and it is worth having both pictures from the start. Off the diagonal, $g_{ij}$ is the rate at which the chain jumps from $i$ to $j$ . On the diagonal, $-g_{ii}$ is the total rate of leaving $i$ , which is the parameter of the exponential time the chain waits there. So a continuous-time chain is a discrete-time chain (where it goes) run on an exponential clock (when it goes): a map of where it can go, together with a speed for each place.

Definition(The chain, its semigroup and its generator)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.9, Definitions (6.9.1), (6.9.2), (6.9.3) and the display (6.9.7), pp. 288-290

Let $S$ be a countable set, which we take to be a set of integers. A family $X = \{X(t) : t \ge 0\}$ of random variables with values in $S$ satisfies the Markov property if

\mathbb{P}\bigl(X(t_n) = j \mid X(t_1) = i_1, \dots, X(t_{n-1}) = i_{n-1}\bigr) = \mathbb{P}\bigl(X(t_n) = j \mid X(t_{n-1}) = i_{n-1}\bigr)

for all states and all increasing sequences $t_1 < t_2 < \cdots < t_n$ of times. A sample path of $X$ is right-continuous if for each $t$ it is constant on some interval $[t, t+\epsilon)$ ; equivalently, since $S$ is discrete, the paths are step functions. We call $X$ a continuous-time Markov chain when both hold: its paths are right-continuous, and the Markov property is satisfied.

Its transition probabilities are $p_{ij}(s,t) = \mathbb{P}(X(t) = j \mid X(s) = i)$ for $s \le t$ , and $X$ is homogeneous if these depend only on $t - s$ ; we assume homogeneity throughout and write $p_{ij}(t)$ , and $P_t$ for the matrix $\bigl(p_{ij}(t)\bigr)$ . The family $\{P_t : t \ge 0\}$ is the transition semigroup of the chain.

Suppose $X(0) = i$ , and write $U_0 = \inf\{t : X(t) \ne i\}$ for the instant at which the chain first moves away from $i$ ; right-continuity forces $U_0 > 0$ . Write $g_i \in [0,\infty)$ for the parameter of the exponential distribution of $U_0$ (that it is exponential is part 3 of the statement below), and define the jump matrix $Y = (y_{ij})$ by $y_{ij} = \delta_{ij}$ if $g_i = 0$ , and by

y_{ij} = \mathbb{P}_i\bigl(X(U_0) = j\bigr) \quad\text{if } g_i > 0 .

The generator of the chain is the matrix $G = (g_{ij} : i,j \in S)$ with

g_{ij} = \begin{cases} g_i\,y_{ij} , & j \ne i , \\[2pt] -g_i , & j = i .\end{cases}

Since $Y$ has row sums $1$ , $G$ has row sums $0$ : $\sum_{j} g_{ij} = 0$ for every $i$ , equivalently $\sum_{j \ne i} g_{ij} = g_i$ . More generally, any matrix with non-negative off-diagonal entries and row sums $0$ is called a generator on $S$ . The off-diagonal entries are rates, in the sense that for $j \ne i$

\mathbb{P}\bigl(X(t+h) = j \mid X(t) = i\bigr) = g_{ij}h + o(h) ,

together with $p_{ii}(h) = 1 + g_{ii}h + o(h)$ on the diagonal. Grimmett and Stirzaker state this as their equation (6.9.9), set the finite case as an exercise, and warn that it must be handled carefully when $S$ is infinite; it is part 1 of the Kolmogorov statement below and is proved there, since everything in that statement rests on it.

Let $T_m$ be the instant of the $m$ th move ( $T_0 = 0$ ), put $U_m = T_{m+1} - T_m$ , and set $T_\infty = \lim_mT_m$ . We call the $U_m$ the holding times. The successive values taken by $X$ form a discrete-time chain $Y = \{Y_n\}$ , governed by the matrix $Y$ ; this is the jump chain. Following the source, the single letter $Y$ carries both meanings — the matrix $Y = (y_{ij})$ and the chain $Y = \{Y_n\}$ it runs — and we keep that economy here; the subscripts tell the two apart. The chain explodes from $i$ if $\mathbb{P}_i(T_\infty < \infty) > 0$ , and one says explosion occurs if it explodes from some state. When explosion is possible one adjoins a state $\infty$ to $S$ and sets $X(t) = \infty$ for $t \ge T_\infty$ ; the process so defined is the minimal process, because it runs as a chain in $S$ for the shortest possible time.

Definition(The architecture of a continuous-time chain)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.9, Theorem (6.9.4), p. 289, Theorem (6.9.5) and Theorem (6.9.6), p. 290, Theorem (6.9.11), p. 291, the construction (6.9.12) and Proposition (6.9.14), p. 292, and Theorem (6.9.17), p. 293

Let $X$ be a homogeneous continuous-time Markov chain with state space $S$ .

The Markov property extends. Fix $t > 0$ . Conditioning on $\{X(t) = j\}$ together with any event decided by the chain strictly before $t$ assigns any event decided strictly after $t$ the same probability as conditioning on $\{X(t)=j\}$ alone: this is the extended Markov property. In its strong form, for a stopping time $T$ and conditional on $\{T < T_\infty\} \cap \{X(T) = i\}$ , the shifted process $X^{*}(u) = X(T+u)$ is again a continuous-time Markov chain with the transition probabilities of $X$ , started at $i$ and independent of $\{X(s) : s < T\}$ . Grimmett and Stirzaker state both and omit the proof, referring to Norris; we do the same, and flag below the three places where the extended form rather than the bare Markov property is what is being used.
The semigroup property. $P_0 = I$ ; each $P_t$ has non-negative entries and row sums $1$ ; and the Chapman-Kolmogorov equations hold, $P_{s+t} = P_sP_t , \qquad s, t \ge 0 .$ (For the minimal process built from a generator this still holds, because the chain can reach the adjoined state $\infty$ but cannot come back.)
Holding times are exponential. Started at $X(0) = i$ , the variable $U_0$ defined above is exponentially distributed, with some parameter $g_i \in [0,\infty)$ , and is a stopping time. The value $g_i = 0$ means $\mathbb{P}_i(U_0 = \infty) = 1$ , that is, $i$ is absorbing; the value $g_i = \infty$ is impossible, because right-continuity forces $\mathbb{P}_i(U_0 > 0) = 1$ .
Destination and duration are independent, and the chain restarts. If $g_i > 0$ then $X(U_0)$ is independent of $U_0$ , and, conditional on $X(U_0) = j$ , the shifted process $X^{*}(s) = X(U_0 + s)$ is again a chain of the same kind, carrying the transition probabilities of $X$ , begun at $j$ , and independent both of $U_0$ and of the starting state $i$ .
The chain is exactly this description. Conversely, given the two ingredients of the construction — a transition matrix $Y$ with $y_{ii} = 0$ , together with non-negative constants $g_i$ , which is the same data as a generator matrix $G$ in the sense defined above — the recipe "sit in $Y_0$ for an exponential time of parameter $g_{Y_0}$ , jump to $Y_1$ , sit for an exponential time of parameter $g_{Y_1}$ , and so on" defines a process, which is a Markov chain provided it does not exhaust $S$ in finite time.
Sufficient conditions for non-explosion. The chain does not explode from $i$ if any one of the following holds: $S$ is finite; $\sup_j g_j < \infty$ ; or $i$ is recurrent for the jump chain $Y$ . None of these is necessary: a pure birth process with rates $\lambda_n = n$ , started from a state $i \ge 1$ , has an infinite state space, unbounded rates and a transient jump chain, yet it never explodes, because $\sum_{n \ge 1} n^{-1}$ diverges. (Started from $0$ it is a different matter, since $\lambda_0 = 0$ makes $0$ absorbing, and then condition (c) holds there anyway.)

The chain is irreducible if for every pair $i,j$ there is some $t > 0$ with $p_{ij}(t) > 0$ ; in that case $p_{ij}(t) > 0$ for all $t > 0$ , a dichotomy with no analogue in discrete time, where periodicity makes some $n$ -step probabilities vanish.

Definition(The Kolmogorov equations)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.9, equation (6.9.9), p. 291 and Exercise (6.9.11), p. 296, and §6.10, equations (6.10.1)-(6.10.5), pp. 297-298, and Theorem (6.10.7), p. 299

Let $X$ be a chain as above, with generator $G$ and semigroup $\{P_t\}$ , and let $P_t'$ denote the matrix whose entries are the derivatives $p_{ij}'(t)$ .

The short-time expansion. If $S$ is finite then, as $h \downarrow 0$ , $p_{ij}(h) = g_{ij}h + o(h) \ \ (j \ne i) , \qquad p_{ii}(h) = 1 + g_{ii}h + o(h) ,$ the error terms being uniform in $i$ and $j$ ; in matrix form $P_h = I + hG + o(h)$ , so that $G = P_0'$ is the derivative of the semigroup at the origin. This is the statement that makes the generator a matrix of rates, and everything below depends on it.
Finite state space. If $S$ is finite then $\{P_t\}$ satisfies the forward equation $P_t' = P_t G , \qquad t \ge 0 ,$ and the backward equation $P_t' = G P_t , \qquad t \ge 0 ,$ both subject to $P_0 = I$ . Each is a finite system of first-order linear differential equations with constant coefficients, so each has a unique solution, namely $P_t \;=\; \sum_{n=0}^{\infty}\frac{t^{n}}{n!}G^{n} \;=\; e^{tG} .$
Countable state space. If $S$ is countable and $X$ is the minimal chain with generator $G$ , then, among all non-negative solutions of the backward system with $P_0 = I$ , the family $\{P_t\}$ is the smallest; and the same is true for the forward system. ("Minimal" means $p_{ij}(t) \le \pi_{ij}(t)$ for any other non-negative solution $\{\Pi_t\}$ with the same boundary condition.) Any solution carrying that boundary value has row sums at most $1$ , that is $\sum_j p_{ij}(t)\le 1$ . When explosion does not occur the row sums equal $1$ , and then $\{P_t\}$ is the unique solution of either system.

Both equations come from the same two-step decomposition $P_{t+h} = P_tP_h = P_hP_t$ ; the forward equation conditions on the state at time $t$ and looks at the last short interval, the backward equation conditions on the state at time $h$ and looks at the first.

Definition(Stationary distributions, the limit theorem, and birth-death chains)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.9, the definitions (6.9.21) of recurrence and transience and of positive recurrence, with Theorem (6.9.22), p. 295, §6.10, Definition (6.10.13), equation (6.10.14), Theorem (6.10.15) and Theorem (6.10.22), pp. 301-304, and §6.11, conditions (2), equation (3), conditions (4) and (6) and Theorem (6.11.7), pp. 307-308

Evolution of the distribution. If $X(0)$ has distribution $\mu^{(0)}$ , written as a row vector, then $X(t)$ has distribution $\mu^{(t)} = \mu^{(0)}P_t$ .

Recurrence. Grimmett and Stirzaker call a state $i$ recurrent for $X$ when $\mathbb{P}_i(\{t : X(t) = i\}\text{ is unbounded}) = 1$ , and transient when that probability is $0$ ; when $g_i > 0$ , the state $i$ is recurrent for $X$ precisely when the jump chain is recurrent at $i$ , and transient for $X$ precisely when the jump chain is transient there. A recurrent $i$ is positive recurrent if either $S = \{i\}$ or the first return time $R_i = \inf\{t > U_0 : X(t) = i\}$ has finite mean $m_i = \mathbb{E}_i(R_i)$ .

Stationarity. For an irreducible, non-explosive $X$ , a measure $\pi$ is a stationary measure if $\pi = \pi P_t$ for all $t \ge 0$ , and a stationary distribution if moreover $\sum_j \pi_j = 1$ . Now let $X$ be irreducible with $\lvert S\rvert \ge 2$ .

Should one state be positive recurrent, a stationary distribution $\pi$ exists and is unique, it is the only distribution satisfying $\pi G = 0$ , and every state is then positive recurrent too.
Suppose instead that $X$ does not explode and that $\pi G = 0$ for some distribution $\pi$ . Then every state is positive recurrent, $\pi$ is stationary, and $\pi_k = 1/(m_kg_k)$ .

So the infinite system $\pi = \pi P_t$ , one equation for every $t$ , reduces to the single linear system $\pi G = 0$ .

The limit theorem. Assume once more that $X$ is irreducible and does not explode. Where a stationary distribution $\pi$ exists, it is unique and $p_{ij}(t)\to\pi_j$ as $t\to\infty$ for every pair $i,j$ . Where none exists, $p_{ij}(t)\to 0$ for every pair. No aperiodicity hypothesis appears, in contrast with the discrete-time theorem of Convergence of Markov chains.

Birth-death chains. A birth-death process is a chain on $\{0,1,2,\dots\}$ whose generator has $g_{n,n+1} = \lambda_n$ , $g_{n,n-1} = \mu_n$ , $g_{nn} = -(\lambda_n+\mu_n)$ and all other off-diagonal entries zero, where $\lambda_i \ge 0$ , $\mu_i \ge 0$ and $\mu_0 = 0$ . Assume

\lambda_i > 0 \ (i \ge 0), \qquad \mu_i > 0 \ (i \ge 1), \qquad \mu_0 = 0 ,

so that the chain is irreducible. Then:

the measure $x_n = \frac{\lambda_0\lambda_1\cdots\lambda_{n-1}}{\mu_1\mu_2\cdots\mu_n}\,x_0 , \qquad n \ge 1 ,$ satisfies $xG = 0$ ;
a distribution $\pi$ with $\pi G = 0$ exists if and only if $\sum_{n=0}^{\infty}\frac{\lambda_0\lambda_1\cdots\lambda_{n-1}}{\mu_1\mu_2\cdots\mu_n} < \infty$ (the $n = 0$ term being read as $1$ ), and then $x_0$ is the reciprocal of that sum;
that $\pi$ is a stationary distribution of $X$ if, in addition, $\sum_{n=0}^{\infty}\frac{\lambda_0\lambda_1\cdots\lambda_{n-1}}{\mu_1\mu_2\cdots\mu_n}\,(\lambda_n + \mu_n) < \infty .$

The extra condition 3 is not idle: Grimmett and Stirzaker exhibit rates for which $\pi G = 0$ has a normalisable solution while the chain is transient, and therefore explosive.

Idea of the proofOriginal

Why there is a generator at all. The semigroup property $P_{s+t} = P_sP_t$ with $P_0 = I$ is the matrix version of the functional equation $f(s+t) = f(s)f(t)$ , whose solutions are exponentials. Differentiating the semigroup property at $t=0$ produces a single matrix $G = P_0'$ , and the same differentiation at a general $t$ turns the family into a differential equation driven by $G$ . Which side $G$ appears on depends on which of the two factorisations $P_{t+h} = P_tP_h$ or $P_{t+h}=P_hP_t$ one differentiates, and that is the only difference between the forward and the backward equations.

Why the holding times are exponential. Starting afresh in state $i$ at time $s$ , given that the chain has not yet left $i$ , is the same as starting in $i$ at time $0$ . The conditioning event $\{U_0 > s\}$ is a statement about the whole of $[0,s]$ and not just about $X(s)$ , so what is used here is the extended Markov property of part 1 of the architecture statement, together with homogeneity; the same remark applies to part 4 of that statement, which needs the strong form at the stopping time $U_0$ . So $Q(t) = \mathbb{P}_i(U_0 > t)$ satisfies $Q(s+t) = Q(s)Q(t)$ , the same functional equation again, now for a scalar; with $Q(0) = 1$ , and its only right-continuous solutions are $Q(t) = e^{-g_it}$ for some $g_i \in [0,\infty)$ .

Why $\pi G = 0$ replaces $\pi P = \pi$ . Stationarity should mean that the distribution does not move, so $\frac{d}{dt}\bigl(\pi P_t\bigr) = 0$ . By the backward equation this derivative is $\pi GP_t$ , and since $P_0 = I$ it vanishes for every $t$ exactly when $\pi G = 0$ — one linear system in place of a continuum of them.

Why the birth-death measure is a product. A birth-death generator is tridiagonal, so the equation $xG = 0$ links only consecutive components and can be unwound one step at a time; what comes out is the detailed-balance relation $x_n\lambda_n = x_{n+1}\mu_{n+1}$ — the mean rate of upward crossings of the gap between $n$ and $n+1$ must equal the rate of downward crossings — and iterating that relation gives the product formula.

Where the care is needed. Every step above interchanges a limit with a sum. For finite $S$ the sums are finite and nothing can go wrong. For infinite $S$ the chain may run through infinitely many states in finite time, and then the forward and backward systems may have many solutions; the transition probabilities of the chain are the smallest of them.

Proof(The forward and backward equations, and the matrix exponential)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.9, equation (6.9.9) and Exercise (6.9.11), pp. 291 and 296, and §6.10, the derivation of (6.10.1)-(6.10.5), pp. 297-298; Riley, Hobson & Bence (2006), §30.8.2, eq. (30.101), p. 1176

Take $S$ finite, and fix $i, j \in S$ . Write $g = \max_{k \in S}g_k$ , which is finite because $S$ is finite.

Step 1: the short-time expansion, part 1 of the statement. Start the chain at $i$ . By parts 3, 4 and 5 of the architecture statement the chain sits at $i$ for an exponential time $U_0$ of parameter $g_i$ , moves to a state $j \ne i$ with probability $y_{ij}$ , and then restarts there; so all three probabilities below can be computed from the holding times and the jump chain.

(a) Two moves in a short time are unlikely, and this is where finiteness enters first. Conditional on the jump chain, $U_0$ and $U_1$ are independent exponential variables with parameters $g_i$ and $g_{Y_1}$ , both at most $g$ ; since $\mathbb{P}(U > t) = e^{-g_{\bullet}t} \ge e^{-gt}$ for each of them, the pair is stochastically larger than a pair of independent exponential variables of parameter $g$ . The sum of the latter has the gamma law $\Gamma(g,2)$ (The Poisson process, part 3), so

\mathbb{P}_i(T_2 \le h) \;\le\; 1 - e^{-gh} - ghe^{-gh} \;=\; \tfrac12 g^{2}h^{2} + O(h^{3}) \;=\; O(h^{2}) ,

with a bound depending only on $g$ , hence uniform over the states.

(b) The diagonal entry. Either the chain has not moved by time $h$ , which has probability $e^{-g_ih}$ , or it has moved at least once. If it has moved and is nevertheless at $i$ at time $h$ , it has moved at least twice: the first move cannot end at $i$ , because right-continuity makes the path a step function and $U_0$ is the first instant at which the value differs from $i$ . So $p_{ii}(h)$ differs from $e^{-g_ih}$ by at most $\mathbb{P}_i(T_2 \le h) = O(h^{2})$ , and since $e^{-g_ih} = 1 - g_ih + O(h^{2})$ and $g_{ii} = -g_i$ ,

p_{ii}(h) = 1 + g_{ii}h + O(h^{2}) .

(c) The off-diagonal entries. Fix $j \ne i$ and split $\{X(h) = j\}$ by the number of moves made by time $h$ . The part with two or more moves contributes at most $O(h^{2})$ by (a). The part with exactly one move requires that move to occur at some instant $s \le h$ , to land on $j$ , and to be followed by no further move in the remaining time $h - s$ ; integrating over $s$ , and using the independence of destination from duration and the restart at $j$ ,

\mathbb{P}_i\bigl(X(h) = j,\ T_1 \le h < T_2\bigr) = y_{ij}\int_0^{h}g_ie^{-g_is}\,e^{-g_j(h-s)}\,ds .

The two exponentials together lie between $e^{-gh}$ and $1$ , so the integral lies between $g_ihe^{-gh}$ and $g_ih$ , and therefore equals $g_ih + O(h^{2})$ . Multiplying by $y_{ij}$ and recalling $g_{ij} = g_iy_{ij}$ ,

p_{ij}(h) = g_{ij}h + O(h^{2}) .

All the error bounds are controlled by the single constant $g$ , so they are uniform in $i$ and $j$ , which is what the next two steps need. $\square$

Step 2: condition on where the chain is at time $t$ . By the Chapman-Kolmogorov equations,

p_{ij}(t+h) = \sum_{k \in S} p_{ik}(t)\,p_{kj}(h) .

Step 3: expand the short-time factor. Step 1 gives $p_{kj}(h) = g_{kj}h + o(h)$ for $k \ne j$ and $p_{jj}(h) = 1 + g_{jj}h + o(h)$ . Substituting,

p_{ij}(t+h) = p_{ij}(t)\bigl(1 + g_{jj}h\bigr) + h\sum_{k \ne j}p_{ik}(t)g_{kj} + o(h) = p_{ij}(t) + h\sum_{k\in S}p_{ik}(t)g_{kj} + o(h) .

The $o(h)$ terms may be collected into one because there are finitely many of them, each bounded by the uniform error of step 1. This is the second place finiteness is used, and it is exactly the step Grimmett and Stirzaker flag as needing justification when $S$ is infinite, where the sum is infinite and a limit is being taken through it.

Step 4: divide and let $h \downarrow 0$ . We obtain

p_{ij}'(t) = \sum_{k\in S}p_{ik}(t)g_{kj} = (P_tG)_{ij} ,

which is the forward equation $P_t' = P_tG$ . Conditioning instead on the state at time $h$ , that is writing $p_{ij}(t+h) = \sum_k p_{ik}(h)p_{kj}(t)$ and expanding the first factor in the same way, gives the backward equation $P_t' = GP_t$ . In both cases $P_0 = I$ because $p_{ij}(0) = \delta_{ij}$ .

Step 5: solve. Each system is a linear constant-coefficient system of $\lvert S\rvert^{2}$ scalar equations, so by the existence and uniqueness theorem for linear systems it has exactly one solution with the given value at $t=0$ . Define $E_t = \sum_{n\ge 0}t^{n}G^{n}/n!$ , the matrix exponential of $tG$ ; the series converges for every $t$ because $\lVert G^{n}\rVert \le \lVert G\rVert^{n}$ in any submultiplicative norm. Differentiating term by term, which is legitimate because the differentiated series converges uniformly on compact $t$ -intervals by the same bound,

\frac{d}{dt}E_t = \sum_{n\ge 1}\frac{t^{n-1}}{(n-1)!}G^{n} = E_tG = GE_t ,

the last equality because $G$ commutes with every power of itself. Also $E_0 = I$ . So $E_t$ solves both systems with the right boundary value, and by uniqueness

P_t = e^{tG} . \qquad\square

Step 6: what survives when $S$ is infinite. Steps 1 and 3 break, and with them uniqueness. Grimmett and Stirzaker's Theorem (6.10.7) salvages the following: the transition semigroup of the minimal chain is still a non-negative solution of both systems, and it is the smallest one. Their proof of the backward half starts from the decomposition

p_{ij}(t) = \delta_{ij}e^{-g_it} + \sum_{k \ne i}\int_0^{t}g_ie^{-g_is}y_{ik}\,p_{kj}(t-s)\,ds ,

obtained by splitting according to whether the first jump happens after $t$ or at some time $s \le t$ ; substituting $u = t-s$ , exchanging sum and integral by Fubini's theorem, and differentiating recovers $p_{ij}'(t) = \sum_k g_{ik}p_{kj}(t)$ . Minimality then follows by showing, by induction on $n$ , that any non-negative solution dominates $\mathbb{P}_i(X(t)=j,\ T_n > t)$ , and letting $n \to \infty$ .

Proof(The stationary measure of a birth-death chain)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.11, the derivation of (3) from (2) and the footnote on detailed balance, p. 307, with the discussion of condition (6) and Example (6.11.8), p. 308

Let $G$ be a birth-death generator satisfying $\lambda_i > 0$ for $i \ge 0$ , $\mu_i > 0$ for $i \ge 1$ and $\mu_0 = 0$ , and let $x$ be a measure with $xG = 0$ . Reading off the columns of $G$ , the equation $xG = 0$ is

-\lambda_0x_0 + \mu_1x_1 = 0 , \qquad \lambda_{n-1}x_{n-1} - (\lambda_n+\mu_n)x_n + \mu_{n+1}x_{n+1} = 0 \quad (n \ge 1) .

Step 1: recognise a telescoping quantity. Define $D_n = \lambda_nx_n - \mu_{n+1}x_{n+1}$ for $n \ge 0$ . The $n$ th equation above says precisely $D_{n-1} = D_n$ for $n \ge 1$ , and the $n = 0$ equation says $D_0 = 0$ . Hence $D_n = 0$ for every $n$ :

\lambda_nx_n = \mu_{n+1}x_{n+1} , \qquad n \ge 0 .

This is the detailed-balance relation; it holds here not by assumption but because the generator is tridiagonal, so that the only way probability can cross the gap between $\{0,\dots,n\}$ and $\{n+1,n+2,\dots\}$ is by a single step.

Step 2: unwind it. Since $\mu_{n+1} > 0$ we may divide, and induction on $n$ gives

x_n = \frac{\lambda_{n-1}}{\mu_n}x_{n-1} = \frac{\lambda_0\lambda_1\cdots\lambda_{n-1}}{\mu_1\mu_2\cdots\mu_n}\,x_0 , \qquad n \ge 1 .

Conversely this $x$ satisfies both displayed equations, as substituting back shows, so it is the general solution of $xG = 0$ up to the scalar $x_0$ .

Step 3: normalise. $x$ can be scaled to a distribution exactly when $\sum_n x_n < \infty$ , which with $x_0 = 1$ is the summability condition of the statement; and then $x_0$ is the reciprocal of the sum. $\square$

Why the second condition is needed for stationarity. Having a normalisable solution of $\pi G = 0$ is not the same as having a stationary distribution, because part 2 of the stationarity statement also requires the chain to be non-explosive. The route Grimmett and Stirzaker take is to pass to the jump chain: the measure $\nu_n = x_ng_n = x_n(\lambda_n+\mu_n)$ is stationary for the jump chain $Y$ , and if it is summable then $Y$ is recurrent, whence $X$ is recurrent and so non-explosive. That summability is the second condition. Their Example (6.11.8), with $\lambda_i = \lambda g_i$ and $\mu_i = \mu g_i$ where $\lambda + \mu = 1$ , taking $g_n = 2^{n}$ and $\lambda/\mu \in (1,2)$ , satisfies the first condition but not the second; there the jump chain is transient, so $X$ is transient, and since a normalisable $\pi$ with $\pi G = 0$ exists, $X$ must be explosive.

Remark(Where the hypotheses are used, and the master equation)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.9, Theorem (6.9.6) and the footnote on (6.9.9), pp. 290-291, Theorem (6.9.17), p. 293, and Example (6.9.16), p. 293, and §6.10, equations (6.10.1)-(6.10.3) and Theorem (6.10.22), pp. 297-298 and 304

Right-continuity is used in three places. It makes the jump times $T_m$ well defined, so that the holding times, the jump chain and the construction of part 5 of the architecture statement all make sense. It forces $\mathbb{P}_i(U_0 > 0) = 1$ and hence $g_i < \infty$ in part 3 of that statement, which we take from Grimmett and Stirzaker rather than prove: if the paths were not right-continuous the chain could leave $i$ immediately, $U_0$ would be $0$ almost surely, and $g_i$ would be $\infty$ . And it is what makes the path a step function, so that the first move out of $i$ cannot end at $i$ — the fact used in step 1(b) of the proof to show that leaving $i$ and returning within a window of length $h$ costs $O(h^{2})$ . Grimmett and Stirzaker adopt right-continuity precisely to avoid the measure-theoretic complications of the general theory, and note that without it one must look elsewhere for a treatment.

Finiteness of $S$ is used twice in the proof. In step 1 it bounds the jump rates, $g = \max_kg_k < \infty$ , which is what makes two moves in a window of length $h$ cost $O(h^{2})$ and makes the errors in the short-time expansion uniform over the states. In step 3 it makes the sum over $k$ finite, so that finitely many uniform $o(h)$ terms can be collected into one and the limit taken through the sum. Everything downstream of those steps — uniqueness of the solution, $P_t = e^{tG}$ , the equivalence of the forward and backward systems — is conditional on it. For infinite $S$ the two systems can have different solution sets, and the chain supplies the minimal one.

Explosion is the obstruction, and it is real. For a birth process with rates $\lambda_n$ , Grimmett and Stirzaker's criterion is that the chain is non-explosive if and only if $\sum_n\lambda_n^{-1} = \infty$ . So birth rates $\lambda_n = (1+n)^{2}$ give a population that becomes infinite in finite time with positive probability, and then $\sum_j p_{ij}(t) < 1$ , the deficit being the probability of having escaped to infinity by time $t$ . The three sufficient conditions in part 6 of the architecture statement are the ones worth checking in practice, and the first — $S$ finite — covers most applications.

Irreducibility and non-explosion are both needed in the limit theorem, and nothing else is. Aperiodicity, which the discrete-time theorem cannot do without, has no analogue here: by the dichotomy noted above, an irreducible continuous-time chain has $p_{ij}(t) > 0$ for every $t > 0$ , so there is no parity obstruction for a limit to overcome. A chain that merely satisfies $\pi G = 0$ with $\pi$ a distribution, but explodes, is not covered, and the birth-death example above shows the conclusion can then fail.

The master equation. Physicists usually meet the forward equation not as a matrix identity but as a balance of gains against losses. Let $\mu_j(t) = \mathbb{P}(X(t) = j)$ . Multiplying $P_t' = P_tG$ on the left by the initial distribution gives $\frac{d}{dt}\mu(t) = \mu(t)G$ , that is $\mu_j'(t) = \sum_i \mu_i(t)g_{ij}$ . Splitting off the diagonal term and using $g_{jj} = -\sum_{k \ne j}g_{jk}$ ,

\frac{d\mu_j}{dt} \;=\; \underbrace{\sum_{i \ne j}\mu_i(t)\,g_{ij}}_{\text{flow into } j} \;-\; \underbrace{\mu_j(t)\sum_{k \ne j}g_{jk}}_{\text{flow out of } j} .

This gain-loss form is what is called the master equation of the process; the rates $g_{ij}$ are then usually written $W(i \to j)$ or $W_{ji}$ . Stationarity, $\pi G = 0$ , says that the two flows balance for every state when summed over the other states; detailed balance, $\pi_ig_{ij} = \pi_jg_{ji}$ for each pair, is the strictly stronger statement that they balance pair by pair, and for a birth-death chain the two coincide, because the only pairs joined by a direct transition are neighbours and step 1 of the proof above showed that the neighbour balances then hold automatically. Note also that the diagonal of $G$ carries no information beyond the off-diagonal entries: it is fixed by the row-sum condition, which is why a master equation is specified by its rates alone.

Method(Setting up and solving a continuous-time chain)Original

List the states and the transitions. For each ordered pair $i \ne j$ of states between which a single event can carry the system, write down the rate $g_{ij}$ at which that event occurs when the system is in $i$ . Rates, not probabilities: $g_{ij}$ has dimensions of inverse time.
Assemble the generator. Put the $g_{ij}$ off the diagonal, zeros where no direct transition exists, and fill the diagonal with $g_{ii} = -\sum_{j\ne i}g_{ij}$ so that every row sums to zero. Check that every off-diagonal entry is non-negative; a sign error here is the commonest mistake.
Read off the holding times and the jump chain if you want the sample paths. The chain stays in $i$ for an exponential time of mean $1/g_i$ with $g_i = -g_{ii}$ , and then moves to $j$ with probability $y_{ij} = g_{ij}/g_i$ . This is also the recipe for simulating the chain.
Check for explosion if $S$ is infinite. If $S$ is finite, or $\sup_ig_i < \infty$ , or the jump chain is recurrent, there is nothing to do. Otherwise the transition probabilities are the minimal solution of the Kolmogorov equations and may not sum to $1$ .
For the equilibrium behaviour, solve the linear system $\pi G = 0$ together with $\sum_j\pi_j = 1$ . For a birth-death chain do not solve it as a system: use the product formula, then normalise. If the chain is irreducible and non-explosive and the system has a solution, that solution is the unique stationary distribution and $p_{ij}(t)\to\pi_j$ .
For the time dependence with $S$ finite, exponentiate. If $G$ has a full set of eigenvectors, diagonalise $G = B\Lambda B^{-1}$ ; then $P_t = Be^{t\Lambda}B^{-1}$ . A generator need not be diagonalisable, and when it is defective there are two standard ways on: put $G$ in Jordan form, which replaces $e^{t\theta}$ by $e^{t\theta}$ times a polynomial in $t$ of degree one less than the block size, or invert the Laplace transform, $P_t = \mathcal{L}^{-1}\bigl[(\sigma I - G)^{-1}\bigr]$ , which needs no eigenvectors at all (The matrix exponential). Either way the spectrum is the same, and it always contains the eigenvalue $0$ , since $G$ has row sums $0$ and therefore the right eigenvector $(1,1,\dots,1)^{\mathsf T}$ . For a finite irreducible generator the eigenvalue $0$ is simple and all the other eigenvalues have negative real parts, and the slowest of them, $-\lvert\operatorname{Re}\theta\rvert$ with $\theta$ the non-zero eigenvalue closest to the imaginary axis, sets the relaxation time $1/\lvert\operatorname{Re}\theta\rvert$ towards equilibrium.
For an infinite $S$ with structure, use generating functions. Multiply the master equation for $\mu_n(t)$ by $s^{n}$ and sum, turning the infinite system into one partial differential equation for $G(s,t) = \sum_n\mu_n(t)s^{n}$ , exactly as for the Poisson process in The Poisson process.

Example(Two instruments, two technicians)Original

A laboratory runs two identical instruments. Each working instrument fails at rate $\varphi = \tfrac12$ per hour; each failed instrument is repaired at rate $r = \tfrac32$ per hour, and there are two technicians, so repairs proceed in parallel. All four clocks are independent and exponential. Let $X(t) \in \{0,1,2\}$ be the number of failed instruments.

Step 1: the generator. From state $0$ (both working) either instrument may fail, so the total failure rate is $2\varphi = 1$ . From state $1$ the one working instrument fails at rate $\varphi = \tfrac12$ and the one failed instrument is repaired at rate $r = \tfrac32$ . From state $2$ both repairs are under way, at total rate $2r = 3$ . This is a birth-death chain with $\lambda_0 = 1$ , $\lambda_1 = \tfrac12$ , $\mu_1 = \tfrac32$ , $\mu_2 = 3$ , so

G = \begin{pmatrix} -1 & 1 & 0 \\[2pt] \tfrac32 & -2 & \tfrac12 \\[2pt] 0 & 3 & -3 \end{pmatrix} ,

whose rows sum to $0$ and whose off-diagonal entries are non-negative.

Step 2: the stationary distribution, by the product formula. The birth-death recursion $x_n = (\lambda_{n-1}/\mu_n)x_{n-1}$ terminates here after two steps, since the state space is finite:

x_0 = 1 , \qquad x_1 = \frac{\lambda_0}{\mu_1} = \frac{1}{3/2} = \frac23 , \qquad x_2 = \frac{\lambda_0\lambda_1}{\mu_1\mu_2} = \frac{1\cdot\frac12}{\frac32\cdot 3} = \frac19 .

The total is $1 + \tfrac23 + \tfrac19 = \tfrac{16}{9}$ , so

\pi = \Bigl(\tfrac{9}{16},\ \tfrac{6}{16},\ \tfrac{1}{16}\Bigr) = (0.5625,\ 0.375,\ 0.0625) .

Check: $\pi G = \bigl(-\tfrac{9}{16}+\tfrac32\cdot\tfrac{6}{16},\ \tfrac{9}{16}-2\cdot\tfrac{6}{16}+3\cdot\tfrac{1}{16},\ \tfrac12\cdot\tfrac{6}{16}-3\cdot\tfrac{1}{16}\bigr) = (0,0,0)$ .

Step 3: the time dependence. The eigenvalues of $G$ are $0$ , $-2$ and $-4$ (their sum is the trace, $-6$ ). Diagonalising and exponentiating,

p_{00}(t) = \frac{9}{16} + \frac38 e^{-2t} + \frac{1}{16}e^{-4t} , \qquad p_{02}(t) = \frac{1}{16}\bigl(1 - e^{-2t}\bigr)^{2} .

At $t = 1$ hour, starting from both instruments working,

p_{00}(1) = 0.5625 + 0.375\,e^{-2} + 0.0625\,e^{-4} = 0.6143954586 ,

and $p_{02}(1) = \tfrac{1}{16}(1-e^{-2})^{2} = 0.04672782$ . As $t\to\infty$ these tend to $\tfrac{9}{16}$ and $\tfrac{1}{16}$ , as the limit theorem requires. The slowest non-zero eigenvalue is $-2$ , so the relaxation time is $\tfrac12$ hour.

Step 4: an independent check. Because the two instruments never interact — each has its own clock and its own technician — $X(t)$ is the sum of two independent two-state chains. A single instrument has generator $\bigl(\begin{smallmatrix}-1/2 & 1/2\\ 3/2 & -3/2\end{smallmatrix}\bigr)$ , hence stays up with probability $\tfrac34 + \tfrac14e^{-2t}$ and has stationary up-probability $\tfrac34$ . Therefore $X(t)$ should be binomial, and indeed

\Bigl(\tfrac34 + \tfrac14e^{-2t}\Bigr)^{2} = \frac{9}{16} + \frac38e^{-2t} + \frac{1}{16}e^{-4t} = p_{00}(t) ,

while $\pi$ is the binomial distribution with two trials and success probability $\tfrac14$ : $(\tfrac9{16},\tfrac6{16},\tfrac1{16})$ . The eigenvalues $0,-2,-4$ are $0$ and $-2$ from one factor added to $0$ and $-2$ from the other.

Step 5: a number the laboratory cares about. In equilibrium both instruments are down a fraction $\pi_2 = \tfrac{1}{16}$ of the time, that is $24/16 = 1.5$ hours per day. The mean number down is $2\times\tfrac14 = 0.5$ . Finally, the mean time to reach the state "both down" starting from "both up" solves the hitting-time system $1 + \sum_k g_{jk}m_k = 0$ for $j \notin\{2\}$ , with $m_2 = 0$ ; this is the system of Grimmett and Stirzaker's Exercise (6.9.6) (Grimmett & Stirzaker (2020), Exercise (6.9.6), p. 296), whose minimal non-negative solution gives the mean hitting times. Here it reads

1 - m_0 + m_1 = 0 , \qquad 1 + \tfrac32m_0 - 2m_1 = 0 ,

whose solution is $m_1 = 5$ and $m_0 = 6$ . Starting from a fully working laboratory, the expected wait until both instruments are simultaneously down is exactly $6$ hours.

Application(Master equations in physics, chemistry and queueing)AdaptedAdapted from Grimmett & Stirzaker (2020), §6.9, Exercise (6.9.3), p. 296, and §6.11, Example (6.11.12) and Theorem (6.11.14), pp. 308-309, Example (6.11.15) with Theorem (6.11.16), pp. 309-310, and Corollary (6.11.18), p. 310

Rate equations for populations of states. Whenever a physical system occupies one of a discrete set of configurations and hops between them at rates fixed by the configuration alone, its probability distribution obeys a master equation, and the system is a continuous-time Markov chain. Nuclear decay chains, the populations of atomic levels under incoherent pumping and spontaneous emission, the open and closed states of an ion channel, and the conformations of a macromolecule are all of this kind. In each case the physics supplies the generator, the equilibrium populations follow from $\pi G = 0$ , and the relaxation rates are the non-zero eigenvalues of $G$ . Where the rates satisfy detailed balance with respect to a Boltzmann distribution, $g_{ij}e^{-\beta E_i} = g_{ji}e^{-\beta E_j}$ , that Boltzmann distribution is automatically stationary — which is how a kinetic model is made thermodynamically consistent.

Chemical kinetics with small numbers of molecules. In a well-stirred vessel containing few copies of each species, the state is the vector of molecule counts and each reaction channel contributes a transition at a rate proportional to the number of available reactant combinations. The resulting master equation is the chemical master equation; the deterministic rate law is its large-volume limit. A single species that is produced at constant rate $\lambda$ and destroyed independently per molecule at rate $\mu$ is exactly a birth-death chain with $\lambda_n = \lambda$ and $\mu_n = n\mu$ , and Grimmett and Stirzaker's Theorem (6.11.14) gives the equilibrium: the molecule number is Poisson with mean $\rho = \lambda/\mu$ . The fluctuation is therefore $\sqrt\rho$ , so the relative noise in a species held at a copy number of a hundred is about ten per cent — the basic reason that stochastic effects matter in gene expression.

Population growth and extinction. The simple birth-death chain with $\lambda_n = n\lambda$ and $\mu_n = n\mu$ models a population in which individuals reproduce and die independently. Its generating function is computed in Grimmett and Stirzaker's Theorem (6.11.16); differentiating that generating function gives the mean population $\mathbb{E}_I(X(t)) = Ie^{(\lambda-\mu)t}$ , which they display just after the theorem rather than inside it, and their Corollary (6.11.18) gives an extinction probability tending to $1$ if $\lambda \le \mu$ and to $(\mu/\lambda)^{I}$ if $\lambda > \mu$ — the continuous-time counterpart of the extinction criterion for branching processes.

Queues. Jobs arriving at a processor in the manner of a Poisson process of rate $\lambda$ , each requiring an independent exponential service time of rate $\mu$ , make the number in the system a birth-death chain with $\lambda_n = \lambda$ for all $n$ and $\mu_n = \mu$ for $n \ge 1$ . The product formula gives $x_n = (\lambda/\mu)^{n}$ , which is summable exactly when $\lambda < \mu$ ; Grimmett and Stirzaker set this as Exercise (6.9.3), with the conclusion that a stationary distribution exists if and only if $\lambda < \mu$ , in which case $\pi_n = (1-\rho)\rho^{n}$ with $\rho = \lambda/\mu$ . The mean number in the system is then $\rho/(1-\rho)$ , which blows up as the arrival rate approaches the service rate: the familiar fact that a server loaded to ninety per cent of capacity carries, on average, nine jobs rather than one.

Diffusions and stochastic calculus

Theorem · advanced

Stationary processes and spectral densities

Adapted

Add to PDF

Defines strong and weak stationarity and the autocovariance function, proves that an autocorrelation function is non-negative definite and therefore, when continuous at the origin, is the characteristic function of a distribution on the frequency axis, and identifies that distribution's density as the spectral density of the process.

Prerequisites: Expectation, variance and covariance, Characteristic functions, Fourier series, The Fourier transform, Fourier inversion

Note(What a spectral density is for)Original

A stationary process has no preferred instant: the statistics you measure do not depend on when you start the clock. That single invariance has a surprisingly rigid consequence. The correlation between the process now and the process a lag $t$ later is a function $c(t)$ of the lag alone, and $c$ cannot be an arbitrary function: it must be the Fourier transform of a positive measure.

That measure is the spectral distribution of the process, and where it has a density, that density is the spectral density. It answers the question an experimentalist actually asks of a noisy signal — how much of the variance sits at each frequency — and it is the object a spectrum analyser estimates. The theorem below, usually named after Wiener and Khinchin, is the statement that the question always has an answer and that the answer is determined by the autocorrelation function alone.

Three warnings before we start. First, there is a second, purely deterministic theorem of the same name, about the Fourier transform of the correlation of two square-integrable functions; it is stated in the remark below so that the two are not confused. Second, several incompatible normalisations of the spectral density are in use; the conversion table is also in the remark. Third, the second name is a transliteration from Russian and is spelt differently by our two sources — Khinchin by Grimmett and Stirzaker, Kinchin by Riley, Hobson and Bence, Khintchine elsewhere again. We write it as the source we are reporting writes it, so both spellings appear below; they are the same man and, where the theorems differ, the difference is in the theorems and not in the name.

Definition(Stationarity)AdaptedAdapted from Grimmett & Stirzaker (2020), §8.2, Definitions (8.2.1) and (8.2.2) with (8.2.3) and the discussion after them, pp. 407-408, with Example (8.2.5), p. 408, and §9.1, Definitions (9.1.1)-(9.1.3), p. 434

Let $X = \{X(t)\}$ be a random process, with $t$ ranging over $[0,\infty)$ , over $\mathbb{R}$ , or over the integers.

$X$ is strongly stationary if, for every $n$ , all times $t_1, \dots, t_n$ and every $h > 0$ , the families

\bigl(X(t_1), \dots, X(t_n)\bigr) \qquad\text{and}\qquad \bigl(X(t_1+h), \dots, X(t_n+h)\bigr)

have the same joint distribution. In particular $X(t)$ then has the same distribution for every $t$ .

$X$ is weakly stationary — the alternative names are second-order stationary and covariance stationary — when, for every $t_1$ , every $t_2$ and every $h > 0$ ,

\mathbb{E}\bigl(X(t_1)\bigr) = \mathbb{E}\bigl(X(t_2)\bigr) \qquad\text{and}\qquad \operatorname{cov}\bigl(X(t_1), X(t_2)\bigr) = \operatorname{cov}\bigl(X(t_1+h), X(t_2+h)\bigr) .

Equivalently, the mean is constant and the autocovariance $c(t, t+h) = \operatorname{cov}(X(t), X(t+h))$ depends only on $h$ .

Neither notion contains the other. A strongly stationary process without a finite second moment need not be weakly stationary, and Grimmett and Stirzaker's Example (8.2.5) — $X_n = A\cos(\lambda n) + B\sin(\lambda n)$ , where $\lambda \in [0,\pi]$ is fixed and $A$ , $B$ are uncorrelated with mean $0$ and variance $1$ — is weakly stationary but, without further conditions on the joint law of $A$ and $B$ , not strongly so. Following them, we read "stationary", unqualified, as "weakly stationary" from here on, because weak stationarity is both the easier hypothesis to verify and the one the spectral theory needs.

We allow $X$ to take values in $\mathbb{C}$ , which costs nothing and is the natural setting. For complex-valued random variables the covariance is defined with a conjugate in the second slot,

\operatorname{cov}(C_1, C_2) = \mathbb{E}\Bigl[(C_1 - \mathbb{E}C_1)\overline{(C_2 - \mathbb{E}C_2)}\Bigr] , \qquad \operatorname{var}(C) = \operatorname{cov}(C,C) = \mathbb{E}\bigl(\lvert C - \mathbb{E}C\rvert^{2}\bigr) ,

so that $\operatorname{cov}$ is not symmetric: $\operatorname{cov}(C_2, C_1) = \overline{\operatorname{cov}(C_1, C_2)}$ . Two complex variables with zero covariance are called orthogonal rather than uncorrelated.

Definition(Autocovariance, autocorrelation, spectral distribution and density)AdaptedAdapted from Grimmett & Stirzaker (2020), §9.3, the opening of the section with (9.3.1), Definition (9.3.3), Definition (9.3.9) and Definition (9.3.10), pp. 438-441

Let $X = \{X(t) : -\infty < t < \infty\}$ be a weakly stationary complex-valued process. Its autocovariance function is

c(t) \;=\; \operatorname{cov}\bigl(X(s), X(s+t)\bigr) ,

which by stationarity does not depend on $s$ . Taking $t = 0$ shows that the variance is the same at every time: $\operatorname{var}(X(t)) = c(0)$ . Replacing $X$ by $X - \mathbb{E}X(0)$ changes nothing in $c$ , so we may and do assume $\mathbb{E}X(t) = 0$ whenever it is convenient.

If $c(0) > 0$ , the autocorrelation function is the normalised autocovariance

\rho(t) \;=\; \frac{c(t)}{c(0)} \;=\; \frac{\operatorname{cov}(X(0), X(t))}{\sqrt{\operatorname{var}(X(0))\operatorname{var}(X(t))}} ,

the correlation between the process and its own value a lag $t$ later. Note $\rho(0) = 1$ .

If $\rho$ admits a representation

\rho(t) \;=\; \int_{-\infty}^{\infty} e^{it\lambda}\,dF(\lambda)

with $F$ a distribution function, then $F$ is the spectral distribution function of $X$ , and where $F$ has a density, that density $f = F'$ is the spectral density function. The spectrum of $X$ is the set of real $\lambda$ with $F(\lambda+\epsilon) - F(\lambda-\epsilon) > 0$ for every $\epsilon > 0$ : the frequencies that actually carry weight.

The reading of this is that $\rho$ is a mixture of pure oscillations $e^{it\lambda}$ , the weight attached to frequencies in $[\lambda, \lambda+d\lambda]$ being $dF(\lambda)$ ; equivalently, $\rho(t) = \mathbb{E}(e^{it\Lambda})$ for a random frequency $\Lambda$ with distribution function $F$ . Because $\rho$ is normalised, so is $F$ : the spectral distribution describes how the fraction of the variance is shared out among frequencies. The variance itself carried by a band $B$ of frequencies is $c(0)\int_B dF$ , and where $F$ has a density $f$ the variance per unit frequency — the quantity a spectrum analyser plots — is $c(0)f(\lambda)$ .

Theorem(The autocovariance function is Hermitian and non-negative definite)AdaptedAdapted from Grimmett & Stirzaker (2020), §9.3, Theorem (9.3.2), p. 438, and §9.6, Theorem (9.6.1), p. 464

Let $X$ be weakly stationary with autocovariance function $c$ . Then

$c(-t) = \overline{c(t)}$ for all $t$ ; and
$c$ is non-negative definite: for all real $t_1, \dots, t_n$ and all complex $z_1, \dots, z_n$ , $\sum_{j,k} c(t_k - t_j)\,z_j\overline{z_k} \;\ge\; 0 .$

These two properties characterise autocovariance functions. For a real-valued function $c : \mathbb{R}\to\mathbb{R}$ with $c(-t) = c(t)$ and $c$ non-negative definite, Grimmett and Stirzaker construct a real-valued strongly stationary process with autocovariance $c$ , by declaring its finite-dimensional distributions to be multivariate normal with zero means and covariance matrix $\bigl(c(t_k - t_j)\bigr)$ and appealing to the Kolmogorov consistency conditions. So nothing is lost and nothing is gained by asking which functions can be autocovariances: exactly these.

Theorem(The spectral theorem for autocorrelation functions)AdaptedAdapted from Grimmett & Stirzaker (2020), §9.3, Theorem (9.3.4), equations (9.3.11)-(9.3.14) and Theorem (9.3.15) with (9.3.16)-(9.3.18), pp. 439-442

Continuous time. Let $X = \{X(t) : -\infty < t < \infty\}$ be a weakly stationary process with strictly positive variance and autocorrelation function $\rho$ . Suppose $\rho$ is continuous at the single point $t = 0$ . Then there is a distribution function $F$ whose characteristic function is $\rho$ :

\rho(t) \;=\; \int_{-\infty}^{\infty} e^{it\lambda}\,dF(\lambda) , \qquad t \in \mathbb{R} .

This is the result usually named after Wiener and Khinchin. The distribution function $F$ is unique, by the uniqueness theorem for characteristic functions.

Discrete time. If $X = \{X_n\}$ is indexed by the integers, $\rho$ is defined only on $\mathbb{Z}$ and cannot be a characteristic function until its domain is extended; the theorem survives in the form

\rho(n) \;=\; \int_{(-\pi,\pi]} e^{in\lambda}\,d\widetilde F(\lambda) , \qquad n \in \mathbb{Z} ,

for a distribution function $\widetilde F$ with $\widetilde F(-\pi) = 0$ and $\widetilde F(\pi) = 1$ ; no continuity hypothesis is appropriate here. The restriction to one period is possible because $e^{in\lambda}$ is unchanged by $\lambda \mapsto \lambda + 2\pi$ : if $\rho$ on $\mathbb{Z}$ is represented by any distribution on the whole of $\mathbb{R}$ , the mass of that distribution may be folded into $(-\pi,\pi]$ by shifting it through multiples of $2\pi$ , and the integrals at integer $n$ are unchanged. If moreover $X$ is real-valued then $\rho(n) = \rho(-n)$ and the representation may be taken in the real symmetric form

\rho(n) \;=\; \int_{[-\pi,\pi]} \cos(n\lambda)\,dG(\lambda)

for a distribution function $G$ of a symmetric distribution on $[-\pi,\pi]$ ; and conversely, a function $\rho$ with $\rho(0)=1$ admitting such a representation is the autocorrelation function of some real-valued stationary sequence.

Inversion. Grimmett and Stirzaker's Theorem (9.3.15) states: if the function $\widetilde F$ above is differentiable with derivative $f$ , then

f(\lambda) \;=\; \frac{1}{2\pi}\sum_{n=-\infty}^{\infty} e^{-in\lambda}\rho(n)

and they attach to this the qualification that it holds wherever $f$ is itself differentiable. For a real-valued sequence the formula may be written $f(\lambda) = (2\pi)^{-1}\sum_n \rho(n)\cos(n\lambda)$ . They record $\sum_{n}\lvert\rho(n)\rvert < \infty$ as a sufficient, but not necessary, condition for the spectral density to exist.

Proposition(A clean sufficient condition in discrete time)AdaptedAdapted from Grimmett & Stirzaker (2020), §9.3, the inversion formula (9.3.16) and the sufficient condition (9.3.18), p. 442

Let $X = \{X_n : n \in \mathbb{Z}\}$ be weakly stationary with $c(0) > 0$ and autocorrelation function $\rho$ , and suppose

\sum_{n=-\infty}^{\infty}\lvert\rho(n)\rvert \;<\; \infty .

Then the series

f(\lambda) \;=\; \frac{1}{2\pi}\sum_{n=-\infty}^{\infty}\rho(n)\,e^{-in\lambda}

converges absolutely and uniformly on $(-\pi,\pi]$ , its sum $f$ is continuous and non-negative, and it is the spectral density of $X$ :

\rho(n) \;=\; \int_{-\pi}^{\pi} e^{in\lambda}f(\lambda)\,d\lambda , \qquad n \in \mathbb{Z} , \qquad \int_{-\pi}^{\pi}f(\lambda)\,d\lambda = 1 .

The formula and the summability hypothesis are Grimmett and Stirzaker's (9.3.16) and (9.3.18); what is ours is the packaging of the two into one statement with the conclusions that make it usable — convergence, continuity, non-negativity and the normalisation — and the proof below, which reaches them without Bochner's theorem and without the differentiability hypothesis their Theorem (9.3.15) carries. This is the version used in practice.

Idea of the proofOriginal

Why a correlation function must be a Fourier transform. The variance of any linear combination $Z = \sum_j z_jX(t_j)$ is non-negative, and expanding that variance writes it as the double sum $\sum_{j,k}c(t_k-t_j)z_j\overline{z_k}$ . So non-negative definiteness of $c$ is not an extra assumption: it is the statement that variances are non-negative, rewritten. Bochner's theorem says that this property, together with continuity and the normalisation $\rho(0)=1$ , is exactly what characterises characteristic functions. The spectral theorem is therefore Bochner's theorem applied to $\rho$ , and the only work is to check that $\rho$ is continuous everywhere, not merely at the origin.

Why continuity at one point gives continuity everywhere. The increment $c(t+h)-c(t)$ is an expectation of $X(0)$ against the increment $X(t+h)-X(t)$ . The Cauchy-Schwarz inequality bounds it by the product of two root-mean-squares, and stationarity makes the second of them $\sqrt{2c(0) - c(h) - c(-h)}$ , a quantity that involves $h$ alone. So the modulus of continuity of $c$ at a general $t$ is controlled by its behaviour at $0$ — uniformly in $t$ . A function that is non-negative definite and continuous at the origin is automatically uniformly continuous.

Why the discrete-time statement lives on an interval. Sampling at integer times cannot distinguish a frequency $\lambda$ from $\lambda + 2\pi$ : the two oscillations agree at every integer. The spectral mass at all these frequencies is therefore indistinguishable and may be piled up at one representative in $(-\pi, \pi]$ . That is aliasing, and it is the reason a sampled signal has a spectrum only up to the Nyquist frequency.

Why absolute summability is enough, and why it gives non-negativity for free. With $\sum_n\lvert\rho(n)\rvert<\infty$ the Fourier series for $f$ converges uniformly, so it may be integrated term by term against $e^{in\lambda}$ , and orthogonality returns $\rho(n)$ . Non-negativity of the sum is less obvious, but it comes from the same source as non-negative definiteness: the Cesàro means of the series are, up to a constant, the expected squared modulus of $\sum_{n\le N}X_ne^{in\lambda}$ , which cannot be negative.

Proof(Hermitian symmetry and non-negative definiteness)AdaptedAdapted from Grimmett & Stirzaker (2020), §9.3, proof of Theorem (9.3.2), p. 439

Part 1. By stationarity we may compute $c(-t)$ at the time $s = t$ :

c(-t) = \operatorname{cov}\bigl(X(t), X(0)\bigr) = \overline{\operatorname{cov}\bigl(X(0), X(t)\bigr)} = \overline{c(t)} ,

the middle step being the antisymmetry $\operatorname{cov}(C_2,C_1) = \overline{\operatorname{cov}(C_1,C_2)}$ of the complex covariance.

Part 2. Fix $t_1,\dots,t_n$ and $z_1,\dots,z_n$ and set $Z = \sum_j z_jX(t_j)$ . Covariance is linear in its first slot and conjugate-linear in its second, so

\operatorname{var}(Z) = \operatorname{cov}(Z,Z) = \sum_{j,k}\operatorname{cov}\bigl(z_jX(t_j),\, z_kX(t_k)\bigr) = \sum_{j,k} z_j\overline{z_k}\,\operatorname{cov}\bigl(X(t_j), X(t_k)\bigr) = \sum_{j,k}c(t_k-t_j)z_j\overline{z_k} ,

where the last step uses stationarity in the form $\operatorname{cov}(X(t_j),X(t_k)) = c(t_k - t_j)$ . A variance is non-negative, so the double sum is non-negative. $\square$

Since $\rho = c/c(0)$ and $c(0) = \operatorname{var}(X(0)) > 0$ , the function $\rho$ inherits both properties and satisfies $\rho(0) = 1$ .

Proof(The spectral theorem)AdaptedAdapted from Grimmett & Stirzaker (2020), §9.3, proof of Theorem (9.3.4), p. 439, and §5.7, Theorem (5.7.3) and the statement of Bochner's theorem, p. 204

Assume, as we may, that $\mathbb{E}X(t) = 0$ , so that $c(t) = \mathbb{E}\bigl(X(0)\overline{X(t)}\bigr)$ .

Step 1: an estimate for the increments of $c$ . For real $t$ and $h$ ,

c(t+h) - c(t) = \mathbb{E}\Bigl[X(0)\,\overline{\bigl(X(t+h) - X(t)\bigr)}\Bigr] ,

by linearity of the expectation. Taking moduli inside and applying the Cauchy-Schwarz inequality,

\bigl\lvert c(t+h) - c(t)\bigr\rvert \;\le\; \mathbb{E}\Bigl[\lvert X(0)\rvert\,\bigl\lvert X(t+h)-X(t)\bigr\rvert\Bigr] \;\le\; \sqrt{\mathbb{E}\bigl(\lvert X(0)\rvert^{2}\bigr)\;\mathbb{E}\bigl(\lvert X(t+h)-X(t)\rvert^{2}\bigr)} .

Step 2: evaluate the second factor by stationarity. Expand the square and use $\mathbb{E}\lvert X(u)\rvert^{2} = c(0)$ , valid for every $u$ , together with $\operatorname{cov}(X(t),X(t+h)) = c(h)$ and $\operatorname{cov}(X(t+h),X(t)) = c(-h)$ :

\mathbb{E}\bigl(\lvert X(t+h) - X(t)\rvert^{2}\bigr) = 2c(0) - c(h) - c(-h) .

Only $h$ survives on the right. Therefore

\bigl\lvert c(t+h) - c(t)\bigr\rvert \;\le\; \sqrt{c(0)\bigl(2c(0) - c(h) - c(-h)\bigr)} \qquad\text{for all } t .

Step 3: continuity at $0$ upgrades to uniform continuity. By hypothesis $\rho$ , and therefore $c = c(0)\rho$ , is continuous at $0$ , so $c(h)\to c(0)$ and $c(-h) = \overline{c(h)}\to c(0)$ as $h \to 0$ . The bound of step 2 therefore tends to $0$ as $h\to 0$ , uniformly in $t$ . So $c$ is uniformly continuous on $\mathbb{R}$ , and so is $\rho = c/c(0)$ .

Step 4: apply Bochner's theorem. By the previous proof, $\rho$ is non-negative definite in the form $\sum_{j,k}\rho(t_k-t_j)z_j\overline{z_k}\ge 0$ ; replacing each $z_j$ by $\overline{z_j}$ and exchanging the names of the summation indices turns this into $\sum_{j,k}\rho(t_j-t_k)z_j\overline{z_k}\ge0$ , which is the form in which Bochner's condition is stated. We also have $\rho(0) = 1$ and $\lvert\rho(t)\rvert\le 1$ , the latter by the Cauchy-Schwarz inequality, and uniform continuity from step 3. Bochner's theorem — quoted by Grimmett and Stirzaker without proof, as the converse half of their Theorem (5.7.3) — says that any function carrying those three properties arises as the characteristic function of a distribution function $F$ . That is the assertion:

\rho(t) = \int_{-\infty}^{\infty}e^{it\lambda}\,dF(\lambda) . \qquad\square

Uniqueness of $F$ is the uniqueness theorem for characteristic functions (see Characteristic functions).

Proof(The proposition in discrete time)Original

Assume $\mathbb{E}X_n = 0$ and $\sum_n\lvert\rho(n)\rvert < \infty$ .

Step 1: convergence and continuity. Each term of $\sum_n \rho(n)e^{-in\lambda}$ has modulus $\lvert\rho(n)\rvert$ , so by the Weierstrass test the series converges absolutely and uniformly in $\lambda$ . A uniform limit of continuous functions is continuous, so $f$ is continuous on $(-\pi,\pi]$ , and $2\pi f$ is bounded by $\sum_n\lvert\rho(n)\rvert$ .

Step 2: the coefficients come back. Uniform convergence permits term-by-term integration, so for each $m \in \mathbb{Z}$

\int_{-\pi}^{\pi}e^{im\lambda}f(\lambda)\,d\lambda = \frac{1}{2\pi}\sum_{n}\rho(n)\int_{-\pi}^{\pi}e^{i(m-n)\lambda}\,d\lambda = \rho(m) ,

because $\int_{-\pi}^{\pi}e^{ik\lambda}d\lambda$ equals $2\pi$ when $k = 0$ and $0$ otherwise: this is the orthogonality of the exponentials over one period that underlies Fourier series, and the display says that $\rho(-m)$ , read the other way round, is the $m$ th Fourier coefficient of $2\pi f$ . Taking $m = 0$ gives $\int_{-\pi}^{\pi}f = \rho(0) = 1$ .

Step 3: non-negativity, through a Cesàro mean. For $N \ge 1$ and $\lambda$ fixed, put $Z_N = \sum_{n=1}^{N}X_ne^{in\lambda}$ and consider

S_N(\lambda) \;=\; \frac{1}{2\pi N\,c(0)}\,\mathbb{E}\bigl(\lvert Z_N\rvert^{2}\bigr) \;\ge\; 0 .

Expanding the square,

\mathbb{E}\bigl(\lvert Z_N\rvert^{2}\bigr) = \sum_{n=1}^{N}\sum_{m=1}^{N}e^{i(n-m)\lambda}\,\mathbb{E}\bigl(X_n\overline{X_m}\bigr) = \sum_{n,m}e^{i(n-m)\lambda}c(m-n) ,

using $\mathbb{E}(X_n\overline{X_m}) = \operatorname{cov}(X_n,X_m) = c(m-n)$ . The summand depends on $n$ and $m$ only through $k = m-n$ , and for each $k$ with $\lvert k\rvert < N$ there are exactly $N - \lvert k\rvert$ pairs, so

S_N(\lambda) = \frac{1}{2\pi}\sum_{\lvert k\rvert < N}\Bigl(1 - \frac{\lvert k\rvert}{N}\Bigr)\rho(k)\,e^{-ik\lambda} .

Step 4: let $N \to \infty$ . Each term converges to $\rho(k)e^{-ik\lambda}/2\pi$ , and the terms are dominated in modulus by $\lvert\rho(k)\rvert/2\pi$ , which is summable; so by dominated convergence for series, $S_N(\lambda)\to f(\lambda)$ . A limit of non-negative numbers is non-negative, so $f(\lambda)\ge 0$ for every $\lambda$ .

Step 5: identify $f$ as the spectral density. By steps 1-4, $f$ is a continuous probability density on $(-\pi,\pi]$ , and by step 2 the distribution function $\widetilde F(\lambda) = \int_{-\pi}^{\lambda}f$ satisfies $\rho(n) = \int_{(-\pi,\pi]}e^{in\lambda}d\widetilde F(\lambda)$ for all $n$ . This is the discrete-time spectral representation, and $\widetilde F$ is unique, so $f$ is the spectral density. $\square$

Remark(Where the hypotheses are used, conventions, and the other Wiener-Kinchin theorem)AdaptedAdapted from Grimmett & Stirzaker (2020), §9.3, Theorem (9.3.4) and the discussion of (9.3.11)-(9.3.14) and (9.3.18), pp. 439-442, and Examples (9.3.19) and (9.3.20), p. 442; Riley, Hobson & Bence (2006), §13.1.8, eqs. (13.40)-(13.43) and the energy spectrum, pp. 449-450

Strict positivity of the variance is needed only to divide by $c(0)$ . A process that is almost surely constant has $c \equiv 0$ and no autocorrelation function; there is nothing to normalise and no spectral distribution.

Continuity of $\rho$ at the origin is used exactly once, in step 3 of the proof, and it is not a technicality. Two different objects go by the name "white noise" in continuous time, and only the first of them is a process of the kind defined above; it is worth keeping them apart, because the hypothesis fails for one and the definition fails for the other.

A process with orthogonal values. Take $c(0) = \sigma^{2} > 0$ and $c(t) = 0$ for every $t \ne 0$ . This $c$ is non-negative definite — for distinct times the double sum of the statement above collapses to $\sigma^{2}\sum_j\lvert z_j\rvert^{2}$ — so by the characterisation quoted there it really is the autocovariance of a process, and that process satisfies the definition of weak stationarity given here: the mean is constant and the covariance depends on the lag alone. What fails is the hypothesis, and with it the conclusion. Its $\rho$ is $1$ at the origin and $0$ elsewhere, no characteristic function jumps like that, and so this process has no spectral distribution at all — not an unusual one, none. Nor does a formal transform rescue it: $\rho$ vanishes off a single point, so $\int\rho(t)e^{-i\lambda t}\,dt = 0$ , which is not a density. The lesson is that weak stationarity by itself buys nothing; the spectral theory begins at the hypothesis of continuity.

The white noise of physics. The delta-correlated object $c(t) = \sigma^{2}\delta(t)$ is a different thing and is not covered by the definitions above at all, because $c(0)$ is infinite: it has no variance, so it is not a second-order process and has no autocorrelation function to be continuous or discontinuous. It is this object, not the one in the previous paragraph, that carries the flat spectrum $S(\omega) = \sigma^{2}$ of infinite total mass; the infinite mass is precisely the infinite variance, seen in the frequency domain. White noise in this sense has to be handled as a generalised process, as the formal derivative of Brownian motion — which is what Brownian motion and The Ito integral and Ito's formula do instead. In the laboratory neither idealisation is met: a real noise source is flat only up to some bandwidth, beyond which $S$ rolls off and the total variance is finite.

Only weak stationarity is used anywhere above. The spectral theory never looks beyond the mean and the covariance. This is the practical reason for preferring the weaker hypothesis: to fit a spectrum one needs to assume only that the first two moments are time-invariant, not that the whole law is.

In discrete time the spectral distribution is unique only on one period. The representation on $\mathbb{R}$ is not unique, since $e^{in\lambda}$ is $2\pi$ -periodic in $\lambda$ ; uniqueness is restored by folding all the mass into $(-\pi,\pi]$ . A sampled process therefore cannot tell a frequency from its aliases, and a spectral density estimated from samples at spacing $\Delta$ is defined only for $\lvert\omega\rvert \le \pi/\Delta$ .

Absolute summability is sufficient but not necessary for a spectral density to exist. Grimmett and Stirzaker say so explicitly at their (9.3.18). Their Example (9.3.20) shows the opposite extreme: for $X_n \equiv Y$ with $\operatorname{var}Y = 1$ , one has $\rho(n) = 1$ for all $n$ , which is not summable; the spectral distribution is the unit mass at $\lambda = 0$ and there is no density. Their Example (9.3.19), discrete white noise with $\rho(n) = \delta_{n0}$ , is the other extreme: the spectral density is the uniform density on $[-\pi,\pi]$ , and the spectrum is all of $[-\pi,\pi]$ .

The hypothesis of the inversion theorem, as printed. Grimmett and Stirzaker's Theorem (9.3.15) assumes that $\widetilde F$ is differentiable with derivative $f$ , and then asserts the inversion formula "at every point $\lambda$ at which $f$ is differentiable". We state it exactly as they do. The proposition above avoids the question by assuming absolute summability outright, which yields a continuous $f$ and the formula at every $\lambda$ .

Conventions. Three normalisations are in common use, and they differ by factors of $2\pi$ and by whether the variance is divided out.

Object	Definition here	Relation
Spectral density $f$ (as above)	$\rho(t) = \int e^{it\lambda}f(\lambda)\,d\lambda$	$\int f = 1$
Power spectral density $S$ (physics)	$S(\omega) = \int_{-\infty}^{\infty}c(t)e^{-i\omega t}\,dt$	$S = 2\pi c(0)f$ , and $\frac{1}{2\pi}\int S(\omega)\,d\omega = c(0)$
One-sided spectrum (engineering)	$S_{+}(\omega) = 2S(\omega)$ for $\omega > 0$ , $S_{+}(0) = S(0)$	defined for real processes only

In the handbook's Fourier convention, $\hat g(k) = \int g(t)e^{-ikt}\,dt$ with the $2\pi$ on the inverse transform (see The Fourier transform), the first two lines read $f = \hat\rho/2\pi$ and $S = \hat c$ . Always check which of these a quoted "spectral density" means before comparing numbers.

The other theorem of the same name. (On the spelling, see the third warning in the opening note: Riley, Hobson and Bence write Kinchin where Grimmett and Stirzaker write Khinchin, and we follow whichever source is being reported.) Riley, Hobson and Bence prove, and call the Wiener-Kinchin theorem, a purely deterministic statement: for square-integrable $f$ and $g$ with cross-correlation $C(z) = \int\overline{f(x)}g(x+z)\,dx$ , one has $\hat C(k) = \sqrt{2\pi}\,\overline{\hat f(k)}\,\hat g(k)$ in their symmetric convention; specialising to $g = f$ makes the transform of the autocorrelation proportional to $\lvert\hat f(k)\rvert^{2}$ , which they call the energy spectrum of $f$ . No probability appears, and no stationarity: the two results are different theorems sharing a name and a shape. The probabilistic one above concerns an ensemble average that exists because the process is stationary; the deterministic one concerns a single square-integrable function, which a stationary process's sample path is not. The bridge between them, which is what makes the shared name defensible, is that a time average of $\lvert\int_{-T}^{T}X(t)e^{-i\omega t}dt\rvert^{2}/2T$ estimates $S(\omega)$ — the content of the periodogram, treated in The periodogram.

Method(Finding the spectral density of a stationary process)Original

Check weak stationarity. Confirm that the mean is constant and that $\operatorname{cov}(X(s),X(s+t))$ does not depend on $s$ . If the process is defined by a recursion or a differential equation driven by noise, this is usually a condition on the coefficients: for an autoregression $X_n = \sum_{j=1}^{r}\alpha_jX_{n-j} + Z_n$ the condition is that the roots of $z^{r} - \alpha_1z^{r-1} - \cdots - \alpha_r$ all lie inside the unit circle.
Compute the autocovariance $c(t)$ , then $\rho(t) = c(t)/c(0)$ . For a process built linearly from orthogonal noise, expand the product $\mathbb{E}(X_n\overline{X_{n+m}})$ and discard every term in which two different noise variables meet.
Check continuity at $t = 0$ in continuous time; in discrete time there is nothing to check.
Transform. In discrete time, if $\sum\lvert\rho(n)\rvert<\infty$ , use $f(\lambda) = (2\pi)^{-1}\sum_n\rho(n)e^{-in\lambda}$ on $(-\pi,\pi]$ ; for a real process this is $(2\pi)^{-1}\bigl(1 + 2\sum_{n\ge1}\rho(n)\cos n\lambda\bigr)$ . In continuous time use $f(\lambda) = (2\pi)^{-1}\int\rho(t)e^{-it\lambda}\,dt$ when $\rho$ is integrable, which is Fourier inversion applied to the representation $\rho(t) = \int e^{it\lambda}f(\lambda)\,d\lambda$ ; otherwise recognise $\rho$ as a known characteristic function and read off its distribution, which is often quicker than inverting an integral.
Check the answer three ways. $f$ must be non-negative; $\int f$ must be $1$ ; and $f$ must be symmetric under $\lambda\mapsto-\lambda$ if the process is real.
Convert to the convention you need using the table in the remark, and record which one you used.
Read off what you wanted. The fraction of the variance carried by a frequency band $B$ is $\int_Bf$ ; a spectral density with a sharp peak at $\lambda_0$ means a quasi-periodic component of period $2\pi/\lambda_0$ ; a spectral density that is flat near $\lambda=0$ and rolls off at $\lambda_c$ means a correlation time of order $1/\lambda_c$ .

Example(A moving average of order two)Original

Let $\{Z_n : n \in \mathbb{Z}\}$ be orthogonal random variables, each of mean $0$ and variance $1$ . Define a real-valued process by

X_n \;=\; Z_n + \tfrac12 Z_{n-1} - \tfrac14 Z_{n-2} , \qquad n \in \mathbb{Z} .

Write $a_0 = 1$ , $a_1 = \tfrac12$ , $a_2 = -\tfrac14$ and $a_k = 0$ otherwise.

Step 1: stationarity and the autocovariance. Each $X_n$ has mean $0$ . Since $\mathbb{E}(Z_jZ_k) = \delta_{jk}$ ,

c(m) = \mathbb{E}(X_nX_{n+m}) = \sum_{j,k}a_ja_k\,\mathbb{E}\bigl(Z_{n-j}Z_{n+m-k}\bigr) = \sum_{k}a_ka_{k+m} ,

which does not depend on $n$ ; so $X$ is weakly stationary. Evaluating,

c(0) = 1 + \tfrac14 + \tfrac1{16} = \tfrac{21}{16} , \qquad c(1) = \tfrac12 + \tfrac12\!\cdot\!\bigl(-\tfrac14\bigr) = \tfrac38 , \qquad c(2) = -\tfrac14 ,

and $c(m) = 0$ for $\lvert m\rvert\ge3$ , because then no two of the shifted index sets overlap. Hence

\rho(1) = \frac{3/8}{21/16} = \frac27 , \qquad \rho(2) = \frac{-1/4}{21/16} = -\frac{4}{21} .

Step 2: the spectral density. Only five terms of the series are non-zero, so absolute summability is immediate and the proposition applies. For a real process $\rho(-n)=\rho(n)$ , so

f(\lambda) = \frac{1}{2\pi}\Bigl(1 + 2\rho(1)\cos\lambda + 2\rho(2)\cos2\lambda\Bigr) = \frac{1}{2\pi}\Bigl(1 + \tfrac47\cos\lambda - \tfrac{8}{21}\cos2\lambda\Bigr) , \qquad -\pi \le \lambda \le \pi .

Step 3: the three checks. Symmetry in $\lambda$ is visible. The integral is $1$ , since $\int_{-\pi}^{\pi}\cos(n\lambda)\,d\lambda = 0$ for $n = 1, 2$ and $\int_{-\pi}^{\pi}d\lambda = 2\pi$ . For non-negativity, note the general identity for a moving average: with $A(z) = \sum_ka_kz^{k}$ ,

\sum_m c(m)e^{-im\lambda} = \sum_{k,m}a_ka_{k+m}e^{-im\lambda} = \overline{A(e^{-i\lambda})}\,A(e^{-i\lambda}) = \bigl\lvert A(e^{-i\lambda})\bigr\rvert^{2} ,

so that $f(\lambda) = \lvert A(e^{-i\lambda})\rvert^{2}/\bigl(2\pi c(0)\bigr) \ge 0$ . Here $A(z) = 1 + \tfrac12z - \tfrac14z^{2}$ . As a numerical check at $\lambda = 0$ : the formula gives $f(0) = (1 + \tfrac47 - \tfrac8{21})/(2\pi) = \tfrac{25}{42\pi} = 0.1894702$ , while $A(1) = \tfrac54$ gives $\lvert A(1)\rvert^{2}/(2\pi c(0)) = (25/16)/(2\pi\cdot 21/16) = 25/(42\pi)$ , the same. At the other end, $f(\pi) = (1 - \tfrac47 - \tfrac8{21})/(2\pi) = 1/(42\pi) = 0.0075788$ , which is the minimum of $f$ and is positive.

Step 4: a number to check. What fraction of the variance of $X$ lies at frequencies $\lvert\lambda\rvert\le\pi/2$ , that is, in periods of four sampling intervals or longer? It is

\int_{-\pi/2}^{\pi/2}f(\lambda)\,d\lambda = \frac{1}{2\pi}\Bigl(\pi + \tfrac47\bigl[\sin\lambda\bigr]_{-\pi/2}^{\pi/2} - \tfrac8{21}\Bigl[\tfrac{\sin2\lambda}{2}\Bigr]_{-\pi/2}^{\pi/2}\Bigr) = \frac{1}{2\pi}\Bigl(\pi + \tfrac87 - 0\Bigr) = \frac12 + \frac{4}{7\pi} .

Numerically this is $0.681891$ . The filter has tilted the flat spectrum of the input noise towards low frequencies: white noise would put exactly half the variance in this band, and the moving average has moved about $18$ per cent of the total into it. That is the signature of the positive coefficient on $Z_{n-1}$ , which makes consecutive values of $X$ positively correlated, $\rho(1) = 2/7 > 0$ .

Application(Noise spectra in the laboratory)Original

What a spectrum analyser measures. Point an instrument at a fluctuating voltage, a photocurrent, a seismometer trace, or a sea-surface elevation record, and what comes back is an estimate of the power spectral density $S(\omega) = \int c(t)e^{-i\omega t}\,dt$ . The theorem above is what guarantees that this is a well-posed thing to measure: provided the record is stationary and the correlation function is continuous at the origin, a non-negative $S$ exists and determines, and is determined by, the correlation function. The estimator itself is the periodogram (The periodogram), and the standard worry about it — that it is noisy however long the record, and must be smoothed — is a statistical matter, not a failure of the theorem.

Lorentzian line shapes. A process whose correlations decay exponentially, $c(t) = \sigma^{2}e^{-\lvert t\rvert/\tau}$ , has $\rho(t) = e^{-\lvert t\rvert/\tau}$ , the characteristic function of a Cauchy distribution scaled by $1/\tau$ ; its spectral density is therefore the Lorentzian

f(\lambda) = \frac{1}{\pi}\,\frac{1/\tau}{(1/\tau)^{2} + \lambda^{2}} , \qquad\text{equivalently}\qquad S(\omega) = \frac{2\sigma^{2}\tau}{1 + \omega^{2}\tau^{2}} .

This single formula is behind the shape of the pressure-broadened spectral line, the roll-off of the Ornstein-Uhlenbeck velocity spectrum (see Stochastic differential equations), the noise spectrum of a single relaxing dipole, and the Lorentzian fitted to a dynamic light-scattering correlogram to extract a diffusion coefficient. The correlation time is read off as the inverse of the half-width: a spectrum flat up to $\omega \sim 1/\tau$ and falling as $\omega^{-2}$ beyond it.

White noise and its colouring. A flat spectral density means an autocorrelation concentrated at zero lag — no memory. Johnson-Nyquist noise in a resistor, and shot noise in a current, are white over the bandwidths usually of interest, and the linear circuit or detector that follows multiplies the spectral density by $\lvert H(\omega)\rvert^{2}$ , the squared modulus of the transfer function. That multiplication rule is the continuous-time twin of the identity $f = \lvert A(e^{-i\lambda})\rvert^{2}/(2\pi c(0))$ verified in the example above, and it is the content of Linear filters: filtering a stationary process reshapes its spectrum pointwise and does nothing else.

Transport coefficients from correlation functions. In statistical mechanics the zero-frequency value of a spectral density is a transport coefficient. For the velocity of a tagged particle in three dimensions, the Green-Kubo relation

D = \frac13\int_0^{\infty}\bigl\langle v(0)\cdot v(t)\bigr\rangle\,dt

expresses the diffusion coefficient as half the zero-frequency spectral density of the velocity, divided by three for the dimensions; the integral converges precisely because the velocity autocorrelation decays, which is the same condition that gives the velocity process a spectral density. Similar expressions give the shear viscosity from the stress autocorrelation and the thermal conductivity from the heat-flux autocorrelation. In every case the quantity actually computed in a molecular-dynamics run is the autocovariance function of a stationary process, and the transport coefficient is read off its transform.

A caution about stationarity. Climate records, financial series and detector baselines drift, and a drifting series is not stationary; its sample "spectrum" will show a spurious rise at low frequency, because a trend looks like an oscillation of period longer than the record. The usual remedies — removing a fitted trend, differencing, or working with increments — are all attempts to recover a process to which this theorem applies.

Definition

Brownian motion

Adapted

Add to PDF

Defines the Wiener process, shows that it is the scaling limit of a random walk in the sense that the rescaled walk's finite-dimensional distributions converge to it, sketches the proof that a version with continuous paths exists, and establishes the path properties - covariance $\sigma^2\min\{s,t\}$ , quadratic variation $\sigma^2 t$ , and total variation infinite on every interval.

Prerequisites: Random variables and distributions, Expectation, variance and covariance, The multivariate normal distribution, Characteristic functions, Markov, Chebyshev, Jensen and Chernoff inequalities, Convergence of random variables, The central limit theorem

Note(What has to be supplied, and what cannot be)Original

A random walk is specified by one number per step. A process in continuous time has uncountably many times, so a list of distributions is no longer enough to pin it down: two processes can have exactly the same distribution at every finite collection of times and yet have completely different paths. One may be continuous and the other wild.

This is the one structural difficulty in defining Brownian motion, and it shapes everything below. The distributional half of the definition is the easy half, and it is strongly suggested - though, as the idea block explains, not forced - by subdividing a time interval and looking at the central limit theorem: the increments ought to be normal, with variance proportional to elapsed time. That is where both of our sources stop arguing and start assuming. The path half - continuity - has to be imposed as a separate requirement as well, and then one must prove that the two halves are compatible. They are, and that is the existence theorem.

Two warnings about names. Grimmett and Stirzaker use Wiener process for the mathematical object and keep Brownian motion for the physical phenomenon it models; Folland uses the two names interchangeably for the process. We follow the first convention, and say "Wiener process" whenever precision matters.

Definition(Gaussian processes, and the Wiener process)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.3, Definition (13.3.1) with the standardisation paragraph, pp. 580-581, and §8.5, pp. 416-417; Folland (1999), §10.5, postulates (10.20)-(10.22), pp. 330-331

A random process $X = \{X(t) : t \ge 0\}$ is a family of random variables on one probability space $(\Omega, \mathcal{F}, \mathbb{P})$ indexed by time. For a fixed $\omega \in \Omega$ the function $t \mapsto X(t;\omega)$ is a sample path. The finite-dimensional distributions of $X$ are the joint distributions of the vectors $(X(t_1), \dots, X(t_k))$ over all finite sets of times.

$X$ is a Gaussian process if every one of its finite-dimensional distributions is multivariate normal (see The multivariate normal distribution). The process $X$ is said to have independent increments when, for any finite list of pairwise disjoint time intervals, the increments of $X$ across those intervals form an independent family; it is enough to require this for two intervals $[u,v]$ and $[s,t]$ with $u \le v \le s \le t$ and then for finitely many at a time.

Definition. Fix a constant $\sigma^{2} > 0$ . A real-valued Gaussian process $W = \{W(t) : t \ge 0\}$ with $W(0) = w$ is called a Wiener process with variance parameter $\sigma^{2}$ when it has all three of these properties.

(Independent increments.) Increments of $W$ over disjoint time intervals are independent.
(Stationary normal increments.) Each increment $W(s+t) - W(s)$ , with $s, t \ge 0$ , is distributed as $N(0, \sigma^{2}t)$ .
(Continuity.) Every sample path of $W$ is a continuous function of time.

$W$ is standard if $w = 0$ and $\sigma^2 = 1$ . Any Wiener process is reduced to a standard one by $W_1(t) = \bigl(W(t) - W(0)\bigr)/\sigma$ . Because the law of $W(s+t) - W(s)$ depends on $t$ and not on $s$ , the increments are called stationary.

Condition 2 forces $\mathbb{E}\bigl(W(s+t) - W(s)\bigr) = 0$ : a Wiener process has no drift. Condition 3 is a statement about the functions $t \mapsto W(t;\omega)$ , not about any distribution, and is the only part of the definition that the finite-dimensional distributions do not see.

Folland reaches the same three conditions by a different route, reading each as a physical postulate about a particle bombarded by molecules: it starts at a known place (his (10.20)); a collision has no lasting memory, so increments over disjoint time intervals are independent (his (10.21)); the mechanism is unchanging in time, so the law of $W(t) - W(s)$ depends only on $t-s$ ; and the particle is pushed left and right equally, so the mean increment is zero. Normality and the proportionality of the variance to $t-s$ are not deduced from these. Folland subdivides $[s,t]$ into $n$ equal pieces, says that the central limit theorem "suggests" that the increment should be normal and that additivity of the variance "strongly indicates" that it should be proportional to $t-s$ , and then writes: "we are led to the third assumption", namely his (10.22), that $W(t)-W(s)$ is normal with mean $0$ and variance $C(t-s)$ . So normality and the variance law are a postulate for him too, as they are condition 2 for us; the subdivision argument recalled in the idea below is its motivation, not its proof.

Definition(Covariance, finite-dimensional distributions, and existence)AdaptedAdapted from Grimmett & Stirzaker (2020), §8.5, Lemma (8.5.1), p. 417, and §13.3, equation (13.3.2), transition density (13.3.4) and Theorem (13.3.19), pp. 581-587; Folland (1999), §10.5, equation (10.23) and Theorem 10.28, pp. 332-334

Let $W$ be a Wiener process with variance parameter $\sigma^2$ and $W(0) = 0$ .

Covariance. For all $s, t \ge 0$ , $\mathrm{cov}\bigl(W(s), W(t)\bigr) \ = \ \sigma^{2}\min\{s,t\} .$
Finite-dimensional distributions. For $0 \le t_1 < t_2 < \cdots < t_k$ , the vector $(W(t_1), \dots, W(t_k))$ is multivariate normal with zero mean vector and covariance matrix $V$ with entries $v_{ij} = \sigma^{2}\min\{t_i, t_j\}$ . Conditions 1 and 2 of the definition therefore determine every finite-dimensional distribution, and so determine the conditional (transition) density of $W(t)$ given $W(s) = x$ , which for the standard process is $f(t, y \mid s, x) \ = \ \frac{1}{\sqrt{2\pi(t-s)}}\, \exp\left( -\frac{(y-x)^{2}}{2(t-s)} \right) , \qquad s < t .$
Existence. There is a probability space carrying a process that satisfies all three conditions of the definition. Equivalently, the probability measure determined by the finite-dimensional distributions in part 2 can be realised on the space of continuous paths.

Part 3 is the content of the definition that is not free. A Gaussian process with the finite-dimensional distributions of part 2 exists by general constructions; what has to be proved is that one of them may be chosen with every sample path continuous.

Definition(The Wiener process as the scaling limit of a random walk)Original

Let $X_1, X_2, \dots$ be independent and identically distributed with $\mathbb{E}(X_1) = 0$ and $\mathrm{var}(X_1) = \sigma^{2}$ , where $0 < \sigma^{2} < \infty$ . Put $S_0 = 0$ and $S_m = X_1 + \cdots + X_m$ , and for each $n \ge 1$ define the rescaled walk

W_n(t) \ = \ \frac{S_{\lfloor nt \rfloor}}{\sqrt{n}} , \qquad t \ge 0 ,

in which space is compressed by $\sqrt n$ while time is compressed by $n$ .

Claim. For every finite set of times $0 \le t_1 < t_2 < \cdots < t_k$ ,

\bigl(W_n(t_1), \dots, W_n(t_k)\bigr) \ \xrightarrow{\mathrm{D}} \ \bigl(W(t_1), \dots, W(t_k)\bigr) \qquad \text{as } n \to \infty ,

where $W$ is a Wiener process with variance parameter $\sigma^{2}$ and $W(0) = 0$ .

That is: the finite-dimensional distributions of the rescaled walk converge to those of the Wiener process, whatever the step distribution, provided only that its mean is zero and its variance is finite and non-zero. No other scaling works: see the remark below.

Definition(Path properties)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.2, p. 579 (nowhere differentiability), Exercise (13.2.1), p. 580, §8.5, Exercise (8.5.4), p. 418, and §8.8, Definition (8.8.5), p. 427

Let $W$ be a Wiener process with variance parameter $\sigma^{2}$ , and fix $t > 0$ . Write

Q_n \ = \ \sum_{j=0}^{n-1} \Bigl( W\bigl((j+1)t/n\bigr) - W\bigl(jt/n\bigr) \Bigr)^{2}

for the sum of squared increments over the uniform partition of $[0,t]$ into $n$ pieces.

Quadratic variation. $Q_n \to \sigma^{2} t$ in mean square as $n \to \infty$ , and almost surely along the subsequence $n = 2^{k}$ . (The mean-square statement is the one Grimmett and Stirzaker set as Exercise (8.5.4); the almost sure statement along the dyadic subsequence is an addition of ours, and both are proved below.)
Unbounded total variation. Almost surely, the sample path of $W$ has infinite total variation on every interval $[a,b]$ with $a < b$ ; that is, $\sup \ \sum_{r} \bigl\lvert W(s_{r+1}) - W(s_r) \bigr\rvert \ = \ \infty ,$ the supremum being over all finite increasing sequences $a = s_0 < s_1 < \cdots < s_n = b$ .
Nowhere differentiability. Almost every sample path is a continuous function of $t$ that is differentiable at no point. Grimmett and Stirzaker record this without proof, and we follow them. Part 2, which we do prove, gives a weaker statement of the same kind by itself: a function that is Lipschitz on an interval has finite total variation there, so almost surely no sample path is Lipschitz on any interval, and in particular none has a derivative that exists and is bounded throughout an interval.

Part 2 is why the Ito integral of The Ito integral and Ito's formula cannot be a Stieltjes integral taken path by path, and part 1 is what replaces it: the correct " $dW^2$ " is not negligible but equals $\sigma^2\,dt$ .

Idea of the proofOriginal

Three separate ideas are at work, one for each statement.

Why the increments are taken to be normal. Split $[s,t]$ into $n$ equal pieces. By stationarity and independence, $W(t) - W(s)$ is a sum of $n$ independent identically distributed increments, for every $n$ ; in a word, its law is infinitely divisible. Two things should be separated here.

What the subdivision really settles is the variance. Write $v(h)$ for the variance of an increment across an interval of length $h$ , assumed finite. Independence and stationarity give $v(s+t) = v(s) + v(t)$ for all $s,t \ge 0$ , and $v$ is non-decreasing because a longer interval contains a shorter one and the extra variance is non-negative; a non-decreasing solution of that additive equation is $v(h) = v(1)h$ . So once the increments are assumed to have finite variance, proportionality to elapsed time is a genuine consequence of conditions 1 and stationarity, with $\sigma^{2} = v(1)$ .

Normality is not. Being an $n$ -fold convolution of identical pieces for every $n$ , even with finite variance, does not make a law normal: the Poisson and gamma laws, and compound Poisson laws generally, are infinitely divisible and have finite variance. The missing ingredient is a negligibility (Lindeberg) condition saying that no single piece of the subdivision carries a non-negligible share of the sum; a Poisson increment fails it, because a single jump is a macroscopic part of the total. With such a condition the central limit theorem for triangular arrays applies and the limit is normal, and that is the honest content of the heuristic. Folland supplies no such condition and claims none: he says the central limit theorem "suggests" normality and then assumes it. Condition 2 is therefore a postulate - very well motivated, and the physically right one for a particle hit by enormously many enormously small kicks, which is exactly a negligibility hypothesis - but a postulate.

Why the rescaled walk converges. Over disjoint time intervals the rescaled walk accumulates disjoint blocks of steps, so its increments are exactly independent at every $n$ , not just in the limit. Each single increment is a normalised sum of about $n(t_j - t_{j-1})$ steps, so the central limit theorem sends it to $N(0, \sigma^2(t_j - t_{j-1}))$ . A joint characteristic function that factorises into pieces, each of which converges, converges to the product of the limits; that product is the characteristic function of the Wiener vector.

Why a continuous version exists. Build the path by successive refinement. Fix its values at the times $m/2^{n}$ for $n = 1$ , then $n = 2$ , and so on, joining them by straight lines; passing from level $n-1$ to level $n$ only moves the midpoints of the existing segments. The midpoint displacement at level $j$ is normal with variance $2^{-j-1}$ , and there are $2^{j-1}$ of them, so the largest of them is of order $\sqrt{j\,2^{-j}}$ - small enough that the total displacement over all levels converges. Summable corrections produce a uniformly convergent series of continuous functions, so the limit is continuous; the first Borel-Cantelli lemma turns "summable" into "almost surely".

Why the quadratic variation survives. Each squared increment has mean $\sigma^2 t/n$ and variance of the same order squared, so the sum has mean exactly $\sigma^{2}t$ and variance of order $1/n$ : adding $n$ independent small terms averages the fluctuation away but not the mean. Since the increments themselves are of order $n^{-1/2}$ and there are $n$ of them, the sum of their absolute values is of order $\sqrt n$ , which is why the total variation diverges.

Proof(Covariance, and the scaling limit)Original

Step 1: the covariance. Take $0 \le s \le t$ and $W(0) = 0$ . Write $W(t) = W(s) + \bigl(W(t) - W(s)\bigr)$ and multiply by $W(s) = W(s) - W(0)$ :

\mathbb{E}\bigl(W(s)W(t)\bigr) \ = \ \mathbb{E}\bigl(W(s)^{2}\bigr) + \mathbb{E}\Bigl( \bigl(W(s)-W(0)\bigr)\bigl(W(t)-W(s)\bigr) \Bigr) .

The two factors in the second term are increments over the disjoint intervals $[0,s]$ and $[s,t]$ , hence independent by condition 1, and each has mean zero by condition 2; so that term vanishes. The first term is $\mathrm{var}(W(s)) = \sigma^{2}s$ by condition 2. Since all means are zero, $\mathrm{cov}(W(s),W(t)) = \sigma^{2}s = \sigma^{2}\min\{s,t\}$ , and the same holds with $s$ and $t$ exchanged.

The finite-dimensional distributions follow: $(W(t_1),\dots,W(t_k))$ is multivariate normal because $W$ is a Gaussian process, its mean vector is zero, and its covariance matrix has just been computed.

Step 2: increments of the rescaled walk. Fix $0 \le t_1 < \cdots < t_k$ and put $t_0 = 0$ , so $W_n(t_0) = 0$ . Define the increments

D_{n,j} \ = \ W_n(t_j) - W_n(t_{j-1}) \ = \ \frac{1}{\sqrt n}\sum_{i = \lfloor n t_{j-1}\rfloor + 1}^{\lfloor n t_j \rfloor} X_i , \qquad j = 1, \dots, k .

For distinct $j$ these sums use disjoint blocks of the $X_i$ , so $D_{n,1}, \dots, D_{n,k}$ are independent for every $n$ - this is where the independence of the steps is used, and it is exact, not asymptotic.

Step 3: the characteristic function of each increment converges. Write $d_j = t_j - t_{j-1}$ . We show that for every $c \in \mathbb{R}$

\mathbb{E}\bigl( e^{i c D_{n,j}} \bigr) \ \longrightarrow \ \exp\left( -\tfrac12 c^{2}\sigma^{2} d_j \right) .

The statement allows $t_1 = 0$ , and then $d_1 = 0$ and the sum defining $D_{n,1}$ is empty, so $D_{n,1} = 0$ for every $n$ and both sides above equal $1$ ; that case is settled, and we assume $d_j > 0$ from here. Put $m_n = \lfloor n t_j\rfloor - \lfloor n t_{j-1}\rfloor$ , so that $m_n/n \to d_j > 0$ and hence $m_n \to \infty$ . Then

D_{n,j} \ = \ a_n Y_n , \qquad a_n = \sqrt{\frac{m_n}{n}} , \qquad Y_n = \frac{1}{\sqrt{m_n}} \sum_{i=\lfloor n t_{j-1}\rfloor+1}^{\lfloor n t_j\rfloor} X_i .

Here $Y_n$ is the sum of $m_n$ independent copies of $X_1$ divided by $\sqrt{m_n}$ , which is $\sigma$ times the sum standardised in the usual way (divided by $\sigma\sqrt{m_n}$ ). Since $X_1$ is centred and has finite non-zero variance, the central limit theorem sends that standardised sum to $N(0,1)$ , so $Y_n \xrightarrow{\mathrm{D}} N(0,\sigma^{2})$ and, by the direct half of the continuity theorem for characteristic functions (Characteristic functions), $\mathbb{E}(e^{i\theta Y_n}) \to e^{-\theta^{2}\sigma^{2}/2}$ for each fixed $\theta$ .

The factor $a_n$ is deterministic and tends to $\sqrt{d_j}$ , but it moves with $n$ , so the convergence just quoted cannot be applied at the argument $c a_n$ directly. Two elementary bounds remove the difficulty. First, $Y_n$ has mean $0$ and variance exactly $\sigma^{2}$ for every $n$ , so $\mathbb{E}\lvert Y_n\rvert \le \sigma$ by the Cauchy-Schwarz inequality. Second, $\lvert e^{iu} - 1\rvert \le \lvert u\rvert$ for real $u$ . Hence

\Bigl\lvert \mathbb{E}\bigl(e^{i c a_n Y_n}\bigr) - \mathbb{E}\bigl(e^{i c \sqrt{d_j}\, Y_n}\bigr) \Bigr\rvert \ \le \ \mathbb{E}\Bigl\lvert e^{i c (a_n - \sqrt{d_j}) Y_n} - 1 \Bigr\rvert \ \le \ \lvert c\rvert\,\bigl\lvert a_n - \sqrt{d_j}\bigr\rvert\, \mathbb{E}\lvert Y_n\rvert \ \le \ \lvert c\rvert\,\sigma\,\bigl\lvert a_n - \sqrt{d_j}\bigr\rvert \ \longrightarrow \ 0 ,

while $\mathbb{E}(e^{i c \sqrt{d_j} Y_n}) \to e^{-c^{2}d_j\sigma^{2}/2}$ by the previous paragraph with the fixed argument $\theta = c\sqrt{d_j}$ . Adding the two statements gives the display at the head of this step, and with it $D_{n,j} \xrightarrow{\mathrm{D}} N(0,\sigma^{2}d_j)$ .

Step 4: assembling the joint limit. Let $\theta_1, \dots, \theta_k$ be real and set $c_j = \theta_j + \theta_{j+1} + \cdots + \theta_k$ . Writing each $W_n(t_j)$ as the sum of the increments up to $j$ and collecting terms,

\sum_{j=1}^{k} \theta_j W_n(t_j) \ = \ \sum_{j=1}^{k} c_j D_{n,j} ,

so by the independence of step 2 and the limit of step 3,

\mathbb{E}\exp\left( i\sum_{j=1}^{k}\theta_j W_n(t_j) \right) = \prod_{j=1}^{k} \mathbb{E}\bigl(e^{i c_j D_{n,j}}\bigr) \ \longrightarrow \ \exp\left( -\tfrac12 \sum_{j=1}^{k} c_j^{2}\sigma^{2} d_j \right) .

The identical algebra applied to a Wiener process $W$ with parameter $\sigma^2$ , whose increments over $[t_{j-1}, t_j]$ are independent and $N(0, \sigma^2 d_j)$ , shows that the right-hand side is exactly the characteristic function of $(W(t_1),\dots,W(t_k))$ evaluated at $(\theta_1,\dots,\theta_k)$ . Pointwise convergence of characteristic functions to a characteristic function gives convergence in distribution, by the continuity theorem. $\square$

Proof(Existence of a version with continuous paths)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.3, Theorem (13.3.19) and its proof, pp. 587-589

We construct the standard process on $[0,1]$ ; the general case follows by scaling ( $W(t) = \sigma W_1(t) + w$ ) and by joining copies on $[n, n+1]$ end to end.

Start from a Gaussian process $X$ on $[0,1]$ with $X(0) = 0$ , independent increments and $X(s+t) - X(s) \sim N(0,t)$ . Such a process exists; the issue is its paths. Let $\mathcal{D}_n = \{m2^{-n} : 0 \le m \le 2^{n}\}$ be the level- $n$ dyadic points and $\mathcal{D} = \bigcup_n \mathcal{D}_n$ .

Step 1: the approximating polygons. Let $X_n$ agree with $X$ on $\mathcal{D}_n$ and be linear in between, so $X_n$ is a continuous piecewise-linear function with $2^n$ segments. Put $X_0 \equiv 0$ and $Z_j = X_j - X_{j-1}$ , so that

X_n \ = \ \sum_{j=1}^{n} Z_j .

Since $X_j$ and $X_{j-1}$ agree at every point of $\mathcal{D}_{j-1}$ , the function $Z_j$ vanishes there; it is piecewise linear with corners only at points of $\mathcal{D}_j$ . A piecewise-linear function attains its extreme values at its corners, so

\sup_{0 \le t \le 1} \lvert Z_j(t)\rvert \ = \ \max\bigl\{ \lvert Z_j(m2^{-j})\rvert : m \text{ odd},\ 1 \le m \le 2^{j}-1 \bigr\} ,

a maximum over $2^{j-1}$ points.

(One level is exceptional, and we follow Grimmett and Stirzaker in passing over it. With the convention $X_0 \equiv 0$ the function $Z_1 = X_1$ does not vanish on $\mathcal{D}_0 = \{0,1\}$ , since $Z_1(1) = X(1)$ ; so at $j=1$ the supremum is a maximum over $\{\tfrac12, 1\}$ rather than over the single odd point, and the two values have variances $\tfrac12$ and $1$ rather than the $2^{-j-1} = \tfrac14$ that step 2 will give. The slip is harmless, because a single level contributes the one almost surely finite quantity $\sup_t\lvert Z_1(t)\rvert$ to the sum in step 4, and the estimates of steps 2 and 3 are needed only for $j \ge 2$ , where the description above is exact. Taking $X_0$ to be the linear interpolant of $X$ on $\mathcal{D}_0$ instead of $0$ would remove the exception altogether.)

Step 2: the size of one correction. For odd $m$ , $X_{j-1}$ interpolates linearly between the neighbouring even dyadics, so

Z_j(m2^{-j}) = X(m2^{-j}) - \tfrac12\Bigl( X\bigl((m-1)2^{-j}\bigr) + X\bigl((m+1)2^{-j}\bigr) \Bigr) = \tfrac12 (A - B) ,

where $A = X(m2^{-j}) - X((m-1)2^{-j})$ and $B = X((m+1)2^{-j}) - X(m2^{-j})$ are increments over adjacent intervals of length $2^{-j}$ . By conditions 1 and 2 they are independent and each is $N(0, 2^{-j})$ , so $Z_j(m2^{-j})$ is normal with mean $0$ and variance $\tfrac14(2^{-j} + 2^{-j}) = 2^{-j-1}$ .

Step 3: a tail bound, level by level. For $Y \sim N(0,v)$ and $x > 0$ the standard tail estimate gives $\mathbb{P}(\lvert Y\rvert > x) \le (2\sqrt{v}/(x\sqrt{2\pi}))e^{-x^{2}/(2v)}$ . With $v = 2^{-j-1}$ this is at most $x^{-1}2^{-j/2}e^{-x^{2}2^{j}}$ . Combining with step 1 and the union bound over the $2^{j-1}$ odd points,

\mathbb{P}\left( \sup_{0\le t\le 1} \lvert Z_j(t)\rvert > x \right) \ \le \ \frac{2^{j-1}}{x\,2^{j/2}}\, e^{-x^{2}2^{j}} .

Choose the threshold $x_j = c\sqrt{j\,2^{-j}\log 2}$ with $c > 1$ fixed. Then $x_j^{2}2^{j} = c^{2}j\log 2$ , so $e^{-x_j^2 2^j} = 2^{-c^{2}j}$ and $x_j 2^{j/2} = c\sqrt{j\log 2}$ , whence

\mathbb{P}\left( \sup_{0 \le t \le 1} \lvert Z_j(t)\rvert > x_j \right) \ \le \ \frac{2^{\,j(1-c^{2})-1}}{c\sqrt{j\log 2}} .

Because $c > 1$ the exponent $j(1-c^{2})$ is negative and linear in $j$ , so these probabilities form a convergent geometric-type series.

Step 4: Borel-Cantelli, and uniform convergence. By the first Borel-Cantelli lemma (Convergence of random variables), almost surely only finitely many $j$ have $\sup_t\lvert Z_j(t)\rvert > x_j$ . Since $\sum_j x_j = c\sqrt{\log 2}\sum_j \sqrt{j}\,2^{-j/2} < \infty$ , it follows that

\sum_{j\ge1} \ \sup_{0\le t\le 1} \lvert Z_j(t)\rvert \ < \ \infty \qquad \text{almost surely.}

On that event the series $\sum_j Z_j$ converges uniformly on $[0,1]$ by the Weierstrass criterion, so $X_n$ converges uniformly to a limit $Y$ , and $Y$ is continuous as a uniform limit of continuous functions.

Step 5: $Y$ has the right law. If $t \in \mathcal{D}$ then $X_n(t) = X(t)$ for all large $n$ , so $Y(t) = X(t)$ . For general $t$ , take dyadics $s \to t$ ; then $\mathbb{E}\bigl((X(s)-X(t))^{2}\bigr) = \lvert s-t\rvert \to 0$ , so $X(s) \to X(t)$ in mean square, while $X(s) = Y(s) \to Y(t)$ almost surely by continuity of $Y$ . Mean-square and almost-sure limits agree almost surely, so $Y(t) = X(t)$ almost surely for every fixed $t$ . Hence $Y$ has the same finite-dimensional distributions as $X$ , so conditions 1 and 2 hold for $Y$ .

Step 6: removing the exceptional set. Let $\Omega'$ be the set of $\omega$ for which $t\mapsto Y(t;\omega)$ is continuous; steps 4 and 5 give $\mathbb{P}(\Omega') = 1$ . Replace $\Omega$ by $\Omega'$ with the induced $\sigma$ -field and probability measure. No probability changes, and now every path is continuous, so condition 3 holds as stated. $\square$

Proof(Quadratic variation, and the divergence of the total variation)Original

Fix $t > 0$ and write $\Delta_j = W((j+1)t/n) - W(jt/n)$ for $j = 0,\dots,n-1$ , so $Q_n = \sum_j \Delta_j^{2}$ .

Step 1: mean and variance of $Q_n$ . The $\Delta_j$ are increments over disjoint intervals of equal length $t/n$ , hence independent and each $N(0, \sigma^{2}t/n)$ . Therefore $\mathbb{E}(\Delta_j^{2}) = \sigma^{2}t/n$ and, using $\mathbb{E}(Y^{4}) = 3v^{2}$ for $Y \sim N(0,v)$ ,

\mathrm{var}(\Delta_j^{2}) = 3\left(\frac{\sigma^{2}t}{n}\right)^{2} - \left(\frac{\sigma^{2}t}{n}\right)^{2} = \frac{2\sigma^{4}t^{2}}{n^{2}} .

Summing over the $n$ independent terms,

\mathbb{E}(Q_n) = \sigma^{2}t , \qquad \mathrm{var}(Q_n) = \frac{2\sigma^{4}t^{2}}{n} .

Step 2: mean-square convergence. $\mathbb{E}\bigl((Q_n - \sigma^{2}t)^{2}\bigr) = \mathrm{var}(Q_n) = 2\sigma^{4}t^{2}/n \to 0$ , which is convergence in mean square to the constant $\sigma^{2}t$ . Note that the limit is deterministic although every $Q_n$ is random: the fluctuation is averaged away, the mean is not.

Step 3: almost sure convergence along $n = 2^{k}$ . By Chebyshev's inequality (Markov, Chebyshev, Jensen and Chernoff inequalities), for $\epsilon > 0$ ,

\mathbb{P}\bigl( \lvert Q_{2^{k}} - \sigma^{2}t\rvert > \epsilon \bigr) \ \le \ \frac{2\sigma^{4}t^{2}}{\epsilon^{2}\,2^{k}} ,

and these are summable in $k$ . By the first Borel-Cantelli lemma the event on the left occurs for only finitely many $k$ , almost surely, for each fixed $\epsilon$ ; taking $\epsilon = 1/i$ and intersecting the countably many full-probability events gives $Q_{2^{k}} \to \sigma^{2}t$ almost surely. (Summability is what is needed here; mean-square convergence alone would only give a subsequence.)

Step 4: infinite total variation on one fixed interval. Work on $[0,t]$ ; the argument on a general $[a,b]$ is the same with the origin shifted, because $s \mapsto W(a+s) - W(a)$ is again a Wiener process with the same parameter, by conditions 1 and 2. Let $V$ be the total variation of the path over $[0,t]$ and bound the quadratic variation by factoring out the largest increment:

Q_n \ = \ \sum_{j} \Delta_j^{2} \ \le \ \Bigl( \max_{j} \lvert \Delta_j\rvert \Bigr) \sum_{j} \lvert \Delta_j \rvert \ \le \ \Bigl( \max_{j} \lvert \Delta_j\rvert \Bigr) \, V .

Suppose $V < \infty$ on a path. The path is continuous on the compact interval $[0,t]$ by condition 3, hence uniformly continuous there, so $\max_j \lvert\Delta_j\rvert \to 0$ as the mesh $t/n$ tends to $0$ . The displayed bound then forces $Q_n \to 0$ . But by step 3, almost surely $Q_{2^{k}} \to \sigma^{2}t > 0$ . So $V = \infty$ almost surely.

Step 5: all intervals at once. Step 4 gives, for each fixed interval, a full-probability event on which the variation over it is infinite; part 2 of the statement claims one full-probability event that works for every interval simultaneously, and the two are not the same thing, since there are uncountably many intervals. Let $\Omega_{a,b}$ be the event that the variation over $[a,b]$ is infinite, and put

\Omega^{\ast} \ = \ \bigcap \bigl\{ \Omega_{a,b} \ : \ a < b, \ a, b \in \mathbb{Q}_{\ge 0} \bigr\} ,

an intersection of countably many events of probability $1$ , so $\mathbb{P}(\Omega^{\ast}) = 1$ . Fix $\omega \in \Omega^{\ast}$ and let $[a,b]$ be any interval with $a < b$ . Choose rationals $a'$ and $b'$ with $a \le a' < b' \le b$ . Total variation is monotone under inclusion of intervals, because every partition of $[a',b']$ extends to a partition of $[a,b]$ and adjoining points can only increase the sum of absolute increments; hence the variation of this path over $[a,b]$ is at least its variation over $[a',b']$ , which is infinite. So on $\Omega^{\ast}$ every interval carries infinite variation, which is part 2. $\square$

Remark(Where the hypotheses are used, and what fails without them)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.2, equation (13.2.1) with properties (a)-(d), pp. 578-579, §13.3, p. 581, and §8.7, p. 423

Independent increments is used three times and in three different ways: to kill the cross term in the covariance computation, to make the blocks $D_{n,j}$ of the rescaled walk independent (which is what lets a product of characteristic functions appear at all), and to make the midpoint corrections $Z_j$ of the existence proof normal with a known variance. Drop it and nothing in this unit survives. Fractional Brownian motion with Hurst exponent $H \ne \tfrac12$ is Gaussian and self-similar with stationary increments, and Grimmett and Stirzaker report that its sample paths "may be taken to be" almost surely continuous and nowhere differentiable; but the only value of the exponent at which its increments come out independent is the Wiener value $H = \tfrac12$ , and its covariance is not $\min\{s,t\}$ .

Finite non-zero variance of the step distribution is where the scaling limit is decided. The compression $S_{\lfloor nt\rfloor}/\sqrt n$ is tuned to $\sigma^2 < \infty$ : if the steps had infinite variance the central limit theorem would not apply and a different exponent, with a non-Gaussian stable limit, would be needed. Grimmett and Stirzaker make the same point in physical terms for the lattice walk of their §13.2: with lattice spacing $\delta$ and inter-jump time $\tau$ the limit is non-degenerate only if $\tfrac13\delta^{2}/\tau$ is held at a constant $\sigma^{2}$ as $\delta,\tau \downarrow 0$ . If $\delta^{2}/\tau \to 0$ the displacement collapses to the constant $0$ ; if $\delta^{2}/\tau \to \infty$ its variance diverges. The factor $\tfrac13$ there is the probability of stepping along the chosen axis in their three-dimensional lattice; in our one-dimensional statement the corresponding constant is $\sigma^{2}$ itself.

Zero mean of the steps keeps the limit centred. With $\mathbb{E}(X_1) = \mu \ne 0$ the same calculation applied to $(S_{\lfloor nt\rfloor} - \mu\lfloor nt\rfloor)/\sqrt n$ gives a Wiener process, while $S_{\lfloor nt\rfloor}/n \to \mu t$ : the walk is then dominated by its drift, and the Wiener process describes only the fluctuation around it.

Continuity of paths is an extra condition, not a consequence. The finite-dimensional distributions do not determine it. Take a Wiener process $W$ and an independent random time $U$ , uniform on $[0,1]$ , and define $\widetilde W(t) = W(t)$ for $t \ne U$ and $\widetilde W(U) = W(U) + 1$ . For each fixed $t$ we have $\mathbb{P}(U = t) = 0$ and hence $\mathbb{P}(\widetilde W(t) = W(t)) = 1$ , so every finite-dimensional distribution is unchanged; yet every path of $\widetilde W$ jumps at $U$ , so no path of it is continuous. This is exactly why part 3 of the existence statement has content, and why the proof of it is the long one.

Normality, not merely independence, fixes the size of the fluctuation. Step 1 of the last proof uses $\mathbb{E}(Y^{4}) = 3v^{2}$ , a property of the normal distribution; it is the normality supplied by condition 2, not merely independence, that gives $\mathrm{var}(Q_n) = O(1/n)$ .

What the scaling-limit statement does not say. It is a statement about finitely many times at once. Convergence of the whole random path - as an element of the space of continuous functions, which would let one pass to the limit in quantities such as $\sup_{0\le s\le 1}W_n(s)$ - is Donsker's invariance principle, and it is not treated in the sources used here; our claim should not be read as asserting it.

Method(Computing with a Wiener process)Original

Almost every elementary computation reduces to four moves.

Re-express in increments. Rewrite the quantity in terms of $W(t_1) - W(0), W(t_2) - W(t_1), \dots$ over disjoint intervals. These are independent and normal, so the joint law is a product of one-dimensional normals.
Use the covariance for anything quadratic. For means, variances and covariances of linear or quadratic functionals, do not integrate densities; use linearity together with $\mathrm{cov}(W(s),W(t)) = \sigma^{2}\min\{s,t\}$ . For instance $\mathrm{var}(W(t)-W(s)) = \sigma^2 t - 2\sigma^2\min\{s,t\} + \sigma^2 s = \sigma^{2}\lvert t-s\rvert$ .
Use linearity to stay Gaussian. Any finite linear combination $\sum_j \theta_j W(t_j)$ is a one-dimensional normal with mean $0$ and variance $\sum_{i,j}\theta_i\theta_j\sigma^{2}\min\{t_i,t_j\}$ ; a limit in distribution of such combinations (for example a limit of Riemann sums) is normal too, since the normal family is closed under limits in distribution. So probabilities about such quantities are read off a single standard normal table.
Use the scaling law to remove parameters. For standard $W$ and $c > 0$ , the process $t \mapsto c^{-1/2}W(ct)$ is again a standard Wiener process, because it is Gaussian with mean zero and covariance $c^{-1}\min\{cs,ct\} = \min\{s,t\}$ and has continuous paths. (For a centred Gaussian process that covariance already forces the other two conditions: increments over disjoint intervals $[u,v]$ and $[s,t]$ with $u\le v\le s\le t$ have covariance $\min\{v,t\}-\min\{v,s\}-\min\{u,t\}+\min\{u,s\} = v-v-u+u = 0$ , and jointly normal variables with zero covariance are independent, while $\mathrm{var}(W(t)-W(s)) = \lvert t-s\rvert$ by move 2.) Use it to reduce a question on $[0,T]$ to one on $[0,1]$ .

A fifth move is to use the transition density when a genuine distribution function is wanted: conditionally on $W(s) = x$ , $W(t)$ is $N(x, \sigma^{2}(t-s))$ for $t > s$ .

Example(The time-average of a Brownian path)Original

Let $W$ be a standard Wiener process ( $\sigma^{2} = 1$ , $W(0) = 0$ ) and let

A \ = \ \int_0^1 W(s)\,ds .

This integral makes sense path by path: every path is continuous by condition 3, hence Riemann integrable on $[0,1]$ . We find the law of $A$ , its correlation with the endpoint $W(1)$ , and a numerical probability.

Step 1: $A$ is normal. Let $A_n = \frac1n\sum_{r=1}^{n} W(r/n)$ be the right-endpoint Riemann sum. Each $A_n$ is a finite linear combination of the $W(r/n)$ , hence normal with mean $0$ by move 3 of the method. For each path, $A_n \to A$ as $n \to \infty$ by Riemann integrability, so $A_n \to A$ almost surely and therefore in distribution. The variances converge, since $\mathrm{var}(A_n) = n^{-2}\sum_{r,s \le n}\min\{r/n, s/n\}$ is a Riemann sum for the double integral evaluated in step 2, and a limit in distribution of $N(0,v_n)$ variables with $v_n \to v$ is $N(0,v)$ . So $A$ is $N(0,v)$ with $v$ as computed next.

Step 2: the variance. Using $\mathbb{E}(W(s)W(u)) = \min\{s,u\}$ and exchanging expectation with the integrals,

v = \mathbb{E}(A^{2}) = \int_0^1\!\!\int_0^1 \min\{s,u\}\,ds\,du = 2\int_0^1\!\!\int_0^{u} s \,ds\,du = 2\int_0^1 \frac{u^{2}}{2}\,du = \frac13 ,

where the factor $2$ comes from the symmetry of $\min\{s,u\}$ in the two variables. So

A \ \sim \ N\!\left(0, \tfrac13\right) , \qquad \mathrm{sd}(A) = 1/\sqrt3 = 0.577350 .

Step 3: correlation with the endpoint. Likewise

\mathrm{cov}\bigl(A, W(1)\bigr) = \int_0^1 \min\{s,1\}\,ds = \int_0^1 s\,ds = \tfrac12 ,

so, since $\mathrm{sd}(W(1)) = 1$ ,

\mathrm{corr}\bigl(A, W(1)\bigr) = \frac{1/2}{\sqrt{1/3}\cdot 1} = \frac{\sqrt3}{2} = 0.866025 .

The average over $[0,1]$ and the final value are strongly but not perfectly correlated, as one expects: the average is dominated by the later part of the path, where the path has already moved.

Step 4: a probability. The chance that the path spends its first unit of time above the level $\tfrac12$ on average is

\mathbb{P}\left( A > \tfrac12 \right) = \mathbb{P}\left( \frac{A}{1/\sqrt3} > \frac{\sqrt3}{2} \right) = 1 - \Phi\left(\tfrac{\sqrt3}{2}\right) = 1 - \Phi(0.866025) = 0.193238 ,

with $\Phi$ the standard normal distribution function.

Step 5: a check on the quadratic variation. For the same process on $[0,3]$ with $n = 1000$ equal subintervals, the proof above gives $\mathbb{E}(Q_{1000}) = \sigma^{2}t = 3$ exactly and $\mathrm{sd}(Q_{1000}) = \sqrt{2\sigma^{4}t^{2}/n} = \sqrt{18/1000} = 0.134164$ . A simulated path should return a value of $Q_{1000}$ within about $0.27$ of $3$ roughly $95\%$ of the time - a sharp and easily checked prediction, and one that fails for any process with differentiable paths, for which $Q_n \to 0$ .

Application(Diffusion, the heat equation, and measuring molecules)Original

The transition density is the diffusion kernel. Let $u(y,t)$ be the probability density of the position at time $t$ of a particle whose displacement is a Wiener process with parameter $\sigma^{2}$ , started from a known point. By part 2 of the first statement, $u$ is the Gaussian kernel with variance $\sigma^{2}t$ , which is exactly the fundamental solution of

\frac{\partial u}{\partial t} = \frac{\sigma^{2}}{2}\,\frac{\partial^{2}u}{\partial y^{2}} ,

the diffusion equation with diffusion coefficient $D = \sigma^{2}/2$ (see The heat kernel). Strauss states the correspondence in the other direction: the probability that a particle started at $x$ lies in $(a,b)$ at time $t$ is the integral of the source function over that interval, so a concentration profile transported by Brownian motion obeys the diffusion equation. The general version of this dictionary, for processes with drift and with a position-dependent diffusion coefficient, is The Fokker-Planck equation; the particular case in which the velocity rather than the position diffuses gives the Ornstein-Uhlenbeck process of Stochastic differential equations.

Harmonic functions and the Dirichlet problem. Strauss gives the companion steady-state statement: for Brownian motion in a container $D$ , stopped the moment it reaches the boundary, the probability $u(x)$ that the particle stops on a chosen boundary piece $C_1$ satisfies $\Delta u = 0$ in $D$ with $u = 1$ on $C_1$ and $u = 0$ on the rest. This is a Dirichlet problem (Harmonic functions), and it is the basis of "walk on spheres" Monte Carlo solvers, which estimate the potential at a single point of a complicated domain by simulating paths rather than meshing the domain.

Einstein's relation and the size of a molecule. The physically measurable content of the model is $\mathbb{E}\bigl((W(t)-W(0))^{2}\bigr) = \sigma^{2}t = 2Dt$ : the mean square displacement grows linearly in time, not quadratically as it would for ballistic motion. For a sphere of radius $a$ in a fluid of viscosity $\eta$ at absolute temperature $T$ , the Stokes-Einstein relation gives $D = k_{\mathrm{B}}T/(6\pi\eta a)$ . For a bead of radius $a = 0.5\,\mu\mathrm{m}$ in water at $20\,^{\circ}\mathrm{C}$ ( $\eta = 1.002\times10^{-3}\,\mathrm{Pa\,s}$ , $T = 293\,\mathrm{K}$ , $k_{\mathrm{B}} = 1.380649\times10^{-23}\,\mathrm{J\,K^{-1}}$ ),

D = \frac{(1.380649\times10^{-23})(293)}{6\pi(1.002\times10^{-3})(0.5\times10^{-6})} = 4.2836\times10^{-13}\ \mathrm{m^{2}\,s^{-1}} = 0.4284\ \mu\mathrm{m^{2}\,s^{-1}} ,

so after $10$ seconds the root-mean-square displacement along one axis is $\sqrt{2Dt} = \sqrt{8.567\times10^{-12}} = 2.93\,\mu\mathrm{m}$ . Running this backwards - measure the displacements, infer $D$ , and read off $k_{\mathrm{B}}$ and hence Avogadro's number - is Perrin's experiment, and it is why a theorem about path regularity has a place in a physics handbook.

Where the irregularity matters. The infinite total variation of part 2 of the path statement is not a curiosity. It means that integrals $\int H(s)\,dW(s)$ cannot be defined path by path as Stieltjes integrals, which is why a stochastic integral must be built as a limit in mean square instead, and why its chain rule carries the extra second-order term supplied by the quadratic variation $\sigma^{2}t$ . That is the subject of The Ito integral and Ito's formula. It also means that a numerical simulation on a time grid of size $h$ reproduces increments of size $\sqrt h$ , not $h$ : halving the step size reduces the step length only by a factor $\sqrt2$ , which is also why the error analysis of a Monte Carlo estimate of such a path functional behaves differently from that of an ordinary quadrature (Monte Carlo methods).

Definition

Martingales

Adapted

Add to PDF

Defines filtrations, martingales, sub- and supermartingales and stopping times, proves that a stopped martingale is a martingale and then the optional stopping theorem in two usable forms, shows by the doubling strategy exactly which hypothesis fails when the conclusion does, and uses the theorem to compute the mean waiting time for a pattern in coin tosses.

Prerequisites: Probability spaces, Random variables and distributions, Expectation, variance and covariance, Conditional expectation, Markov, Chebyshev, Jensen and Chernoff inequalities, Convergence of random variables, Monotone convergence and Fatou's lemma, Dominated convergence

Note(What a martingale is for)Original

A martingale is the mathematics of a fair game: whatever has happened so far, the expected change at the next step is zero. That single property is weaker than independence and weaker than the Markov property, and it is nonetheless enough to prove convergence theorems, concentration inequalities and - the subject of this unit - a rule for exchanging a random stopping time with an expectation.

The use is almost always the same. One wants a quantity attached to a random time: the chance of hitting one barrier before another, the mean number of steps until a pattern appears, the probability that a running statistic ever crosses a threshold. Direct computation means summing over all the ways the history could have gone. Instead one finds a martingale, argues that its mean cannot have changed by the time one stops, and reads off the answer from a single equation. The catch is that the mean can change: a gambler who can borrow without limit and who stops as soon as he is ahead is certain to end ahead. The theorems below say precisely which hypothesis that gambler violates.

Definition(Filtrations, martingales, and stopping times)AdaptedAdapted from Grimmett & Stirzaker (2020), Ch. 12, Definitions (12.1.1), (12.1.8) and (12.1.9), pp. 533-536, and Definition (12.4.1) with the construction of $\mathcal{F}_T$, p. 550

Throughout, $(\Omega,\mathcal{F},\mathbb{P})$ is a probability space and time runs over $n = 0,1,2,\dots$ .

Filtration, adapted. A filtration is a family $F = \{\mathcal{F}_n : n \ge 0\}$ of sub- $\sigma$ -fields of $\mathcal{F}$ that increases, $\mathcal{F}_n \subseteq \mathcal{F}_{n+1}$ . Read $\mathcal{F}_n$ as everything an observer knows at time $n$ . A sequence $Y = \{Y_n\}$ is adapted to $F$ if each $Y_n$ is $\mathcal{F}_n$ -measurable, that is, if $Y_n$ is known at time $n$ . The natural filtration of a sequence $X = \{X_n\}$ is $\mathcal{F}_n = \sigma(X_0, X_1,\dots,X_n)$ .

Martingale. The adapted pair $(Y,F)$ is a martingale if, for every $n \ge 0$ ,

\mathbb{E}\lvert Y_n \rvert < \infty , \qquad \mathbb{E}\bigl( Y_{n+1} \mid \mathcal{F}_n \bigr) = Y_n .

Here $\mathbb{E}(\,\cdot \mid \mathcal{F}_n)$ is conditional expectation given a $\sigma$ -field (Conditional expectation); it is a random variable, and the identity, like every identity between conditional expectations in this unit, is asserted up to an event of probability zero. Taking expectations in the defining identity gives $\mathbb{E}(Y_n) = \mathbb{E}(Y_0)$ for every $n$ : the mean of a martingale never moves.

When the filtration is the natural one of some observed sequence $X$ , one says that $Y$ is a martingale with respect to $X$ ; the condition is then $\mathbb{E}(Y_{n+1} \mid X_0,\dots,X_n) = Y_n$ . A finite family $\{Y_n : 0 \le n \le N\}$ satisfying the same conditions is also called a martingale.

Submartingale, supermartingale. The adapted pair $(Y,F)$ is a submartingale if $\mathbb{E}(Y_n^{+}) < \infty$ and $\mathbb{E}(Y_{n+1}\mid\mathcal{F}_n) \ge Y_n$ for all $n$ , and a supermartingale if $\mathbb{E}(Y_n^{-}) < \infty$ and $\mathbb{E}(Y_{n+1}\mid\mathcal{F}_n) \le Y_n$ for all $n$ , where $x^{+} = \max\{0,x\}$ and $x^{-} = -\min\{0,x\}$ . Note that these moment conditions are one-sided, and so weaker than $\mathbb{E}\lvert Y_n\rvert<\infty$ . A sequence is a martingale exactly when it is both; $Y$ is a submartingale exactly when $-Y$ is a supermartingale. The names are the opposite way round from what the inequalities suggest; they come from the theory of harmonic functions, where a subharmonic function lies below the harmonic one with the same boundary values.

Stopping time. A random variable $T$ with values in $\{0,1,2,\dots\}\cup\{\infty\}$ is a stopping time for $F$ if

\{T = n\} \in \mathcal{F}_n \qquad \text{for every } n \ge 0 ,

equivalently if $\{T \le n\} \in \mathcal{F}_n$ for every $n$ , and hence also $\{T > n\} \in \mathcal{F}_n$ . The content is that the decision to stop at time $n$ may use the first $n$ observations and nothing later. A stopping time is allowed to be infinite: "never" is a permitted value.

The standard example is a first passage time: if $X$ is adapted and $B \subseteq \mathbb{R}$ , then $T_B = \min\{n : X_n \in B\}$ (with $T_B = \infty$ if $X$ never enters $B$ ) is a stopping time, since $\{T_B = n\}$ is decided by $X_0,\dots,X_n$ . By contrast "the last time $X$ visits $B$ " is not a stopping time: deciding it requires the whole future.

The $\sigma$ -field at a stopping time. $\mathcal{F}_T$ is the collection of events $A$ with $A \cap \{T \le n\} \in \mathcal{F}_n$ for every $n$ . It is a $\sigma$ -field, and it is the information available at the random time $T$ .

We write $a \wedge b = \min\{a,b\}$ , so $Y_{T\wedge n}$ is the process watched until $T$ and then held fixed.

Definition(Stopping a martingale)AdaptedAdapted from Grimmett & Stirzaker (2020), §12.4, Theorem (12.4.5) and the optional sampling theorem (12.4.11), pp. 551-552, and §12.5, Theorems (12.5.1) and (12.5.9) with Example (12.5.12), pp. 554-556

Throughout this statement $T$ is a stopping time for the filtration $F$ , and $(Y,F)$ is a martingale.

A. The stopped process. $Z_n = Y_{T \wedge n}$ defines a martingale $(Z,F)$ ; if $(Y,F)$ is only a submartingale, the same holds with "submartingale" in place of "martingale". In particular

\mathbb{E}\bigl( Y_{T\wedge n} \bigr) \ = \ \mathbb{E}(Y_0) \qquad \text{for every finite } n .

If moreover $\mathbb{P}(T \le N) = 1$ for some deterministic $N$ , then $T\wedge N = T$ and so $\mathbb{E}(Y_T) = \mathbb{E}(Y_0)$ : for a bounded stopping time no further hypothesis is needed. This last statement is the optional sampling theorem, and it has a sharper form that is where the $\sigma$ -field $\mathcal{F}_T$ earns its place: if $T_1 \le T_2 \le \cdots$ are stopping times, each bounded by a deterministic constant, then the sampled pairs $(Y_{T_j}, \mathcal{F}_{T_j})$ again form a martingale. Watching a fair game at a sequence of bounded, non-anticipating times gives a fair game, with $\mathcal{F}_{T_j}$ as the information available at the $j$ th observation. The boundedness is essential: with $Y$ the symmetric simple random walk and $T_1$ its first passage to level $1$ , a single unbounded sampling time already gives $\mathbb{E}(Y_{T_1}) = 1 \ne 0 = \mathbb{E}(Y_0)$ .

B. Optional stopping theorem. $\mathbb{E}(Y_T) = \mathbb{E}(Y_0)$ provided all three of

$\mathbb{P}(T<\infty) = 1$ ,
$\mathbb{E}\lvert Y_T\rvert < \infty$ ,
$\mathbb{E}\bigl( Y_n \mathbf{1}_{\{T>n\}} \bigr) \to 0$ as $n \to \infty$ .

C. A sufficient condition that is easier to check. $\mathbb{E}(Y_T) = \mathbb{E}(Y_0)$ provided

$\mathbb{P}(T<\infty)=1$ and $\mathbb{E}(T) < \infty$ , and
there is a constant $c$ with $\mathbb{E}\bigl( \lvert Y_{n+1}-Y_n \rvert \mid \mathcal{F}_n \bigr) \le c$ for all $n < T$ .

D. Wald's equation. Let $X_1,X_2,\dots$ be independent and identically distributed with $\mathbb{E}\lvert X_1\rvert < \infty$ and mean $\mu$ , let $S_n = X_1+\cdots+X_n$ , and let $T$ be a stopping time with $\mathbb{E}(T)<\infty$ for the filtration $\mathcal{F}_n = \sigma(X_1,\dots,X_n)$ . Then

\mathbb{E}(S_T) \ = \ \mu\, \mathbb{E}(T) .

Grimmett and Stirzaker state this with the filtration generated by the martingale $Y_n = S_n - n\mu$ rather than by the data, that is with $\sigma(Y_1,\dots,Y_n)$ in place of $\sigma(X_1,\dots,X_n)$ . The two are the same $\sigma$ -field: $Y_n$ is a function of $X_1,\dots,X_n$ , and conversely $S_n = Y_n + n\mu$ and $X_n = S_n - S_{n-1}$ recover the data from the $Y$ 's. So nothing turns on the choice, and we use the natural filtration of the data throughout.

Version C is the one used in practice: both of its hypotheses are statements about the step sizes and about $T$ alone, and neither mentions $Y_T$ .

Idea of the proofOriginal

Stopping does not break fairness. Between times $n$ and $n+1$ the stopped process either moves exactly as $Y$ does, on the event $\{T>n\}$ , or does not move at all. Which of the two happens is decided by time $n$ , because $T$ is a stopping time; so, conditionally on $\mathcal{F}_n$ , the increment is either a fair increment or zero, and in both cases has conditional mean zero. Nothing is needed beyond the definition of a stopping time, which is exactly why this part is free.

Why that is not yet the theorem. Part A controls $Y$ at the time $T\wedge n$ , not at $T$ . The two differ only on the event $\{T>n\}$ , whose probability tends to zero; but "the event is unlikely" is not "the contribution is small", because $Y_n$ may be huge precisely there. The one-line identity

Y_T \ = \ Y_{T\wedge n} + \bigl(Y_T - Y_n\bigr)\mathbf{1}_{\{T>n\}}

isolates the whole difficulty in the two terms $\mathbb{E}(Y_T \mathbf{1}_{\{T>n\}})$ and $\mathbb{E}(Y_n\mathbf{1}_{\{T>n\}})$ . The first dies automatically under hypothesis 2, being the tail of a convergent sum; the second is hypothesis 3, and the examples show it is a genuine extra demand.

Why bounded increments and a finite mean suffice. Hypothesis 3 is awkward to verify, so one looks for a single dominating random variable. The total movement of the stopped process never exceeds the sum of all the increments up to $T$ , so $\lvert Y_{T\wedge n}\rvert \le \lvert Y_0\rvert + \sum_{i\le T}\lvert Y_i - Y_{i-1}\rvert$ . Bounding each conditional increment by $c$ and summing over $i$ with the weights $\mathbb{P}(T \ge i)$ turns the right-hand side into $c\,\mathbb{E}(T)$ , which is finite. A dominated sequence may be passed to the limit, and that is version C.

Proof(A: the stopped process is a martingale)AdaptedAdapted from Grimmett & Stirzaker (2020), §12.4, proof of Theorem (12.4.5), p. 551

Let $Z_n = Y_{T\wedge n}$ .

Step 1: $Z_n$ is $\mathcal{F}_n$ -measurable. Split according to the value of $T$ :

Z_n \ = \ \sum_{t=0}^{n-1} Y_t \mathbf{1}_{\{T=t\}} + Y_n \mathbf{1}_{\{T \ge n\}} .

Each $Y_t$ with $t \le n$ is $\mathcal{F}_n$ -measurable because $Y$ is adapted and the filtration increases; each indicator is $\mathcal{F}_n$ -measurable because $T$ is a stopping time, using $\{T \ge n\} = \{T \le n-1\}^{c}$ . Hence so is the sum.

Step 2: $Z_n$ satisfies the moment condition. Here the two cases must be kept apart, because the definitions impose different moment conditions. If $Y$ is a martingale, the display shows $\lvert Z_n\rvert \le \sum_{t=0}^{n}\lvert Y_t\rvert$ , a finite sum of integrable variables, so $\mathbb{E}\lvert Z_n\rvert<\infty$ . If $Y$ is only a submartingale we may not assume $\mathbb{E}\lvert Y_t\rvert<\infty$ - a submartingale is required to satisfy only the one-sided condition $\mathbb{E}(Y_t^{+})<\infty$ - and the bound above is unavailable. But $Z_n$ equals one of $Y_0,\dots,Y_n$ pointwise, so $Z_n^{+} \le \sum_{t=0}^{n} Y_t^{+}$ , whence

\mathbb{E}\bigl( Z_n^{+} \bigr) \ \le \ \sum_{t=0}^{n} \mathbb{E}\bigl( Y_t^{+} \bigr) \ < \ \infty ,

which is exactly the one-sided condition a submartingale must satisfy. This is the bound Grimmett and Stirzaker use, and it is the reason their theorem is stated for submartingales in the first place.

Step 3: the increment. Subtracting the displays for $n+1$ and $n$ , every term cancels except on the event that $T$ has not yet occurred:

Z_{n+1} - Z_n \ = \ \bigl( Y_{n+1} - Y_n \bigr)\, \mathbf{1}_{\{T > n\}} .

The indicator is $\mathcal{F}_n$ -measurable, so it may be taken outside the conditional expectation:

\mathbb{E}\bigl( Z_{n+1} - Z_n \mid \mathcal{F}_n \bigr) \ = \ \mathbf{1}_{\{T>n\}}\, \mathbb{E}\bigl( Y_{n+1}-Y_n \mid \mathcal{F}_n \bigr) \ = \ 0 ,

the last step by the martingale property of $Y$ . (If $Y$ is only a submartingale the final bracket is non-negative, and since the indicator is non-negative the product is too, which gives the submartingale version.) Taking expectations in $\mathbb{E}(Z_{n+1}\mid\mathcal{F}_n) = Z_n$ gives $\mathbb{E}(Y_{T\wedge n}) = \mathbb{E}(Z_0) = \mathbb{E}(Y_0)$ .

Step 4: bounded stopping times. If $\mathbb{P}(T\le N)=1$ then $Y_{T\wedge N} = Y_T$ almost surely, and step 3 at $n=N$ gives $\mathbb{E}(Y_T) = \mathbb{E}(Y_0)$ . $\square$

Proof(B and C: the optional stopping theorem)AdaptedAdapted from Grimmett & Stirzaker (2020), §12.5, proofs of Theorems (12.5.1) and (12.5.9), pp. 554-556

Part B. Assume hypotheses 1-3. Because $\mathbb{P}(T<\infty)=1$ , the variable $Y_T$ is defined almost surely, and splitting according to whether $T$ exceeds $n$ ,

Y_T \ = \ Y_{T\wedge n} \ + \ \bigl( Y_T - Y_n \bigr)\mathbf{1}_{\{T>n\}} .

Every term is integrable - $Y_T$ by hypothesis 2, $Y_n$ and $Y_{T\wedge n}$ by step 2 of the previous proof - so expectations may be taken term by term, and part A replaces $\mathbb{E}(Y_{T\wedge n})$ by $\mathbb{E}(Y_0)$ :

\mathbb{E}(Y_T) \ = \ \mathbb{E}(Y_0) + \mathbb{E}\bigl( Y_T\mathbf{1}_{\{T>n\}} \bigr) - \mathbb{E}\bigl( Y_n\mathbf{1}_{\{T>n\}} \bigr) .

This holds for every $n$ , so it holds in the limit. The third term tends to $0$ : that is hypothesis 3. For the second, decompose over the value of $T$ ,

\mathbb{E}\bigl( Y_T \mathbf{1}_{\{T>n\}} \bigr) \ = \ \sum_{k=n+1}^{\infty} \mathbb{E}\bigl( Y_T \mathbf{1}_{\{T=k\}}\bigr) ,

which is the $n$ th tail of the series $\sum_{k\ge0}\mathbb{E}(Y_T\mathbf{1}_{\{T=k\}})$ . Hypothesis 2 makes that series absolutely convergent (its absolute terms sum to $\mathbb{E}\lvert Y_T\rvert$ ), and hypothesis 1 makes it sum to $\mathbb{E}(Y_T)$ , so its tails tend to $0$ . Letting $n\to\infty$ leaves $\mathbb{E}(Y_T)=\mathbb{E}(Y_0)$ .

Part C. Now assume instead that $\mathbb{E}(T)<\infty$ and that the conditional increments are bounded by $c$ before $T$ . Write $R_i = \lvert Y_i - Y_{i-1}\rvert$ for $i \ge 1$ and

R \ = \ \sum_{i=1}^{T} R_i \ = \ \sum_{i=1}^{\infty} R_i \mathbf{1}_{\{T \ge i\}} .

Step 1: $R$ has finite mean. The event $\{T\ge i\} = \{T\le i-1\}^{c}$ lies in $\mathcal{F}_{i-1}$ , so the indicator comes outside the conditional expectation and hypothesis 2 applies on it:

\mathbb{E}\bigl( R_i \mathbf{1}_{\{T\ge i\}} \mid \mathcal{F}_{i-1}\bigr) \ = \ \mathbf{1}_{\{T \ge i\}}\, \mathbb{E}\bigl( R_i \mid \mathcal{F}_{i-1}\bigr) \ \le \ c\,\mathbf{1}_{\{T\ge i\}} .

Taking expectations and summing over $i$ , with monotone convergence (Monotone convergence and Fatou's lemma) to exchange the sum and the expectation of non-negative terms,

\mathbb{E}(R) \ \le \ c\sum_{i\ge1} \mathbb{P}(T\ge i) \ = \ c\,\mathbb{E}(T) \ < \ \infty .

Step 2: domination. For every $n$ , the stopped process has moved from $Y_0$ by at most the total of the increments it has used, so $\lvert Y_{T\wedge n}\rvert \le \lvert Y_0\rvert + R$ , and the dominating variable $\lvert Y_0\rvert + R$ is integrable and does not depend on $n$ .

Step 3: pass to the limit. Since $\mathbb{E}(T)<\infty$ we have $T<\infty$ almost surely, so $Y_{T\wedge n}\to Y_T$ almost surely. Dominated convergence (Dominated convergence) applies by step 2, so $\mathbb{E}(Y_{T\wedge n}) \to \mathbb{E}(Y_T)$ , while part A says every term of that sequence equals $\mathbb{E}(Y_0)$ . Hence $\mathbb{E}(Y_T)=\mathbb{E}(Y_0)$ .

Part D. Put $Y_n = S_n - n\mu$ with $Y_0 = 0$ and $\mathcal{F}_n = \sigma(X_1,\dots,X_n)$ . Then $\mathbb{E}(Y_{n+1}-Y_n\mid\mathcal{F}_n) = \mathbb{E}(X_{n+1}) - \mu = 0$ by independence, so $(Y,F)$ is a martingale; and

\mathbb{E}\bigl(\lvert Y_{n+1}-Y_n\rvert \mid \mathcal{F}_n\bigr) \ = \ \mathbb{E}\lvert X_{n+1}-\mu\rvert \ = \ \mathbb{E}\lvert X_1 - \mu\rvert \ \le \ 2\,\mathbb{E}\lvert X_1\rvert \ < \ \infty ,

a constant, so hypothesis C2 holds with that constant. With $\mathbb{E}(T)<\infty$ , part C gives $\mathbb{E}(Y_T) = 0$ , which is $\mathbb{E}(S_T) = \mu\,\mathbb{E}(T)$ . $\square$

Remark(Where the hypotheses are used: the doubling strategy)AdaptedAdapted from Grimmett & Stirzaker (2020), §12.1, Example (12.1.3), pp. 533-534, and §12.4, the symmetric random walk stopped at level 1 as an instance of failed optional sampling, p. 552

One example breaks every version of the theorem at once, and it is worth tracking exactly which hypothesis it violates.

A gambler stakes $1$ on an even-money bet. After each loss he doubles: having lost the first $n$ plays, he stakes $2^{n}$ on play $n+1$ . He stops at $T$ , the first play he wins. Let $Y_n$ be his accumulated gain after $n$ plays, with $Y_0 = 0$ .

$(Y,F)$ is a martingale: before stopping, the next play adds $+2^{n}$ or $-2^{n}$ with probability $\tfrac12$ each, so the conditional mean increment is zero; after stopping it is zero outright.
$T$ is a stopping time, $\mathbb{P}(T=n) = 2^{-n}$ for $n \ge 1$ , so $\mathbb{P}(T<\infty)=1$ and even $\mathbb{E}(T) = \sum_n n2^{-n} = 2 < \infty$ . Hypotheses B1 and C1 hold.
$Y_T = 2^{T-1} - (1+2+\cdots+2^{T-2}) = 1$ with probability $1$ , so $\mathbb{E}\lvert Y_T\rvert = 1 < \infty$ and hypothesis B2 holds.
And yet $\mathbb{E}(Y_T) = 1 \ne 0 = \mathbb{E}(Y_0)$ .

So hypothesis B3 is what fails, and it fails by exactly the right amount. On $\{T>n\}$ the gambler has lost $n$ times, so $Y_n = -(1+2+\cdots+2^{n-1}) = -(2^{n}-1)$ there, while $\mathbb{P}(T>n)=2^{-n}$ . Hence

\mathbb{E}\bigl( Y_n \mathbf{1}_{\{T>n\}}\bigr) \ = \ -\bigl(2^{n}-1\bigr)2^{-n} \ = \ -\bigl(1 - 2^{-n}\bigr) \ \longrightarrow \ -1 ,

not $0$ ; substituting into the identity in the proof of part B gives $\mathbb{E}(Y_T) = 0 + 0 - (-1) = 1$ , the right answer. A vanishingly unlikely event carrying an exponentially large debt contributes a finite amount in the limit, which is the whole phenomenon.

Hypothesis C2 fails too, and only C2. On $\{T>n\}$ the next stake is $2^{n}$ , so $\mathbb{E}(\lvert Y_{n+1}-Y_n\rvert \mid \mathcal{F}_n) = 2^{n}$ , which no constant $c$ bounds. Since C1 holds, this isolates the bounded-increment condition as the one doing the work in version C. The practical reading is familiar: the strategy is a certainty only for a gambler with unbounded credit, and Grimmett and Stirzaker note that his mean accumulated loss just before the winning play is infinite.

A second counterexample, where C2 holds and C1 fails. Let $Y$ be a symmetric simple random walk started at $0$ and let $T = \min\{n : Y_n = 1\}$ (Random walks). The walk is recurrent, so $T<\infty$ almost surely and B1 holds; $Y_T = 1$ , so B2 holds; and the increments are bounded by $1$ , so C2 holds with $c = 1$ . Yet $\mathbb{E}(Y_T) = 1 \ne 0 = \mathbb{E}(Y_0)$ . What fails is $\mathbb{E}(T)<\infty$ , that is C1, and with it B3: since $Y_{T\wedge n}$ has mean $0$ and equals $1$ on $\{T\le n\}$ , we get $\mathbb{E}(Y_n\mathbf{1}_{\{T>n\}}) = -\mathbb{P}(T\le n) \to -1$ . Put beside the doubling strategy, where C1 held and C2 failed, this shows that neither of the two hypotheses of version C can be dropped in favour of the other.

Condition B1 is of a different kind. It is not traded against B2 and B3; it is what makes the conclusion mean anything, because if $\mathbb{P}(T=\infty)>0$ then $Y_T$ is undefined on an event of positive probability. The gap is not always repairable by a convention. Take $X_1,X_2,\dots$ independent with $\mathbb{P}(X_i=1) = p < \tfrac12$ and $\mathbb{P}(X_i=-1) = q = 1-p$ , let $S_n = X_1+\cdots+X_n$ with $S_0=0$ , and let $M_n = S_n - (p-q)n$ , a martingale with $M_0 = 0$ . Let $T$ be the first passage of $S$ to level $1$ . This walk drifts downwards, and the first-passage probability is $\mathbb{P}(T<\infty) = p/q < 1$ , so $\{T=\infty\}$ has probability $1 - p/q > 0$ . Now $M$ is itself a random walk whose steps $X_i - (p-q)$ are bounded, non-degenerate and of mean zero, and such a walk oscillates: almost surely $\limsup_n M_n = +\infty$ and $\liminf_n M_n = -\infty$ . Hence on $\{T=\infty\}$ the sequence $M_n$ has no limit, no value can be assigned to $M_T$ there by any limiting convention, and the statement $\mathbb{E}(M_T)=\mathbb{E}(M_0)$ cannot even be formulated, let alone rescued. B1 is a condition of well-posedness, while B2, B3, C1 and C2 are conditions that make a well-posed conclusion true.

What is never needed. Nothing in part A: for a bounded stopping time the conclusion is unconditional. Every counterexample must therefore use a stopping time that is unbounded, even if it is finite with probability one.

Method(Using optional stopping to compute a hitting quantity)Original

To find a hitting probability, a mean hitting time, or the mean value of some functional at a random time:

Name the filtration and the stopping time. Usually $\mathcal{F}_n$ is generated by the observations up to time $n$ , and $T$ is a first passage time, which is automatically a stopping time.
Find a martingale whose value at $T$ encodes the answer. For a hitting probability, look for a function $\psi$ of the state with $\mathbb{E}(\psi(X_{n+1})\mid\mathcal{F}_n) = \psi(X_n)$ - a harmonic function for the dynamics - so that $\psi(X_T)$ takes one known value on each possible stopping set. For a mean time, look for a compensated quantity such as $\psi(X_n) - \alpha n$ , whose value at $T$ contains $T$ itself.
Verify one sufficient condition. Check $\mathbb{E}(T)<\infty$ and a uniform bound $c$ on the conditional absolute increments before $T$ (version C). The most common way to get $\mathbb{E}(T)<\infty$ is a geometric tail: exhibit $m$ and $\epsilon>0$ such that, from any history, the event occurs within the next $m$ steps with probability at least $\epsilon$ ; then $\mathbb{P}(T > km) \le (1-\epsilon)^{k}$ and so $\mathbb{E}(T)<\infty$ .
Write $\mathbb{E}(Y_T) = \mathbb{E}(Y_0)$ and expand the left side over the possible values of $Y_T$ . One martingale gives one linear equation; if there are two unknowns, find a second martingale.
Sanity-check the answer in a limiting or symmetric case where it is known independently, and - if the hypotheses were only sufficient - against a direct recursion.

Example(How long until HH appears?)Original

A fair coin is tossed repeatedly, giving independent outcomes $C_1, C_2, \dots$ , each equal to $\mathrm{H}$ or $\mathrm{T}$ with probability $\tfrac12$ . Let

T \ = \ \min\{ n \ge 2 : C_{n-1}=C_n=\mathrm{H} \}

be the first time the block $\mathrm{HH}$ is complete. We compute $\mathbb{E}(T)$ .

Step 1: build the martingale. Imagine a casino that pays fair odds: a stake of $s$ on a correctly guessed letter returns $2s$ , and nothing otherwise. Before toss $k$ , and only while the game is still running (that is, if $k \le T$ ), a new gambler $G_k$ arrives, stakes $1$ on " $C_k = \mathrm{H}$ ", and thereafter follows one rule: if she wins, she stakes her entire holding on the next letter of the pattern $\mathrm{HH}$ ; if she loses, she is broke and plays no more; if she has matched the whole pattern, she keeps her holding and retires.

Let $\mathcal{F}_n = \sigma(C_1,\dots,C_n)$ , let $H_n$ be the total holding of all gamblers after toss $n$ (so $H_0=0$ ), and set

Y_n \ = \ H_n - (T\wedge n) ,

the gamblers' total wealth minus the total $1$ per gambler they have paid in.

$Y$ is a martingale. The event $\{T>n\}$ lies in $\mathcal{F}_n$ , since $T$ is a first passage time and hence a stopping time, so the two cases may be treated separately. On $\{T \le n\}$ no new gambler arrives, no stake is placed and nothing is paid in, so $H_{n+1}=H_n$ and $T\wedge(n+1) = T\wedge n$ : both terms of $Y$ are frozen and the increment is zero. On $\{T>n\}$ gambler $G_{n+1}$ adds $1$ to the holdings and $1$ to the amount paid in, retired gamblers' holdings are unchanged, and each active gambler's holding is doubled or wiped out with probability $\tfrac12$ each, so its conditional mean is unchanged. Hence

\mathbb{E}\bigl( H_{n+1} - H_n \mid \mathcal{F}_n \bigr) \ = \ \mathbf{1}_{\{T>n\}} \ = \ \mathbb{E}\bigl( (T\wedge(n+1)) - (T\wedge n) \mid \mathcal{F}_n \bigr) ,

the two displayed quantities being equal because $T\wedge(n+1) - T\wedge n$ is the indicator of $\{T>n\}$ outright. Subtracting, $\mathbb{E}(Y_{n+1}\mid\mathcal{F}_n)=Y_n$ , with $Y_0 = 0$ . (The unconditional identity $\mathbb{E}(H_{n+1}\mid\mathcal{F}_n) = H_n+1$ would be wrong: it holds only while the game is still running.)

Step 2: check the hypotheses of version C. At most two gamblers can hold anything at once (one who has matched $\mathrm{H}$ , holding $2$ , and one who has just matched $\mathrm{HH}$ , holding $4$ ), so $0 \le H_n \le 6$ and $\lvert Y_{n+1}-Y_n\rvert \le 7$ always; condition C2 holds with $c = 7$ . For C1, split the tosses into disjoint pairs: each pair is $\mathrm{HH}$ with probability $\tfrac14$ independently of the others, so $\mathbb{P}(T>2m) \le (3/4)^{m}$ and $\mathbb{E}(T)<\infty$ . Also $\mathbb{E}\lvert Y_n\rvert \le 6+n<\infty$ .

Step 3: evaluate at $T$ . At the moment $\mathrm{HH}$ is complete, gambler $G_{T-1}$ guessed both letters and holds $2^{2}=4$ ; gambler $G_{T}$ guessed the first letter $\mathrm{H}$ correctly (it is the second $\mathrm{H}$ of the block) and holds $2^{1}=2$ ; every other gambler is broke. So $H_T = 4+2 = 6$ , a constant, and $T\wedge T = T$ . Optional stopping gives

0 \ = \ \mathbb{E}(Y_0) \ = \ \mathbb{E}(Y_T) \ = \ \mathbb{E}(H_T) - \mathbb{E}(T) \ = \ 6 - \mathbb{E}(T) ,

\boxed{\ \mathbb{E}(T) = 6 \ }

Step 4: the same calculation for $\mathrm{HT}$ . Let $T'$ be the first completion of the block $\mathrm{HT}$ . Now $G_{T'-1}$ has matched both letters and holds $4$ , but $G_{T'}$ staked on " $C_{T'}=\mathrm{H}$ " and the toss was $\mathrm{T}$ , so she is broke: $H_{T'}=4$ and $\mathbb{E}(T')=4$ . Two patterns of the same length have different mean waiting times, and the difference is caused entirely by whether the pattern overlaps itself - $\mathrm{HH}$ does, $\mathrm{HT}$ does not. Conditioning on the first step gives the same answers, but needs a separate recursion for each pattern; the martingale reads the answer straight off the overlap structure.

Step 5: check against a direct recursion. Track the length of the current match: let $e_0$ and $e_1$ be the mean remaining number of tosses when the current match has length $0$ and $1$ . For $\mathrm{HH}$ ,

e_0 = 1 + \tfrac12 e_1 + \tfrac12 e_0 , \qquad e_1 = 1 + \tfrac12\cdot 0 + \tfrac12 e_0 ,

whose solution is $e_1 = 4$ , $e_0 = 6$ , confirming step 3. The corresponding pair for $\mathrm{HT}$ gives $e_0 = 4$ .

Step 6: a biased check. If the coin shows $\mathrm{H}$ with probability $p$ , fair odds return $s/p$ on a correct guess of $\mathrm{H}$ and $s/q$ on a correct guess of $\mathrm{T}$ , where $q = 1-p$ . The argument is unchanged, and the holdings at $T$ become $p^{-2}+p^{-1}$ for $\mathrm{HH}$ and $(pq)^{-1}$ for $\mathrm{HT}$ , so

\mathbb{E}(T_{\mathrm{HH}}) = \frac{1}{p^{2}}+\frac1p , \qquad \mathbb{E}(T_{\mathrm{HT}}) = \frac{1}{pq} .

At $p=\tfrac12$ these give $6$ and $4$ as before. At $p=\tfrac13$ they give $9+3=12$ and $1/(\tfrac13\cdot\tfrac23) = 4.5$ , and the recursion of step 5 with $p=\tfrac13$ returns $e_0 = 12$ and $e_0 = 4.5$ respectively. The ranking of the two patterns is not universal. At $p=\tfrac23$ the formulas give $\tfrac94+\tfrac32 = 3.75$ for $\mathrm{HH}$ against $1/(\tfrac23\cdot\tfrac13) = 4.5$ for $\mathrm{HT}$ , so with a coin biased towards heads the self-overlapping pattern is the faster of the two.

Application(Anytime-valid testing, and fair games)AdaptedAdapted from Grimmett & Stirzaker (2020), §12.8, Example (12.8.7), p. 568, §12.4, Exercise (12.4.4), p. 553, and §12.5, Example (12.5.5), p. 555

A non-negative martingale rarely gets large, ever. Let $(Y,F)$ be a non-negative martingale with $\mathbb{E}(Y_0)=1$ and let $a>0$ (the interesting case being $a>1$ ). Put $T = \min\{n : Y_n \ge a\}$ , a stopping time. By part A, $\mathbb{E}(Y_{T\wedge n}) = 1$ for every $n$ ; since $Y\ge0$ we may throw away the part of that expectation on $\{T>n\}$ and bound the rest below by $a$ :

1 \ = \ \mathbb{E}\bigl(Y_{T\wedge n}\bigr) \ \ge \ \mathbb{E}\bigl( Y_T \mathbf{1}_{\{T\le n\}}\bigr) \ \ge \ a\,\mathbb{P}(T \le n) .

For each fixed $n$ the event $\{T \le n\}$ is $\{\max_{0\le m\le n} Y_m \ge a\}$ , which is the form in which Grimmett and Stirzaker record the bound, with $\mathbb{E}(Y_0)/x$ on the right, for non-negative supermartingales. Letting $n\to\infty$ and using that $\{T\le n\}$ increases to $\{T<\infty\}$ ,

\mathbb{P}\bigl( Y_n \ge a \ \text{for some } n \ge 0 \bigr) \ = \ \mathbb{P}(T<\infty) \ \le \ \frac1a .

This is the martingale form of Markov's inequality (Markov, Chebyshev, Jensen and Chernoff inequalities): it bounds the chance of ever reaching the level $a$ , over all time, by what Markov's inequality would give at a single time.

The level must be attained on the left: the event $\{\sup_n Y_n \ge a\}$ is strictly larger, since a path may climb towards $a$ without ever reaching it, and the two probabilities can differ. (Take $Y_0=1$ and let $Y$ either jump to $0$ and stay there or survive, with the surviving value at time $n$ equal to $c_n = 2 - 2^{-n}$ and the survival probability from time $n$ chosen as $c_n/c_{n+1}$ to keep the mean at $1$ . The total survival probability is $\prod_{n\ge0} c_n/c_{n+1} = c_0/\lim c_n = \tfrac12$ , and on the survival event $Y_n < 2$ for every $n$ while $\sup_n Y_n = 2$ ; so with $a = 2$ we have $\mathbb{P}(T<\infty) = 0$ but $\mathbb{P}(\sup_n Y_n \ge 2) = \tfrac12$ .) The supremum form of the bound is nevertheless true, by one extra line: for $0 < b < a$ the event $\{\sup_n Y_n \ge a\}$ is contained in $\{Y_n \ge b$ for some $n\}$ , whose probability is at most $1/b$ , and letting $b \uparrow a$ gives $\mathbb{P}(\sup_{n\ge0} Y_n \ge a) \le 1/a$ as well. The example above, with $\mathbb{P}(\sup_n Y_n \ge 2) = \tfrac12 = 1/a$ , shows that this bound is attained.

Why this matters for sequential experiments. Suppose data $X_1,X_2,\dots$ arrive one at a time and are independent and identically distributed with density either $q$ (the null hypothesis) or $p$ . Assume $q > 0$ wherever $p > 0$ , so that the ratio $p/q$ is defined and finite at every point the data can reach under either hypothesis, and the set $\{q = 0\}$ is null under both. The likelihood ratio

L_n \ = \ \prod_{i=1}^{n} \frac{p(X_i)}{q(X_i)} , \qquad L_0 = 1 ,

is a martingale under the null, because the $X_i$ are independent and, almost surely under $q$ ,

\mathbb{E}_q\left( \frac{p(X_{n+1})}{q(X_{n+1})} \,\Bigm|\, \mathcal{F}_n \right) = \int_{\{q>0\}} \frac{p(x)}{q(x)}\,q(x)\,dx = \int_{\{q>0\}} p(x)\,dx = 1 ,

the cancellation being legitimate exactly because the integral runs over $\{q>0\}$ , and the last equality holding because $p$ vanishes off that set. It is also non-negative, so the bound above applies: if the rule is "reject the null the first time $L_n \ge 100$ ", then the probability of ever rejecting a true null is at most $0.01$ no matter how long one watches, and no matter how the decision to stop is made. This is the formal content of the modern "anytime-valid" or "safe" testing methods, and it is why they are immune to the usual objection to repeatedly peeking at accumulating data: a fixed-sample test loses its guarantee when the stopping rule is data-dependent, and a martingale test does not.

A concrete instance: to detect a unit shift in the mean of standard normal noise, $q = N(0,1)$ and $p = N(1,1)$ , so $\log L_n = \sum_{i\le n} X_i - n/2$ , and the rule "stop and reject when $\sum_{i\le n} X_i \ge n/2 + \log 100 = n/2 + 4.6052$ " has false-alarm probability at most $0.01$ over an unlimited run. The same quantity, watched under $p$ instead, grows at rate $\tfrac12$ per observation, so the detection delay is about $9.2$ observations.

Elsewhere. Martingales are the natural language wherever a quantity is conserved in conditional mean. For a Markov chain with a bounded harmonic function $\psi$ (one satisfying $\sum_j p_{ij}\psi(j)=\psi(i)$ ), the sequence $\psi(X_n)$ is a martingale, and optional stopping at a first passage time shows that on an irreducible recurrent chain such a $\psi$ must be constant - the discrete Liouville theorem, which is Grimmett and Stirzaker's Example (12.5.5); see Discrete-time Markov chains. In continuous time the Wiener process and its compensated square $W(t)^{2}-t$ are martingales, and the stochastic integral of The Ito integral and Ito's formula is built so as to be one; the pricing of a financial derivative, and the "fluctuation-dissipation" bookkeeping of a driven physical system, are both statements that a suitably discounted quantity is a fair game.

Theorem

The Ito integral and Ito's formula

Adapted

Add to PDF

Builds the Ito integral against a Wiener process by extending the elementary sum over predictable step functions along the Ito isometry, shows that the resulting integral is a martingale with second moment equal to the mean integrated square of the integrand, and states and sketches Ito's formula, the chain rule with its extra second-derivative term.

Prerequisites: Brownian motion, Martingales, Conditional expectation, Convergence of random variables, Hilbert spaces

Note(What goes wrong with the ordinary chain rule)Original

Write $\delta W = W_{t+\delta t} - W_t$ for an increment of a standard Wiener process. The whole of this unit turns on one fact about that increment: it is of size $\sqrt{\delta t}$ , not $\delta t$ , because $\mathbb{E}\bigl[(\delta W)^2\bigr] = \delta t$ .

In ordinary calculus a second-order Taylor term is negligible because $(\delta x)^2$ is small compared with $\delta x$ . Here $(\delta W)^2$ is comparable with $\delta t$ , so the second-order term of a Taylor expansion survives the limit and contributes a drift. Everything below is the bookkeeping that makes this precise: a definition of $\int \psi \, dW$ that is stable under the limit, and a chain rule that carries the extra term.

A reader who wants only the operational rule can take away one line, to be used inside any smooth function of an Itô process: $(dW)^2 = dt$ , $dW\,dt = 0$ , $(dt)^2 = 0$ .

Definition(Admissible integrands, predictable step functions, and the integral)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.8, pp. 605-608, including (13.8.1), (13.8.2), (13.8.3) and the definition of $I(\psi)$

Let $W$ be a standard Wiener process on a complete probability space $(\Omega, \mathcal{F}, \mathbb{P})$ . For $t \ge 0$ let $\mathcal{F}_t$ be the smallest sub- $\sigma$ -field of $\mathcal{F}$ that contains every null event and with respect to which each of $W_s$ , $0 \le s \le t$ , is measurable, and write $\mathbf{F} = \{\mathcal{F}_t : t \ge 0\}$ for the filtration so obtained.

A process $\psi$ is measurable if the map $(t, \omega) \mapsto \psi_t(\omega)$ is measurable for the product $\sigma$ -field $\mathcal{B} \otimes \mathcal{F}$ , where $\mathcal{B}$ is the Borel $\sigma$ -field on $[0,\infty)$ ; a measurable $\psi$ is adapted if $\psi_t$ is $\mathcal{F}_t$ -measurable for every $t$ . Let

\mathcal{A} = \Bigl\{\, \psi \text{ adapted} \ :\ \mathbb{E}\Bigl(\int_0^\infty |\psi_t|^2 \, dt\Bigr) < \infty \,\Bigr\} , \qquad \|\psi\| = \sqrt{\mathbb{E}\Bigl(\int_0^\infty |\psi_t|^2\, dt\Bigr)} . \tag{1}

With $\|\cdot\|$ as its norm, $\mathcal{A}$ is a Hilbert space, and therefore complete. (Strictly, $\|\cdot\|$ is a norm on the equivalence classes of processes that agree with probability one, since $\|\psi\| = 0$ only forces $\psi$ to vanish almost everywhere.)

A predictable step function is a process of the form

\varphi_t = \sum_{j=0}^{n-1} C_j \, \mathbf{1}_{(a_j,\, a_{j+1}]}(t) , \qquad 0 = a_0 < a_1 < \cdots < a_n , \tag{2}

where each $C_j$ has a finite second moment and is $\mathcal{F}_{a_j}$ -measurable — the height on an interval is decided before the interval begins. Its integral is the obvious sum,

I(\varphi) = \sum_{j=0}^{n-1} C_j \,\bigl(W_{a_{j+1}} - W_{a_j}\bigr) , \tag{3}

and $I$ is linear on predictable step functions.

For general $\psi \in \mathcal{A}$ the Itô integral $I(\psi)$ is the mean-square limit of $I(\varphi^{(n)})$ along any sequence $\{\varphi^{(n)}\}$ of predictable step functions with $\|\varphi^{(n)} - \psi\| \to 0$ . The statement below says that such a sequence exists and that the limit does not depend on which one is used. We write

\int_0^\infty \psi_s \, dW_s := I(\psi) , \qquad \int_0^t \psi_s \, dW_s := I\bigl(\psi \,\mathbf{1}_{(0,t]}\bigr) .

Finally, a continuous process $X$ is an Itô process, or a diffusion, with infinitesimal mean $\mu(t,x)$ and infinitesimal variance $\sigma(t,x)^2$ if

X_t = X_0 + \int_0^t \mu(s, X_s)\, ds + \int_0^t \sigma(s, X_s)\, dW_s , \tag{4}

for which the shorthand is $dX_t = \mu(t,X_t)\,dt + \sigma(t,X_t)\,dW_t$ . Grimmett and Stirzaker state that the existence of such a process is beyond the scope of their account; see Stochastic differential equations.

Theorem(The Ito isometry, and the integral it defines)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.8, Lemma (13.8.4) and Theorem (13.8.6), p. 606, the uniqueness argument on p. 608, Theorem (13.8.11), p. 608, Theorem (13.8.13), p. 609, and Exercises (13.8.4) and (13.8.6), p. 610

Let $W$ be a standard Wiener process and $\mathbf{F}$ the filtration above. Write $\|U\|_2 = \sqrt{\mathbb{E}|U|^2}$ for a random variable $U$ .

Isometry on step functions. If $\varphi$ is a predictable step function then $\|I(\varphi)\|_2 = \|\varphi\|$ .
Approximation. If $\psi \in \mathcal{A}$ has continuous sample paths, there is a sequence $\{\varphi^{(n)}\}$ of predictable step functions with $\|\varphi^{(n)} - \psi\| \to 0$ . (Grimmett and Stirzaker prove this under the continuity assumption and record that continuity is not necessary.)
The limit exists and is unique. For such a $\psi$ the random variables $I(\varphi^{(n)})$ converge in mean square, and any two approximating sequences give limits that agree with probability one. This limit is $I(\psi)$ .
The isometry persists. For an adapted $\psi$ with $\|\psi\| < \infty$ , $\|I(\psi)\|_2 = \|\psi\|$ . The source also records the bilinear version $\mathbb{E}\bigl(I(\psi_1) I(\psi_2)\bigr) = \mathbb{E}\Bigl(\int_0^\infty \psi_1(t)\psi_2(t)\,dt\Bigr) ,$ whenever both sides exist, for $\psi_1$ and $\psi_2$ predictable step functions. It extends to $\psi_1,\psi_2 \in \mathcal{A}$ at once by polarisation, since $I$ is linear and the norm identity holds throughout $\mathcal{A}$ : four applications of $\|I(\cdot)\|_2 = \|\cdot\|$ to $\psi_1\pm\psi_2$ give it. (That last deduction is ours, not the source's.) In particular $\mathbb{E}\,I(\psi) = 0$ .
Martingale property. If $\psi \in \mathcal{A}$ has continuous sample paths then $J_t = \int_0^t \psi_u\,dW_u$ is a martingale with respect to $\mathbf{F}$ .
Converse (martingale representation). If $M$ is a martingale with respect to $\mathbf{F}$ , there is an adapted process $\psi$ with $M_t = M_0 + \int_0^t \psi_u \, dW_u$ for all $t \ge 0$ . Grimmett and Stirzaker state this without proof.

Item 4 is the identity usually called the Itô isometry. It is the workhorse: it turns a question about the second moment of a stochastic integral into an ordinary deterministic integral of a second moment.

Theorem(Ito's formula)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.9, Theorem (13.9.2), p. 610, and Theorem (13.9.4), p. 611

General form. Let $dX = \mu(t,X)\,dt + \sigma(t,X)\,dW$ and let $Y_t = f(t, X_t)$ , where $f : [0,\infty) \times \mathbb{R} \to \mathbb{R}$ is twice continuously differentiable. Then $Y$ is again an Itô process, and

dY = \Bigl[ f_x(t,X)\,\mu(t,X) \;+\; f_t(t,X) \;+\; \tfrac12 f_{xx}(t,X)\,\sigma^2(t,X) \Bigr]\,dt \;+\; f_x(t,X)\,\sigma(t,X)\,dW . \tag{5}

Grimmett and Stirzaker state (5) with the proof omitted, and record that it extends to multivariate diffusions.

The case $X = W$ . Under the name Itô's simple formula, the source states the following. Suppose $W$ is standard, and let the function $f(s,w)$ possess continuous partial derivatives up to the third order on the half-plane $s \ge 0$ . Then $Y_t = f(t, W_t)$ is an Itô process, and

dY = \Bigl[ f_t(t,W) + \tfrac12 f_{ww}(t,W) \Bigr]\,dt \;+\; f_w(t,W)\,dW . \tag{6}

This is (5) with $\mu = 0$ and $\sigma = 1$ . Grimmett and Stirzaker give a sketch proof of (6) and note that its differentiability assumption may be weakened; the extra derivative is an artefact of that proof, not of the result. The term $\tfrac12 f_{ww}$ , absent from the Newton-Leibniz chain rule, is the Itô correction.

Idea of the proofOriginal

The integral is built by isometric extension, exactly as the Fourier transform is extended from nice functions to $L^2$ . For a step function whose heights are fixed before each interval starts, the sum (3) is a sum of uncorrelated terms, so its second moment is the sum of the individual second moments — which is precisely $\|\varphi\|^2$ . So $I$ is an isometry from a subset of the Hilbert space $\mathcal{A}$ into $L^2(\Omega)$ . An isometry is uniformly continuous, so it carries Cauchy sequences to Cauchy sequences; since both spaces are complete, $I$ extends uniquely and isometrically to the closure of the step functions, and the remaining work is to show that this closure is all of $\mathcal{A}$ .

The martingale property is the same cancellation looked at from one step back. Condition on $\mathcal{F}_s$ . Each term $C_j(W_{a_{j+1}} - W_{a_j})$ of the sum that lies wholly after $s$ contributes nothing, because the increment is independent of $\mathcal{F}_{a_j}$ and has zero mean while $C_j$ is known at time $a_j$ ; each term wholly before $s$ is already known. This is exactly where adaptedness is spent.

Itô's formula is a second-order Taylor expansion in which the second-order term does not die. Expand $f$ over a short interval: the first-order terms reproduce the ordinary chain rule, and the second-order term in $w$ carries $(\delta W)^2$ . That quantity has mean $\delta t$ and fluctuations small enough that, summed over $t/\delta t$ intervals, it may be replaced by $\delta t$ in the limit. Third-order terms carry $(\delta W)^3 \sim (\delta t)^{3/2}$ and do vanish. So one extra term survives, and only one.

Proof(The isometry, and the construction of the integral)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.8, proof of Lemma (13.8.4), p. 606, and the argument following Theorem (13.8.6), pp. 607-608

We prove items 1 and 3, and then item 5; items 2, 4 and 6 are quoted from the source as stated.

Step 1: square the elementary sum. For a predictable step function $\varphi$ as in (2), write $\Delta_j = W_{a_{j+1}} - W_{a_j}$ . Multiplying out,

\mathbb{E}\bigl(|I(\varphi)|^2\bigr) = \sum_{j=0}^{n-1} \mathbb{E}\bigl(C_j^2 \Delta_j^2\bigr) \;+\; 2 \sum_{0 \le j < k \le n-1} \mathbb{E}\bigl(C_j C_k \Delta_j \Delta_k\bigr) .

Step 2: the diagonal terms. Fix $j$ and condition on $\mathcal{F}_{a_j}$ . Because $\varphi$ is a predictable step function, $C_j$ is $\mathcal{F}_{a_j}$ -measurable, so it comes out of the conditional expectation; and $\Delta_j$ is independent of $\mathcal{F}_{a_j}$ with $\mathbb{E}(\Delta_j^2) = a_{j+1} - a_j$ , since $W$ has independent increments of variance equal to their length. By the tower property,

\mathbb{E}\bigl(C_j^2\Delta_j^2\bigr) = \mathbb{E}\Bigl( C_j^2 \,\mathbb{E}\bigl(\Delta_j^2 \mid \mathcal{F}_{a_j}\bigr)\Bigr) = \mathbb{E}\bigl(C_j^2\bigr)\,(a_{j+1} - a_j) .

Step 3: the off-diagonal terms vanish. Let $j < k$ . Condition on $\mathcal{F}_{a_k}$ . The three factors $C_j$ , $C_k$ and $\Delta_j$ are all $\mathcal{F}_{a_k}$ -measurable, again by predictability and because $a_j < a_{j+1} \le a_k$ ; the remaining factor $\Delta_k$ is independent of $\mathcal{F}_{a_k}$ and has mean zero. Hence each such term has expectation $0$ .

Step 4: recognise the answer. Combining Steps 1-3,

\mathbb{E}\bigl(|I(\varphi)|^2\bigr) = \sum_{j} \mathbb{E}\bigl(C_j^2\bigr)(a_{j+1}-a_j) = \mathbb{E}\Bigl(\int_0^\infty |\varphi_t|^2\,dt\Bigr) = \|\varphi\|^2 ,

the middle equality because $\varphi$ takes the constant value $C_j$ on an interval of length $a_{j+1}-a_j$ . This is item 1.

Step 5: the approximating integrals form a Cauchy sequence. Let $\psi \in \mathcal{A}$ have continuous sample paths and take $\{\varphi^{(n)}\}$ from item 2. The difference $\varphi^{(m)} - \varphi^{(n)}$ is again a predictable step function, so by linearity of $I$ , by Step 4 applied to that difference, and by the triangle inequality in $\mathcal{A}$ ,

\bigl\|I(\varphi^{(m)}) - I(\varphi^{(n)})\bigr\|_2 = \bigl\|\varphi^{(m)} - \varphi^{(n)}\bigr\| \le \bigl\|\varphi^{(m)} - \psi\bigr\| + \bigl\|\varphi^{(n)} - \psi\bigr\| \ \longrightarrow\ 0

as $m, n \to \infty$ . The space $L^2(\Omega)$ is complete, so $I(\varphi^{(n)})$ converges in mean square to some random variable.

Step 6: the limit does not depend on the sequence. Let $\{\rho^{(n)}\}$ be a second sequence of predictable step functions with $\|\rho^{(n)} - \psi\| \to 0$ , with limit $I(\rho)$ . Insert both approximations:

\|I(\varphi) - I(\rho)\|_2 \le \|I(\varphi) - I(\varphi^{(n)})\|_2 + \|I(\varphi^{(n)}) - I(\rho^{(n)})\|_2 + \|I(\rho^{(n)}) - I(\rho)\|_2 .

The outer two terms tend to $0$ by the definition of the limits. The middle term equals $\|\varphi^{(n)} - \rho^{(n)}\|$ by Step 4 and linearity, and that is at most $\|\varphi^{(n)} - \psi\| + \|\rho^{(n)} - \psi\| \to 0$ . So the left-hand side, which does not depend on $n$ , is $0$ : the two limits agree with probability one. This is item 3.

Step 7: the martingale property. Let $0 < s < t$ and choose a partition $0 = a_0 < \cdots < a_m = s < \cdots < a_n = t$ that includes $s$ as a partition point. Let $\psi^{(n)}_u = \sum_j \psi_{a_j}\mathbf{1}_{(a_j, a_{j+1}]}(u)$ and $J^{(n)}_v = \int_0^v \psi^{(n)}_u\,dW_u$ . Conditioning the sum (3) on $\mathcal{F}_s$ term by term:

a term with $j < m$ lies entirely in $[0,s]$ , and is $\mathcal{F}_s$ -measurable, so it is unchanged;
a term with $j \ge m$ has $a_j \ge s$ ; conditioning first on $\mathcal{F}_{a_j}$ and using that $\psi_{a_j}$ is $\mathcal{F}_{a_j}$ -measurable while $W_{a_{j+1}} - W_{a_j}$ is independent of $\mathcal{F}_{a_j}$ with zero mean, the term has conditional expectation $0$ .

Hence $\mathbb{E}(J^{(n)}_t \mid \mathcal{F}_s) = J^{(n)}_s$ . Now refine the partition so that its mesh tends to $0$ . By the approximation of item 2, $\|\psi^{(n)}\mathbf{1}_{(0,v]} - \psi \mathbf{1}_{(0,v]}\| \to 0$ for $v = s$ and $v = t$ , so $J^{(n)}_s \to J_s$ and $J^{(n)}_t \to J_t$ in mean square. Mean-square convergence is preserved by conditional expectation, so the identity passes to the limit: $\mathbb{E}(J_t \mid \mathcal{F}_s) = J_s$ almost surely. $\square$

Proof(Sketch of Ito's simple formula)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.9, sketch proof of Theorem (13.9.4), p. 611, with Exercise (13.7.2), p. 604

This is a sketch; the source itself gives only a summary, and the estimates behind items (i), (ii) and (iv) below are omitted there and here. Fix $t > 0$ , put $\delta = t/n$ , write $t_j = j\delta$ and $\Delta_j = W_{t_{j+1}} - W_{t_j}$ .

Step 1: telescope, splitting each step into a time move and a space move. Write the total increment as

f(t, W_t) - f(0, W_0) = \sum_{j=0}^{n-1}\Bigl[ f(t_{j+1}, W_{t_{j+1}}) - f(t_j, W_{t_{j+1}}) \Bigr] \;+\; \sum_{j=0}^{n-1}\Bigl[ f(t_j, W_{t_{j+1}}) - f(t_j, W_{t_j}) \Bigr] .

Each bracket in the first sum moves only in time, each in the second only in space; the two sums are handled by different arguments, which is the reason for the split.

Step 2: Taylor-expand each sum. By the mean value theorem there are random $\theta_j \in [t_j, t_{j+1}]$ making the first sum $\sum_j f_t(\theta_j, W_{t_{j+1}})\delta$ . By Taylor's theorem with Lagrange remainder in the space variable there are random $\Xi_j$ between $W_{t_j}$ and $W_{t_{j+1}}$ making the second sum

\sum_{j} f_w(t_j, W_{t_j})\Delta_j \;+\; \tfrac12 \sum_{j} f_{ww}(t_j, W_{t_j})\Delta_j^2 \;+\; \tfrac16 \sum_{j} f_{www}(t_j, \Xi_j)\Delta_j^3 .

The third derivative appears here, and nowhere else; this is why the hypothesis of (6) asks for three derivatives.

Step 3: the time sum becomes an ordinary integral. $W$ has continuous sample paths and $f_t$ is continuous, so along almost every path the integrand is continuous and the sum is a Riemann sum: $\sum_j f_t(\theta_j, W_{t_{j+1}})\delta \to \int_0^t f_t(s,W_s)\,ds$ almost surely.

Step 4: the first space sum becomes the Itô integral. The summand is the elementary integral (3) of the predictable step function with heights $f_w(t_j, W_{t_j})$ , which is exactly the approximating step function of the construction above; it converges in mean square to $\int_0^t f_w(s, W_s)\,dW_s$ .

Step 5: the quadratic sum becomes a time integral — the Itô correction. Here is the one genuinely stochastic step. Each $\Delta_j^2$ has mean $\delta$ , and $\Delta_j^2$ and $\Delta_k^2$ are independent for $j \ne k$ because $W$ has independent increments. Replacing $\Delta_j^2$ by its mean $\delta$ therefore introduces an error that is a sum of $n$ independent-in-the-relevant-sense, mean-zero terms each of size $\delta$ ; the variance of that error is of order $n\delta^2 = t\delta \to 0$ . (For $f_{ww} \equiv 1$ this is the statement that $\sum_j \Delta_j^2 \to t$ in mean square, the quadratic variation of the Wiener process.) Hence

\tfrac12\sum_j f_{ww}(t_j, W_{t_j})\Delta_j^2 - \tfrac12\sum_j f_{ww}(t_j, W_{t_j})\,\delta \ \longrightarrow\ 0

in mean square, and the second of these is a Riemann sum converging to $\tfrac12\int_0^t f_{ww}(s,W_s)\,ds$ .

Step 6: the cubic sum is negligible. $\mathbb{E}|\Delta_j|^3$ is of order $\delta^{3/2}$ , so $n$ such terms total order $n\delta^{3/2} = t\,\delta^{1/2} \to 0$ ; the source states that this term converges to zero in mean square.

Step 7: assemble. Adding the limits of Steps 3-6 gives the integrated form of (6), and $t$ was arbitrary. $\square$

Remark(Where the hypotheses are used)Original

Predictability (the height $C_j$ is $\mathcal{F}_{a_j}$ -measurable) is what makes the isometry true, and it is used twice, in Steps 2 and 3. Drop it and both fail. Take the single-interval step function with $a_0 = 0$ , $a_1 = t$ and height $C_0 = W_t$ , which is $\mathcal{F}_t$ -measurable but not $\mathcal{F}_0$ -measurable. Then $I(\varphi) = W_t(W_t - W_0) = W_t^2$ , whose mean is $t$ , not $0$ , and whose second moment is $\mathbb{E}(W_t^4) = 3t^2$ , while $\|\varphi\|^2 = \mathbb{E}\int_0^t W_t^2\,ds = t^2$ . Both item 4 and the martingale property are destroyed.

This is the same thing as the dependence of the limit on the evaluation point. Partition $(0,t]$ into $n$ equal pieces and form $\sum_j W_{\theta_j}(W_{t_{j+1}} - W_{t_j})$ . Taking $\theta_j = t_j$ , the left endpoint, gives $\tfrac12(W_t^2 - t)$ in mean square — the Itô value; taking $\theta_j = t_{j+1}$ gives $\tfrac12(W_t^2 + t)$ . A Riemann integral would not notice the difference. The Wiener process has unbounded total variation on every non-trivial interval, and that is exactly why it does. The midpoint rule gives a third answer and a different calculus, discussed in Stochastic differential equations.

The square-integrability condition (1) is what makes $J_t$ a martingale rather than only a local martingale. Grimmett and Stirzaker record in a footnote that without it one obtains a local martingale, and that in a more general theory the correct condition on the integrand is not adaptedness but local boundedness and predictability, with $W$ replaced by a semimartingale.

Continuity of the sample paths of $\psi$ is used only in item 2 and in item 5, and only because the proof given for the density of step functions uses uniform continuity on compact time intervals. The source states that this condition is not necessary.

Two derivatives in Itô's formula (5) are not a technicality. The term $\tfrac12 f_{xx}\sigma^2$ is a leading term, not a remainder. With $f$ only once continuously differentiable there is no formula of this shape at all, because the first-order Taylor remainder is of size $(\delta W)^2 \sim \delta t$ and does not vanish on summation. The simplest instance is $f(w) = w^2$ : the Newton-Leibniz rule would give $d(W^2) = 2W\,dW$ and hence $\int_0^t W_s\,dW_s = \tfrac12 W_t^2$ , whereas the correct value is $\tfrac12(W_t^2 - t)$ . The two differ by a quantity, $t/2$ , that is not small.

The simple formula (6) asks for three derivatives where the general formula (5) asks for two. This looks backwards, since (6) is a special case of (5). It is an honest feature of the source: (5) is quoted there without proof, while (6) is given the sketch proof above, and that sketch spends a third derivative on the Taylor remainder in Step 2. Grimmett and Stirzaker say explicitly that the assumption may be weakened. Use (5) when two derivatives are all you have.

Method(Evaluating a stochastic integral by reversing Ito's formula)Original

There is no fundamental theorem of calculus for $\int \psi\,dW$ with a general adapted $\psi$ . When the integrand has the form $g(s, W_s)$ , however, the following works.

Find an antiderivative in the space variable. Choose $f(s,w)$ with $f_w(s,w) = g(s,w)$ , three times continuously differentiable (or use (5) if you can justify two).
Compute the drift $f_t + \tfrac12 f_{ww}$ .
Write down Itô's formula in integrated form. $f(t, W_t) - f(0, W_0) = \int_0^t \Bigl[f_t(s,W_s) + \tfrac12 f_{ww}(s,W_s)\Bigr]ds + \int_0^t g(s,W_s)\,dW_s .$
Rearrange to isolate the stochastic integral. If the drift in Step 2 vanishes identically, the stochastic integral is simply $f(t,W_t) - f(0,W_0)$ and is a martingale.
Check admissibility before using the isometry or the martingale property. Both need $g(\cdot, W_\cdot)\mathbf{1}_{(0,t]} \in \mathcal{A}$ , that is $\int_0^t \mathbb{E}\bigl(g(s,W_s)^2\bigr)\,ds < \infty$ . For a polynomial $g$ this is automatic, because all moments of a normal variable are finite.
Take second moments with the isometry. $\mathbb{E}\bigl[\bigl(\int_0^t g\,dW\bigr)^2\bigr] = \int_0^t \mathbb{E}\bigl(g(s,W_s)^2\bigr)\,ds$ , an ordinary one-dimensional integral. Step 1 has already given the first moment, which is $0$ .
Check the answer independently whenever the left-hand side has a known distribution; see the example.

Example(A cubic martingale, and its second moment two ways)Original

Take $f(t,w) = w^3 - 3tw$ , a polynomial and so as smooth as required, and let $W$ be a standard Wiener process with $W_0 = 0$ .

Step 1: the derivatives.

f_t = -3w , \qquad f_w = 3w^2 - 3t , \qquad f_{ww} = 6w .

Step 2: the drift cancels.

f_t + \tfrac12 f_{ww} = -3w + \tfrac12(6w) = 0 .

So by Itô's simple formula (6), $Y_t = W_t^3 - 3tW_t$ has $dY = (3W_t^2 - 3t)\,dW_t$ : a pure stochastic integral, with no drift.

Step 3: read off the integral. Since $f(0,0) = 0$ ,

\int_0^t \bigl(3W_s^2 - 3s\bigr)\,dW_s = W_t^3 - 3t\,W_t .

Step 4: check admissibility. With $g(s,w) = 3w^2 - 3s$ and $W_s \sim N(0,s)$ ,

\mathbb{E}\bigl(g(s,W_s)^2\bigr) = 9\,\mathbb{E}\bigl(W_s^4 - 2sW_s^2 + s^2\bigr) = 9\bigl(3s^2 - 2s^2 + s^2\bigr) = 18 s^2 ,

using $\mathbb{E}(W_s^2) = s$ and $\mathbb{E}(W_s^4) = 3s^2$ . This is integrable on $[0,t]$ , so $g(\cdot,W_\cdot)\mathbf{1}_{(0,t]} \in \mathcal{A}$ and items 4 and 5 of the statement apply. In particular $Y$ is a martingale, so $\mathbb{E}(W_t^3 - 3tW_t) = 0$ for every $t$ .

Step 5: second moment by the isometry.

\mathbb{E}\Bigl[\bigl(W_t^3 - 3tW_t\bigr)^2\Bigr] = \int_0^t 18 s^2\,ds = 6t^3 .

Step 6: the same number directly. Expand the square and use the moments of $W_t \sim N(0,t)$ , namely $\mathbb{E}(W_t^2) = t$ , $\mathbb{E}(W_t^4) = 3t^2$ and $\mathbb{E}(W_t^6) = 15t^3$ :

\mathbb{E}\bigl(W_t^6 - 6t\,W_t^4 + 9t^2 W_t^2\bigr) = 15t^3 - 18t^3 + 9t^3 = 6t^3 .

The two routes agree, which checks the isometry on this example.

A number. At $t = 2$ , $\ \mathbb{E}\bigl[(W_2^3 - 6W_2)^2\bigr] = 6 \cdot 2^3 = \boxed{48}$ .

Note how badly the naive chain rule would do here. Treating $dW$ as an ordinary differential would give $\int_0^t 3W_s^2\,dW_s = W_t^3$ , and $W_t^3$ is not a martingale in any useful sense — it has the right mean, $0$ , but the compensating $-3tW_t$ is exactly what the Itô correction supplies.

Application(Multiplicative noise and the gap between the mean and the typical path)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.9, Example (13.9.9), p. 612, and §13.10, equation (13.10.2), p. 614

A quantity that grows by random proportional increments — a population, the size of an aerosol droplet under random coagulation, the price of a traded asset — is modelled by geometric Brownian motion,

dS_t = S_t\bigl(\mu\,dt + \sigma\,dW_t\bigr) .

Apply Itô's simple formula to $Y_t = \exp(\mu t + \sigma W_t)$ : with $f(t,w) = e^{\mu t + \sigma w}$ one has $f_t = \mu f$ , $f_w = \sigma f$ , $f_{ww} = \sigma^2 f$ , so $dY = (\mu + \tfrac12\sigma^2)Y\,dt + \sigma Y\,dW$ . Therefore the solution of the displayed equation with $S_0 = 1$ is not $e^{\mu t + \sigma W_t}$ but

S_t = \exp\Bigl[\bigl(\mu - \tfrac12\sigma^2\bigr)t + \sigma W_t\Bigr] ,

which is the form in which the Black-Scholes model of §13.10 of the source states it, $\sigma$ there being called the volatility.

The consequence is one that is easy to get wrong in practice. Since $\mathbb{E}\,e^{\sigma W_t} = e^{\sigma^2 t/2}$ , the mean grows at rate $\mu$ :

\mathbb{E}(S_t) = e^{\mu t} .

But $\log S_t$ is normal with mean $(\mu - \tfrac12\sigma^2)t$ , so the median, and the almost sure long-run growth rate of a single realisation, is governed by $\mu - \tfrac12\sigma^2$ . The two differ by exactly the Itô correction.

Take $\mu = 0.05$ and $\sigma = 0.4$ per year. Then $\mu - \tfrac12\sigma^2 = 0.05 - 0.08 = -0.03$ . After ten years the mean of the ensemble is $e^{0.5} = 1.6487$ times its initial value, while the median path is at $e^{-0.3} = 0.7408$ of it, and

\mathbb{P}(S_{10} < S_0) = \Phi\!\left(\frac{0.03 \times 10}{0.4\sqrt{10}}\right) = \Phi(0.2372) = 0.5937 ,

so a clear majority of realisations have gone down while the ensemble mean has gone up by nearly two thirds. The mean is carried by a thin tail of very large values. Whenever a quantity fluctuates multiplicatively, the average over an ensemble and the behaviour of a typical member are different questions, and Itô's formula is what converts between them. The same term reappears as the drift of the Fokker-Planck equation for $\log S$ ; see The Fokker-Planck equation.

Method

Stochastic differential equations

Adapted

Add to PDF

Gives a recipe for solving a linear stochastic differential equation with additive noise by an integrating factor, proves that its solution is Gaussian with an explicit mean, variance and autocovariance, applies this to the Ornstein-Uhlenbeck process and the Langevin equation, and explains how the Ito and Stratonovich readings of the same equation differ and how to convert between them.

Prerequisites: The Ito integral and Ito's formula, Brownian motion, Stationary processes and spectral densities, The central limit theorem, Solving first-order equations, Dependence on initial data and Gronwall's inequality

Note(What an equation driven by noise can mean)Original

A physicist writes $m\dot v = -\gamma v + \xi(t)$ and calls $\xi$ "white noise". Nothing in ordinary analysis supports this: the intended $\xi$ is the derivative of a Wiener process, and Brownian motion has nowhere differentiable paths. The repair is to multiply through by $dt$ and read the equation as an integral equation in which the noise appears only inside a stochastic integral, never differentiated.

Once that is done, a second question appears that has no analogue in ordinary differential equations: which stochastic integral? The Itô integral and the Stratonovich integral give different answers to the same written equation. This unit gives the method for the case that is solvable in closed form — linear drift and additive noise — and then says what the two readings are and how to pass between them.

Definition(Stochastic differential equations, and the two readings)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.8, equations (13.8.9)-(13.8.10), p. 608, §13.9, equation (13.9.1), p. 610, and §13.7, pp. 603-604

Let $W$ be a standard Wiener process, with the filtration $\mathbf{F}$ built in The Ito integral and Ito's formula. A stochastic differential equation (an SDE) is written

dX_t = \mu(t, X_t)\,dt + \sigma(t, X_t)\,dW_t \tag{1}

and is shorthand for the integral equation

X_t = X_0 + \int_0^t \mu(s,X_s)\,ds + \int_0^t \sigma(s,X_s)\,dW_s , \qquad t \ge 0 , \tag{2}

in which the second integral is an Itô integral. A continuous adapted process $X$ satisfying (2) is called a solution, and is an Itô process, or diffusion, with infinitesimal mean $\mu$ and infinitesimal variance $\sigma^2$ .

The last integral in (2) is the mean-square limit of sums $\sum_j \sigma(t_j, X_{t_j})(W_{t_{j+1}} - W_{t_j})$ in which the integrand is evaluated at the left endpoint of each partition interval. Evaluating instead at the midpoint produces a different limit, and that limit defines the Stratonovich integral, written $\int_0^t \sigma \circ dW$ . An equation written with $\circ\,dW$ in place of $dW$ ,

dX_t = \tilde\mu(t,X_t)\,dt + \sigma(t,X_t)\circ dW_t , \tag{3}

is a Stratonovich SDE: a different equation, with different solutions, not a different notation for (1). The remark below gives the dictionary between them.

Throughout, linear with additive noise means the special case

dX_t = \bigl(\alpha(t)X_t + \nu(t)\bigr)\,dt + \delta(t)\,dW_t , \tag{4}

in which $\alpha, \nu, \delta$ are deterministic functions of $t$ alone. (The letter $\nu$ is used for the affine forcing because $\beta$ is reserved below for the relaxation rate of the Ornstein-Uhlenbeck process.) Because $\sigma = \delta$ does not depend on $x$ , equations (1) and (3) coincide for (4) — the two readings differ only when the noise is multiplicative.

Method(Existence and uniqueness: what the source asserts)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.9, opening paragraph, p. 610, and §13.12, Problem (13.12.21), p. 629

Grimmett and Stirzaker assert the following about equation (1). Provided $\mu$ , $\sigma$ and $X_0$ satisfy conditions that they describe as rather weak, there is exactly one solution; it is a Markov process, and its sample paths are continuous. They do not list the conditions, and they declare the proof beyond the scope of their book. We report the assertion in that form rather than substituting the Lipschitz and linear-growth hypotheses familiar from other accounts, which are not the hypotheses the source gives.

That some condition is needed is shown by their Problem (13.12.21), Tanaka's example. Let $\operatorname{sign}(x) = 1$ for $x > 0$ and $-1$ otherwise, and consider

dX_t = \operatorname{sign}(X_t)\,dW_t , \qquad X_0 = 0 .

The process $V_t = \int_0^t \operatorname{sign}(W_s)\,dW_s$ is itself a standard Wiener process and solves this equation; so does $-V$ . (The source writes the same letter $W$ for the process used to build $V$ and for the process driving the equation; the driving process is the one reconstructed from $V$ itself, namely $\int_0^\cdot \operatorname{sign}(V_s)\,dV_s$ , which is again a standard Wiener process. Nothing below depends on which reading is taken.) Solutions are therefore not unique path by path, although every solution has the finite-dimensional distributions of a Wiener process. The coefficient $\operatorname{sign}$ is bounded, measurable and non-zero, but discontinuous at the one point the process keeps returning to.

For the linear equation (4) there is no such difficulty, and the next statement settles existence, uniqueness and the law of the solution outright.

Method(Linear SDE with additive noise: closed form and law)Original

Let $\nu$ and $\delta$ be continuous real functions on $[0,\infty)$ , let $\alpha$ be a continuously differentiable real function on $[0,\infty)$ , let $X_0$ be a constant, and set

\Phi_t = \exp\left(-\int_0^t \alpha(r)\,dr\right) .

Then equation (4) has one and only one continuous adapted solution, namely

X_t = \Phi_t^{-1}\left( X_0 + \int_0^t \Phi_s\,\nu(s)\,ds + \int_0^t \Phi_s\,\delta(s)\,dW_s \right) . \tag{5}

The solution is a Gaussian process, with the usual convention that a constant counts as normal with variance $0$ (if $\delta \equiv 0$ the equation has no noise and the solution is the deterministic function (5)), and

\mathbb{E}(X_t) = \Phi_t^{-1}\left(X_0 + \int_0^t \Phi_s \nu(s)\,ds\right) , \qquad \operatorname{var}(X_t) = \Phi_t^{-2}\int_0^t \Phi_s^2\,\delta(s)^2\,ds , \tag{6}

and, for $s \le t$ ,

\operatorname{cov}(X_s, X_t) = \Phi_s^{-1}\Phi_t^{-1}\int_0^{s} \Phi_u^2\,\delta(u)^2\,du . \tag{7}

A random initial value. Suppose instead that $X_0$ is a random variable with finite variance, independent of $W$ . Then no process starting at $X_0$ is adapted to $\mathbf{F}$ at all, because $\mathcal{F}_0$ contains only the null events and their complements and so cannot carry a non-degenerate $X_0$ . The filtration must be enlarged to $\mathbf{G}$ , where $\mathcal{G}_t = \mathcal{F}_t \vee \sigma(X_0)$ . This costs nothing: since $X_0$ is independent of $W$ , the increments of $W$ after time $t$ are still independent of $\mathcal{G}_t$ , so $W$ is a Wiener process for $\mathbf{G}$ too and the Itô integral in (5) is the same object as before. With "adapted" read as adapted to $\mathbf{G}$ , formula (5) is again the unique continuous adapted solution, and (6) and (7) hold conditionally on $X_0$ . Unconditionally, $\operatorname{var}(X_0)$ is added to the integral in each of (6) and (7):

\operatorname{var}(X_t) = \Phi_t^{-2}\left(\operatorname{var}(X_0) + \int_0^t\Phi_s^2\delta(s)^2\,ds\right) , \qquad \operatorname{cov}(X_s,X_t) = \Phi_s^{-1}\Phi_t^{-1}\left(\operatorname{var}(X_0) + \int_0^{s}\Phi_u^2\delta(u)^2\,du\right)

for $s \le t$ (both follow from (5) and the independence of $X_0$ and $W$ , which makes the two contributions uncorrelated), and $X$ is a Gaussian process whenever $X_0$ is normal.

This is our own statement; it is proved in full below, and the special case $\alpha \equiv -\beta$ , $\nu \equiv 0$ , $\delta \equiv \sigma$ recovers the law of the Ornstein-Uhlenbeck process that the source states in Problem (13.12.4).

Method(The Ornstein-Uhlenbeck process)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.3, Example (13.3.11), p. 584, the representations on p. 603, Exercise (13.8.5), p. 610, and §13.12, Problems (13.12.3) and (13.12.4), p. 627

Fix $\beta > 0$ and $\sigma > 0$ . The Ornstein-Uhlenbeck process is the diffusion $V$ with instantaneous mean and variance

a(t,x) = -\beta x , \qquad b(t,x) = \sigma^2 ,

that is, the solution of

dV_t = -\beta V_t\,dt + \sigma\,dW_t . \tag{8}

Items 1-4, and the two representations displayed in item 5, are what the source records; the covariance computation in item 5 and the identifications of $\sigma$ that it yields are ours.

Law at time $t$ . If $V_0 = u$ then $V_t \ \sim\ N\!\left(u e^{-\beta t},\ \frac{\sigma^2\bigl(1 - e^{-2\beta t}\bigr)}{2\beta}\right) .$
Limit law. $V_t$ is asymptotically $N\bigl(0, \tfrac12\sigma^2/\beta\bigr)$ as $t \to \infty$ .
Stationarity. If $V_0 \sim N\bigl(0, \tfrac12\sigma^2/\beta\bigr)$ then $V$ is strongly stationary; see Stationary processes and spectral densities.
Characterisation. Among processes that are at once Gaussian, Markov, stationary and equipped with an autocovariance function that is continuous, this one is the only one.
Representations. With $W$ standard, each of $U_1(t) = e^{-\beta t}W\bigl(e^{2\beta t} - 1\bigr) , \qquad U_3(t) = W_t - \beta\int_0^t e^{-\beta(t-s)}W_s\,ds$ is an Ornstein-Uhlenbeck process, but not with the same $\sigma$ ; the source does not say which $\sigma$ , and the rest of this item, which settles it, is ours. For $s \le t$ , $\operatorname{cov}\bigl(U_1(s),U_1(t)\bigr) = e^{-\beta(s+t)}\min\bigl(e^{2\beta s}-1, e^{2\beta t}-1\bigr) = e^{-\beta(t-s)} - e^{-\beta(t+s)}$ , which by (7) is the case $\sigma^2 = 2\beta$ , $V_0 = 0$ ; $U_3$ is the case $\sigma = 1$ , $V_0 = 0$ , and is what integrating (8) with those values produces. Because these are built from $W$ by continuous operations, the sample paths of $V$ inherit continuity from those of $W$ .

Idea of the proofOriginal

The integrating factor of ordinary linear theory still works, and Itô's formula says why nothing extra appears. For the deterministic equation $\dot x = \alpha x + \nu$ one multiplies by the integrating factor $\Phi_t = e^{-\int\alpha}$ , which turns the terms in $x$ and $\dot x$ into the derivative of the single product $\Phi x$ . Here we must apply Itô's formula, not the product rule, to $\Phi_t X_t$ . Itô's formula adds a term $\tfrac12 f_{xx}\sigma^2$ ; but $f(t,x) = \Phi_t x$ is linear in $x$ , so $f_{xx} = 0$ and the correction is absent. That is the whole reason the classical recipe survives for additive noise, and the whole reason it fails for multiplicative noise.

Once the equation is integrated, Gaussianity is free. What is left on the right is a deterministic function plus an Itô integral of a deterministic integrand. Such an integral is, at each stage of its construction, a linear combination of independent normal increments of $W$ , hence normal; and a mean-square limit of normals is normal. Its mean is zero and its variance is given by the Itô isometry, which here is an ordinary integral.

For the Ornstein-Uhlenbeck process the two terms in (5) have a clean physical reading. The factor $e^{-\beta t}$ damps the memory of the initial value; the stochastic integral accumulates kicks, each discounted by $e^{-\beta(t-s)}$ according to how long ago it arrived. The variance is a balance between the two, and settles at $\sigma^2/(2\beta)$ : more noise raises it, more friction lowers it.

Proof(Of the linear statement)Original

Write $\mathcal{F}_t$ for the filtration of $W$ , and assume throughout that $X_0$ is constant. For a random $X_0$ independent of $W$ , every step below is read with $\mathbf{F}$ replaced by the enlarged filtration $\mathbf{G}$ of the statement, which changes nothing in Steps 1-4 and 6, because $W$ is a Wiener process for $\mathbf{G}$ as well and the Itô integral of a deterministic integrand is built from the same increments; Step 5 is the one step that needs $X_0$ to be constant (or normal).

Step 1: the integrating factor is smooth and deterministic. $\alpha$ is continuously differentiable, so $t \mapsto \int_0^t\alpha$ is twice continuously differentiable and $\Phi$ is a positive, twice continuously differentiable, deterministic function with $\Phi'_t = -\alpha(t)\Phi_t$ , $\Phi''_t = \bigl(\alpha(t)^2 - \alpha'(t)\bigr)\Phi_t$ and $\Phi_0 = 1$ . The same holds for $\Phi^{-1}$ , with $(\Phi^{-1})' = \alpha\Phi^{-1}$ and $(\Phi^{-1})'' = (\alpha^2+\alpha')\Phi^{-1}$ . This is the only place $\alpha'$ is wanted, and the remark below says why it is wanted at all.

Step 2: apply Itô's formula to the product. Suppose $X$ solves (4) and put $f(t,x) = \Phi_t x$ , so that $f_t = \Phi'_t x = -\alpha(t)\Phi_t x$ , $f_x = \Phi_t$ and $f_{xx} = 0$ . By Step 1 the function $f$ is twice continuously differentiable on $[0,\infty)\times\mathbb{R}$ — $f_{tt} = \Phi''_tx$ exists and is continuous because $\alpha \in C^1$ , and $f_{tx} = \Phi'_t$ , $f_{xx} = 0$ — so Itô's formula in the general form of The Ito integral and Ito's formula, which asks for exactly that, applies, with $\mu(t,x) = \alpha(t)x + \nu(t)$ and $\sigma(t,x) = \delta(t)$ :

d\bigl(\Phi_t X_t\bigr) = \Bigl[\underbrace{\Phi_t\bigl(\alpha X_t + \nu\bigr)}_{f_x\mu} \underbrace{- \alpha\Phi_t X_t}_{f_t} + \underbrace{\tfrac12 \cdot 0 \cdot \delta^2}_{\frac12 f_{xx}\sigma^2}\Bigr]dt + \Phi_t\,\delta(t)\,dW_t = \Phi_t\nu(t)\,dt + \Phi_t\delta(t)\,dW_t .

The two terms containing $X_t$ cancel, which is what the factor was chosen for, and the Itô correction is zero because $f$ is linear in $x$ .

Step 3: integrate and solve. In integrated form, $\Phi_tX_t - X_0 = \int_0^t\Phi_s\nu\,ds + \int_0^t\Phi_s\delta\,dW_s$ . The Itô integral on the right is defined because $\Phi\delta$ is continuous and deterministic, so $\int_0^t \Phi_s^2\delta(s)^2\,ds < \infty$ and the admissibility condition of The Ito integral and Ito's formula holds. Dividing by $\Phi_t > 0$ gives (5). Conversely — and this is a second application of Itô's formula, not Step 2 run backwards — set

Y_t = X_0 + \int_0^t\Phi_s\nu(s)\,ds + \int_0^t\Phi_s\delta(s)\,dW_s ,

a continuous adapted Itô process with $dY_t = \Phi_t\nu(t)\,dt + \Phi_t\delta(t)\,dW_t$ , and apply Itô's formula to $g(t,y) = \Phi_t^{-1}y$ along $Y$ , which is again twice continuously differentiable by Step 1. Again $g_{yy} = 0$ , so there is no Itô correction, and since $\partial_t\bigl(\Phi_t^{-1}\bigr) = \alpha(t)\Phi_t^{-1}$ ,

d\bigl(\Phi_t^{-1}Y_t\bigr) = \alpha(t)\Phi_t^{-1}Y_t\,dt + \Phi_t^{-1}\bigl(\Phi_t\nu(t)\,dt + \Phi_t\delta(t)\,dW_t\bigr) = \bigl(\alpha(t)X_t + \nu(t)\bigr)dt + \delta(t)\,dW_t ,

where $X_t = \Phi_t^{-1}Y_t$ is exactly the process (5). So (5) is continuous, adapted and solves (4): a solution exists.

Step 4: uniqueness. Let $X$ and $\tilde X$ both solve (4) with the same $X_0$ and put $Z = X - \tilde X$ . Subtracting the two copies of (2), the stochastic integrals cancel — they do not involve the solution, since $\delta$ does not depend on $x$ — and $Z_t = \int_0^t \alpha(s)Z_s\,ds$ for every $t$ , pathwise, with $Z_0 = 0$ . Fix $T$ and let $A = \max_{[0,T]}|\alpha|$ , finite by continuity. Then $|Z_t| \le A\int_0^t|Z_s|\,ds$ on $[0,T]$ . Both solutions are continuous, so $K = \max_{[0,T]}|Z|$ is finite, and feeding the bound into itself gives

|Z_t| \le K\,\frac{(At)^n}{n!} , \qquad 0 \le t \le T ,

for every $n \ge 0$ : the case $n = 0$ is the definition of $K$ , and if the bound holds for $n$ then $|Z_t| \le A\int_0^t K(As)^n/n!\,ds = K(At)^{n+1}/(n+1)!$ . Letting $n \to \infty$ gives $Z_t = 0$ . (This is the zero-constant case of Gronwall's inequality, which Dependence on initial data and Gronwall's inequality proves in general; the iteration is spelled out here so that the step needs nothing beyond calculus.) Hence $Z \equiv 0$ on $[0,T]$ for every $T$ .

Step 5: the solution is Gaussian. Only the term $M_t = \int_0^t\Phi_s\delta(s)\,dW_s$ is random. By construction it is the mean-square limit of the elementary sums $\sum_j c_j (W_{t_{j+1}} - W_{t_j})$ with deterministic coefficients $c_j$ , and such a sum is a linear combination of independent normal variables, hence normal with mean $0$ . Mean-square convergence implies convergence in distribution, and the normal family is closed under limits in distribution (move 3 of the method of Brownian motion), so $M_t$ is normal; its variance is $\lim_n v_n$ , since mean-square convergence carries second moments with it. The same argument applied to a finite linear combination $\sum_k \lambda_k M_{t_k}$ shows every such combination is normal, which is to say that $(M_{t_1},\dots,M_{t_n})$ is multivariate normal; so $M$ , and therefore $X$ , is a Gaussian process.

Step 6: mean, variance and autocovariance. Taking expectations in (5), and using $\mathbb{E}(M_t) = 0$ , gives the first half of (6). The Itô isometry gives

\operatorname{var}(M_t) = \mathbb{E}\bigl(M_t^2\bigr) = \int_0^t \Phi_s^2\delta(s)^2\,ds ,

and dividing by $\Phi_t^2$ gives the second half of (6). For $s \le t$ , apply the bilinear form of the isometry to $\psi_1 = \Phi\delta\mathbf{1}_{(0,s]}$ and $\psi_2 = \Phi\delta\mathbf{1}_{(0,t]}$ , whose pointwise product is $\Phi^2\delta^2\mathbf{1}_{(0,s]}$ :

\operatorname{cov}(M_s, M_t) = \mathbb{E}(M_sM_t) = \int_0^{s}\Phi_u^2\delta(u)^2\,du .

Dividing by $\Phi_s\Phi_t$ gives (7). $\square$

Deduction of the Ornstein-Uhlenbeck law. Put $\alpha \equiv -\beta$ , $\nu \equiv 0$ , $\delta \equiv \sigma$ and $V_0 = u$ . Then $\Phi_t = e^{\beta t}$ , and (5) reads $V_t = ue^{-\beta t} + \sigma\int_0^te^{-\beta(t-s)}\,dW_s$ . By (6) the mean is $ue^{-\beta t}$ and the variance is

\sigma^2e^{-2\beta t}\int_0^te^{2\beta s}\,ds = \sigma^2 e^{-2\beta t}\cdot\frac{e^{2\beta t}-1}{2\beta} = \frac{\sigma^2\bigl(1-e^{-2\beta t}\bigr)}{2\beta} ,

which is item 1 of the statement above. Letting $t \to \infty$ gives item 2, and (7) gives, for $s \le t$ ,

\operatorname{cov}(V_s,V_t) = \frac{\sigma^2}{2\beta}\Bigl(e^{-\beta(t-s)} - e^{-\beta(t+s)}\Bigr) \ \longrightarrow\ \frac{\sigma^2}{2\beta}e^{-\beta(t-s)}

as $s,t \to \infty$ with $t - s$ fixed, which is the stationary autocovariance of item 3.

Remark(Where the hypotheses are used)Original

Additive noise ( $\delta$ independent of $x$ ) is used in Steps 2 and 4, and nowhere can it be dropped. In Step 2 it is what makes $f_{xx}\sigma^2 = 0$ ; in Step 4 it is what makes the stochastic integrals cancel between two solutions. For multiplicative noise the integrating factor must itself be corrected: the equation $dS = S(\mu\,dt+\sigma\,dW)$ is solved not by $\exp(\mu t + \sigma W_t)$ but by $\exp\bigl((\mu-\tfrac12\sigma^2)t+\sigma W_t\bigr)$ , as Grimmett and Stirzaker (Grimmett & Stirzaker (2020), §13.9, Example (13.9.9), p. 612) show. The missing $-\tfrac12\sigma^2 t$ is exactly the term that Step 2 was able to discard.

Deterministic coefficients are used only in Step 5 — but Steps 1-2 have to be reread. If $\alpha, \nu, \delta$ are adapted processes with suitable integrability then (5) is still the solution and uniqueness still holds, but the route to it changes: $\Phi_t = \exp(-\int_0^t\alpha)$ is then a continuous adapted process, not a deterministic function, so Step 1 gives it as a process of finite variation rather than a differentiable function, and Step 2 cannot apply Itô's formula to a deterministic $f$ . One uses instead the product rule for a finite-variation factor times an Itô process, $d(\Phi_tX_t) = \Phi_t\,dX_t + X_t\,d\Phi_t$ , in which there is no cross-variation term precisely because $\Phi$ has finite variation; the two terms in $X_t$ cancel as before. Step 4 goes through with $A = \max_{[0,T]}|\alpha|$ replaced by a pathwise bound. What does not survive is Step 5: $M_t$ is no longer a combination of normals, and nothing in (6)-(7) holds.

Continuous differentiability of $\alpha$ is used in Step 1 only, and only to make $f(t,x) = \Phi_tx$ twice continuously differentiable in both variables, which is what the general Itô formula of The Ito integral and Ito's formula demands. Continuity of $\alpha$ alone makes $\Phi$ merely $C^1$ , so $f_{tt}$ need not exist; the conclusions are unchanged under that weaker hypothesis, but reaching them needs the $C^{1,2}$ form of Itô's formula (once continuously differentiable in $t$ , twice in $x$ ), which this handbook does not state, so we assume $\alpha \in C^1$ rather than appeal to it. Mere continuity of $\alpha$ is all that Step 4 wants, where it bounds $|\alpha|$ on a compact interval and so makes the iterated bound converge; local integrability and a local bound would suffice there.

$\int_0^t\Phi_s^2\delta(s)^2\,ds < \infty$ is the admissibility condition of the Itô integral, used in Step 3. Without it the stochastic integral in (5) need not exist and (6) is meaningless.

$\beta > 0$ in the Ornstein-Uhlenbeck process is what makes items 1-4 of that statement true. At $\beta = 0$ equation (8) is $dV = \sigma\,dW$ , whose solution is a Wiener process: the variance $\sigma^2 t$ grows without bound, there is no limit law and no stationary version. For $\beta < 0$ the mean $ue^{-\beta t}$ and the variance $\sigma^2(e^{2|\beta|t}-1)/(2|\beta|)$ both diverge. Only a restoring drift produces a balance.

The existence and uniqueness assertion for the general equation (1) rests on conditions the source does not state. Tanaka's example above shows that the pathwise part of the conclusion genuinely fails for a discontinuous $\sigma$ , so "rather weak" cannot be read as "none".

Remark(Ito versus Stratonovich)Original

Partition $(0,t]$ into $n$ equal intervals with endpoints $t_j = jt/n$ and write $\Delta_j = W_{t_{j+1}}-W_{t_j}$ . For the integrand $W$ itself, the three natural evaluation points give three different limits in mean square:

evaluation point	limit of $\sum_j W_{\theta_j}\Delta_j$
left, $\theta_j = t_j$ (Itô)	$\tfrac12\bigl(W_t^2 - t\bigr)$
midpoint, $\theta_j = \tfrac12(t_j+t_{j+1})$ (Stratonovich)	$\tfrac12 W_t^2$
right, $\theta_j = t_{j+1}$	$\tfrac12\bigl(W_t^2 + t\bigr)$

The midpoint row is checked as follows. Split $\Delta_j = A_j + B_j$ into its halves, which are independent and each $N(0, t/(2n))$ . Then $W_{\theta_j} = \tfrac12\bigl(W_{t_j}+W_{t_{j+1}}\bigr) + \tfrac12(A_j - B_j)$ . The first part telescopes exactly:

\sum_j \tfrac12\bigl(W_{t_j}+W_{t_{j+1}}\bigr)\bigl(W_{t_{j+1}}-W_{t_j}\bigr) = \tfrac12\sum_j\bigl(W_{t_{j+1}}^2 - W_{t_j}^2\bigr) = \tfrac12 W_t^2 .

The second part contributes $\sum_j\tfrac12(A_j-B_j)(A_j+B_j) = \tfrac12\sum_j(A_j^2-B_j^2)$ , a sum of $n$ independent mean-zero terms each of variance of order $(t/n)^2$ , so of total variance of order $t^2/n \to 0$ . Hence the midpoint limit is $\tfrac12W_t^2$ in mean square. A note on the source: the prose of Grimmett and Stirzaker (Grimmett & Stirzaker (2020), §13.7, p. 604, the paragraph after (13.7.5)) names the midpoint value as $W_t^2$ ; their own Exercise (13.7.3)(c) on the same page, which asks for the limit of $\sum_j\tfrac12(V_{j+1}+V_j)\Delta_j$ , telescopes to $\tfrac12W_t^2$ , and that is the value consistent with the two flanking entries of the table. We use $\tfrac12 W_t^2$ .

The dictionary. Let $X$ solve $dX = \mu\,dt + \sigma(t,X)\,dW$ with $\sigma$ continuously differentiable in $x$ , and let the integrand be $h(X_s)$ for a continuously differentiable $h$ . Repeat the splitting above with $h(X_{\theta_j})$ in place of $W_{\theta_j}$ . Expanding

h(X_{\theta_j}) \approx h(X_{t_j}) + h'(X_{t_j})\bigl(X_{\theta_j}-X_{t_j}\bigr) , \qquad X_{\theta_j}-X_{t_j} \approx \sigma(t_j, X_{t_j})\,A_j

to leading order, the midpoint sum exceeds the left-endpoint sum by

\sum_j \bigl(h'\sigma\bigr)(t_j,X_{t_j})\,A_j\bigl(A_j+B_j\bigr) ,

whose conditional mean is $\sum_j(h'\sigma)(t_j,X_{t_j})\cdot\tfrac12\bigl(t/n\bigr)$ , since $\mathbb{E}(A_j^2) = t/(2n)$ and $\mathbb{E}(A_jB_j) = 0$ . In the limit this is $\tfrac12\int_0^t(h'\sigma)(s,X_s)\,ds$ . So, formally,

\int_0^t h(X_s)\circ dW_s \;=\; \int_0^t h(X_s)\,dW_s \;+\; \tfrac12\int_0^t \bigl(h'\sigma\bigr)(s,X_s)\,ds .

Two checks and one consequence.

The table above is the case $h(x) = x$ and $X = W$ , so that $h' \equiv 1$ and $\sigma \equiv 1$ . The correction is $\tfrac12\int_0^t ds = t/2$ , which is exactly the gap $\tfrac12W_t^2 - \tfrac12(W_t^2-t)$ between the midpoint and left rows.
If the integrand does not depend on the state then $h' \equiv 0$ and there is no correction at all. Additive noise is unambiguous.
In the Stratonovich equation (3) the integrand is the diffusion coefficient, $h = \sigma$ . So the correction is $\tfrac12\int_0^t(\sigma_x\sigma)(s,X_s)\,ds$ , and (3) describes the same process as the Itô equation (1) with $\mu = \tilde\mu + \tfrac12\,\sigma_x\sigma .$

This derivation is formal — it discards remainders without estimating them — and is ours, not the source's; the source gives only the case tabulated above. Where it matters, verify the conversion against a case with a known answer.

Which to use. Three practical points. The Itô integral is a martingale and obeys the isometry, which is why every result in this unit is stated for it. The Stratonovich integral obeys the ordinary chain rule, which makes it the convenient one when a change of variables or a coordinate-free formulation is wanted. And when the noise is additive, $\sigma_x = 0$ and the two coincide — so the Langevin equation below, and every linear equation (4), is unambiguous.

Method(Solving a linear SDE with additive noise)Original

To solve $dX_t = (\alpha(t)X_t+\nu(t))\,dt + \delta(t)\,dW_t$ with $X_0$ given:

Check the form. The diffusion coefficient $\delta$ must not depend on $X$ . If it does, the method does not apply; try instead a substitution such as $Y = \log X$ , remembering that Itô's formula, not the chain rule, governs the substitution.
Build the integrating factor $\Phi_t = \exp\bigl(-\int_0^t\alpha(r)\,dr\bigr)$ , exactly as for the deterministic equation $\dot x = \alpha x + \nu$ .
Differentiate the product with Itô's formula, not with the product rule. Confirm that $f(t,x) = \Phi_tx$ has $f_{xx} = 0$ , so that the Itô correction vanishes and $d(\Phi_tX_t) = \Phi_t\nu\,dt + \Phi_t\delta\,dW_t$ .
Integrate and divide to get (5).
Read off the mean by deleting the stochastic integral, whose expectation is zero.
Read off the variance with the Itô isometry: square the coefficient of $dW$ inside the integral, integrate it against $ds$ , and divide by $\Phi_t^2$ . For the autocovariance at two times, integrate only up to the earlier one, as in (7).
Declare the law. With deterministic coefficients and constant $X_0$ , the answer is Gaussian, so steps 5 and 6 determine it completely; any probability is then a normal-tail evaluation.
Sanity-check the limits. As $t \to 0$ the mean should return $X_0$ and the variance should vanish; as $t \to \infty$ a stable equation ( $\int^t\alpha \to -\infty$ ) should give a finite limiting variance.

Example(An Ornstein-Uhlenbeck process, with numbers)Original

Take $\beta = \tfrac12$ , $\sigma^2 = 3$ and $X_0 = 2$ in equation (8); the time unit is whatever makes the relaxation rate $\tfrac12$ .

Step 1: integrating factor. $\alpha \equiv -\tfrac12$ , so $\Phi_t = e^{t/2}$ .

Step 2: solve. By (5) with $\nu \equiv 0$ ,

X_t = 2e^{-t/2} + \sqrt3\int_0^t e^{-(t-s)/2}\,dW_s .

Step 3: mean.

\mathbb{E}(X_t) = 2e^{-t/2} , \qquad \text{so at } t = 4 : \quad 2e^{-2} = 0.270671 .

Step 4: variance by the isometry.

\operatorname{var}(X_t) = 3\int_0^te^{-(t-s)}\,ds = 3\bigl(1-e^{-t}\bigr) , \qquad \text{so at } t=4 : \quad 3\bigl(1-e^{-4}\bigr) = 2.945053 ,

using $e^{-4} = 0.0183156$ . The standard deviation is $\sqrt{2.945053} = 1.716116$ .

Step 5: check against the general formula. Item 1 of the Ornstein-Uhlenbeck statement predicts variance $\sigma^2(1-e^{-2\beta t})/(2\beta) = 3(1-e^{-4})/1$ , which agrees. The limiting variance is $\sigma^2/(2\beta) = 3$ , and $2.945053$ is already short of it by only $1.83\%$ : after four time units, that is two relaxation times, the process has nearly forgotten where it started.

Step 6: a probability. $X_4$ is normal, so

\mathbb{P}(X_4 > 3) = 1 - \Phi\!\left(\frac{3 - 0.270671}{1.716116}\right) = 1 - \Phi(1.590411) = 0.055871 .

In the stationary state the same question would give $1 - \Phi(3/\sqrt3) = 1-\Phi(1.732051) = 0.041632$ ; the process is still slightly displaced towards its starting value $2$ , which raises the probability.

Step 7: autocovariance. By (7), $\operatorname{cov}(X_s,X_t) = \tfrac{3}{1}\bigl(e^{-(t-s)/2} - e^{-(t+s)/2}\bigr)$ for $s \le t$ . At $s=4$ , $t=6$ this is $3(e^{-1}-e^{-5}) = 1.083424$ , and with $\operatorname{var}(X_6) = 3(1-e^{-6}) = 2.992564$ the correlation is

\frac{1.083424}{\sqrt{2.945053 \times 2.992564}} = 0.364947 ,

against the stationary value $e^{-\beta\cdot 2} = e^{-1} = 0.367879$ . The correlation time, the lag at which the stationary correlation falls to $e^{-1}$ , is $1/\beta = 2$ time units.

Application(The Langevin equation, equipartition and a colloidal bead)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.2, pp. 579-580

Grimmett and Stirzaker introduce the Ornstein-Uhlenbeck process as a repair of the Wiener process as a model of Brownian motion. The objection is physical: a Wiener path is nowhere differentiable, so it is not a path a particle of non-zero mass can follow. Their remedy is to let the velocity, not the position, be the fluctuating variable, with the increment of velocity made of two parts, a frictional drift proportional to the current velocity and a random kick from collisions; this is what they call the Langevin equation. The position is then recovered by integrating the velocity, and so is continuously differentiable.

In continuum form, for a particle of mass $m$ in a fluid of drag coefficient $\gamma$ ,

m\,dV_t = -\gamma V_t\,dt + c\,dW_t , \qquad\text{that is}\qquad dV_t = -\frac{\gamma}{m}V_t\,dt + \frac{c}{m}\,dW_t ,

which is (8) with $\beta = \gamma/m$ and $\sigma = c/m$ . The noise is additive, so there is no Itô-Stratonovich ambiguity, and the method above applies unchanged. By item 2 of the Ornstein-Uhlenbeck statement the velocity settles to

V_\infty \sim N\!\left(0, \frac{\sigma^2}{2\beta}\right) = N\!\left(0, \frac{c^2}{2\gamma m}\right) .

Fixing the noise strength (this part is ours, not the source's). Thermal equilibrium at temperature $T$ requires equipartition, $\tfrac12 m\langle V^2\rangle = \tfrac12 k_BT$ , so $\langle V^2\rangle = k_BT/m$ . Comparing with the display above forces

c^2 = 2\gamma k_B T .

The strength of the random kicks is tied to the strength of the friction: the same molecular collisions cause both. This is the simplest fluctuation-dissipation relation, and it is a consequence of the Ornstein-Uhlenbeck variance formula plus one thermodynamic input.

A number. Take a silica sphere of radius $a = 0.5\,\mu\mathrm{m}$ and density $2200\,\mathrm{kg\,m^{-3}}$ in water at $T = 300\,\mathrm{K}$ , with viscosity $\eta = 8.9\times10^{-4}\,\mathrm{Pa\,s}$ . Then

m = \tfrac43\pi a^3\rho = 1.152\times10^{-15}\,\mathrm{kg} , \qquad \gamma = 6\pi\eta a = 8.388\times10^{-9}\,\mathrm{kg\,s^{-1}} ,

by Stokes' law. The velocity relaxation time is

\frac1\beta = \frac m\gamma = 1.373\times10^{-7}\,\mathrm{s} \approx 0.14\,\mu\mathrm{s} ,

and the root-mean-square thermal velocity is

\sqrt{\frac{k_BT}{m}} = \sqrt{\frac{1.380649\times10^{-23}\times 300}{1.152\times10^{-15}}}\ \mathrm{m\,s^{-1}} = 1.896\times10^{-3}\,\mathrm{m\,s^{-1}} \approx 1.9\,\mathrm{mm\,s^{-1}} .

Both numbers explain why the Wiener process is nevertheless an excellent model of what a microscope sees. The velocity decorrelates in a tenth of a microsecond; on any observable timescale the position is the integral of an already-decorrelated velocity, which by The central limit theorem is approximately a Wiener process, with diffusion constant $k_BT/\gamma$ . Resolving the ballistic regime below $0.14\,\mu\mathrm{s}$ , and so seeing the Ornstein-Uhlenbeck velocity directly, needs an optical trap and nanosecond timing.

The evolution of the probability density of such a process is treated in The Fokker-Planck equation.

Theorem

The Fokker-Planck equation

Adapted

Add to PDF

States the forward (Fokker-Planck) and backward Kolmogorov equations satisfied by the transition density of a one-dimensional diffusion, derives each from Ito's formula - the forward equation by testing against compactly supported functions and integrating by parts, the backward equation from the martingale property of the conditional expectation - and gives the zero-flux recipe for the stationary density, which produces the Boltzmann distribution.

Prerequisites: Stochastic differential equations, The heat kernel, The Ito integral and Ito's formula

Definition(Diffusions, transition densities, and the two operators)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.3, pp. 581 and 583

Let $D = \{D(t) : t \ge 0\}$ be a process with almost surely continuous sample paths. Suppose there are functions $a(t,x)$ and $b(t,x)$ such that, for every $\epsilon > 0$ ,

\mathbb{P}\bigl(|D(t+h) - D(t)| > \epsilon \;\big|\; D(t) = x\bigr) = o(h) , \tag{1}

\mathbb{E}\bigl(D(t+h) - D(t) \;\big|\; D(t) = x\bigr) = a(t,x)\,h + o(h) , \tag{2}

\mathbb{E}\bigl([D(t+h) - D(t)]^2 \;\big|\; D(t) = x\bigr) = b(t,x)\,h + o(h) . \tag{3}

Then $a$ is the instantaneous mean, or drift, and $b$ the instantaneous variance of $D$ . For $s \le t$ the transition density is

f(t,y \mid s,x) = \frac{\partial}{\partial y}\,\mathbb{P}\bigl(D(t) \le y \;\big|\; D(s) = x\bigr) ,

a function of four variables: the pair $(t,y)$ is forward data, the pair $(s,x)$ backward data.

A solution of the stochastic differential equation $dX_t = \mu(t,X_t)\,dt + \sigma(t,X_t)\,dW_t$ of Stochastic differential equations is such a process, with $a = \mu$ and $b = \sigma^2$ ; conditions (2) and (3) are the infinitesimal mean and variance of the increment read off from that equation. Condition (1) is not read off in the same way, but it holds as soon as $\mu$ and $\sigma$ are bounded, say by $K$ . Split the increment as $X_{t+h}-X_t = A_h + M_h$ with $A_h = \int_t^{t+h}\mu\,dr$ and $M_h = \int_t^{t+h}\sigma\,dW$ . The drift part is bounded pathwise, $|A_h| \le Kh$ , so $A_h^4 \le K^4h^4$ . For the Itô part, apply Itô's formula to $y \mapsto y^4$ along $M$ : $M_h^4 = 4\int_0^h M_r^3\sigma\,dW + 6\int_0^hM_r^2\sigma^2\,dr$ . The stochastic integral has mean zero, and the isometry gives $\mathbb{E}(M_r^2) \le K^2r$ , so $\mathbb{E}(M_h^4) \le 6K^2\int_0^hK^2r\,dr = 3K^4h^2$ . Since $(u+v)^4 \le 8(u^4+v^4)$ , this gives $\mathbb{E}\bigl([X_{t+h}-X_t]^4 \mid X_t = x\bigr) = O(h^2)$ , and Markov's inequality applied to the fourth power gives $\mathbb{P}(|X_{t+h}-X_t| > \epsilon \mid X_t = x) \le O(h^2)/\epsilon^4 = o(h)$ . Associated with $(a,b)$ is the second-order differential operator

(\mathcal{L}_t\,g)(x) = a(t,x)\,g'(x) + \tfrac12\,b(t,x)\,g''(x) , \tag{4}

called the generator, and its formal adjoint

(\mathcal{L}^{*}_t\,p)(y) = -\frac{\partial}{\partial y}\bigl[a(t,y)\,p(y)\bigr] + \frac12\frac{\partial^2}{\partial y^2}\bigl[b(t,y)\,p(y)\bigr] . \tag{5}

"Formal adjoint" means that $\int (\mathcal{L}_tg)\,p\,dy = \int g\,(\mathcal{L}^*_tp)\,dy$ for $g$ smooth and compactly supported — integration by parts, twice, with no boundary terms. The discrete-state analogue of this pair is the generator of a continuous-time chain and its transpose; see Continuous-time Markov chains.

Theorem(The forward and backward Kolmogorov equations)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.3, equations (13.3.7) and (13.3.8) with the surrounding hypotheses, pp. 583-584

Let $D$ be a diffusion satisfying (1), (2) and (3), and suppose in addition that $D$ has the Markov property, so that the conditional law of $D(t)$ given the whole history up to time $s$ depends on that history only through $D(s)$ ; let $f(t,y\mid s,x)$ be its transition density. The Markov property is a hypothesis of both equations, not a by-product: it is what makes $f$ a function of the four variables alone, it is the route by which the source reaches (6) and (7), and it is used in Step B2 of the proof below. Subject to further technical conditions, for which Grimmett and Stirzaker refer to Feller (1971), pp. 332-335, and which they do not reproduce, $f$ satisfies the following two partial differential equations.

Forward equation (Fokker-Planck). A note on the name before the equation: Grimmett and Stirzaker call (6) simply the forward equation, and the words "Fokker-Planck" do not appear in their chapter on diffusions. The name is the one used in physics and in the partial-differential-equations literature, where Evans (Evans (2010), §1.2, items 7 and 8, p. 4) lists it as a standard linear equation alongside Kolmogorov's equation; both names are used here, for the same equation. In the forward variables, with $(s,x)$ held fixed,

\frac{\partial f}{\partial t} = -\frac{\partial}{\partial y}\bigl[a(t,y)\,f\bigr] + \frac12\frac{\partial^2}{\partial y^2}\bigl[b(t,y)\,f\bigr] , \qquad\text{that is}\qquad \partial_t f = \mathcal{L}^*_t f . \tag{6}

Backward equation. In the backward variables, with $(t,y)$ held fixed,

\frac{\partial f}{\partial s} = -a(s,x)\,\frac{\partial f}{\partial x} - \frac12\,b(s,x)\,\frac{\partial^2 f}{\partial x^2} , \qquad\text{that is}\qquad \partial_s f = -\mathcal{L}_s f . \tag{7}

Two consequences are worth stating separately.

The pair $(a,b)$ determines everything. The source emphasises that no further information about the distribution of an increment is needed: once the instantaneous mean and variance are known, the transition density is pinned down (subject to boundary conditions, which the source does not specify at this point). This is convenient in applications, where $a$ and $b$ are usually what the physical description supplies.
The density of the process itself obeys (6). Fix the initial state, write $p(t,y) = f(t,y\mid 0,x_0)$ ; then $p$ solves (6) with initial condition the point mass at $x_0$ . More generally, if $D(0)$ has density $p_0$ , then $p(t,\cdot) = \int f(t,\cdot\mid 0,x)p_0(x)\,dx$ solves (6) with $p(0,\cdot) = p_0$ , by linearity.

Three instances, as the source gives them. With $a \equiv 0$ , $b \equiv \sigma^2$ the process is a Wiener process and (6) is the heat equation $\partial_t f = \tfrac12\sigma^2\partial_y^2 f$ , whose solution is the Gaussian transition density; see The heat kernel. With $a \equiv m$ , $b \equiv \sigma^2$ it is a Wiener process with drift, $D(t) = \sigma W(t) + mt$ . With $a(t,x) = -\beta x$ , $b \equiv \sigma^2$ it is the Ornstein-Uhlenbeck process and the forward equation reads $\partial_t f = \beta\,\partial_y(yf) + \tfrac12\sigma^2\partial^2_yf$ .

Idea of the proofOriginal

The two equations answer two different questions, and that is the whole content of the words "forward" and "backward". The forward equation propagates a distribution: given where the process is now, where will the probability be later? The backward equation propagates an observable: given a payoff to be collected at a fixed future time, what is its expected value as a function of where and when you start? The first acts on densities and uses $\mathcal{L}^*$ ; the second acts on functions of the state and uses $\mathcal{L}$ . They are adjoint because total probability is conserved.

Both come out of Itô's formula, in one line each, once the right thing is differentiated. Apply Itô's formula to $g(X_t)$ for a smooth compactly supported $g$ and take expectations. The stochastic integral drops out, because it is a martingale with mean zero, and what remains says

\frac{d}{dt}\,\mathbb{E}\,g(X_t) = \mathbb{E}\bigl[(\mathcal{L}_tg)(X_t)\bigr] .

Written out with the density this is $\int g\,\partial_tp = \int(\mathcal{L}_tg)\,p$ . Move the derivatives off $g$ and onto $p$ by integrating by parts — which is legitimate precisely because $g$ has compact support — and the identity becomes $\int g\,\partial_t p = \int g\,\mathcal{L}^*_tp$ for every test function $g$ . Two functions with the same integral against every test function are equal, and that is the forward equation.

The backward equation is the statement that a conditional expectation of a fixed future quantity is a martingale. Freeze a terminal time $t$ and a payoff $g$ , and let $u(s,x)$ be the expected payoff started from $x$ at time $s$ . By the tower property and the Markov property, $u(s,X_s)$ is a martingale in $s$ . Apply Itô's formula to it: a martingale that is also an Itô process can have no $dt$ term, so its drift $u_s + \mathcal{L}_su$ must vanish identically. That is (7), tested against $g$ .

Proof(Of the forward and backward equations)AdaptedAdapted from Grimmett & Stirzaker (2020), §13.9, Theorem (13.9.2), p. 610, §13.8, Theorem (13.8.11), p. 608, and §13.12, Problem (13.12.24), p. 629

We derive (6) and (7) for a diffusion given as the solution of $dX_t = a(t,X_t)\,dt + \sigma(t,X_t)\,dW_t$ with $b = \sigma^2$ . Grimmett and Stirzaker instead obtain the equations directly from the Markov property and the moment conditions (1)-(3), deferring the technical conditions to Feller; the route below buys concreteness by assuming more, and we list the assumptions rather than hide them.

Standing assumptions. $a$ and $\sigma$ are continuous and bounded, with bounded continuous first and second derivatives in $x$ ; $X_t$ has a density $p(t,\cdot)$ for each $t > 0$ , jointly continuously differentiable once in $t$ and twice in $y$ , together with $a p$ and $bp$ ; differentiation under the integral sign in $t$ is permitted; the equation has, for every starting point $(s_0,x_0)$ , a solution on $[s_0,\infty)$ with $X_{s_0} = x_0$ , and that solution is a Markov process with transition density $f$ ; and the function $u$ of Part B below is twice continuously differentiable on $[0,t]\times\mathbb{R}$ , with bounded derivatives — twice in $s$ as well as twice in $x$ , because Step B3 applies the general Itô formula of The Ito integral and Ito's formula to $u(v,X_v)$ and that formula asks for a $C^2$ function of the pair $(s,x)$ ; the weaker $C^{1,2}$ regularity that the equation itself delivers would do, but this handbook states Itô's formula only in the $C^2$ form. These are the sort of conditions Feller's list supplies; none of them is proved here.

Part A: the forward equation.

Step A1: apply Itô's formula to a test function. Let $g \in C_c^\infty(\mathbb{R})$ , that is, smooth with compact support, and set $Y_t = g(X_t)$ . Since $g$ does not depend on $t$ , Itô's formula gives

g(X_t) = g(X_0) + \int_0^t \Bigl[a(r,X_r)g'(X_r) + \tfrac12 b(r,X_r)g''(X_r)\Bigr]dr + \int_0^t \sigma(r,X_r)g'(X_r)\,dW_r .

Step A2: the stochastic integral has mean zero. Its integrand $\sigma(r,X_r)g'(X_r)$ is adapted and bounded, because $\sigma$ is bounded by assumption and $g'$ is continuous with compact support; so $\int_0^t\mathbb{E}\bigl[(\sigma g')^2\bigr]dr < \infty$ and the integrand is admissible. By the martingale property of the Itô integral, the last term has expectation $0$ for every $t$ .

Step A3: take expectations and differentiate in $t$ . Take expectations in Step A1 and interchange $\mathbb{E}$ with $\int_0^t dr$ . The interchange is Fubini's theorem, licensed because the integrand $(\mathcal{L}_rg)(X_r)$ is bounded — $a$ and $b$ are bounded by the standing assumptions and $g'$ , $g''$ are continuous with compact support — and the region $[0,t]\times\Omega$ has finite measure. Writing $\mathcal{L}_r$ for the generator (4),

\mathbb{E}\,g(X_t) = \mathbb{E}\,g(X_0) + \int_0^t \mathbb{E}\bigl[(\mathcal{L}_rg)(X_r)\bigr]\,dr .

The integrand is bounded and continuous in $r$ , so the right-hand side is differentiable in $t$ and

\frac{d}{dt}\,\mathbb{E}\,g(X_t) = \mathbb{E}\bigl[(\mathcal{L}_tg)(X_t)\bigr] . \tag{8}

Step A4: write (8) with the density. By the definition of $p$ , equation (8) says

\int_{-\infty}^{\infty} g(y)\,\frac{\partial p}{\partial t}(t,y)\,dy = \int_{-\infty}^{\infty}\Bigl[a(t,y)g'(y) + \tfrac12 b(t,y)g''(y)\Bigr]\,p(t,y)\,dy ,

differentiation under the integral sign on the left being permitted by assumption.

Step A5: move the derivatives onto $p$ . Integrate the first term on the right by parts once and the second twice. The support of $g$ is compact, so every boundary term vanishes, and we get

\int g(y)\,\partial_t p\,dy = \int g(y)\left[-\frac{\partial}{\partial y}\bigl(a(t,y)p\bigr) + \frac12\frac{\partial^2}{\partial y^2}\bigl(b(t,y)p\bigr)\right]dy = \int g(y)\,\bigl(\mathcal{L}^*_tp\bigr)(y)\,dy .

Step A6: remove the test function. The difference $h = \partial_tp - \mathcal{L}^*_tp$ is continuous and satisfies $\int gh = 0$ for every $g \in C_c^\infty$ . If $h(y_0) \ne 0$ for some $y_0$ , then by continuity $h$ keeps its sign on an interval around $y_0$ , and choosing a non-negative $g$ supported in that interval and positive at $y_0$ makes $\int gh \ne 0$ , a contradiction. Hence $h \equiv 0$ , which is (6). Applying the argument to the conditional law given $X_s = x$ gives the same equation for $f(t,y\mid s,x)$ .

Part B: the backward equation.

Step B1: the conditional expectation of a terminal payoff. Fix $t$ and $g \in C^\infty_c(\mathbb{R})$ and put, for $0 \le s \le t$ and every $x \in \mathbb{R}$ ,

u(s,x) = \mathbb{E}\bigl(g(X_t) \mid X_s = x\bigr) = \int g(y)\,f(t,y\mid s,x)\,dy .

This defines $u$ at every point of $[0,t]\times\mathbb{R}$ , because the transition density is given for every starting point, not only for the points one particular trajectory visits.

Step B2: fix a starting point and run the diffusion from it. Fix $(s_0,x_0)$ with $0 \le s_0 < t$ , and let $X$ be the solution on $[s_0,t]$ with the deterministic initial value $X_{s_0} = x_0$ , which the standing assumptions provide. Put $M_v = u(v,X_v)$ for $v \in [s_0,t]$ . By the Markov property, $\mathbb{E}(g(X_t)\mid\mathcal{F}_v) = u(v,X_v)$ , so for $r \le v$ the tower property gives

\mathbb{E}\bigl(M_v\mid\mathcal{F}_r\bigr) = \mathbb{E}\bigl(\mathbb{E}(g(X_t)\mid\mathcal{F}_v)\mid\mathcal{F}_r\bigr) = \mathbb{E}\bigl(g(X_t)\mid\mathcal{F}_r\bigr) = M_r ,

that is, $M$ is a martingale on $[s_0,t]$ .

Step B3: compute its drift with Itô's formula. On $[s_0,t]$ ,

dM_v = \Bigl[u_s(v,X_v) + a(v,X_v)u_x(v,X_v) + \tfrac12 b(v,X_v)u_{xx}(v,X_v)\Bigr]dv + \sigma(v,X_v)u_x(v,X_v)\,dW_v .

Write $\alpha_v$ for the bracket; by the standing assumptions it is bounded, and it is continuous in $v$ along almost every path.

Step B4: the drift vanishes at the starting point. The stochastic integral in Step B3 has a bounded adapted integrand, so it is a martingale and has mean zero. Taking expectations in Step B3 between $s_0$ and $s$ , and using $\mathbb{E}M_s = \mathbb{E}M_{s_0} = u(s_0,x_0)$ ,

\mathbb{E}\left(\frac{1}{s-s_0}\int_{s_0}^{s}\alpha_v\,dv\right) = 0 , \qquad s_0 < s \le t .

Now let $s \downarrow s_0$ . Along almost every path the average inside tends to $\alpha_{s_0}$ , because $v \mapsto \alpha_v$ is continuous at $v = s_0$ ; and $X_{s_0} = x_0$ is deterministic, so that limit is the number $\bigl(u_s + au_x + \tfrac12 bu_{xx}\bigr)(s_0,x_0)$ . The averages are bounded uniformly, so bounded convergence lets the limit pass through the expectation. (The source states this zero-drift criterion for the special case of the heat equation in its Problem (13.12.24).)

Step B5: identify the equation. Step B4 says

u_s(s_0,x_0) + a(s_0,x_0)u_x(s_0,x_0) + \tfrac12 b(s_0,x_0)u_{xx}(s_0,x_0) = 0 .

The point $(s_0,x_0)$ was fixed before the process was started and was arbitrary in $[0,t)\times\mathbb{R}$ — this is why Step B2 starts the diffusion afresh from each point instead of following one trajectory, which would deliver the identity only where the law of $X_{s_0}$ lives. So the identity holds at every $(s,x)$ with $s < t$ , and at $s = t$ by continuity in $s$ .

Step B6: remove the test function. Substituting $u(s,x) = \int g(y)f(t,y\mid s,x)\,dy$ and differentiating under the integral sign,

\int g(y)\left[\frac{\partial f}{\partial s} + a(s,x)\frac{\partial f}{\partial x} + \tfrac12 b(s,x)\frac{\partial^2f}{\partial x^2}\right]dy = 0

for every $g \in C_c^\infty$ . By the argument of Step A6 the bracket vanishes, which is (7). $\square$

Remark(Where the hypotheses are used, and two conventions)Original

Compact support of the test function is used three times, in Steps A2, A3 and A5. In A2 and A3 it is one half of the boundedness of $g$ and its first two derivatives, which is what makes the integrand of the stochastic integral admissible and what licenses the Fubini interchange. In A5 it is what kills the boundary terms of the integration by parts, and that is the use that shapes the statement. On a bounded interval with reflecting or absorbing barriers the boundary terms do not vanish, and (6) must be supplemented by boundary conditions; Grimmett and Stirzaker note at this point that they avoid specifying boundary conditions. Zero flux at the ends is the reflecting condition, $f = 0$ at the ends the absorbing one.

Boundedness of $\sigma$ and the compact support of $g'$ are used in Step A2 to put the integrand in the admissible class, so that the stochastic integral is a martingale and not merely a local martingale. Without this the mean of the last term in Step A1 need not be zero, and (8) fails.

The continuity of the drift $\alpha$ at the starting time is used in Step B4. The zero-drift conclusion is read off from a limit of averages over $[s_0,s]$ as $s \downarrow s_0$ , so $v \mapsto \alpha_v$ must be right-continuous at $v = s_0$ ; it is, because $u$ has continuous derivatives, $a$ and $b$ are continuous, and $X$ has continuous paths. For merely measurable coefficients one gets the identity only for almost every $s$ , which is enough for (7) to hold almost everywhere but not pointwise.

Starting the process afresh from each fixed point is what lets Step B5 reach every $x$ . Had we followed one trajectory, Step B4 would give the identity at the random points $(r,X_r)$ , that is, on the support of the law of $X_r$ , and continuity would extend it only to the closure of that support. Nothing here makes that support the whole line — the next paragraph points out that $b$ may vanish, and then the law of $X_r$ can be confined to part of the line. Fixing $(s_0,x_0)$ in advance avoids the question.

Non-degeneracy of $b$ is not needed for the derivation but is needed for the equations to behave like diffusion equations. In the language of The heat kernel and of Evans (Evans (2010), §7.1, equations (2)-(4), pp. 349-350), the operator $\partial_t - \mathcal{L}$ is uniformly parabolic exactly when $\tfrac12 b(t,x) \ge \theta > 0$ for some constant $\theta$ . Where $b$ vanishes the equation degenerates: for Feller's branching diffusion, $a(t,x) = \alpha x$ and $b(t,x) = \beta x$ (Example (13.3.12) of the source, which calls the two constants $a$ and $b$ ; those letters are already spoken for here, as the instantaneous mean and variance, so we write $\alpha$ and $\beta$ for them), $b$ vanishes at the origin, which is exactly the absorbing state of the branching process, and no smoothing occurs there.

Two sign conventions, and the dictionary between them. The partial-differential-equations literature writes the same pair of equations with different letters and a different sign. Evans (Evans (2010), §1.2, items 7 and 8, p. 4) lists, among the standard linear equations, Kolmogorov's equation

u_t - \sum_{i,j} a^{ij}u_{x_ix_j} + \sum_i b^iu_{x_i} = 0

and the Fokker-Planck equation

u_t - \sum_{i,j}\bigl(a^{ij}u\bigr)_{x_ix_j} - \sum_i\bigl(b^iu\bigr)_{x_i} = 0 .

In one dimension these become $u_t = a\,u_{xx} - b\,u_x$ and $u_t = (au)_{xx} + (bu)_x$ . Comparing the second with (6) gives the dictionary

a^{\text{Evans}} = \tfrac12\,b^{\text{G\&S}} , \qquad b^{\text{Evans}} = -\,a^{\text{G\&S}} ,

so that Evans's first-order coefficient is the negative of the probabilists' drift, and his second-order coefficient is half the instantaneous variance. With that substitution his Kolmogorov equation is (7) run forward in time, $\tau = -s$ . The two sources agree; only the normalisation differs. Everything in this unit follows the probabilists' convention of (6) and (7). The $n$ -dimensional forms are Evans's as displayed, with $a^{ij} = \tfrac12\sigma\sigma^{\mathsf T}$ built from the diffusion matrix.

Method(Finding a stationary density)Original

Suppose $a$ and $b$ do not depend on $t$ and $b > 0$ . A stationary density is a probability density $p_\infty$ with $\mathcal{L}^*p_\infty = 0$ ; a process started from it has the same law at every time.

Write the forward equation as a conservation law. Define the probability flux
$J(y) = a(y)p(y) - \tfrac12\frac{d}{dy}\bigl[b(y)p(y)\bigr] ,$
so that (6) reads $\partial_tp = -\partial_yJ$ . A stationary density therefore has $J$ constant in $y$ .
Set the constant to zero when the state space is the whole line, $p$ is to be integrable, and
$a(y)p(y) \to 0 \quad\text{and}\quad b(y)p(y) \to 0 \qquad (y \to \pm\infty) .$
Suppose $J \equiv c$ . Then $\tfrac12(bp)' = ap - J = ap - c \to -c$ , so a non-zero $c$ would force $bp$ to grow linearly, contradicting $bp \to 0$ . Hence $c = 0$ . (On a finite interval with a current driven through it, keep the constant; that is the non-equilibrium case.)

The decay hypothesis, not boundedness of $a$ and $b$ , is what the argument needs. When $a$ and $b$ are bounded it is automatic, since an integrable $p$ with integrable derivative has limits at $\pm\infty$ and those limits must be $0$ . When they are not — the Ornstein-Uhlenbeck drift $a(y) = -\beta y$ of step 5 is unbounded — it has to be checked on the candidate density produced by step 3, and there it holds, because that density decays like a Gaussian and so kills any polynomial factor.
Solve the first-order equation $J \equiv 0$ . Put $q = bp$ . Then $\tfrac12 q' = (a/b)\,q$ , a separable linear equation, so
$q(y) = C\exp\left(\int^y \frac{2a(z)}{b(z)}\,dz\right) , \qquad\text{hence}\qquad p_\infty(y) = \frac{C}{b(y)}\exp\left(\int^y\frac{2a(z)}{b(z)}\,dz\right) . \tag{9}$
Normalise. Fix $C$ by $\int p_\infty = 1$ . If the integral diverges there is no stationary density, and that is the correct answer: a Wiener process with or without drift has none.
Check against a known case. For the Ornstein-Uhlenbeck process, $a = -\beta y$ and $b = \sigma^2$ , the exponent is $-\beta y^2/\sigma^2$ and (9) gives a centred normal density of variance $\sigma^2/(2\beta)$ , matching the limit law quoted in the statement above.
Recognise the Boltzmann form. Whenever $b \equiv 2\kappa$ is constant and $a = -V'$ is a gradient, (9) collapses to
$p_\infty(y) = \frac1Z\,e^{-V(y)/\kappa} ,$
with $Z$ the normalising constant. The letter $\kappa$ is used here, not $D$ , because $D$ already names the process. The next block identifies $\kappa$ with $k_BT$ divided by the friction, and the exponent with the Boltzmann factor.

Example(A Wiener process with drift, from the equation)Original

Take $a \equiv m$ and $b \equiv \sigma^2$ , both constant, with $D(0) = 0$ . We solve the forward equation rather than quoting the process.

Step 1: the equation. By (6), with $a$ and $b$ constant the derivatives pass through:

\frac{\partial p}{\partial t} = -m\,\frac{\partial p}{\partial y} + \frac{\sigma^2}{2}\,\frac{\partial^2 p}{\partial y^2} , \qquad p(0,\cdot) = \delta_0 .

Step 2: remove the drift by moving with it. Put $z = y - mt$ and $\tilde p(t,z) = p(t,z+mt)$ . By the chain rule $\partial_t\tilde p = \partial_tp + m\,\partial_yp$ , so the first-order term cancels and

\frac{\partial\tilde p}{\partial t} = \frac{\sigma^2}{2}\frac{\partial^2\tilde p}{\partial z^2} ,

the heat equation with diffusivity $\kappa = \sigma^2/2$ .

Step 3: quote the heat kernel. By The heat kernel, the solution from a point mass at the origin is $\tilde p(t,z) = (4\pi\kappa t)^{-1/2}e^{-z^2/(4\kappa t)}$ . With $\kappa = \sigma^2/2$ this is $(2\pi\sigma^2t)^{-1/2}\exp\bigl(-z^2/(2\sigma^2t)\bigr)$ .

Step 4: transform back.

p(t,y) = \frac{1}{\sqrt{2\pi\sigma^2t}}\exp\left(-\frac{(y-mt)^2}{2\sigma^2 t}\right) ,

that is, $D(t) \sim N(mt,\ \sigma^2t)$ — which is the law of $\sigma W(t) + mt$ , as the source records in its Example (13.3.10). Substituting this $p$ back into the equation of Step 1 confirms it. Writing $u = y - mt$ , one finds $\partial_yp = -p\,u/(\sigma^2t)$ and $\partial_y^2p = p\bigl[u^2/(\sigma^4t^2) - 1/(\sigma^2t)\bigr]$ , while differentiating in $t$ (remembering that $u$ depends on $t$ ) gives the same thing: both sides equal

p\left[\frac{mu}{\sigma^2 t} + \frac{u^2}{2\sigma^2t^2} - \frac{1}{2t}\right] .

Step 5: no stationary density. Run the method: $a/b = m/\sigma^2$ is constant, so (9) gives $p_\infty \propto e^{2my/\sigma^2}$ , which is not integrable for any $m$ , including $m = 0$ . There is no stationary density, as there should not be: the process wanders off.

Step 6: a number. Take $m = 0.8$ and $\sigma^2 = 0.25$ in their respective units. At $t = 4$ the mean is $mt = 3.2$ and the variance is $\sigma^2t = 1.0$ , so

\mathbb{P}\bigl(D(4) > 5\bigr) = 1 - \Phi\!\left(\frac{5 - 3.2}{1.0}\right) = 1 - \Phi(1.8) = 0.035930 .

Application(Boltzmann statistics and a bistable colloid)Original

An overdamped particle at position $X$ in a potential $U$ , in a fluid of friction coefficient $\gamma$ at temperature $T$ , obeys the Smoluchowski equation: the drift is the force divided by the friction and the diffusion constant is $\kappa = k_BT/\gamma$ , so

dX_t = -\frac{U'(X_t)}{\gamma}\,dt + \sqrt{2\kappa}\,dW_t , \qquad a(y) = -\frac{U'(y)}{\gamma} ,\quad b(y) = 2\kappa .

The forward equation is $\partial_tp = \partial_y\bigl[(U'/\gamma)p + \kappa\,\partial_yp\bigr]$ , whose bracket is minus the flux: the first term is the drift down the potential, the second is Fick diffusion. Step 6 of the method applies with $V = U/\gamma$ and gives

p_\infty(y) = \frac1Z\exp\left(-\frac{U(y)}{\gamma \kappa}\right) = \frac1Z\exp\left(-\frac{U(y)}{k_BT}\right) ,

the Boltzmann distribution, once $\kappa = k_BT/\gamma$ is inserted. This is worth pausing on: equilibrium statistical mechanics is here a one-line consequence of the stationary Fokker-Planck equation plus the Einstein relation between diffusion and mobility. It is also a consistency check on that relation — any other choice of $\kappa$ would give the wrong equilibrium.

A bistable trap, with numbers. A colloidal bead in a double optical trap sees, in reduced units, the potential

U(y) = y^4 - 2y^2 ,

with minima at $y = \pm1$ of depth $U(\pm1) = -1$ and a barrier at $y = 0$ of height $1$ . Take $\gamma\kappa = k_BT = 0.25$ , that is a barrier of four times the thermal energy. Then $p_\infty(y) = Z^{-1}e^{-4(y^4-2y^2)}$ with, by numerical quadrature, $Z = 51.7502$ .

Barrier suppression. $\dfrac{p_\infty(0)}{p_\infty(1)} = e^{-[U(0)-U(1)]/(\gamma\kappa)} = e^{-4} = 0.018316$ : the bead is found at the barrier about one fiftieth as often as at a minimum.
Occupation of the barrier region. $\mathbb{P}\bigl(|X| < 0.5\bigr) = Z^{-1}\int_{-0.5}^{0.5}e^{-4(y^4-2y^2)}\,dy = 0.041655$ , so the bead spends about $4\%$ of its time within half a unit of the barrier top, and the rest in the two wells.
Mean square displacement. $\mathbb{E}\bigl(X^2\bigr) = 0.917671$ in the stationary state, close to the $1$ it would be if the bead sat exactly in one well or the other.
Symmetry. $\mathbb{P}(X > 0) = \tfrac12$ exactly, by the evenness of $U$ .

The last point is the one to be careful with in practice. The stationary density is symmetric, but a single experimental run of finite length is not: the bead sits in one well for a long time, hops, and sits in the other. The time between hops grows like $e^{\Delta U/(k_BT)}$ — Kramers' law, which the stationary density alone does not give, since it is a statement about first passage rather than about the equilibrium distribution. With the barrier only four thermal units high the hops are frequent enough to see; at ten thermal units they are not, and the measured histogram over any feasible run would be one well, not two.