Metric, normed and Hilbert spaces

Completeness, fixed points and the geometry of Hilbert space.

In this chapter 8

Tick a unit to add it to your PDF basket.

Spaces

Spaces

Definition

Metric spaces and completeness

Adapted

Add to PDF

Gives the distance axioms that let convergence be defined on any set, the Cauchy criterion that tests for convergence without naming the limit, and the proofs that the continuous functions on a closed interval are complete in the sup metric but not in the integral metric.

Prerequisites: Limits of sequences and completeness, Uniform convergence

Definition(Metric space)AdaptedAdapted from Kreyszig (1978), §1.1, Def. 1.1-1, p. 3; Hassani (2013), §1.3, Def. 1.3.1, p. 8; Folland (1999), §0.6, p. 13; Stakgold & Holst (2011), §4.3, (4.3.1), p. 234

A metric on a set $X$ is a function $d$ of two points of $X$ that returns a real number, subject to four conditions. For all $x,y,z \in X$ :

(M1) $d(x,y)$ is real, finite and $\ge 0$ ;
(M2) $d(x,y) = 0$ if and only if $x = y$ ;
(M3) $d(x,y) = d(y,x)$ ;
(M4) $d(x,z) \le d(x,y) + d(y,z)$ — the triangle inequality.

The pair $(X,d)$ is a metric space, its elements are points, and $d(x,y)$ is the distance from $x$ to $y$ . We write $X$ alone when $d$ is clear. The labels (M1)–(M4) are Kreyszig's (Kreyszig (1978), §1.1, Def. 1.1-1, p. 3).

Induction on (M4) gives the generalized triangle inequality: $d(x_0,x_n) \le \sum_{j=1}^{n} d(x_{j-1},x_j)$ for any finite chain of points.

If $Y \subseteq X$ , then $Y$ with $d$ restricted to $Y \times Y$ is a subspace of $X$ , carrying the induced metric. The open ball of radius $r>0$ about $x$ is $B(x;r) = \{y \in X : d(x,y) < r\}$ ; a set $U \subseteq X$ is open if every point of $U$ has some ball about it inside $U$ , and closed if its complement is open. The closure $\overline{E}$ of $E \subseteq X$ is the intersection of all the closed sets that contain $E$ , and $E$ is dense in $X$ if $\overline{E} = X$ . A set $E$ is bounded if its diameter $\operatorname{diam} E = \sup\{d(x,y) : x,y \in E\}$ is finite. These are the metric cases of the notions in topological spaces.

$X$ needs no algebraic structure whatever: a metric is defined on a bare set. When $X$ also carries a norm the two structures are linked by $d(x,y) = \lVert x-y\rVert$ , and a function of one variable then does the work of a function of two; that is the subject of normed and Banach spaces.

Note(Axiom lists that look different and are not)AdaptedAdapted from Folland (1999), §0.6, p. 13; Hassani (2013), §1.3, Def. 1.3.1, p. 8; Stakgold & Holst (2011), §4.3, (4.3.1), p. 234; Kreyszig (1978), §1.1, Def. 1.1-1, p. 3

The library's four statements of this definition are packaged differently and say the same thing.

Folland (Folland (1999), §0.6, p. 13) declares $\rho : X\times X \to [0,\infty)$ , so (M1) is absorbed into the codomain, and then lists only (M2), (M3), (M4).
Hassani (Hassani (2013), §1.3, Def. 1.3.1, p. 8) lists three conditions, the first combining nonnegativity with (M2).
Stakgold and Holst (Stakgold & Holst (2011), §4.3, (4.3.1), p. 234) split (M2) into $d(u,v)>0$ for distinct points and $d(u,u)=0$ .
Kreyszig (Kreyszig (1978), §1.1, Def. 1.1-1, p. 3) keeps all four, which is the list used above because each axiom is then available by name.

Nonnegativity is in fact forced by (M2)–(M4): putting $z = x$ in (M4) and using (M3) gives $0 = d(x,x) \le 2d(x,y)$ . It is listed anyway, as all four books do in one form or another, because it is used constantly and a reader should not have to rederive it.

Hassani adds a warning worth repeating: in relativity the squared interval between two events can be negative, so it is not a metric in this sense, and none of what follows applies to it.

Definition(Convergence, Cauchy sequences, completeness)AdaptedAdapted from Kreyszig (1978), §1.4, Def. 1.4-1, p. 25, and Def. 1.4-3, p. 28; Hassani (2013), §1.3, Defs. 1.3.3–1.3.5, pp. 9–10; Folland (1999), §0.6, p. 14; Stakgold & Holst (2011), §4.3, p. 236

Let $(X,d)$ be a metric space and $(x_n)_{n\ge1}$ a sequence in $X$ .

$(x_n)$ converges to $x \in X$ if $d(x_n,x) \to 0$ as $n\to\infty$ : for every $\varepsilon>0$ there is $N$ with $d(x_n,x) < \varepsilon$ for all $n>N$ . We write $x_n \to x$ and call $x$ the limit. A sequence with no limit in $X$ is divergent.
$(x_n)$ is a Cauchy sequence (also fundamental) if for every $\varepsilon>0$ there is $N$ with $d(x_m,x_n) < \varepsilon \qquad \text{for all } m,n > N .$
$X$ is complete when each of its Cauchy sequences has a limit lying in $X$ .

The definition of convergence names the limit; the Cauchy condition does not. That is the entire point of completeness: in a complete space a sequence may be certified convergent by an estimate involving only its own terms, before anything is known about what it converges to. The price is that the certificate says nothing about which point the limit is.

Convergence is not a property of the sequence alone but of the sequence together with the space it is regarded as living in. The sequence $1, \tfrac12, \tfrac13, \dots$ converges in $[0,1]$ and diverges in $(0,1]$ , with no term changed.

Definition(The basic facts)AdaptedAdapted from Kreyszig (1978), §1.4, Lemma 1.4-2, p. 26, Thm 1.4-4, p. 28, Thms 1.4-5 and 1.4-6, pp. 29–30, Thm 1.4-7, p. 30, and Thm 1.4-8, pp. 30–31; Folland (1999), §0.6, Prop. 0.22 and Prop. 0.24, p. 14; Stakgold & Holst (2011), §4.3, p. 236

Let $(X,d)$ and $(Y,\tilde d)$ be metric spaces.

Uniqueness and boundedness. A convergent sequence in $X$ has exactly one limit, and its set of terms is bounded. If $x_n\to x$ and $y_n\to y$ then $d(x_n,y_n)\to d(x,y)$ .
Convergent implies Cauchy. Every convergent sequence in $X$ is Cauchy. The converse holds for every sequence in $X$ precisely when $X$ is complete.
A Cauchy sequence with a convergent subsequence converges, to the same limit.
Closure by sequences. For $E \subseteq X$ , a point $x$ lies in $\overline{E}$ if and only if some sequence in $E$ converges to $x$ . Hence $E$ is closed if and only if it contains the limit of every convergent sequence of its own points.
Completeness and closedness. A subspace $M$ of a complete metric space $X$ is complete exactly when $M$ is closed in $X$ . In an arbitrary metric space, a complete subspace is closed.
Sequential continuity. A map $T : X \to Y$ is continuous at the point $x_0$ exactly when every sequence with $x_n \to x_0$ satisfies $Tx_n \to Tx_0$ .
Which spaces are complete. $\mathbb{R}$ and $\mathbb{C}$ are complete, and so are $\mathbb{R}^n$ and $\mathbb{C}^n$ in the Euclidean metric; so are the sequence spaces $\ell^\infty$ and $\ell^p$ for $1\le p<\infty$ , and so is $C[a,b]$ , the real- or complex-valued continuous functions on a closed bounded interval, in the sup metric $d_\infty(x,y) = \max_{t\in[a,b]} \lvert x(t)-y(t)\rvert .$ Convergence in $d_\infty$ is uniform convergence on $[a,b]$ . Of these, only the case of $C[a,b]$ is proved below. Completeness of $\mathbb{R}$ and $\mathbb{C}$ is taken as known from limits of sequences, and the cases $\mathbb{R}^n$ , $\mathbb{C}^n$ , $\ell^\infty$ and $\ell^p$ are taken on the authority of Kreyszig (Kreyszig (1978), §1.5, Examples 1.5-1, 1.5-2 and 1.5-4, pp. 33–36) and Stakgold and Holst (Stakgold & Holst (2011), §4.3, Examples 1–4, pp. 238–239); each of them runs the three-step scheme of the method block below.
Which are not. $\mathbb{Q}$ with $d(x,y)=\lvert x-y\rvert$ is incomplete, as is any bounded open interval $(a,b)$ with $a<b$ , as is the set of polynomials on $[a,b]$ in the sup metric. Most important here: $C[a,b]$ in the integral metric $d_1(x,y) = \int_a^b \lvert x(t)-y(t)\rvert \,\mathrm{d}t$ is not complete, and neither is it complete in the mean-square metric $d_2(x,y) = \bigl(\int_a^b \lvert x-y\rvert^2\bigr)^{1/2}$ .

Idea of the proofAdaptedAdapted from Kreyszig (1978), §1.5, p. 32

Every estimate in parts 1–6 comes from the triangle inequality applied to one interpolating point. To compare $x$ with $z$ when the hypothesis is about $x$ and $y$ , insert $y$ ; the cost is one extra term, which the hypothesis makes small.

Completeness proofs for concrete spaces all run the same way, and the reason is that the metric is built out of real or complex numbers whose completeness is already known. A Cauchy sequence in the space forces, coordinate by coordinate or point by point, a Cauchy sequence of scalars; those converge, and their limits assemble into a candidate. Two things then remain, and they are where the work is: the candidate must lie in the space, and the convergence must hold in the metric of the space, not merely coordinatewise.

Incompleteness proofs run the other way. One exhibits a sequence whose terms are visibly in the space and whose pointwise limit is visibly not, then shows that no other element of the space can be the metric limit either — because a metric limit would have to agree with the pointwise limit, which it cannot.

Proof(Parts 1–6)AdaptedAdapted from Kreyszig (1978), §1.4, pp. 26–31; Folland (1999), §0.6, p. 14

1. Suppose $x_n\to x$ and $x_n \to z$ . By (M4), $0 \le d(x,z) \le d(x,x_n)+d(x_n,z)$ , and both terms on the right tend to $0$ ; so $d(x,z)=0$ and $x=z$ by (M2). For boundedness, take $\varepsilon=1$ in the definition of convergence to get $N$ with $d(x_n,x)<1$ for $n>N$ ; then $d(x_n,x) \le 1 + \max\{d(x_1,x),\dots,d(x_N,x)\}$ for every $n$ , a finite bound, so all terms lie in one ball. For the third claim, the generalized triangle inequality gives $d(x_n,y_n) \le d(x_n,x)+d(x,y)+d(y,y_n)$ , hence $d(x_n,y_n)-d(x,y) \le d(x_n,x)+d(y,y_n)$ ; interchanging the roles of the two pairs gives the same bound for $d(x,y)-d(x_n,y_n)$ , so $\lvert d(x_n,y_n)-d(x,y)\rvert \to 0$ .

2. If $x_n\to x$ , pick $N$ with $d(x_n,x) < \varepsilon/2$ for $n>N$ . For $m,n>N$ , (M4) and (M3) give $d(x_m,x_n) \le d(x_m,x)+d(x,x_n) < \varepsilon$ . The second sentence is the definition of completeness restated.

3. Let $(x_n)$ be Cauchy with $x_{n_k} \to x$ . Given $\varepsilon>0$ choose $N$ with $d(x_m,x_n)<\varepsilon/2$ for $m,n>N$ , then choose $k$ with $n_k>N$ and $d(x_{n_k},x)<\varepsilon/2$ . For any $n>N$ , $d(x_n,x) \le d(x_n,x_{n_k})+d(x_{n_k},x) < \varepsilon$ . Here the Cauchy hypothesis is what transports the good behaviour of one subsequence to the whole sequence; without it a sequence may have a convergent subsequence and still not converge.

4. If $x \in E$ , the constant sequence works. If $x \in \overline{E}\setminus E$ , then every ball $B(x;1/n)$ meets $E$ — otherwise the complement of that ball would be a closed set containing $E$ but not $x$ , contradicting minimality of the closure — so choosing $x_n \in B(x;1/n)\cap E$ gives $x_n \to x$ . Conversely, if $x_n \in E$ and $x_n \to x$ but $x \notin \overline{E}$ , then the open set $X\setminus\overline{E}$ contains a ball about $x$ that contains no point of $E$ , contradicting $x_n\to x$ . The second sentence is the case $\overline{E}=E$ .

5. Let $X$ be any metric space and $M \subseteq X$ complete as a subspace; completeness of $X$ is not assumed in this half. If $x\in\overline{M}$ , part 4 supplies $x_n \in M$ with $x_n \to x$ ; by part 2 the sequence is Cauchy, by completeness of $M$ it converges in $M$ , and by uniqueness of limits (part 1) that limit is $x$ , so $x \in M$ . Thus $M$ is closed. Conversely let $M$ be closed in the complete space $X$ and let $(x_n)$ be Cauchy in $M$ . It is Cauchy in $X$ , so it converges to some $x \in X$ ; by part 4 and closedness, $x \in M$ . Completeness of the ambient space is used only in this second direction, which is why the first direction — a complete subspace is closed — is stated in part 5 for an arbitrary ambient space.

6. Suppose $T$ is continuous at $x_0$ and $x_n\to x_0$ . Given $\varepsilon>0$ take $\delta>0$ with $\tilde d(Tx,Tx_0)<\varepsilon$ whenever $d(x,x_0)<\delta$ , then $N$ with $d(x_n,x_0)<\delta$ for $n>N$ ; so $\tilde d(Tx_n,Tx_0)<\varepsilon$ for $n>N$ . Conversely, if $T$ is not continuous at $x_0$ there is an $\varepsilon>0$ for which no $\delta$ works; taking $\delta = 1/n$ produces $x_n$ with $d(x_n,x_0)<1/n$ and $\tilde d(Tx_n,Tx_0)\ge\varepsilon$ , so $x_n\to x_0$ while $Tx_n \not\to Tx_0$ . $\square$

Proof(Part 7 for

C[a,b]

, and part 8)AdaptedAdapted from Kreyszig (1978), §1.5, Example 1.5-5 and Thm 1.5-6, pp. 36–37, and Example 1.5-9, pp. 38–39; Stakgold & Holst (2011), §4.3, Example 5, p. 239, and Example 6, pp. 239–240

$C[a,b]$ is complete in $d_\infty$ . Let $(x_m)$ be Cauchy in $d_\infty$ and write $J=[a,b]$ .

A candidate. Fix $t\in J$ . Since $\lvert x_m(t)-x_n(t)\rvert \le d_\infty(x_m,x_n)$ , the numbers $x_1(t),x_2(t),\dots$ form a Cauchy sequence in $\mathbb{R}$ (or $\mathbb{C}$ ), which converges because $\mathbb{R}$ and $\mathbb{C}$ are complete. Call the limit $x(t)$ . This defines a function $x$ on $J$ .
The convergence is uniform. Given $\varepsilon>0$ take $N$ with $d_\infty(x_m,x_n)\le\varepsilon$ for $m,n>N$ . Fix $m>N$ and $t\in J$ and let $n\to\infty$ in $\lvert x_m(t)-x_n(t)\rvert\le\varepsilon$ : the bound survives the limit, giving $\lvert x_m(t)-x(t)\rvert\le\varepsilon$ for every $t\in J$ at once. The bound is independent of $t$ because the sup metric made it so, and that is exactly uniform convergence.
The candidate is in the space. A uniform limit of continuous functions is continuous (uniform convergence), so $x \in C[a,b]$ , and step 2 says $d_\infty(x_m,x)\le\varepsilon$ for $m>N$ , that is $x_m\to x$ in $C[a,b]$ .

Step 2 also proves the parenthetical claim of part 7: convergence in $d_\infty$ is uniform convergence, and conversely.

$C[a,b]$ is not complete in $d_1$ . It is enough to produce one Cauchy sequence with no continuous limit; take $[a,b]=[-1,1]$ and

g_n(t) = \tanh(nt), \qquad n = 1,2,\dots,

each of which is continuous on $[-1,1]$ .

The distances. For $t>0$ the map $n\mapsto\tanh(nt)$ increases, so for $n<m$ we have $\lvert g_m - g_n\rvert = g_m-g_n$ on $(0,1]$ and $= g_n - g_m$ on $[-1,0)$ ; by oddness, $d_1(g_n,g_m) = 2\int_0^1\bigl[\tanh(mt)-\tanh(nt)\bigr]\,\mathrm{d}t = 2\left[\frac{\ln\cosh m}{m} - \frac{\ln\cosh n}{n}\right],$ using $\int_0^1 \tanh(nt)\,\mathrm{d}t = n^{-1}\ln\cosh n$ . Since $\ln\cosh n = n - \ln 2 + \ln(1+e^{-2n})$ , the bracket equals $\ln 2\,(n^{-1}-m^{-1})$ up to a term smaller than $e^{-2n}$ , so $d_1(g_n,g_m) \le 2\ln 2/n \to 0$ and the sequence is Cauchy in $d_1$ .
No continuous limit. Suppose $h\in C[-1,1]$ with $d_1(g_n,h)\to0$ . The same computation with $\operatorname{sgn}$ in place of $g_m$ gives $d_1(g_n,\operatorname{sgn}) = 2(1 - n^{-1}\ln\cosh n) \to 0$ . These two statements are read on the larger set of bounded functions continuous off a finite set, the only place a larger space is used. There the formula $\int_{-1}^{1}\lvert u-v\rvert$ is not a metric but a pseudometric: altering $\operatorname{sgn}$ at the single point $0$ changes no integral, so (M2) fails. Only (M4) is needed, and (M4) does hold, because the pointwise inequality $\lvert u-w\rvert \le \lvert u-v\rvert+\lvert v-w\rvert$ integrates. Hence $\int_{-1}^{1}\lvert h-\operatorname{sgn}\rvert \;\le\; d_1(h,g_n) + d_1(g_n,\operatorname{sgn}) \;\xrightarrow[n\to\infty]{}\; 0 ,$ and the left side does not depend on $n$ , so $\int_{-1}^{1}\lvert h-\operatorname{sgn}\rvert = 0$ . The integrand is nonnegative and continuous except at $0$ , so $h(t)=\operatorname{sgn} t$ for every $t\ne0$ ; letting $t\to0^{\pm}$ forces $h(0)$ to equal both $1$ and $-1$ . No continuous $h$ exists, so $(g_n)$ is Cauchy and divergent in $(C[-1,1],d_1)$ .

The remaining claims of part 8.

A general interval $[a,b]$ . Put $c = \tfrac12(a+b)$ and $\tilde g_n(t) = \tanh\bigl(n(t-c)\bigr)$ . The two steps above run word for word, with $\operatorname{sgn}(t-c)$ as the obstruction, so $(C[a,b],d_1)$ is incomplete for every $a<b$ .
The mean-square metric. Back on $[-1,1]$ : $g_m$ and $g_n$ have the same sign as $t$ and modulus below $1$ , so $\lvert g_m-g_n\rvert\le1$ pointwise, hence $\lvert g_m-g_n\rvert^2 \le \lvert g_m-g_n\rvert$ and $d_2(g_n,g_m)^2 \le d_1(g_n,g_m) \to 0$ . The same sequence is therefore Cauchy in $d_2$ . If some $h \in C[-1,1]$ had $d_2(g_n,h)\to0$ , then Cauchy–Schwarz against the constant function $1$ , in the form $\int_{-1}^{1}\lvert u\rvert\cdot 1 \le \bigl(\int_{-1}^{1}\lvert u\rvert^2\bigr)^{1/2}\bigl(\int_{-1}^{1}1^2\bigr)^{1/2}$ applied to $u=g_n-h$ , would give $d_1(g_n,h) \le \sqrt{2}\,d_2(g_n,h) \to 0$ , which step 2 has just excluded. Boundedness of the interval enters only through the finite factor $\sqrt{b-a}$ , which is $\sqrt2$ here; on a general $[a,b]$ the same two lines work with $\tilde g_n$ .
$\mathbb{Q}$ . The decimal truncations $q_n$ of $\sqrt2$ satisfy $\lvert q_n-q_m\rvert\le 10^{-\min(n,m)}$ , so they are Cauchy in $\mathbb{Q}$ ; a rational limit would also be a limit in $\mathbb{R}$ , and limits there are unique, so it would be $\sqrt2\notin\mathbb{Q}$ .
A bounded open interval $(a,b)$ . The points $a+(b-a)/n$ , $n\ge2$ , lie in $(a,b)$ and are Cauchy; a limit in $(a,b)$ would be the limit taken in $\mathbb{R}$ , namely $a$ , which is not in $(a,b)$ . What the argument actually uses is that the endpoint $a$ is finite, not that the interval is bounded: $(0,\infty)$ is unbounded and incomplete by the same two lines, with the points $1/n$ . Part 8 says bounded because that is the cheapest hypothesis giving one clean statement; what has to be excluded is $\mathbb{R}$ itself, which is an open interval of $\mathbb{R}$ and is complete.
The polynomials on $[a,b]$ in $d_\infty$ . The partial sums $p_n(t)=\sum_{k=0}^{n}t^k/k!$ converge uniformly on $[a,b]$ to $e^t$ , so by step 2 of the first half they are $d_\infty$ -Cauchy. A polynomial limit would have to equal $e^t$ , uniform limits being unique, and $e^t$ is not a polynomial: a polynomial of degree $N$ has vanishing $(N+1)$ -st derivative, while every derivative of $e^t$ is $e^t \ne 0$ .

The sequence $(g_n)$ of the second half, in the same set $C[-1,1]$ , is not Cauchy in $d_\infty$ : $d_\infty(g_n,g_{2n}) \ge \lvert \tanh(2n/n) - \tanh(n/n)\rvert$ evaluated at $t=1/n$ , namely $\tanh 2 - \tanh 1 = 0.2024334\ldots$ , bounded away from $0$ . Completeness is a property of the metric, not of the set. $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Kreyszig (1978), §1.4, Thm 1.4-7, p. 30, §1.5, p. 32, and Examples 1.5-7 to 1.5-9, pp. 37–39; Folland (1999), §0.6, Prop. 0.24 and Thm 0.25, pp. 14–15; Stakgold & Holst (2011), §4.3, pp. 236, 241

(M2) is what makes limits unique. Drop the "only if" half and $d$ becomes a pseudometric: distinct points at distance $0$ are then limits of the same sequences. This is not a pathology to be avoided but the standard situation in integration theory, where $\int\lvert f-g\rvert = 0$ for functions equal almost everywhere; the repair is to pass to equivalence classes, which is exactly what makes $\lVert\cdot\rVert_p$ a norm on $L^p$ rather than a seminorm.
(M4) is the engine of parts 1–6. Every estimate above is one application of the triangle inequality with a single interpolating point. The other axioms enter only to close the arguments those estimates set up: (M2) turns $d(x,z)=0$ into $x=z$ at the uniqueness step of part 1, and (M3) reverses $d(x,x_n)$ to $d(x_n,x)$ in part 2. Nothing about $d$ beyond the four axioms is used anywhere in parts 1–6, which is why those facts hold in every metric space whatever its elements are.
Closedness, in part 5, is relative. $[\tfrac12,1)$ is closed as a subset of $(0,1)$ and not as a subset of $\mathbb{R}$ , as Reed and Simon (Reed & Simon (1980), §I.2, p. 6) warn. Part 5 is therefore a statement about $M$ inside a named $X$ ; "complete" by contrast is intrinsic to $(M,d)$ .
Completeness of the ambient space is needed only one way round in part 5. A complete subspace of any metric space is closed in it; a closed subspace of an incomplete space need not be complete. For instance $\mathbb{Q}$ is closed in $\mathbb{Q}$ and not complete.
Boundedness of the interval, and closedness of it, are used in part 7. On $[a,b]$ a continuous function attains its maximum, so $d_\infty$ is finite and is a maximum rather than a supremum. On $(0,1)$ or on $\mathbb{R}$ one must restrict to the bounded continuous functions for $d_\infty$ to be defined at all; that space is again complete, by the same proof.
Completeness is not inherited by passing to a weaker metric on the same set, as the two metrics on $C[-1,1]$ show, and it is not a topological property: $\mathbb{R}$ and $(0,1)$ are homeomorphic (continuity and homeomorphism), one is complete and the other is not. What is topological is completeness plus total boundedness, which by Folland's Thm 0.25 (Folland (1999), §0.6, p. 15) is exactly compactness.
Every incomplete metric space sits densely inside a complete one, its completion, unique up to a distance-preserving bijection. Stakgold and Holst (Stakgold & Holst (2011), §4.3, Examples 6 and 7, pp. 239–240) take this as the definition of $L_2(a,b)$ : the completion of the continuous functions in the mean-square metric. That is a legitimate route to the space, and it is not the route of $L^p$ spaces, which builds the same space from the Lebesgue integral and then proves completeness. The two descriptions agree; this unit uses neither as a hypothesis.

Note(A subscript convention)AdaptedAdapted from Reed & Simon (1980), §I.2, p. 5

On $C[0,1]$ , Reed and Simon (Reed & Simon (1980), §I.2, p. 5) write $d_1$ for the sup metric and $d_2$ for the integral metric, numbering the two metrics in the order they introduce them. This unit instead indexes by the exponent, so $d_1$ is the integral metric and $d_\infty$ the sup metric, matching $\lVert\cdot\rVert_p$ throughout the handbook. Their observation that $d_2 \le d_1$ reads here as $d_1 \le d_\infty$ on an interval of length $1$ : the integral metric is the weaker of the two, so sup-convergence implies integral-convergence and not conversely.

Method(Proving a space complete, or proving it is not)AdaptedAdapted from Kreyszig (1978), §1.5, p. 32

To show $(X,d)$ is complete, take an arbitrary Cauchy sequence $(x_m)$ and carry out Kreyszig's (Kreyszig (1978), §1.5, p. 32) three steps.

Build a candidate limit $x$ . Use the metric to extract, from the Cauchy condition, a Cauchy sequence of real or complex numbers — one per coordinate, per point of the domain, or per basis element — and let $x$ be assembled from their limits. This step uses completeness of $\mathbb{R}$ and $\mathbb{C}$ and nothing else.
Check $x \in X$ . This is where membership conditions are verified: boundedness for $\ell^\infty$ , summability of the $p$ -th powers for $\ell^p$ , continuity for $C[a,b]$ . It is the step that fails for the spaces in part 8.
Check $d(x_m,x)\to 0$ . Pointwise or coordinatewise convergence is not enough. The standard device is to freeze $m$ in the Cauchy estimate, let the other index go to infinity, and observe that a non-strict inequality survives the limit; finite partial sums are handled first and the number of terms let go to infinity afterwards.

Shortcut: if $X$ is a subspace of a space already known to be complete, do not repeat the argument — show $X$ is closed and quote part 5.

To show $(X,d)$ is not complete, exhibit one Cauchy sequence and rule out every limit:

Choose a sequence in $X$ whose natural limit object (pointwise limit, formal sum, supremum) lies outside $X$ .
Estimate $d(x_m,x_n)$ directly and show it tends to $0$ .
Suppose $x_m \to h$ for some $h\in X$ and derive a contradiction, usually by showing $h$ would have to coincide with the natural limit object.

Example(One set, two metrics, worked numerically)Original

Take $X = C[0,1]$ and the functions $f_n(t) = t^n$ , $n\ge1$ . All lie in $X$ . We compute both distances exactly.

The sup metric. For $n<m$ and $t\in[0,1]$ we have $t^n \ge t^m$ , so

d_\infty(f_n,f_m) = \max_{0\le t\le 1}\bigl(t^n - t^m\bigr) .

Setting the derivative $nt^{n-1} - mt^{m-1}$ to zero gives $t_\ast = (n/m)^{1/(m-n)}$ , the unique interior critical point, and the value there is

d_\infty(f_n,f_m) = \Bigl(\frac{n}{m}\Bigr)^{n/(m-n)}\Bigl(1 - \frac{n}{m}\Bigr).

Take $m = 2n$ . Then $n/m = 1/2$ and $n/(m-n) = 1$ , so

d_\infty(f_n,f_{2n}) = \tfrac12\cdot\tfrac12 = \frac14 \qquad\text{for every } n .

Checking $n=1$ , $m=2$ by hand: $t_\ast = 1/2$ and $t_\ast - t_\ast^2 = \tfrac12-\tfrac14 = \tfrac14$ . The distance between $f_n$ and $f_{2n}$ is therefore the constant $0.25$ , however large $n$ is, so $(f_n)$ is not Cauchy in $d_\infty$ .

The integral metric. For $n<m$ ,

d_1(f_n,f_m) = \int_0^1 (t^n - t^m)\,\mathrm{d}t = \frac{1}{n+1}-\frac{1}{m+1} ,

which is at most $1/(n+1)$ and tends to $0$ : the sequence is Cauchy in $d_1$ . Numerically, $d_1(f_3,f_7) = \tfrac14-\tfrac18 = 0.125$ while $d_\infty(f_3,f_7) = (3/7)^{3/4}(4/7) = 0.3026769\ldots$ .

What each metric sees. In $d_1$ the sequence converges, and its limit is the zero function: $d_1(f_n,0) = 1/(n+1) \to 0$ , and $0 \in C[0,1]$ . In $d_\infty$ it does not converge at all, since a non-Cauchy sequence cannot converge. The pointwise limit is the discontinuous function that is $0$ on $[0,1)$ and $1$ at $t=1$ ; the integral metric cannot distinguish it from $0$ , and the sup metric can.

The checkable number. $d_\infty(f_n,f_{2n}) = 0.25$ exactly, for every $n$ , while $d_1(f_n,f_{2n}) = \dfrac{1}{n+1}-\dfrac{1}{2n+1} = \dfrac{n}{(n+1)(2n+1)}$ , which is $0.0888889$ at $n=4$ and $0.0097069$ at $n=50$ . A single sequence in a single set is Cauchy in one metric and uniformly far from Cauchy in another.

Example(A quantitative version: the rate at which the missing limit is approached)Original

Return to $g_n(t)=\tanh(nt)$ on $[-1,1]$ from the proof, which is $d_1$ -Cauchy in $C[-1,1]$ and has no limit there.

First, the quantity that does not measure this. It is tempting to measure the defect as a distance from the sequence to the space, or from the space to its missing limit. Both are zero. Each $g_n$ is itself a continuous function, so

\inf_{h \in C[-1,1]} d_1(g_n,h) \;=\; 0 \qquad\text{for every } n,

attained at $h = g_n$ . And the missing limit is no further away: in the pseudometric space $P$ of the proof — bounded functions on $[-1,1]$ continuous off a finite set, with $d_1(u,v) = \int_{-1}^{1}\lvert u-v\rvert$ — the continuous functions are $d_1$ -dense. For $0<\eta<1$ let $h_\eta$ equal $\operatorname{sgn}$ off $(-\eta,\eta)$ and be linear across it; then $h_\eta \in C[-1,1]$ and

d_1(\operatorname{sgn},h_\eta) \;=\; 2\int_0^{\eta}\Bigl(1-\frac{t}{\eta}\Bigr)\mathrm{d}t \;=\; \eta \;\longrightarrow\; 0 .

So no positive number separates $C[-1,1]$ from the point it is missing. Incompleteness is never a gap of positive width: it is the absence of a point, and the only thing left to quantify is the rate at which the sequence travels towards the place where that point should be.

The quantity that does. Measure $g_n$ against the limit itself, inside $P$ . For $t>0$ we have $\operatorname{sgn}t = 1$ and $\tanh(nt) < 1$ , and both functions are odd, so with $\int_0^1\tanh(nt)\,\mathrm{d}t = n^{-1}\ln\cosh n$ ,

d_1(g_n,\operatorname{sgn}) \;=\; 2\int_0^1\bigl(1-\tanh(nt)\bigr)\,\mathrm{d}t \;=\; 2\Bigl(1-\frac{\ln\cosh n}{n}\Bigr) .

This is an equality, not a bound, and it is strictly positive for every $n$ : it says by how much $g_n$ still misses the function it is heading for. (Changing $\operatorname{sgn}$ at the single point $0$ changes neither side, which is exactly why $d_1$ is only a pseudometric on $P$ .)

At $n=10$ , $\ln\cosh 10 = 9.3068528\ldots$ and

d_1(g_{10},\operatorname{sgn}) \;=\; 2\bigl(1 - 0.93068528\bigr) \;=\; 0.13862944 ,

in agreement with the asymptotic value $2\ln 2/n = 0.13862944$ to eight decimals; numerical quadrature of $\int_{-1}^{1}\lvert\tanh(10t)-\operatorname{sgn}t\rvert\,\mathrm{d}t$ returns $0.138629436$ , the same number. The gap between the exact value and the asymptotic one is exactly $(2/n)\ln\bigl(1+e^{-2n}\bigr)$ , which at $n=10$ is $4.12\times10^{-10}$ , of order $e^{-2n}/n$ and not of order $e^{-2n}$ . At $n=50$ the distance is $0.027725887$ and at $n=100$ it is $0.013862944$ : the approach is of order $1/n$ , even though away from the origin $g_n \to \operatorname{sgn}$ exponentially fast. The $1/n$ is the width of the window $\lvert t\rvert \lesssim 1/n$ in which $g_n$ has not yet made up its mind, and the integral measures the area of that window.

Meanwhile the mutual distances are $d_1(g_{10},g_{20}) = 2[\,\tfrac{1}{20}\ln\cosh 20 - \tfrac{1}{10}\ln\cosh 10\,] = 0.069314718$ , which is $\ln2/10$ to nine decimals. So by $n=10$ the terms are within $0.07$ of each other and the sequence is going somewhere; the space simply has no point there. Numerical quadrature of $\int_{-1}^{1}\lvert\tanh(10t)-\tanh(20t)\rvert\,\mathrm{d}t$ returns $0.0693147176$ , confirming the closed form.

ApplicationOriginal

Which notion of "close" a calculation needs. Two temperature fields can be close in the mean-square sense and differ by a large amount on a thin layer; two spectra can agree to high accuracy in energy and disagree badly at one frequency. Choosing the metric is choosing which errors count, and the two examples above show how far the choice can go: the same sequence may converge, diverge, or not even be Cauchy. A tolerance quoted without a metric is not a tolerance.

Existence proofs that never write down the answer. The pattern in nearly every existence theorem of analysis is: construct approximations, bound their mutual distances, invoke completeness. The solution is then produced before any formula for it is known. The contraction mapping theorem packages this pattern, and the hypothesis it cannot do without is completeness of the space the iterates live in.

Why $L^2$ and not the continuous functions. Quantum mechanics, signal processing and least-squares estimation all need a space in which a limit of approximations is again a legitimate object. The computation above shows that the continuous functions in a mean-square metric are not such a space: a perfectly well-behaved sequence of smooth states has no continuous limit. Enlarging to $L^2$ is not a technicality but the minimum required for the theory to close, and it is why Hilbert spaces are defined with completeness built in.

Numerical analysis. An iterative solver is stopped when successive iterates are within tolerance — a Cauchy-type test, since the answer is unknown. The test is only meaningful if the space of candidates is complete in the norm being measured, and if that norm is the one in which the answer is required. Convergence of an iteration in a discrete $\ell^2$ norm says nothing about the pointwise error unless the two are compared, and in infinite dimensions they cannot be compared uniformly; see normed and Banach spaces.

Geometry on curved sets. Hassani's (Hassani (2013), §1.3, Example 1.3.2, p. 9) two metrics on a sphere — straight-line chord and great-circle arc — are both legitimate, give the same convergent sequences, and give quite different distances. Navigation uses the second; a linear-algebra embedding uses the first. The general lesson is that a set can carry many metrics, and that theorems about convergence often survive the change while theorems about distance do not.

Theorem

The contraction mapping theorem

Adapted

Add to PDF

A map that shrinks all distances by a fixed factor less than one, on a complete metric space, has exactly one fixed point, reached by iterating from anywhere; the proof supplies two computable error bounds, one usable before the iteration starts and one after each step.

Prerequisites: Metric spaces and completeness

Definition(Fixed point, contraction, iteration)AdaptedAdapted from Kreyszig (1978), §5.1, pp. 299–300, and Def. 5.1-1, p. 300; Stakgold & Holst (2011), §4.4, pp. 245–246

Let $(X,d)$ be a metric space and $T : X \to X$ a map, not assumed linear.

A fixed point of $T$ is a point $x$ with $Tx = x$ .
$T$ is Lipschitz on $X$ with constant $\rho \ge 0$ if $d(Tx,Ty) \le \rho\, d(x,y)$ for all $x,y \in X$ .
$T$ is a contraction on $X$ if it is Lipschitz with some constant $\alpha$ satisfying $\alpha < 1$ . The smallest such $\alpha$ is the contraction constant.
The iteration generated by $T$ from a starting point $x_0 \in X$ is the sequence $x_{n+1} = Tx_n , \qquad n = 0,1,2,\dots,$ so that $x_n = T^n x_0$ .

A contraction is continuous, indeed uniformly so: given $\varepsilon>0$ , $d(x,y)<\varepsilon$ already forces $d(Tx,Ty) \le \alpha\varepsilon < \varepsilon$ .

Solving an equation and finding a fixed point are the same problem in different clothing. Any equation $F(x)=0$ can be rewritten $x = x - \lambda F(x) =: Tx$ for a constant $\lambda \ne 0$ , and more generally as $x = Tx$ in many ways; the art is choosing a rewriting for which $T$ is a contraction.

Note(Three books, three uses of the word)AdaptedAdapted from Reed & Simon (1980), §V.5, p. 151; Stakgold & Holst (2011), §4.4, pp. 245–246; Kreyszig (1978), §5.1, Def. 5.1-1, p. 300; Evans (2010), §9.2.1, p. 502

The terminology is not uniform in the library, and the difference is exactly the strictness of the inequality.

Reed and Simon (Reed & Simon (1980), §V.5, p. 151) call $T$ a contraction when $\rho(Tx,Ty) \le \rho(x,y)$ — no shrinking required — and a strict contraction when there is a $K<1$ . Their Theorem V.18 is about strict contractions.
Evans (Evans (2010), §9.2.1, p. 502) likewise says strict contraction for the condition with $\gamma<1$ .
Kreyszig (Kreyszig (1978), §5.1, Def. 5.1-1, p. 300) and Stakgold and Holst (Stakgold & Holst (2011), §4.4, p. 246) build $\alpha<1$ into the word contraction itself; Stakgold and Holst reserve weak contraction for the strict-inequality-without-a-constant condition $\lVert Tu - Tv\rVert < \lVert u-v\rVert$ , $u \ne v$ .

This unit follows Kreyszig and Stakgold-Holst: contraction means $\alpha<1$ throughout. Readers consulting Reed and Simon or Evans should read their "strict contraction" wherever this unit says "contraction". The distinction is not pedantry — the theorem below is false for a weak contraction, as the remark shows.

Theorem(Banach's fixed point theorem, with error bounds)AdaptedAdapted from Kreyszig (1978), §5.1, Thm 5.1-2, p. 300, Cor. 5.1-3, p. 302, and Thm 5.1-4, p. 303; Stakgold & Holst (2011), §4.4, pp. 246–248; Reed & Simon (1980), §V.5, Thm V.18, p. 151; Evans (2010), §9.2.1, Thm 1, p. 502

1. The theorem. Let $(X,d)$ be a nonempty complete metric space and let $T : X \to X$ be a contraction with constant $\alpha<1$ . Then $T$ has exactly one fixed point $x^\ast \in X$ . For every starting point $x_0 \in X$ the iteration $x_{n+1}=Tx_n$ converges to $x^\ast$ .

2. Error bounds. With $x_0$ arbitrary and $x_n = T^nx_0$ , for every $m \ge 1$ :

\underbrace{d(x_m,x^\ast) \;\le\; \frac{\alpha^{m}}{1-\alpha}\, d(x_0,x_1)}_{\textbf{prior estimate}} , \qquad \underbrace{d(x_m,x^\ast) \;\le\; \frac{\alpha}{1-\alpha}\, d(x_{m-1},x_m)}_{\textbf{posterior estimate}} .

The prior estimate uses only the first step and so predicts, before any computing, how many iterations a given accuracy needs. The posterior estimate uses the step just taken and is never worse than the prior one.

3. Contraction on a ball. Let $(X,d)$ be complete, $x_0 \in X$ , $r>0$ , and let $Y = \{x \in X : d(x,x_0) \le r\}$ . Suppose $T : X \to X$ satisfies $d(Tx,Ty)\le\alpha\, d(x,y)$ for all $x,y \in Y$ , with $\alpha<1$ , and suppose

d(x_0,Tx_0) \;<\; (1-\alpha)\,r .

Then every iterate $x_n = T^nx_0$ lies in $Y$ , the iteration converges to a point $x^\ast \in Y$ , and $x^\ast$ is the unique fixed point of $T$ in $Y$ . (Nothing is claimed about fixed points outside $Y$ .)

4. On a closed subset of a Banach space. Let $X$ be a Banach space, $S \subseteq X$ a closed set, and $T : S \to S$ a contraction. Then $T$ has exactly one fixed point in $S$ , reachable by iteration from any $u_0 \in S$ . This is the form in which the theorem is usually applied, and it is part 1 applied to the complete metric space $S$ (see metric spaces and completeness, part 5).

5. Powers. If $T$ maps a closed subset $S$ of a Banach space into itself and $T^k$ is a contraction for some integer $k\ge1$ , then $T$ has exactly one fixed point in $S$ — even though $T$ itself need not be a contraction, and need not even be continuous.

Idea of the proofOriginal

Each application of $T$ shrinks distances by at least the factor $\alpha$ , so the successive steps of the iteration shrink geometrically: $d(x_m,x_{m+1}) \le \alpha^m d(x_0,x_1)$ . The total distance travelled from step $m$ onward is therefore bounded by the tail of a geometric series, $\alpha^m/(1-\alpha)$ times the first step. A sequence whose remaining journey is summable is Cauchy, and completeness converts that into an actual limit. Continuity of $T$ then moves $T$ inside the limit, which is what makes the limit a fixed point.

Uniqueness is independent of all this and needs no completeness: if two points were both fixed, the map would have to shrink the distance between them while leaving it unchanged, so that distance is zero.

The error bounds are the geometric-series estimate itself, read in two ways. Letting the far index run to infinity in the Cauchy estimate gives the prior bound; restarting the count at $x_{m-1}$ rather than at $x_0$ gives the posterior bound.

Proof(Parts 1 and 2)AdaptedAdapted from Kreyszig (1978), §5.1, Thm 5.1-2, pp. 300–302, and Cor. 5.1-3, p. 302; Stakgold & Holst (2011), §4.4, pp. 246–247

Step 1: uniqueness, which uses only $\alpha<1$ . Suppose $Tx = x$ and $Ty = y$ . Then $d(x,y) = d(Tx,Ty) \le \alpha\, d(x,y)$ , so $(1-\alpha)d(x,y) \le 0$ . Since $\alpha<1$ the factor is positive, so $d(x,y)=0$ and $x=y$ by axiom (M2). Neither completeness nor nonemptiness was used, so a contraction on any metric space has at most one fixed point.

Step 2: consecutive steps shrink geometrically. Fix $x_0 \in X$ (possible because $X \ne \varnothing$ ) and set $x_{n+1}=Tx_n$ . The contraction property applied $n$ times gives

d(x_n,x_{n+1}) = d(Tx_{n-1},Tx_n) \le \alpha\, d(x_{n-1},x_n) \le \cdots \le \alpha^{n} d(x_0,x_1) .

Step 3: the iteration is Cauchy. For $n>m$ , the generalized triangle inequality and step 2 give

d(x_m,x_n) \le \sum_{j=m}^{n-1} d(x_j,x_{j+1}) \le \bigl(\alpha^m + \alpha^{m+1}+\cdots+\alpha^{n-1}\bigr)\,d(x_0,x_1) = \alpha^m\,\frac{1-\alpha^{\,n-m}}{1-\alpha}\,d(x_0,x_1) .

Because $0\le\alpha<1$ we have $1-\alpha^{\,n-m} < 1$ , so

d(x_m,x_n) \;\le\; \frac{\alpha^m}{1-\alpha}\,d(x_0,x_1) \qquad (n>m) . \tag{$\ast$}

The right-hand side does not involve $n$ and tends to $0$ as $m\to\infty$ , and this is exactly where $\alpha<1$ is used a second time: for $\alpha=1$ the sum in step 3 is $(n-m)d(x_0,x_1)$ , which is unbounded. So $(x_m)$ is Cauchy.

Step 4: a limit exists. $X$ is complete, so $x_m \to x^\ast$ for some $x^\ast \in X$ . This is the only use of completeness.

Step 5: the limit is fixed. $T$ is continuous (it is Lipschitz), so by sequential continuity (metric spaces, part 6),

Tx^\ast = T\bigl(\lim_m x_m\bigr) = \lim_m Tx_m = \lim_m x_{m+1} = x^\ast .

With step 1, $x^\ast$ is the unique fixed point, and since $x_0$ was arbitrary every iteration converges to this same point.

Step 6: the prior estimate. Fix $m$ and let $n\to\infty$ in $(\ast)$ . The constant sequence $x_m,x_m,\dots$ converges to $x_m$ and $x_n \to x^\ast$ by step 4, so $d(x_m,x_n) \to d(x_m,x^\ast)$ by the joint continuity of the metric (part 1 of metric spaces, third claim), while the right side of $(\ast)$ does not involve $n$ :

d(x_m,x^\ast) \le \frac{\alpha^m}{1-\alpha}\,d(x_0,x_1) .

Step 7: the posterior estimate. Apply the prior estimate with $x_{m-1}$ in place of $x_0$ . The iteration started at $x_{m-1}$ is $x_{m-1},x_m,x_{m+1},\dots$ and has the same limit $x^\ast$ , so taking the prior estimate at its own first step, that is with exponent $1$ ,

d(x_m,x^\ast) \le \frac{\alpha}{1-\alpha}\, d(x_{m-1},x_m) .

Since $d(x_{m-1},x_m) \le \alpha^{m-1}d(x_0,x_1)$ by step 2, the posterior bound is at most the prior bound; it is usually smaller, because step 2 is an inequality. $\square$

Proof(Parts 3, 4 and 5)AdaptedAdapted from Kreyszig (1978), §5.1, Thm 5.1-4, p. 303; Stakgold & Holst (2011), §4.4, pp. 246–248

Part 3. The one thing to check is that nothing leaves $Y$ , since only there is $T$ known to contract. The hypothesis is a statement about the single point $x_0$ , and the first move is to turn it into a statement about all of $Y$ .

$T$ maps $Y$ into $Y$ . Let $y \in Y$ , so $d(x_0,y)\le r$ . Insert $Tx_0$ and use the contraction bound, which is available because $x_0$ and $y$ both lie in $Y$ :

d(x_0,Ty) \;\le\; d(x_0,Tx_0) + d(Tx_0,Ty) \;\le\; d(x_0,Tx_0) + \alpha\, d(x_0,y) \;<\; (1-\alpha)r + \alpha r \;=\; r .

So $Ty \in Y$ , and $T|_Y$ is a contraction of $Y$ into itself. Since $x_0 \in Y$ , induction gives $x_n = T^nx_0 \in Y$ for every $n$ ; the orbit never escapes, and steps 2 and 3 of the previous proof apply to it verbatim.

The conclusion. $Y$ is closed in $X$ , being where the continuous function $x \mapsto d(x,x_0)$ stays at or below $r$ , and $X$ is complete, so $Y$ is a complete metric space by part 5 of metric spaces. It is nonempty, containing $x_0$ . Part 1 applied to $T|_Y : Y \to Y$ therefore gives a unique fixed point $x^\ast \in Y$ , reached by the iteration from $x_0$ . Nothing has been said about $X \setminus Y$ , where $T$ need not contract.

What the hypothesis buys quantitatively. Taking $m=0$ in the unsimplified estimate of step 3, before $1-\alpha^{\,n}$ was discarded, and using $x_1 = Tx_0$ ,

d(x_0,x_n) \;\le\; \frac{1-\alpha^{n}}{1-\alpha}\, d(x_0,x_1) \;\le\; \frac{1}{1-\alpha}\,d(x_0,Tx_0) \;<\; \frac{(1-\alpha)r}{1-\alpha} = r ,

so the whole orbit, and with it $x^\ast$ , stays strictly inside the ball: the bound $d(x_0,Tx_0)<(1-\alpha)r$ is exactly what leaves room for the geometric series.

Part 4. A closed subset $S$ of a Banach space is a complete metric space in the induced metric $d(u,v) = \lVert u-v\rVert$ (metric spaces, part 5). Apply part 1 to $(S,d)$ . Closedness is what is doing the work: it is the hypothesis that makes the limit of the iterates, which certainly exists in $X$ , belong to $S$ .

Part 5. Let $a$ be the unique fixed point of the contraction $T^k$ . Applying $T$ to $T^k a = a$ gives $T^{k}(Ta) = T(T^k a) = Ta$ , so $Ta$ is also a fixed point of $T^k$ ; uniqueness forces $Ta = a$ , so $a$ is fixed by $T$ . Conversely any fixed point of $T$ is fixed by $T^k$ , hence equals $a$ . Note that continuity of $T$ was never needed — only of $T^k$ , which is automatic. $\square$

Remark(Where the hypotheses are used, and what fails without them)AdaptedAdapted from Kreyszig (1978), §5.1, Thm 5.1-2, pp. 300–302, and Problem 4, p. 304; Stakgold & Holst (2011), §4.4, the remark on p. 247 and Corollary 4.4.2, p. 248

Each hypothesis is used at an identifiable point of the proof, and dropping it kills the theorem.

Completeness enters only at step 4. Without it the iteration still has all the good properties — it is Cauchy, with the same geometric bounds — but may have nowhere to converge to. Stakgold and Holst's (Stakgold & Holst (2011), §4.4, p. 247) example is $S = (0,1)$ with $Tu = u/2$ : a contraction with $\alpha=\tfrac12$ , whose iterates head for $0$ , which is not in $S$ . The failure is of existence only; uniqueness (step 1) survives.
$T$ must map the space into itself. Otherwise the iteration is not defined after the first step. In part 3 this is the content of the hypothesis $d(x_0,Tx_0)<(1-\alpha)r$ , which is a statement about one point and nevertheless keeps the whole orbit inside the ball.
$\alpha<1$ strictly, not $\alpha\le1$ . It is used twice, in step 1 and in step 3, and the second use is the one that breaks. The standard counterexample, Kreyszig's Problem 4 (Kreyszig (1978), §5.1, Problem 4, p. 304) and also Stakgold and Holst's (Stakgold & Holst (2011), §4.4, p. 247), is $X = [1,\infty), \qquad Tx = x + \frac1x .$ $X$ is closed in $\mathbb{R}$ and hence complete. By the mean value theorem $\lvert Tx-Ty\rvert = \lvert 1 - c^{-2}\rvert\,\lvert x-y\rvert$ for some $c$ between $x$ and $y$ , and $0 \le 1-c^{-2} < 1$ for $c \ge 1$ , so $T$ strictly decreases every distance: $\lvert Tx-Ty\rvert < \lvert x-y\rvert$ whenever $x \ne y$ . But $Tx=x$ would need $1/x = 0$ , so there is no fixed point at all. There is no single $\alpha<1$ here, because $1-c^{-2}\to1$ as $c\to\infty$ : uniform shrinking is the hypothesis, not shrinking.
The identity map and a rotation show the two ways $\alpha=1$ can fail: the identity on any $S$ has every point fixed, and a rotation of a closed annulus (Stakgold & Holst (2011), §4.4, p. 247) through a nonzero angle less than $2\pi$ has none, while both are isometries.
Compactness repairs the weak case. A weak contraction on a compact set does have a fixed point, since $x\mapsto d(x,Tx)$ is then continuous on a compact set and attains its minimum, which must be $0$ . Compactness is a strong hypothesis and is what the $[1,\infty)$ example lacks.
Continuity of $T$ is not a separate assumption in parts 1–4; it follows from the Lipschitz bound. It is a real gain in part 5: a wildly discontinuous $T$ with a contractive power still has a unique fixed point, which is why one often checks $T^k$ rather than $T$ .
The error bounds need no extra hypotheses, but they do need the contraction constant $\alpha$ , not merely the existence of one. A bound computed with an overestimate $\alpha' > \alpha$ is still valid (both sides increase), so in practice one uses any $\alpha$ one can certify.

Method(Applying the theorem)Original

Rewrite the problem as $u = Tu$ . For $F(u)=0$ the family $T_\lambda u = u - \lambda F(u)$ gives one rewriting per $\lambda$ ; for a differential equation, integrate it to get an integral equation, because integration maps continuous functions to continuous functions whereas differentiation does not.
Choose the space. Pick a complete metric space, or a closed subset $S$ of a Banach space, and check that $T$ maps it into itself. For an unbounded nonlinearity this usually means shrinking $S$ to a ball and verifying the hypothesis of part 3.
Estimate the contraction constant. On an interval of $\mathbb{R}$ , $\alpha = \sup\lvert T'\rvert$ by the mean value theorem; for an integral operator with kernel $k$ , $\alpha \le \sup_s \int \lvert k(s,t)\rvert\,\mathrm{d}t$ in the sup norm; for a matrix iteration $x \mapsto Cx + b$ , $\alpha = \lVert C\rVert$ in whichever induced norm is being used.
If $\alpha \ge 1$ , do not give up. Three standard repairs: shrink the domain (a small interval, a small time horizon, a small parameter — this is the pattern Evans (Evans (2010), §9.2.1, pp. 502–503) identifies in nonlinear partial differential equations); change to an equivalent metric that is weighted so as to make $\alpha<1$ ; or apply part 5 to a power $T^k$ .
Iterate and stop on the posterior bound. Compute $\alpha\,d(x_{m-1},x_m)/(1-\alpha)$ after each step and stop when it is below tolerance. Use the prior bound beforehand to budget the number of steps: $m \ge \log\bigl[\varepsilon(1-\alpha)/d(x_0,x_1)\bigr]/\log\alpha$ guarantees accuracy $\varepsilon$ .
Remember what is proved. Existence, uniqueness in the set chosen, and a rate. A fixed point outside that set is not excluded, and the convergence rate is only linear, with error multiplied by about $\alpha$ each step — quadratically convergent methods such as Newton's do better precisely because their contraction constant tends to $0$ near the root.

Example(Kepler's equation, with both error bounds)Original

A body on an elliptical orbit of eccentricity $e$ has eccentric anomaly $E$ related to the mean anomaly $M$ (which advances uniformly in time) by Kepler's equation

E - e\sin E = M .

There is no elementary solution for $E$ . Rewrite it as a fixed-point problem,

E = T(E) := M + e\sin E ,

and take $X = \mathbb{R}$ with the usual metric, which is complete, and $T : \mathbb{R}\to\mathbb{R}$ .

Step 1: the contraction constant. $T'(E) = e\cos E$ , so $\lvert T'\rvert \le e$ everywhere, and by the mean value theorem $\lvert T(E)-T(F)\rvert \le e\,\lvert E-F\rvert$ . For any bound orbit $e<1$ , so $T$ is a contraction with $\alpha = e$ , and Kepler's equation has exactly one solution for every $M$ . (For $e \to 1$ the bound degrades, which is the numerical difficulty with near-parabolic orbits.)

Step 2: numbers. Take $e = 0.3$ and $M = 0.5$ radians, so $\alpha = 0.3$ and $\alpha/(1-\alpha) = 3/7 = 0.4285714$ . Start at $x_0 = M = 0.5$ :

$m$	$x_m$
$0$	$0.500000000000$
$1$	$0.643827661581$
$2$	$0.680078363070$
$3$	$0.688656186522$
$4$	$0.690650059101$
$5$	$0.691111520955$
$6$	$0.691218213482$

The first step is $d(x_0,x_1) = 0.143827661581$ .

Step 3: the prior bound at $m=6$ .

d(x_6,E^\ast) \le \frac{\alpha^6}{1-\alpha}\,d(x_0,x_1) = \frac{0.000729}{0.7}\times 0.143827661581 = 1.49786\times10^{-4} .

Step 4: the posterior bound at $m=6$ . $d(x_5,x_6) = 1.066925\times10^{-4}$ , so

d(x_6,E^\ast) \le \tfrac37 \times 1.066925\times10^{-4} = 4.57254\times10^{-5} ,

about three times sharper than the prior bound, as advertised.

Step 5: the true error. Solving $E - 0.3\sin E = 0.5$ to high precision gives

E^\ast = 0.691250289594 ,

so the actual error at $m=6$ is $3.20761\times10^{-5}$ . Both bounds hold, the posterior one by a factor $1.43$ and the prior one by a factor $4.67$ .

Step 6: budgeting in advance. To reach $\varepsilon = 10^{-10}$ the prior bound asks for

m \ \ge\ \frac{\ln\bigl[\varepsilon(1-\alpha)/d(x_0,x_1)\bigr]}{\ln\alpha} = \frac{\ln(4.8669\times10^{-10})}{\ln 0.3} = 17.81 ,

so $m=18$ suffices; the bound there is $7.96\times10^{-11}$ , while $m=17$ gives $2.653\times10^{-10}$ and is not enough. This is the whole practical value of the prior estimate: the iteration count is known before a single sine is evaluated.

ApplicationOriginal

Existence for differential equations. Kreyszig (Kreyszig (1978), §5.3, Thm 5.3-1, pp. 315–317) proves Picard's theorem by this route. For $x'=f(t,x)$ , $x(t_0)=x_0$ with $f$ continuous and bounded by $c$ on a rectangle $\lvert t-t_0\rvert\le a$ , $\lvert x-x_0\rvert\le b$ , and Lipschitz in $x$ with constant $k$ , the map

(Tx)(t) = x_0 + \int_{t_0}^{t} f\bigl(\tau,x(\tau)\bigr)\,\mathrm{d}\tau

sends the closed subset $\{x \in C(J) : \lvert x(t)-x_0\rvert \le c\beta\}$ of $C(J)$ , $J=[t_0-\beta,t_0+\beta]$ , into itself and contracts it with $\alpha = k\beta$ , provided

\beta < \min\{a,\ b/c,\ 1/k\} .

The fixed point is the solution. Existence and uniqueness of solutions proves the same theorem by the telescoping-series route of Coddington and gets the longer interval $\alpha_{\text{C}} = \min\{a, b/M\}$ with no $1/k$ term; Kreyszig flags this himself in a footnote, noting that the classical interval is better and that a weighted metric would recover it. The two statements are both correct and are not the same statement, so a reader needing the largest guaranteed interval should use the ODE unit's version.

Integral equations. For a Fredholm equation $u(s) = g(s) + \lambda\int_a^b k(s,t)u(t)\,\mathrm{d}t$ , the right-hand side defines $T$ on $C[a,b]$ with $\alpha = \lvert\lambda\rvert\,\sup_s\int_a^b\lvert k(s,t)\rvert\,\mathrm{d}t$ . Where $\alpha<1$ the iterates are precisely the partial sums of the Neumann series, so the theorem is the convergence proof for that series and the prior bound is its truncation error. For Volterra equations the factor $t-a$ makes some power $T^k$ a contraction even when $T$ is not, which is part 5 in action.

Nonlinear partial differential equations. Evans (Evans (2010), §9.2.1, pp. 502–503) solves a reaction-diffusion system $\mathbf{u}_t - \Delta\mathbf{u} = \mathbf{f}(\mathbf{u})$ by contraction in $C([0,T];L^2)$ , with the Lipschitz constant of $\mathbf{f}$ times $T$ playing the role of $\alpha$ ; one solves on a short time interval and restarts. The restart loses nothing here: the length $T_1$ of the usable interval depends only on the Lipschitz constant of $\mathbf{f}$ , so finitely many repetitions reach an arbitrary prescribed $T$ , and Evans' Theorem 2 (Evans (2010), §9.2.1, Thm 2, pp. 503–505) concludes with a unique weak solution on all of $[0,T]$ . What the contraction step by itself delivers is short-time existence; it becomes global existence only because the step size does not shrink. Evans notes that a small parameter of this kind is the signature of every such proof, and that the globally Lipschitz hypothesis which keeps $T_1$ fixed is unrealistic — for a polynomial nonlinearity the restart time does shrink, and whether the solution lives forever or blows up is then a genuine question.

Numerical linear algebra. The splitting $Ax=b \Rightarrow x = Cx + d$ underlying the Jacobi and Gauss-Seidel iterations is a contraction exactly when $\lVert C\rVert<1$ in some induced norm, and the theorem then gives both convergence and the stopping rule; see iterative methods for linear systems and relaxation methods. The same bound governs fixed-point iteration for scalar equations and the local convergence of Newton's method, where $T(x)=x-f(x)/f'(x)$ has $T'(x^\ast)=0$ at a simple root, so $\alpha$ can be made as small as one likes by starting close enough.

Orbital mechanics and ephemerides. The worked example is the standard way Kepler's equation is solved in practice for moderate eccentricity; the contraction constant is the eccentricity, which explains both why the iteration is fast for planets ( $e \approx 0.0167$ for Earth, giving an error reduction of about $60$ per step) and why comets need a different method.

Self-similar sets. A finite family of contractions on a complete metric space induces a contraction on the space of compact subsets with the Hausdorff metric; its unique fixed point is the attractor, which is how the Cantor set, the Koch curve and the other objects of fractals are defined and drawn.

Definition

Normed and Banach spaces

Adapted

Add to PDF

Gives the norm axioms and the metric they induce, the series test for completeness, the standard Banach spaces and the standard incomplete ones, and proves that on a finite-dimensional space every norm is equivalent to every other, so that in finite dimensions the choice of norm never matters and in infinite dimensions it always may.

Prerequisites: Metric spaces and completeness, Vector spaces and bases, Compactness

Definition(Norm, Banach space, equivalent norms)AdaptedAdapted from Kreyszig (1978), §2.2, Def. 2.2-1, pp. 58–59, and Def. 2.4-4, p. 75; Folland (1999), §5.1, pp. 151–152; Stakgold & Holst (2011), §4.3, (4.3.2), p. 235; Reed & Simon (1980), §I.2, p. 6; Evans (2010), App. D.1, p. 674

Let $X$ be a vector space over $\mathbb{K}$ , where $\mathbb{K}=\mathbb{R}$ or $\mathbb{C}$ . A norm on $X$ is a real-valued function $x\mapsto\lVert x\rVert$ such that for all $x,y\in X$ and all $\alpha\in\mathbb{K}$ :

(N1) $\lVert x\rVert \ge 0$ ;
(N2) $\lVert x\rVert = 0$ if and only if $x = 0$ ;
(N3) $\lVert \alpha x\rVert = \lvert\alpha\rvert\,\lVert x\rVert$ ;
(N4) $\lVert x+y\rVert \le \lVert x\rVert+\lVert y\rVert$ — the triangle inequality.

The pair $(X,\lVert\cdot\rVert)$ is a normed space. A function satisfying (N1), (N3) and (N4) but not necessarily (N2) is a seminorm; Folland (Folland (1999), §5.1, pp. 151–152) starts from seminorms because $\lVert f\rVert_1 = \int\lvert f\rvert$ is one on the actual functions and becomes a norm only after functions equal almost everywhere are identified.

Every norm induces a metric,

d(x,y) = \lVert x-y\rVert ,

the metric induced by the norm, and the norm is recovered from it as $\lVert x\rVert = d(x,0)$ . The axioms (M1)–(M4) of a metric follow: (M1) from (N1), (M2) from (N2), (M3) from (N3) with $\alpha=-1$ , and (M4) from (N4) applied to $(x-y)+(y-z)$ . A normed space is therefore a metric space, and all the vocabulary of that unit — convergence, Cauchy sequence, open set, closure, density, completeness — applies unchanged.

A Banach space is a normed space that is complete in the induced metric.

Two norms $\lVert\cdot\rVert_a$ and $\lVert\cdot\rVert_b$ on the same vector space $X$ are equivalent if there are constants $c, C>0$ with

c\,\lVert x\rVert_b \;\le\; \lVert x\rVert_a \;\le\; C\,\lVert x\rVert_b \qquad \text{for all } x \in X .

Because the inequality is two-sided it is a genuine equivalence relation. In a normed space the metric comes from a function of one variable rather than two, and the metric is translation-invariant and homogeneous: $d(x+z,y+z)=d(x,y)$ and $d(\alpha x,\alpha y) = \lvert\alpha\rvert\,d(x,y)$ . Not every metric on a vector space has these properties, so not every metric space that happens to be a vector space is a normed space.

Definition(What norms and completeness give)AdaptedAdapted from Kreyszig (1978), §2.2, (2) and the continuity of the norm, p. 60, Examples 2.2-2 to 2.2-7, pp. 60–62, §2.4, Thms 2.4-2, 2.4-3 and 2.4-5, pp. 73–76, §2.5, Thm 2.5-3, pp. 77–78, and Thm 2.5-5, p. 80; Folland (1999), §5.1, Thm 5.1, p. 152, and Exercise 6, p. 155; Stakgold & Holst (2011), §4.3, Examples 3, 5 and 6, pp. 238–240, and p. 241; Evans (2010), App. D.1, p. 675

Let $X$ be a normed space over $\mathbb{K}$ .

The norm is continuous. $\bigl\lvert\,\lVert x\rVert-\lVert y\rVert\,\bigr\rvert \le \lVert x-y\rVert$ , so $x\mapsto\lVert x\rVert$ is $1$ -Lipschitz from $X$ to $\mathbb{R}$ . Addition and scalar multiplication are continuous as well.
Equivalent norms are interchangeable. If $\lVert\cdot\rVert_a$ and $\lVert\cdot\rVert_b$ are equivalent they have the same open sets, the same convergent sequences with the same limits, the same Cauchy sequences, the same bounded sets, and the same closed sets; in particular $X$ is complete in one if and only if it is complete in the other.
The series test for completeness. $X$ is a Banach space if and only if every absolutely convergent series in $X$ converges, that is, $\sum_{n\ge1}\lVert x_n\rVert<\infty$ implies that the partial sums $\sum_{n\le N}x_n$ converge in $X$ .
Banach spaces. $\mathbb{K}^n$ with any of the norms $\lVert x\rVert_p = \Bigl(\sum_{j=1}^{n}\lvert x_j\rvert^{p}\Bigr)^{1/p}\ \ (1\le p<\infty), \qquad \lVert x\rVert_\infty = \max_{j}\lvert x_j\rvert ;$ the sequence spaces $\ell^p$ ( $1\le p<\infty$ ) and $\ell^\infty$ with the corresponding norms; the convergent sequences $c$ and the null sequences $c_0$ with the $\ell^\infty$ norm; $C[a,b]$ with $\lVert x\rVert_\infty = \max_{t\in[a,b]}\lvert x(t)\rvert$ ; the bounded continuous functions on any topological space with the uniform norm; and $L^p(\mu)$ for $1\le p\le\infty$ on any measure space.
Normed spaces that are not Banach spaces. $C[a,b]$ with $\lVert x\rVert_1 = \int_a^b\lvert x(t)\rvert\,\mathrm{d}t$ , or with $\lVert x\rVert_2 = (\int_a^b\lvert x\rvert^2)^{1/2}$ ; the polynomials on $[a,b]$ with the sup norm; and the sequences with only finitely many nonzero terms, with any $\ell^p$ norm. Each is a dense subspace of a Banach space and is not closed in it.
Finite dimensions. Let $X$ $X$ be a normed space.
- (Linear combinations.) If $x_1,\dots,x_n \in X$ are linearly independent, there is a constant $c>0$ , depending on the $x_j$ and the norm, with $\Bigl\lVert \sum_{j=1}^{n}\alpha_jx_j \Bigr\rVert \;\ge\; c\sum_{j=1}^{n}\lvert\alpha_j\rvert \qquad\text{for all scalars } \alpha_1,\dots,\alpha_n .$
- Every finite-dimensional subspace of $X$ is complete, hence closed in $X$ .
- Every norm on a finite-dimensional vector space is equivalent to every other.
- In a finite-dimensional normed space a subset is compact if and only if it is closed and bounded. In an infinite-dimensional normed space this fails: by Kreyszig's Thm 2.5-5 (Kreyszig (1978), §2.5, p. 80), a normed space whose closed unit ball is compact is finite-dimensional, so in infinite dimensions the closed unit ball is never compact.

Part 6 is what makes finite-dimensional linear algebra metrically featureless: convergence, completeness, continuity and compactness there do not depend on the norm, and one may always choose whichever is convenient. The statements in part 5 and the last clause of part 6 are what is lost in infinite dimensions.

Idea of the proofOriginal

A norm is a metric that respects the linear structure, and every statement in part 1 and part 2 is the consequence of that single compatibility.

The series test (part 3) exploits the fact that in a normed space one may add. A Cauchy sequence can be thinned to a subsequence whose consecutive gaps are summable, and such a subsequence is the partial-sum sequence of an absolutely convergent series; conversely a series with summable norms has Cauchy partial sums. So the two conditions say the same thing about the space, and the series form is easier to verify because one controls one number per term.

Finite dimensions (part 6) all rest on one lemma: on the set of coefficient vectors of norm $1$ , the length $\lVert\sum\alpha_jx_j\rVert$ has a positive minimum. The reason is compactness of that set in $\mathbb{K}^n$ , which is Bolzano-Weierstrass, together with linear independence, which forbids the minimum from being $0$ . Once the lemma is in hand, completeness, closedness and the equivalence of all norms are three readings of the same inequality: a norm on a finite-dimensional space is squeezed between two multiples of the coordinate sum $\sum_j\lvert\alpha_j\rvert$ , and so is any competing norm.

Proof(Parts 1, 2 and 3)AdaptedAdapted from Kreyszig (1978), §2.2, p. 60; Folland (1999), §5.1, Thm 5.1, p. 152

Part 1. By (N4), $\lVert x\rVert = \lVert (x-y)+y\rVert \le \lVert x-y\rVert+\lVert y\rVert$ , so $\lVert x\rVert-\lVert y\rVert \le \lVert x-y\rVert$ . Exchanging $x$ and $y$ and using $\lVert y-x\rVert = \lVert x-y\rVert$ from (N3) gives the same bound for $\lVert y\rVert-\lVert x\rVert$ , hence the stated inequality; a $1$ -Lipschitz map is continuous. For addition, $\lVert (x_n+y_n)-(x+y)\rVert \le \lVert x_n-x\rVert+\lVert y_n-y\rVert$ ; for scalar multiplication, $\lVert \alpha_nx_n-\alpha x\rVert \le \lvert\alpha_n\rvert\lVert x_n-x\rVert+\lvert\alpha_n-\alpha\rvert\lVert x\rVert$ , and $(\alpha_n)$ is bounded because it converges.

Part 2. Suppose $c\lVert\cdot\rVert_b \le \lVert\cdot\rVert_a \le C\lVert\cdot\rVert_b$ . Then $\lVert x_n-x\rVert_a\to0$ if and only if $\lVert x_n-x\rVert_b\to0$ , because each quantity is squeezed by a positive multiple of the other; the same two-sided squeeze applies to $\lVert x_m-x_n\rVert$ , giving the same Cauchy sequences, and to $\sup$ over a set, giving the same bounded sets. Every ball of one norm contains a ball of the other with the same centre and a rescaled radius, so the two collections of open sets coincide, and hence so do the closed sets. Completeness is "every Cauchy sequence converges", and both halves of that sentence have just been shown to be norm-independent. Two-sidedness is essential: a one-sided bound $\lVert\cdot\rVert_a \le C\lVert\cdot\rVert_b$ gives only that $b$ -convergence implies $a$ -convergence.

Part 3. Complete $\Rightarrow$ absolutely convergent series converge. Let $\sum\lVert x_n\rVert<\infty$ and $s_N = \sum_{n=1}^{N}x_n$ . For $N>M$ , $\lVert s_N-s_M\rVert \le \sum_{n=M+1}^{N}\lVert x_n\rVert$ , which is a tail of a convergent series of nonnegative numbers and so tends to $0$ as $M,N\to\infty$ . Thus $(s_N)$ is Cauchy and converges.

Converse. Let $(x_n)$ be Cauchy. Choose indices $n_1<n_2<\cdots$ with $\lVert x_m-x_n\rVert<2^{-j}$ for all $m,n\ge n_j$ . Put $y_1 = x_{n_1}$ and $y_j = x_{n_j}-x_{n_{j-1}}$ for $j\ge2$ . Then $\sum_{j\ge1}\lVert y_j\rVert \le \lVert x_{n_1}\rVert + \sum_{j\ge2}2^{-(j-1)} = \lVert x_{n_1}\rVert+1<\infty$ , so by hypothesis the series $\sum_jy_j$ converges; its $k$ -th partial sum telescopes to $x_{n_k}$ , so the subsequence $(x_{n_k})$ converges. A Cauchy sequence with a convergent subsequence converges to the same limit (metric spaces, part 3). $\square$

Proof(Part 6)AdaptedAdapted from Kreyszig (1978), §2.4, Lemma 2.4-1, pp. 72–73, Thm 2.4-2, pp. 73–74, and Thm 2.4-5, pp. 75–76

The linear-combinations lemma. Write $s = \sum_j\lvert\alpha_j\rvert$ . If $s=0$ all the $\alpha_j$ vanish and the inequality holds for any $c$ . If $s>0$ , divide by $s$ : with $\beta_j = \alpha_j/s$ the claim becomes

\Bigl\lVert\sum_{j=1}^{n}\beta_jx_j\Bigr\rVert \ \ge\ c \qquad\text{whenever } \sum_{j=1}^{n}\lvert\beta_j\rvert = 1 . \tag{$\dagger$}

Suppose $(\dagger)$ fails for every $c>0$ . Then there are coefficient vectors $\beta^{(m)} = (\beta^{(m)}_1,\dots,\beta^{(m)}_n)$ with $\sum_j\lvert\beta^{(m)}_j\rvert=1$ and $y_m := \sum_j\beta^{(m)}_jx_j$ satisfying $\lVert y_m\rVert\to0$ . Each coordinate satisfies $\lvert\beta^{(m)}_j\rvert\le1$ , so the vectors $\beta^{(m)}$ lie in a closed bounded subset of $\mathbb{K}^n$ ; by Bolzano-Weierstrass (compactness) a subsequence converges, say $\beta^{(m_i)}\to\beta$ with $\sum_j\lvert\beta_j\rvert=1$ , so not all $\beta_j$ are zero. Set $y = \sum_j\beta_jx_j$ . Then

\lVert y_{m_i}-y\rVert = \Bigl\lVert\sum_j\bigl(\beta^{(m_i)}_j-\beta_j\bigr)x_j\Bigr\rVert \le \sum_j\bigl\lvert\beta^{(m_i)}_j-\beta_j\bigr\rvert\,\lVert x_j\rVert \longrightarrow 0 ,

so $y_{m_i}\to y$ , and by continuity of the norm $\lVert y\rVert = \lim\lVert y_{m_i}\rVert = 0$ , whence $y=0$ by (N2). But $y$ is a nontrivial linear combination of linearly independent vectors, so $y\ne0$ . Contradiction, so $(\dagger)$ holds for some $c>0$ . Linear independence is used exactly once, at the last step, and nowhere else.

Finite-dimensional subspaces are complete. Let $Y\subseteq X$ have $\dim Y = n$ and basis $e_1,\dots,e_n$ , and let $(y_m)$ be Cauchy in $Y$ , $y_m = \sum_j\alpha^{(m)}_je_j$ . By the lemma, with the constant $c$ belonging to $\{e_1,\dots,e_n\}$ ,

\lVert y_m-y_r\rVert \ \ge\ c\sum_{j=1}^{n}\bigl\lvert\alpha^{(m)}_j-\alpha^{(r)}_j\bigr\rvert \ \ge\ c\,\bigl\lvert\alpha^{(m)}_j-\alpha^{(r)}_j\bigr\rvert

for each $j$ . Dividing by $c>0$ shows each coordinate sequence is Cauchy in $\mathbb{K}$ , hence convergent, say $\alpha^{(m)}_j\to\alpha_j$ . Put $y = \sum_j\alpha_je_j \in Y$ ; then $\lVert y_m-y\rVert \le \sum_j\lvert\alpha^{(m)}_j-\alpha_j\rvert\,\lVert e_j\rVert\to0$ . So $Y$ is complete, and by part 5 of metric spaces it is closed in $X$ .

All norms on a finite-dimensional space are equivalent. Let $\dim X = n$ with basis $e_1,\dots,e_n$ , and let $\lVert\cdot\rVert_a$ , $\lVert\cdot\rVert_b$ be two norms. For $x = \sum_j\alpha_je_j$ , the lemma applied in $(X,\lVert\cdot\rVert_a)$ gives a $c>0$ with $\lVert x\rVert_a \ge c\sum_j\lvert\alpha_j\rvert$ , while (N3) and (N4) applied in $(X,\lVert\cdot\rVert_b)$ give

\lVert x\rVert_b \ \le\ \sum_{j=1}^{n}\lvert\alpha_j\rvert\,\lVert e_j\rVert_b \ \le\ k\sum_{j=1}^{n}\lvert\alpha_j\rvert , \qquad k = \max_j\lVert e_j\rVert_b > 0 .

Combining, $\lVert x\rVert_a \ge (c/k)\lVert x\rVert_b$ . Exchanging the roles of $a$ and $b$ gives the other inequality, and the two together are the definition of equivalence.

Compactness. In any metric space, compactness forces closedness and boundedness. Conversely, in a finite-dimensional normed space let $M$ be closed and bounded and $(x_m)\subseteq M$ . Writing $x_m$ in a basis and using the lemma exactly as above, boundedness of $(\lVert x_m\rVert)$ forces boundedness of the coefficient vectors in $\mathbb{K}^n$ ; Bolzano-Weierstrass gives a convergent subsequence of coefficients, which assembles into a convergent subsequence of $(x_m)$ , whose limit lies in $M$ because $M$ is closed. Finite dimension is used in reducing to coefficient vectors in $\mathbb{K}^n$ , and the reduction has no analogue in infinite dimensions: in $\ell^2$ the vectors $e_n = (0,\dots,0,1,0,\dots)$ all have norm $1$ and satisfy $\lVert e_n-e_m\rVert = \sqrt{2}$ for $n\ne m$ , so they form a closed bounded set with no convergent subsequence. $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Kreyszig (1978), §2.2, Example 2.2-7, p. 62, §2.4, p. 74, and §2.5, p. 77 and Thm 2.5-5, p. 80; Folland (1999), §5.1, pp. 151–152; Stakgold & Holst (2011), §4.3, p. 241

(N2) distinguishes a norm from a seminorm, and it is the axiom that fails for $\int\lvert f\rvert$ on actual functions. Everything else in this unit holds for a seminorm, with "metric" replaced by "pseudometric"; passing to the quotient by $\{x : \lVert x\rVert = 0\}$ restores (N2), and that quotient is how $L^p$ is built.
(N3) is what ties the norm to the linear structure and is used in part 1, in the lemma and throughout the finite-dimensional theory. A metric on a vector space without it — for instance $d(x,y)=\min(1,\lvert x-y\rvert)$ on $\mathbb{R}$ , which is a perfectly good metric inducing the usual topology — comes from no norm, because it is bounded and (N3) forbids a bounded nonzero norm.
Completeness is a property of the norm, not of the vector space. The same set $C[a,b]$ is a Banach space under $\lVert\cdot\rVert_\infty$ and is not under $\lVert\cdot\rVert_1$ or $\lVert\cdot\rVert_2$ ; see the examples in metric spaces and completeness. By part 2 this is possible only because those norms are inequivalent.
Finite dimension in part 6 cannot be weakened to separability or to anything else. The polynomials form an infinite-dimensional subspace of $C[0,1]$ that is dense and therefore not closed, which is Kreyszig's (Kreyszig (1978), §2.4, p. 74) example; the finite-dimensional conclusion fails for it in every respect.
The failure of compactness of the closed unit ball is the defining feature of infinite dimensions. Stakgold and Holst (Stakgold & Holst (2011), §4.3, p. 241) make the point with $1,x,x^2,\dots$ in $C[0,1]$ : all have sup norm $1$ and no subsequence converges uniformly. This is why "bounded" is so much weaker than "compact" in analysis, why minimising sequences need an extra argument to converge, and why weak convergence and compact operators exist as subjects.
Every normed space has a completion, a Banach space containing it as a dense subspace, unique up to a linear isometry; Reed and Simon (Reed & Simon (1980), §I.2, p. 7) note that the vector operations and the norm extend to the metric completion in exactly one way. Kreyszig (Kreyszig (1978), §2.2, Example 2.2-7, p. 62) defines $L^p[a,b]$ this way, as the completion of $C[a,b]$ in $\lVert\cdot\rVert_p$ ; $L^p$ spaces constructs the same space from the Lebesgue integral and proves completeness directly. Neither description is used as a hypothesis above.
This unit says nothing about maps between normed spaces. Boundedness, continuity and the operator norm, which Folland (Folland (1999), §5.1, Prop. 5.2 and (5.3), pp. 153–154) and Reed and Simon (Reed & Simon (1980), §I.2, pp. 6–7) introduce alongside the definitions above, are the subject of bounded linear operators.

Method(Working with norms)Original

To prove a candidate is a norm, check (N3) and (N4) and then (N2); (N1) is usually free, being either built into the formula or forced by (N2)–(N4) as $0 = \lVert x-x\rVert \le 2\lVert x\rVert$ . For $\lVert\cdot\rVert_p$ the only hard axiom is (N4), which is Minkowski's inequality; see $L^p$ spaces.
To prove a space is a Banach space, prefer the series test of part 3 to a direct Cauchy-sequence argument: one controls a single number per term and the triangle inequality does the rest.
To prove a space is not a Banach space, exhibit it as a dense non-closed subspace of a known Banach space, or produce one Cauchy sequence whose natural limit leaves the space.
To prove two norms are equivalent, produce the two constants. If the space is finite-dimensional, do not: quote part 6. If it is infinite-dimensional, expect the answer to be no, and look for a sequence with $\lVert x_n\rVert_a = 1$ and $\lVert x_n\rVert_b\to0$ , which rules equivalence out.
To find the sharp constants, normalise one norm to $1$ and minimise or maximise the other. In finite dimensions the extremum is attained, by part 6, so the sharp constants exist and are achieved by an explicit vector.
Choose the norm to match the question. A sup norm controls the worst point, an $L^1$ norm controls the total, an $L^2$ norm controls the energy. They are inequivalent on any infinite-dimensional function space, so the choice is a modelling decision, not a technicality.

Example(Sharp equivalence constants, and their failure one dimension at a time)Original

On $C[0,1]$ compare

\lVert u\rVert_\infty = \max_{0\le t\le1}\lvert u(t)\rvert , \qquad \lVert u\rVert_1 = \int_0^1\lvert u(t)\rvert\,\mathrm{d}t .

One inequality is immediate: $\lVert u\rVert_1 \le \lVert u\rVert_\infty$ , since the interval has length $1$ , with equality for constants. The question is whether a reverse constant exists.

On the two-dimensional subspace $P = \{a+bt\}$ . Here part 6 guarantees one; we find the best. Let $p(t)=a+bt$ with $p \ne 0$ , and write $u = \lvert p(0)\rvert = \lvert a\rvert$ , $v = \lvert p(1)\rvert = \lvert a+b\rvert$ . Since $p$ is affine, $\lVert p\rVert_\infty = \max(u,v)$ .

The $L^1$ norm. If $p$ does not vanish in $(0,1)$ then $\lvert p\rvert$ is affine there and $\lVert p\rVert_1 = (u+v)/2$ . If it vanishes at $t_0 = -a/b \in (0,1)$ , the graph of $\lvert p\rvert$ is two triangles of heights $u$ and $v$ and bases $t_0$ and $1-t_0$ , and since $\lvert b\rvert = u+v$ in that case,

\lVert p\rVert_1 = \frac{u\,t_0}{2}+\frac{v(1-t_0)}{2} = \frac{u^2+v^2}{2(u+v)} .

Minimise the ratio. The sign-changing case is the smaller of the two, since $(u^2+v^2)/(u+v) \le u+v$ . Scale so that $\max(u,v)=1$ , say $v=1$ and $u\in[0,1]$ , and minimise

f(u) = \frac{u^2+1}{2(u+1)} ,\qquad f'(u) = \frac{u^2+2u-1}{2(u+1)^2} ,

which vanishes at $u = \sqrt2-1$ . There

f(\sqrt2-1) = \frac{(\sqrt2-1)^2+1}{2\sqrt2} = \frac{4-2\sqrt2}{2\sqrt2} = \sqrt2-1 .

So on $P$ ,

\boxed{\ (\sqrt2-1)\,\lVert p\rVert_\infty \ \le\ \lVert p\rVert_1 \ \le\ \lVert p\rVert_\infty\ } \qquad \sqrt2-1 = 0.4142135624 ,

and both constants are sharp: the upper one for $p\equiv1$ , the lower one for

p_\ast(t) = (\sqrt2-1) - \sqrt2\,t ,

which has $p_\ast(0)=\sqrt2-1$ , $p_\ast(1)=-1$ , so $\lVert p_\ast\rVert_\infty = 1$ , and vanishes at $t_0 = 1-1/\sqrt2 = 0.2928932$ , giving

\lVert p_\ast\rVert_1 = \tfrac12(\sqrt2-1)(0.2928932)+\tfrac12(1)(0.7071068) = 0.0606602+0.3535534 = 0.4142136 .

A numerical sweep over all $p$ with $\lVert p\rVert_\infty=1$ returns a minimum $L^1$ norm of $0.41421356$ , confirming the algebra.

On all of $C[0,1]$ there is no such constant. Take $q_n(t) = t^n$ . Then $\lVert q_n\rVert_\infty = 1$ for every $n$ , while

\lVert q_n\rVert_1 = \int_0^1 t^n\,\mathrm{d}t = \frac{1}{n+1} \longrightarrow 0 .

A constant $c>0$ with $c\lVert u\rVert_\infty \le \lVert u\rVert_1$ would force $c \le 1/(n+1)$ for every $n$ , so $c=0$ . The two norms are inequivalent on $C[0,1]$ , which is why $C[0,1]$ is complete in one and not in the other.

What happened between the two cases. Restricting to $P$ cut the dimension to $2$ , and the extremal functions $q_n$ — which concentrate their mass near $t=1$ — cannot be approximated by affine functions. The degradation is gradual: on the polynomials of degree $\le N$ the best constant is positive for each $N$ , by part 6, and tends to $0$ as $N\to\infty$ , since $q_N$ lies in that space. Part 6 is a statement for each fixed finite dimension, with a constant that is allowed to depend on it, and nothing more.

A finite-dimensional sanity check. In $\mathbb{K}^3$ take $x=(3,-4,12)$ : $\lVert x\rVert_1 = 19$ , $\lVert x\rVert_2 = \sqrt{9+16+144} = 13$ , $\lVert x\rVert_\infty = 12$ . The sharp general bounds $\lVert x\rVert_\infty \le \lVert x\rVert_2 \le \lVert x\rVert_1 \le \sqrt3\,\lVert x\rVert_2$ read $12\le13\le19\le22.5167$ , all satisfied, with the last slack because $x$ is far from the vector $(1,1,1)$ that makes it an equality.

ApplicationOriginal

Which errors a numerical method controls. A finite-element or finite-difference error bound is always stated in a norm, and in infinite dimensions the norms are inequivalent, so the statement does not transfer. An $L^2$ error bound permits a large pointwise error on a small set; an $L^\infty$ bound forbids it and is correspondingly harder to prove. Grid refinement studies that report one norm and are used to justify a claim in another are making an unproved inference. See finite elements.

Why finite-dimensional linear algebra needs no analysis. Every statement about convergence of matrices or vectors in matrix norms and conditioning is norm-independent, by part 6, which is why one may compute a condition number in whichever norm is cheapest and still say "ill-conditioned" meaningfully. The constants relating the norms depend on the dimension, so the statement degrades as a discretisation is refined, and that degradation is exactly the infinite-dimensional obstruction appearing in finite form.

Function spaces in partial differential equations. Evans (Evans (2010), App. D.1, p. 675) lists $L^p$ , the Hölder spaces and the Sobolev spaces as the Banach spaces of the subject. Existence proofs produce a solution as the limit of approximations and so need completeness in the norm used for the estimates; the choice of that norm is the choice of what a "solution" is. See weak solutions and Sobolev spaces.

Signal and control engineering. Peak amplitude is an $L^\infty$ constraint, total energy an $L^2$ constraint, total variation an $L^1$ -type constraint on the derivative. Because these are inequivalent, a specification must name all the ones that matter; a controller optimal in one can be arbitrarily bad in another, which is the whole content of the distinction between $H^\infty$ and $H^2$ design.

Series of operators and of functions. The series test of part 3 is the reason absolutely convergent expansions may be manipulated at all: the Neumann series for $(I-K)^{-1}$ , the exponential series for the matrix exponential, and the Weierstrass $M$ -test for uniform convergence are the same statement in three Banach spaces.

Definition

Hilbert spaces

Adapted

Add to PDF

A Hilbert space is an inner product space that is complete; this unit says which of the standard inner product spaces are complete and which are not, proves the completeness of $\ell^2$ , and uses the parallelogram law as the test that decides whether a given Banach space is a Hilbert space at all.

Prerequisites: Normed and Banach spaces, Inner products and norms

Definition(Hilbert space)AdaptedAdapted from Kreyszig (1978), §3.1, Def. 3.1-1, pp. 128–129; Reed & Simon (1980), §II.1, pp. 39–40; Hassani (2013), §7.1, Def. 7.1.6, p. 218; Folland (1999), §5.5, p. 172; Stakgold & Holst (2011), §4.5, p. 264

Let $X$ be an inner product space over $\mathbb{K} = \mathbb{R}$ or $\mathbb{C}$ , in the sense of inner products and norms: a vector space carrying a map $(\cdot,\cdot) : X\times X\to\mathbb{K}$ that is linear in its first slot, conjugate-symmetric, and positive definite. It carries the induced norm and induced metric

\lVert x\rVert = (x,x)^{1/2} , \qquad d(x,y) = \lVert x-y\rVert = (x-y,\,x-y)^{1/2} ,

so every inner product space is a normed space and hence a metric space.

A Hilbert space is an inner product space that is complete in that metric. An inner product space that is not assumed complete is a pre-Hilbert space.

Thus the three structures nest:

\text{inner product space} \;\subset\; \text{normed space}, \qquad \text{Hilbert space} \;\subset\; \text{Banach space} ,

and a Hilbert space is exactly a Banach space whose norm happens to come from an inner product. Which Banach spaces those are is decided below.

Completeness is the whole content of the word "Hilbert". Everything geometric — Cauchy-Schwarz, the triangle inequality, Pythagoras, orthogonality, the parallelogram law, polarization — holds in any inner product space and is proved in inner products and norms without completeness. What completeness buys is that limits of Cauchy sequences exist, and from that follow the three theorems that make Hilbert space worth a name: the projection theorem, the convergence of orthonormal expansions, and the Riesz representation theorem. None of them survives in a merely pre-Hilbert space.

Definition(Which inner product spaces are Hilbert spaces)AdaptedAdapted from Kreyszig (1978), §3.1, Examples 3.1-3 to 3.1-8, pp. 131–134, §3.2, Thms 3.2-3 and 3.2-4, pp. 139–140, and §1.5, Example 1.5-4, pp. 35–36; Reed & Simon (1980), §II.1, pp. 39–40; Hassani (2013), §7.1, Prop. 7.1.4 and Example 7.1.5, p. 217; Folland (1999), §5.5, pp. 172–173; Stakgold & Holst (2011), §4.5, p. 264 and Exercises 4.5.1 and 4.5.4, pp. 272–273

Hilbert spaces.
- $\mathbb{R}^n$ and $\mathbb{C}^n$ with $(x,y) = \sum_j x_j\overline{y_j}$ . More generally every finite-dimensional inner product space over $\mathbb{R}$ or $\mathbb{C}$ is a Hilbert space.
- $\ell^2 = \{\,x = (x_1,x_2,\dots) : \sum_j\lvert x_j\rvert^2<\infty\,\}$ with $(x,y) = \sum_j x_j\overline{y_j}$ , the series converging absolutely by Cauchy-Schwarz.
- $L^2(X,\mathcal{M},\mu)$ for any measure space, with $(f,g) = \int_X f\,\overline{g}\,\mathrm{d}\mu$ , the integral being finite because $\lvert fg\rvert \le \tfrac12(\lvert f\rvert^2+\lvert g\rvert^2)$ . Completeness is the $p=2$ case of the Riesz-Fischer theorem of $L^p$ spaces. Taking $\mu$ to be counting measure on a set $A$ recovers $\ell^2(A)$ , so the two previous items are one construction.
- Direct sums: if $H_1,H_2,\dots$ are Hilbert spaces, the sequences $(x_n)$ with $x_n\in H_n$ and $\sum_n\lVert x_n\rVert^2_{H_n}<\infty$ form a Hilbert space $\bigoplus_n H_n$ under the sum of the inner products.
- Any closed subspace of a Hilbert space; and the Sobolev space $H^1(\Omega)$ , the completion of the continuously differentiable functions in the norm from $(u,v) = \int_\Omega(\nabla u\cdot\nabla v + uv)$ .
Pre-Hilbert spaces that are not Hilbert spaces.
- $C[a,b]$ with $(x,y) = \int_a^b x\overline{y}$ . Its completion is $L^2[a,b]$ .
- The sequences with only finitely many nonzero entries, with the $\ell^2$ inner product.
- The polynomials on $[a,b]$ , with any of the inner products above.
Each is a dense, non-closed subspace of a Hilbert space.
Subspaces. If $Y$ is a subspace of a Hilbert space $H$ , then $Y$ is itself a Hilbert space exactly when $Y$ is closed in $H$ ; every finite-dimensional subspace is closed, hence a Hilbert space; and if $H$ is separable so is every subset of it.
Completion. Every inner product space $X$ is isomorphic to a dense subspace of a Hilbert space $H$ , and $H$ is unique up to isomorphism. Here an isomorphism of inner product spaces is a linear bijection $T$ with $(Tx,Ty) = (x,y)$ for all $x,y$ ; such a $T$ is automatically isometric, and between Hilbert spaces it is called unitary.
The test: which norms come from an inner product. By the parallelogram law (inner products and norms, parts 5 and 7), a norm on a vector space $W$ is induced by an inner product if and only if
$\lVert x+y\rVert^2+\lVert x-y\rVert^2 = 2\lVert x\rVert^2+2\lVert y\rVert^2 \qquad\text{for all } x,y \in W,$
and the inner product is then uniquely determined by polarization. Consequently:
- a Banach space is a Hilbert space precisely when its norm satisfies this identity;
- $\ell^p$ and $L^p$ for $p \ne 2$ are Banach spaces and not Hilbert spaces;
- $C[a,b]$ with the sup norm is a Banach space and not a Hilbert space.
$\ell^2$ is the model. Every separable infinite-dimensional Hilbert space is isomorphic to $\ell^2$ ; every $n$ -dimensional one is isomorphic to $\mathbb{K}^n$ . Hilbert spaces are therefore classified by one cardinal number, the size of an orthonormal basis; the proof belongs to orthonormal bases in Hilbert space.

Idea of the proofOriginal

Part 1 is three completeness proofs with one structure, the scheme of metric spaces and completeness: a Cauchy sequence in the space forces a Cauchy sequence of scalars in each coordinate, those converge because $\mathbb{K}$ is complete, and the limits are assembled into a candidate which must then be shown to lie in the space and to be approached in the norm and not merely coordinatewise. The second of those two checks is where the summability condition $\sum\lvert x_j\rvert^2<\infty$ is verified, and the trick is to prove it for every finite partial sum with a bound that does not depend on how many terms are taken, and only then let the number of terms go to infinity.

Part 2 is the same calculation read backwards: an approximating sequence stays inside the subspace while its limit does not.

Part 5 is the only item that is not about completeness. An inner product has more structure than a norm — it knows angles, not just lengths — and the parallelogram law is the exact algebraic shadow of that extra structure on the norm. To refute it for a given norm one needs only two well-chosen vectors, which is why the counterexamples below are two lines each.

Proof(Completeness of

\ell^2

, and the non-examples of part 2)AdaptedAdapted from Kreyszig (1978), §1.5, Example 1.5-4, pp. 35–36, and §3.1, Examples 3.1-6 to 3.1-8, pp. 133–134; Hassani (2013), §7.1, Prop. 7.1.4, p. 217

$\ell^2$ is complete. Let $\bigl(x^{(m)}\bigr)_{m\ge1}$ be Cauchy in $\ell^2$ , with $x^{(m)} = \bigl(x^{(m)}_1,x^{(m)}_2,\dots\bigr)$ .

Coordinates. For every $j$ , $\bigl\lvert x^{(m)}_j - x^{(n)}_j\bigr\rvert \le \bigl\lVert x^{(m)}-x^{(n)}\bigr\rVert$ , so the $j$ -th coordinates form a Cauchy sequence of scalars. By completeness of $\mathbb{K}$ it converges; call the limit $x_j$ , and put $x = (x_1,x_2,\dots)$ .
A bound uniform in the number of terms. Given $\varepsilon>0$ choose $N$ with $\lVert x^{(m)}-x^{(n)}\rVert\le\varepsilon$ for $m,n>N$ . For every $k$ and every $m,n>N$ , the finite sum satisfies $\sum_{j=1}^{k}\bigl\lvert x^{(m)}_j - x^{(n)}_j\bigr\rvert^2 \ \le\ \bigl\lVert x^{(m)}-x^{(n)}\bigr\rVert^2 \ \le\ \varepsilon^2 .$ Let $n\to\infty$ : the sum has finitely many terms, each converging by step 1, so the non-strict inequality passes to the limit and $\sum_{j=1}^{k}\lvert x^{(m)}_j - x_j\rvert^2 \le \varepsilon^2$ for every $k$ and every $m>N$ . It is because the bound $\varepsilon^2$ does not depend on $k$ that we may now let $k\to\infty$ : $\bigl\lVert x^{(m)}-x\bigr\rVert^2 = \sum_{j=1}^{\infty}\bigl\lvert x^{(m)}_j - x_j\bigr\rvert^2 \ \le\ \varepsilon^2 \qquad (m>N) . \tag{$\ast$}$
The limit is in the space. $(\ast)$ says $x^{(m)}-x \in \ell^2$ . Since $x^{(m)}\in\ell^2$ and $\ell^2$ is a vector space — by Minkowski's inequality, the $p=2$ case in $L^p$ spaces — we get $x = x^{(m)}-(x^{(m)}-x) \in \ell^2$ .
Convergence in norm. $(\ast)$ says exactly $\lVert x^{(m)}-x\rVert\le\varepsilon$ for $m>N$ , that is $x^{(m)}\to x$ in $\ell^2$ .

Finite-dimensional inner product spaces are complete. Let $\dim X = n$ and let $e_1,\dots,e_n$ be an orthonormal basis, available by Gram-Schmidt (Gram-Schmidt orthogonalisation). Writing $x^{(m)} = \sum_k\alpha^{(m)}_ke_k$ , orthonormality gives

\bigl\lVert x^{(m)}-x^{(r)}\bigr\rVert^2 = \sum_{k=1}^{n}\bigl\lvert\alpha^{(m)}_k-\alpha^{(r)}_k\bigr\rvert^2 ,

so a Cauchy sequence of vectors is exactly $n$ Cauchy sequences of scalars; their limits $\alpha_k$ assemble into $x = \sum_k\alpha_ke_k \in X$ , and the same identity gives $\lVert x^{(m)}-x\rVert\to0$ . There is no step 3 here, because a finite linear combination of basis vectors is automatically in the space. (This also follows from part 6 of normed and Banach spaces, which needs no inner product.)

The three non-examples of part 2. In each case the space is a subspace of a Hilbert space and the proof is the same: exhibit a sequence in the subspace converging, in the ambient norm, to a point outside it, so the subspace is not closed and part 3 applies.

Finitely supported sequences in $\ell^2$ . Take $x^{(m)} = (1,\tfrac12,\dots,\tfrac1m,0,0,\dots)$ . For $n>m$ , $\lVert x^{(n)}-x^{(m)}\rVert^2 = \sum_{j=m+1}^{n}j^{-2} \le \sum_{j>m}j^{-2}\to0$ , so the sequence is Cauchy; its $\ell^2$ limit is $(1/j)_{j\ge1}$ , which has infinitely many nonzero entries.
$C[a,b]$ in the mean-square norm. Hassani's (Hassani (2013), §7.1, Example 7.1.5, p. 217) ramps $f_k$ on $[-1,1]$ , equal to $0$ for $x\le-1/k$ , to $(kx+1)/2$ in between and to $1$ for $x\ge1/k$ , are continuous and Cauchy in the mean-square norm, and converge to a function with a jump at $0$ . Metric spaces and completeness carries the same conclusion for a different sequence.
Polynomials. They are dense in $L^2(a,b)$ and are not all of it, so they are not closed. $\square$

Proof(Part 5: the test, and its two applications)AdaptedAdapted from Kreyszig (1978), §3.1, Examples 3.1-7 and 3.1-8, pp. 133–134; Stakgold & Holst (2011), §4.5, Exercises 4.5.1, 4.5.3 and 4.5.4, pp. 272–273

The identity itself, and the fact that it characterises inner-product norms, are parts 5 and 7 of inner products and norms; the characterisation is the Jordan-von Neumann theorem, which Stakgold and Holst (Stakgold & Holst (2011), §4.5, Exercise 4.5.4, p. 273) set out in the library as a guided exercise: assuming the law, the polarization formula defines a function $f(u,v)$ , and one checks in turn that it is continuous in $v$ , additive in $u$ , and homogeneous, hence an inner product; by their Exercise 4.5.3 no other inner product can give the same norm. Only the forward direction is needed to refute, and it is what the two computations use.

$\ell^p$ , $p\ne2$ , is not a Hilbert space. Take $x = (1,1,0,0,\dots)$ and $y = (1,-1,0,0,\dots)$ , both in $\ell^p$ for every $p$ . Then $x+y = (2,0,0,\dots)$ and $x-y = (0,2,0,\dots)$ , so

\lVert x+y\rVert_p = \lVert x-y\rVert_p = 2 , \qquad \lVert x\rVert_p = \lVert y\rVert_p = 2^{1/p} .

The parallelogram law would require $4+4 = 2\cdot2^{2/p}+2\cdot2^{2/p}$ , that is $2^{2/p} = 2$ , that is $p = 2$ . So for $p\ne2$ the $\ell^p$ norm is induced by no inner product; since $\ell^p$ is complete, it is a Banach space that is not a Hilbert space.

$C[a,b]$ with the sup norm is not a Hilbert space. It is enough to exhibit two continuous functions violating the law; the worked example below does so on $[0,1]$ with explicit numbers, and the same two functions, rescaled by $t\mapsto(t-a)/(b-a)$ , work on any $[a,b]$ because the sup norm is unchanged by that substitution. $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Kreyszig (1978), §3.1, pp. 130, 133–134, and §3.2, Thms 3.2-3 and 3.2-4, pp. 139–140; Reed & Simon (1980), §II.1, pp. 36, 39–40; Folland (1999), §5.5, pp. 172, 180; Stakgold & Holst (2011), §4.5, Exercise 4.5.3, pp. 272–273

Completeness is used nowhere in the geometry and everywhere in the analysis. Cauchy-Schwarz, Pythagoras and the parallelogram law are identities, true in any pre-Hilbert space. The existence of a nearest point in a closed convex set, of an orthogonal projection, of the sum of an orthonormal expansion, and of the vector representing a bounded functional are all existence statements, and each fails in a pre-Hilbert space. The non-closed subspaces of part 2 are where to look for counterexamples: in the finitely supported sequences, the vector $(1/j)$ has no nearest point in any reasonable sense, because the infimum of the distance is $0$ and is not attained.
Closedness, not completeness, is the hypothesis one checks on a subspace. Part 3 reduces one to the other, and it is the finite-dimensional case that makes approximation arguments work: a best approximation by polynomials of degree $\le N$ exists because that subspace is closed, while a best approximation by polynomials of unrestricted degree need not.
Positive definiteness, $(x,x)=0 \Rightarrow x=0$ , is what makes $d$ a metric rather than a pseudometric. On the actual square-integrable functions $(f,f)=0$ only forces $f=0$ almost everywhere; $L^2$ is a space of equivalence classes for exactly this reason, which is the same repair as for $\lVert\cdot\rVert_p$ in $L^p$ spaces.
The parallelogram test needs only two vectors, and it is a necessary condition used contrapositively. To show a norm is an inner-product norm one must verify the identity for all pairs, which is usually done by exhibiting the inner product instead.
Uniqueness in part 4 is up to isomorphism, not equality. Two completions of $C[a,b]$ in the mean-square norm are different sets of objects related by a unitary map; in particular $L^2[a,b]$ built as a completion (Kreyszig (Kreyszig (1978), §3.1, Example 3.1-5, p. 132)) and $L^2[a,b]$ built from the Lebesgue integral ( $L^p$ spaces) are isomorphic, and this unit uses neither construction as a hypothesis.
Separability in part 6 is a real restriction, even though every Hilbert space arising in physics is separable. Drop it and the classification is by a larger cardinal; the statement "every Hilbert space is $\ell^2$ " is false without it.
The slot that is linear. This handbook, with Kreyszig (Kreyszig (1978), §3.1, p. 130) and Folland (Folland (1999), §5.5, p. 171), takes $(\cdot,\cdot)$ linear in the first argument. Reed and Simon (Reed & Simon (1980), §II.1, p. 36) take it linear in the second and say so explicitly, and Folland (Folland (1999), §5.5, note to §5.5, p. 180) records that the physics literature, writing $\langle x\vert y\rangle$ , does the same. The two conventions are related by $(x,y)_{\text{here}} = \overline{(x,y)_{\text{theirs}}}$ and no statement in this unit changes; what does change is which argument a scalar may be pulled out of unconjugated, so a formula transcribed from one convention into the other must have its conjugations checked.

Method(Deciding what a given space is)Original

Given a vector space $W$ with a candidate structure, settle three questions in order.

Is there an inner product, or only a norm? If a formula $(x,y)$ is given, check conjugate symmetry, linearity in the first slot, and $(x,x)>0$ for $x\ne0$ — positivity is usually the only one with content. If only a norm is given, test the parallelogram law on two vectors chosen to make $x+y$ and $x-y$ simple: for a sequence or function space, take two "bumps" with disjoint supports, which makes $\lVert x+y\rVert = \lVert x-y\rVert$ and reduces the law to an arithmetic identity. If it fails, the space carries no inner product and the matter is closed.
Is it complete? Prefer not to prove it from scratch. Identify $W$ as a subspace of a known Hilbert space — $\ell^2$ , $L^2(\mu)$ , or a direct sum of these — and then by part 3 the question is whether $W$ is closed. If $W$ is defined by finitely many continuous linear conditions, or is finite-dimensional, it is closed. If $W$ is defined by a condition that can be lost in the limit — "finitely many nonzero terms", "polynomial", "continuous", "compactly supported" — expect it to be dense and not closed.
If it is not complete, name its completion. The completion exists and is unique by part 4, and in practice it is already a familiar space: $L^2$ for the continuous functions in the mean-square norm, $\ell^2$ for the finitely supported sequences, $H^1$ for the continuously differentiable functions in the Dirichlet norm. Work in the completion, and recover statements about the original space by density.

To recover the inner product from a norm already known to satisfy the parallelogram law, use polarization: over $\mathbb{R}$ , $4(x,y) = \lVert x+y\rVert^2-\lVert x-y\rVert^2$ ; over $\mathbb{C}$ , add $i\bigl(\lVert x+iy\rVert^2-\lVert x-iy\rVert^2\bigr)$ .

Example(The parallelogram test, run twice)Original

Test 1: the sup norm on $C[0,1]$ . Take

f(t) = t , \qquad g(t) = t^2 .

Then $\lVert f\rVert_\infty = \lVert g\rVert_\infty = 1$ , both maxima at $t=1$ . Next,

(f+g)(t) = t+t^2 , \qquad \lVert f+g\rVert_\infty = 2 \ \ (\text{at } t=1),

since $t+t^2$ increases on $[0,1]$ ; and

(f-g)(t) = t-t^2 , \qquad \lVert f-g\rVert_\infty = \tfrac14 \ \ (\text{at } t=\tfrac12),

since $t-t^2$ has derivative $1-2t$ . Therefore

\lVert f+g\rVert_\infty^2+\lVert f-g\rVert_\infty^2 = 4+\tfrac{1}{16} = \tfrac{65}{16} = 4.0625 , \qquad 2\lVert f\rVert_\infty^2+2\lVert g\rVert_\infty^2 = 4 .

The two sides differ by $0.0625$ , so the parallelogram law fails and the sup norm on $C[0,1]$ comes from no inner product. $C[0,1]$ is a Banach space, and it is not a Hilbert space.

Test 2: $\ell^3$ . Take the first two standard unit vectors $e_1 = (1,0,0,\dots)$ and $e_2 = (0,1,0,\dots)$ . Then $\lVert e_1\rVert_3 = \lVert e_2\rVert_3 = 1$ , while $e_1\pm e_2$ has two entries of modulus $1$ , so

\lVert e_1+e_2\rVert_3 = \lVert e_1-e_2\rVert_3 = 2^{1/3} = 1.2599210 .

Hence

\lVert e_1+e_2\rVert_3^2+\lVert e_1-e_2\rVert_3^2 = 2\cdot2^{2/3} = 2^{5/3} = 3.1748021 , \qquad 2+2 = 4 .

Again the law fails, and $\ell^3$ is a Banach space that is not a Hilbert space. The same computation with a general $p$ gives $2^{(p+2)/p}$ against $4$ , equal only at $p=2$ — the scalar statement behind part 5.

The contrast: $L^2(0,1)$ . Take $f\equiv1$ and $g(t)=t$ with $(f,g) = \int_0^1 t\,\mathrm{d}t = \tfrac12$ . Then

\lVert f\rVert^2 = 1 , \qquad \lVert g\rVert^2 = \int_0^1t^2\,\mathrm{d}t = \tfrac13 ,

\lVert f+g\rVert^2 = \int_0^1(1+t)^2\,\mathrm{d}t = \tfrac73 , \qquad \lVert f-g\rVert^2 = \int_0^1(1-t)^2\,\mathrm{d}t = \tfrac13 ,

and the law holds exactly: $\tfrac73+\tfrac13 = \tfrac83 = 2(1)+2(\tfrac13)$ . Cauchy-Schwarz is also strict here, $\lvert(f,g)\rvert = 0.5$ against $\lVert f\rVert\lVert g\rVert = 3^{-1/2} = 0.5773503$ , as it must be since $f$ and $g$ are linearly independent, and the angle between them is

\theta = \arccos\frac{1/2}{1\cdot3^{-1/2}} = \arccos\frac{\sqrt3}{2} = 30^\circ

exactly. The constant function and the ramp make an angle of thirty degrees on the unit interval: that is the kind of statement an inner product licenses and a norm alone does not.

Example(Completeness made quantitative, and where it stops)Original

Work in $\ell^2$ and let $S \subset \ell^2$ be the subspace of sequences with only finitely many nonzero terms. Put

x = \Bigl(1,\tfrac12,\tfrac13,\dots\Bigr) , \qquad x^{(m)} = \Bigl(1,\tfrac12,\dots,\tfrac1m,0,0,\dots\Bigr) \in S .

$x$ is in $\ell^2$ and $x^{(m)}$ converges to it. By the Basel sum,

\lVert x\rVert^2 = \sum_{j=1}^{\infty}\frac{1}{j^2} = \frac{\pi^2}{6} = 1.6449341 , \qquad \lVert x\rVert = 1.2825499 ,

so $x \in \ell^2$ , and the tail is

\lVert x-x^{(m)}\rVert^2 = \sum_{j>m}\frac{1}{j^2} .

At $m=10$ the partial sum is $\sum_{j\le10}j^{-2} = 1.5497677$ , so

\lVert x-x^{(10)}\rVert^2 = 1.6449341-1.5497677 = 0.0951664 , \qquad \lVert x-x^{(10)}\rVert = 0.3084904 ,

about $24\%$ of $\lVert x\rVert$ . At $m=100$ the distance is $0.0997505$ , about $7.8\%$ . The integral comparison $\sum_{j>m}j^{-2} \in (1/(m+1),\,1/m)$ shows the distance decays like $m^{-1/2}$ , which is slow: to get within $10^{-3}$ one needs $m \approx 10^{6}$ .

Where the two spaces part company. The sequence $\bigl(x^{(m)}\bigr)$ is Cauchy, since $\lVert x^{(n)}-x^{(m)}\rVert^2 = \sum_{m<j\le n}j^{-2} \le 1/m$ for $n>m$ . In $\ell^2$ it converges, to $x$ . In $S$ it cannot converge: a limit in $S$ would also be a limit in $\ell^2$ , hence equal to $x$ by uniqueness of limits, and $x \notin S$ . So $S$ is an inner product space, dense in $\ell^2$ and not complete; it is a pre-Hilbert space and not a Hilbert space, and $\ell^2$ is its completion.

The numerical moral. Every element of $S$ is a vector one can actually store. The theorem that $\ell^2$ is complete is the statement that a sequence of such finite vectors, improving in the sense of a computable error estimate, has something to converge to — and the example shows the limit may be an object with infinitely many nonzero entries, outside every space one can represent exactly. Truncating at $m=10$ costs $0.308$ in norm; the completeness theorem is what entitles one to speak of the error at all.

ApplicationOriginal

Quantum mechanics. The state space of a quantum system is a separable complex Hilbert space, by hypothesis of the theory; for a particle in three dimensions it is $L^2(\mathbb{R}^3)$ . Completeness is what makes an eigenfunction expansion converge to a state rather than to nothing, what makes a projection onto an eigenspace exist, and what makes the time evolution of Stone's theorem well defined. The inner product, not merely the norm, is what supplies the transition amplitude $(\psi,\varphi)$ and hence the Born probability $\lvert(\psi,\varphi)\rvert^2$ — and Cauchy-Schwarz is what keeps that probability at most $1$ . See the mathematical framework of quantum mechanics.

Least squares and estimation. Fitting data by minimising a sum or integral of squares is minimising a distance in an $\ell^2$ or $L^2$ norm, and the normal equations are the orthogonality conditions that the minimiser must satisfy; see projections and least squares. The existence of the minimiser in infinite dimensions is the projection theorem, which needs completeness; this is why "least squares" is a theorem in $L^2$ and only a hope in $L^1$ or $L^\infty$ , where the parallelogram law fails and the nearest point need not exist, nor be unique when it does.

Fourier analysis. Parseval's identity says the Fourier coefficient map is a unitary isomorphism $L^2(-\pi,\pi) \to \ell^2(\mathbb{Z})$ , which is part 6 made explicit, and Plancherel's theorem is its continuous analogue. Every statement that a signal's energy is conserved by a transform is the statement that the transform is unitary between two Hilbert spaces.

Probability. The finite-variance random variables on a probability space form $L^2(\Omega,\mathcal{F},\mathbb{P})$ , with $(X,Y)=\mathbb{E}[X\overline{Y}]$ ; uncorrelated means orthogonal, variance is squared norm, and conditional expectation is an orthogonal projection onto a closed subspace. Pythagoras in this space is the variance decomposition, and the parallelogram law is the identity $\operatorname{Var}(X+Y)+\operatorname{Var}(X-Y) = 2\operatorname{Var}X+2\operatorname{Var}Y$ .

Partial differential equations. Variational methods minimise an energy that is a squared Hilbert norm, usually a Sobolev one; the Lax-Milgram theorem and the Rayleigh-Ritz method of Rayleigh-Ritz both run in a Hilbert space and both use completeness to produce the minimiser. The same completeness underlies the convergence of finite elements, where the approximating subspaces are finite-dimensional and hence closed, and the limit is taken in the ambient Hilbert space.

Why not a Banach space. The worked examples show that $C[0,1]$ in the sup norm and $\ell^p$ for $p\ne2$ are perfectly good complete spaces with no inner product, and therefore no orthogonality, no projection, no Parseval. Whenever a method relies on decomposing a vector into orthogonal pieces — normal modes, principal components, multipoles, spherical harmonics — it is relying on an inner product, and the first question to ask of any new function space is whether its norm passes the parallelogram test.

Theorem

The projection theorem

Adapted

Add to PDF

Shows that a non-empty closed convex subset of a Hilbert space contains exactly one point nearest any given vector, that for a closed subspace the error of that nearest point is orthogonal to the subspace, and hence that $H = M \oplus M^\perp$ , which is what makes "drop a perpendicular" a legitimate move in infinitely many dimensions.

Prerequisites: Hilbert spaces, Metric spaces and completeness, Inner products and norms, Limits of sequences and completeness

Definition(Distance to a set, convexity, orthogonal complement)AdaptedAdapted from Kreyszig (1978), §3.3, Def. 3.3-3, p. 146, and the annihilator, p. 148; Stakgold & Holst (2011), §4.5, p. 266; Hassani (2013), Def. 6.1.1, p. 169

Throughout, $H$ is a Hilbert space over $\mathbb{R}$ or $\mathbb{C}$ , with inner product $(u,v)$ linear in the first slot and norm $\|u\| = (u,u)^{1/2}$ (see Hilbert spaces).

For a non-empty set $S \subseteq H$ and $u \in H$ , the distance from $u$ to $S$ is

d(u,S) = \inf_{z \in S} \|u - z\| .

An element $v \in S$ with $\|u - v\| = d(u,S)$ is a nearest point of $S$ to $u$ ; the infimum of a set of non-negative numbers always exists, but need not be attained.

A set $C \subseteq H$ is convex if $tz_1 + (1-t)z_2 \in C$ whenever $z_1, z_2 \in C$ and $0 \le t \le 1$ . Every linear subspace is convex, and so is every translate of one, every ball, and every set defined by linear inequalities. Only one consequence of convexity is used below: the midpoint $\tfrac12(z_1+z_2)$ of two members of $C$ is again in $C$ .

Two vectors are orthogonal, $u \perp v$ , if $(u,v) = 0$ . For a non-empty $S \subseteq H$ the orthogonal complement is

S^{\perp} = \{\, u \in H : (u,z) = 0 \text{ for every } z \in S \,\} .

$S^\perp$ is always a closed linear subspace, whatever $S$ is: it is a subspace because $(\alpha u_1 + \beta u_2, z) = \alpha(u_1,z) + \beta(u_2,z)$ , and it is closed because if $u_k \to u$ with $(u_k,z) = 0$ then $(u,z) = \lim_k (u_k,z) = 0$ by continuity of the inner product. We write $S^{\perp\perp}$ for $(S^\perp)^\perp$ .

$H$ is the direct sum of two subspaces $M$ and $N$ , written $H = M \oplus N$ , if every $u \in H$ has exactly one representation $u = v + w$ with $v \in M$ and $w \in N$ .

Theorem(The projection theorem)AdaptedAdapted from Kreyszig (1978), Thm 3.3-1, p. 144, Lemma 3.3-2, p. 145, and Thm 3.3-4, p. 146; Stakgold & Holst (2011), §4.5, the projection theorem, p. 266; Folland (1999), Thm 5.24, p. 173

Let $H$ be a Hilbert space.

Nearest point in a closed convex set. If $C \subseteq H$ is non-empty, convex and closed, then for every $u \in H$ there is exactly one $v \in C$ with

\|u - v\| = d(u,C) .

Orthogonality. If moreover $C = M$ is a linear subspace (closed, non-empty), then the error $w = u - v$ satisfies $w \in M^{\perp}$ ; and conversely, if $v \in M$ and $u - v \in M^\perp$ , then $v$ is the nearest point.
Direct sum. Hence for every closed subspace $M \subseteq H$ ,

H = M \oplus M^{\perp} ,

that is, each $u \in H$ is in exactly one way $u = v + w$ with $v \in M$ , $w \in M^\perp$ , and then $\|u\|^2 = \|v\|^2 + \|w\|^2$ . The map $P : H \to H$ , $Pu = v$ , is the orthogonal projection onto $M$ ; it is linear, satisfies $P^2 = P$ , has range $M$ and null space $M^{\perp}$ , and $\|Pu\| \le \|u\|$ . 4. Double complement and density. For a closed subspace $M$ , $M^{\perp\perp} = M$ ; and if $M \ne H$ then $M^\perp$ contains a non-zero vector. For an arbitrary non-empty $S \subseteq H$ , the span of $S$ is dense in $H$ if and only if $S^{\perp} = \{0\}$ .

Kreyszig (Kreyszig (1978), Thm 3.3-1, p. 144) states part 1 in a slightly different frame: $X$ is an inner product space, not assumed complete, and $C$ is a non-empty convex subset which is complete in the induced metric. Part 1 above is the case $X = H$ , where closed and complete agree. Folland (Folland (1999), Thm 5.24, p. 173) states parts 2 and 3 together and adds that $w$ is likewise the unique nearest point of $M^\perp$ to $u$ .

Idea of the proofOriginal

Everything comes from one identity. In an inner product space the parallelogram law

\|a - b\|^2 + \|a + b\|^2 = 2\|a\|^2 + 2\|b\|^2

converts a statement about how close two competitors are to each other into a statement about how close their midpoint is to the target. Apply it to $a = u - z_n$ and $b = u - z_m$ : the term $\|a+b\|^2$ becomes $4\|u - \tfrac12(z_n+z_m)\|^2$ , and convexity says the midpoint is a legal competitor, so that term is at least $4d^2$ . Two competitors that are both nearly optimal therefore have a midpoint that is also nearly optimal, and the identity forces them to be close to each other. A minimising sequence is Cauchy for free, completeness gives it a limit, and closedness keeps the limit in the set. The same identity with two exactly optimal competitors gives uniqueness, with no extra work.

Orthogonality is then a one-variable calculus statement. Once $v$ minimises, the function $t \mapsto \|u - v - tz\|^2$ has a minimum at $t = 0$ for every direction $z \in M$ ; it is a quadratic in $t$ whose linear coefficient is $-2\operatorname{Re}(u-v,z)$ , so that coefficient must vanish. Running the argument again with $iz$ in place of $z$ kills the imaginary part too. This is the only place where $M$ must be a subspace rather than merely convex: we need to be able to move from $v$ in both directions along $z$ and stay inside.

Proof(Part 1: existence and uniqueness of the nearest point)AdaptedAdapted from Kreyszig (1978), Thm 3.3-1, pp. 144–145; Stakgold & Holst (2011), §4.5, proof parts (a) and (c), pp. 266–267

Write $d = d(u,C) \ge 0$ , which is finite since $C \ne \varnothing$ .

Step 1: a minimising sequence. By the definition of an infimum there are $z_n \in C$ with $\|u - z_n\| \to d$ .

Step 2: the parallelogram law on the differences. Apply the parallelogram law to $a = u - z_n$ and $b = u - z_m$ . Since $a - b = z_m - z_n$ and $a + b = 2u - (z_n + z_m)$ ,

\|z_n - z_m\|^2 = 2\|u - z_n\|^2 + 2\|u - z_m\|^2 - 4\Bigl\|u - \tfrac{z_n + z_m}{2}\Bigr\|^2 .

Step 3: convexity bounds the last term from below. $\tfrac12(z_n + z_m) \in C$ because $C$ is convex, so the last norm is at least $d$ , giving

\|z_n - z_m\|^2 \le 2\|u - z_n\|^2 + 2\|u - z_m\|^2 - 4d^2 .

As $n,m \to \infty$ the right-hand side tends to $2d^2 + 2d^2 - 4d^2 = 0$ . Hence $(z_n)$ is Cauchy.

Step 4: pass to the limit. $H$ is complete, so $z_n \to v$ for some $v \in H$ ; and $C$ is closed, so $v \in C$ . The norm is continuous, so $\|u - v\| = \lim_n \|u - z_n\| = d$ . Existence is proved.

Step 5: uniqueness. Let $v$ and $v'$ in $C$ both satisfy $\|u - v\| = \|u - v'\| = d$ . Step 2 with $z_n = v$ , $z_m = v'$ reads

\|v - v'\|^2 = 2d^2 + 2d^2 - 4\Bigl\|u - \tfrac{v + v'}{2}\Bigr\|^2 \le 4d^2 - 4d^2 = 0 ,

again because the midpoint lies in $C$ . So $\|v - v'\| = 0$ , that is $v' = v$ . $\square$

Proof(Part 2: the error is orthogonal to the subspace)AdaptedAdapted from Stakgold & Holst (2011), §4.5, proof part (b), p. 267; Folland (1999), Thm 5.24, p. 174; Reed & Simon (1980), Thm II.3, p. 42

Let $M$ be a closed subspace, $v$ the nearest point given by part 1, and $w = u - v$ .

Step 1: set up a one-parameter competitor. Fix $z \in M$ and $t \in \mathbb{R}$ . Since $M$ is a subspace, $v + tz \in M$ , so by minimality $\|u - (v + tz)\| \ge \|u - v\|$ , that is

\|w - tz\|^2 \ge \|w\|^2 \qquad \text{for all } t \in \mathbb{R} .

Step 2: expand. Using sesquilinearity and $\overline{(z,w)} = (w,z)$ ,

\|w - tz\|^2 = \|w\|^2 - 2t\operatorname{Re}(w,z) + t^2\|z\|^2 .

Step 3: read off the linear term. By step 1, $\varphi(t) = -2t\operatorname{Re}(w,z) + t^2\|z\|^2 \ge 0$ for every real $t$ . A real quadratic in $t$ that vanishes at $t=0$ and is never negative cannot have a non-zero linear coefficient: if $\operatorname{Re}(w,z) \ne 0$ , then taking $t$ small with the sign of $\operatorname{Re}(w,z)$ makes $\varphi(t) < 0$ . Hence $\operatorname{Re}(w,z) = 0$ .

Step 4: the imaginary part. $iz \in M$ as well, so step 3 applies to it: $0 = \operatorname{Re}(w,iz) = \operatorname{Re}\bigl(-i(w,z)\bigr) = \operatorname{Im}(w,z)$ . (Here $(w,iz) = \overline{i}\,(w,z) = -i(w,z)$ , because the inner product is conjugate-linear in the second slot.) Therefore $(w,z) = 0$ for every $z \in M$ , that is $w \in M^\perp$ . For a real Hilbert space step 3 alone finishes the argument.

Step 5: the converse. Suppose $v \in M$ and $u - v \in M^\perp$ . For any $z \in M$ the vector $v - z$ lies in $M$ , so $u - v \perp v - z$ and the Pythagorean identity gives

\|u - z\|^2 = \|(u-v) + (v-z)\|^2 = \|u-v\|^2 + \|v-z\|^2 \ge \|u-v\|^2 ,

with equality only for $z = v$ . So $v$ is the nearest point, and this also re-proves its uniqueness. $\square$

Proof(Parts 3 and 4: the direct sum, the projection, and the complements)AdaptedAdapted from Kreyszig (1978), Thm 3.3-4, pp. 146–147, Lemma 3.3-5, p. 148, Lemma 3.3-6, p. 149, and Lemma 3.3-7, pp. 149–150; Stakgold & Holst (2011), the corollary following the projection theorem, p. 268

Step 1: existence of the decomposition. A closed subspace is non-empty (it contains $0$ ), convex and closed, so parts 1 and 2 give $u = v + w$ with $v \in M$ and $w = u - v \in M^\perp$ .

Step 2: uniqueness. If $u = v + w = v' + w'$ with $v,v' \in M$ and $w,w' \in M^\perp$ , then $v - v' = w' - w$ lies in $M \cap M^\perp$ . A vector $x$ in that intersection satisfies $(x,x) = 0$ , hence $x = 0$ . So $v = v'$ and $w = w'$ . This is what $H = M \oplus M^\perp$ means. Since $v \perp w$ , the Pythagorean identity gives $\|u\|^2 = \|v\|^2 + \|w\|^2$ .

Step 3: properties of $P$ . Linearity: if $u_1 = v_1 + w_1$ and $u_2 = v_2 + w_2$ are the decompositions, then $\alpha u_1 + \beta u_2 = (\alpha v_1 + \beta v_2) + (\alpha w_1 + \beta w_2)$ exhibits a decomposition of $\alpha u_1 + \beta u_2$ , and by uniqueness it is the decomposition; so $P(\alpha u_1 + \beta u_2) = \alpha P u_1 + \beta P u_2$ . If $u \in M$ then $u = u + 0$ is its decomposition, so $Pu = u$ ; applying this to $u = Pv$ gives $P^2 = P$ . The range is $M$ and $Pu = 0$ exactly when $u = w \in M^{\perp}$ , so the null space is $M^\perp$ . Finally $\|u\|^2 = \|Pu\|^2 + \|u - Pu\|^2 \ge \|Pu\|^2$ .

Step 4: $M \ne H$ gives a non-zero vector in $M^\perp$ . Pick $u \in H \setminus M$ and decompose $u = v + w$ . If $w$ were $0$ then $u = v \in M$ , a contradiction; so $w \in M^\perp$ is non-zero.

Step 5: $M^{\perp\perp} = M$ for closed $M$ . The inclusion $M \subseteq M^{\perp\perp}$ holds for any set: if $x \in M$ then $x \perp$ everything in $M^\perp$ by the definition of $M^\perp$ . Conversely let $x \in M^{\perp\perp}$ and write $x = v + w$ with $v \in M$ , $w \in M^\perp$ . Then $w = x - v$ is a difference of two members of $M^{\perp\perp}$ (using $M \subseteq M^{\perp\perp}$ for $v$ ), so $w \in M^{\perp\perp}$ ; but $w \in M^\perp$ as well, and $M^\perp \cap M^{\perp\perp} = \{0\}$ by the argument of step 2. Hence $w = 0$ and $x = v \in M$ .

Step 6: the density criterion. Let $S \ne \varnothing$ and let $V$ be the closure of the span of $S$ , a closed subspace. First, $S^\perp = V^\perp$ : a vector orthogonal to all of $S$ is orthogonal to every finite linear combination, hence, by continuity of the inner product, to every limit of such. Now if $V = H$ then $S^\perp = H^\perp = \{0\}$ . Conversely if $V \ne H$ then step 4 supplies a non-zero vector in $V^\perp = S^\perp$ . So $S^\perp = \{0\}$ if and only if the span of $S$ is dense. $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Stakgold & Holst (2011), §4.5, Remarks 1–3 on the proof, pp. 267–268; Kreyszig (1978), Thm 3.3-1, p. 144; Folland (1999), Exercise 58, p. 177

Convexity is used exactly once, in step 3 of the first proof (and again in step 5), to put the midpoint $\tfrac12(z_n+z_m)$ back into the set. Stakgold and Holst (Stakgold & Holst (2011), §4.5, Remark 3, p. 267) make the point in reverse: the proof of part 1 never needs $C$ to be a subspace, only that midpoints stay inside, which is the defining property of a convex set, and the general convex case is their Exercise 4.5.8.
Closedness of $C$ is used in step 4, to keep the limit of the minimising sequence inside $C$ . Without it the infimum can fail to be attained: their Remark 1 takes $H = L_2(a,b)$ and $C$ the set of all polynomials, which is a non-closed subspace. It is dense (see $L^p$ spaces), so $d(u,C) = 0$ for every $u$ , yet no polynomial equals a non-polynomial $u$ . Here $C^\perp = \{0\}$ while $C \ne H$ , which is why part 4 insists that $M$ be closed.
Completeness of $H$ is used in the same step, to produce the limit at all. A Cauchy minimising sequence in an incomplete inner product space can run off the edge of the space.
The parallelogram law — that is, the norm coming from an inner product — is used in step 2, and it is not a technicality. Remark 2 of Stakgold and Holst (Stakgold & Holst (2011), §4.5, p. 267) gives a Banach-space failure: in $C[0,1]$ with the sup norm let $M$ be the closed subspace of continuous functions vanishing at $x = 0$ ; the function $u(x) = 1$ has distance $1$ from $M$ but no nearest element of $M$ . The theorem is a Hilbert-space theorem, not a Banach-space one.
$M$ must be a subspace, not merely convex, for parts 2–4. Step 1 of the second proof perturbs $v$ by $tz$ for both signs of $t$ ; in a convex set only $t \ge 0$ small is available, and the conclusion degrades from $(w,z) = 0$ to the variational inequality $\operatorname{Re}(u - v, z - v) \le 0$ for all $z \in C$ . The example below shows a convex projection that is not linear.
Orthogonality of the complement is what makes $P$ self-adjoint. Any decomposition $H = M \oplus N$ into complementary subspaces defines an idempotent $P$ , but only the orthogonal choice $N = M^\perp$ gives $\|Pu\| \le \|u\|$ and $(Pu,z) = (u,Pz)$ . Folland's (Folland (1999), Exercise 58, p. 177) Exercise 58 records the converse: a bounded operator with $P^2 = P^* = P$ is the orthogonal projection onto its (then automatically closed) range. See adjoints, self-adjoint and unitary operators.
Finite dimension hides all of this. Hassani (Hassani (2013), Prop. 6.1.3, p. 170) proves $\mathcal{V} = \mathcal{M} \oplus \mathcal{M}^\perp$ for any subspace $\mathcal{M}$ of a finite-dimensional inner product space, with no closedness hypothesis, by extending an orthonormal basis of $\mathcal{M}$ and setting $P = \sum_{i} |e_i\rangle\langle e_i|$ . That is legitimate there because every finite-dimensional subspace is automatically closed; in infinite dimensions the hypothesis has to be imposed.

Method(Computing an orthogonal projection)Original

Identify the subspace and check it is closed. Finite-dimensional subspaces and null spaces of bounded operators are closed; spans of infinite sets usually are not, and must be replaced by their closures.
If $M = \operatorname{span}\{h_1,\dots,h_n\}$ , write $v = \sum_j c_j h_j$ and impose orthogonality. The condition $u - v \perp h_i$ for each $i$ is the normal equations

\sum_{j=1}^{n} (h_j, h_i)\, c_j = (u, h_i) , \qquad i = 1,\dots,n ,

an $n \times n$ linear system whose matrix is the Gram matrix $G_{ij} = (h_j,h_i)$ . It is invertible exactly when the $h_j$ are linearly independent. See projections and least squares. 3. If the $h_j$ are orthonormal, skip the solve: $c_j = (u,h_j)$ , so $Pu = \sum_j (u,h_j)h_j$ . This is why one orthonormalises first (the Gram-Schmidt process), and it extends to infinite orthonormal families in orthonormal bases in Hilbert space. 4. Get the distance without computing $v$ twice: $d(u,M)^2 = \|u\|^2 - \|Pu\|^2 = \|u\|^2 - \sum_i c_i \overline{(u,h_i)}$ when $Pu = \sum_i c_i h_i$ . A negative answer means an arithmetic error. 5. For a convex set that is not a subspace, do not solve a linear system. Minimise directly, using the characterisation $\operatorname{Re}(u - v, z - v) \le 0$ for all $z \in C$ ; for sets defined pointwise, such as $\{f \ge 0\}$ in an $L^2$ space, the minimisation decouples point by point. 6. To prove a family is total (its span dense), do not try to expand a general vector. Use part 4: take an arbitrary $u$ orthogonal to every member of the family and show $u = 0$ .

Example(A projection onto a subspace and a projection onto a cone)Original

Work in the real Hilbert space $H = L^2(0,1)$ with $(f,g) = \int_0^1 f(x)g(x)\,dx$ .

Part A: a closed subspace. Let $M = \operatorname{span}\{1, x\}$ , the polynomials of degree at most $1$ ; it is two-dimensional, hence closed. Project $u(x) = x^3$ onto it.

The inner products needed are $(1,1) = 1$ , $(1,x) = \tfrac12$ , $(x,x) = \tfrac13$ , $(u,1) = \int_0^1 x^3dx = \tfrac14$ and $(u,x) = \int_0^1 x^4dx = \tfrac15$ . Writing $Pu = a + bx$ , the normal equations of step 2 are

a + \tfrac12 b = \tfrac14 , \qquad \tfrac12 a + \tfrac13 b = \tfrac15 .

Multiply the first by $\tfrac12$ and subtract from the second: $\bigl(\tfrac13 - \tfrac14\bigr)b = \tfrac15 - \tfrac18$ , that is $\tfrac{1}{12}b = \tfrac{3}{40}$ , so $b = \tfrac{9}{10}$ and then $a = \tfrac14 - \tfrac{9}{20} = -\tfrac15$ . Hence

Pu(x) = -\tfrac15 + \tfrac{9}{10}x .

By step 4,

d(u,M)^2 = \|u\|^2 - \bigl[a(u,1) + b(u,x)\bigr] = \frac17 - \Bigl[-\frac15\cdot\frac14 + \frac{9}{10}\cdot\frac15\Bigr] = \frac17 - \frac{13}{100} = \frac{9}{700} ,

so the distance from $x^3$ to the straight lines is

d(u,M) = \frac{3}{\sqrt{700}} = 0.1133893419\ldots

Two checks. First, orthogonality: $(u - Pu, 1) = \tfrac14 - \bigl(-\tfrac15 + \tfrac{9}{20}\bigr) = \tfrac14 - \tfrac14 = 0$ , and $(u - Pu, x) = \tfrac15 - \bigl(-\tfrac{1}{10} + \tfrac{3}{10}\bigr) = \tfrac15 - \tfrac15 = 0$ . Second, a competitor: the Taylor-style choice $3x - 2$ , which matches $u$ and $u'$ at $x = 1$ , gives $\int_0^1 (x^3 - 3x + 2)^2dx = \tfrac{33}{35} = 0.942857$ , distance $0.971008$ , worse by a factor $8.5635$ . Nearest in the mean-square sense is not nearest in any other sense.

Part B: a closed convex set that is not a subspace. Let

C = \{\, f \in L^2(0,1) : f \ge 0 \text{ a.e.} \,\} ,

the non-negative cone. It is convex (a convex combination of non-negative functions is non-negative) and closed (if $f_k \to f$ in $L^2$ then a subsequence converges a.e., and an a.e. limit of non-negative functions is non-negative; see modes of convergence). Project $u(x) = x - \tfrac12$ .

Because the constraint is imposed pointwise, so is the minimisation: for each $x$ separately, the number $s \ge 0$ minimising $(u(x) - s)^2$ is $s = \max(u(x),0)$ . Any $f \in C$ therefore satisfies $\int_0^1 (u-f)^2 \ge \int_0^1 (u - u^+)^2$ , with equality only if $f = u^+$ a.e. So the nearest point is the positive part

v(x) = u^+(x) = \begin{cases} 0, & 0 < x \le \tfrac12, \\ x - \tfrac12, & \tfrac12 < x < 1,\end{cases}

and the distance is

d(u,C)^2 = \int_0^{1/2}\Bigl(x - \tfrac12\Bigr)^2 dx = \frac{1}{24} , \qquad d(u,C) = \frac{1}{\sqrt{24}} = 0.2041241452\ldots

This projection is genuinely non-linear: $u^+ + (-u)^+ = |u| \ne 0 = (u + (-u))^+$ . The error $u - v = -u^-$ is not orthogonal to $C$ — indeed $(u - v, v) = 0$ but $(u-v, z) = \int_0^{1/2}(x - \tfrac12)z \le 0$ for every $z \in C$ , which is the variational inequality of the remark, with a strict sign for most $z$ . Convexity buys existence and uniqueness; only a subspace buys orthogonality and linearity.

ApplicationOriginal

Least squares and data fitting. Fitting a model that is linear in its parameters to data means projecting the data vector onto the span of the basis functions, and the normal equations of the method are exactly step 2. Everything a practitioner relies on — that the fit exists, that it is unique when the design matrix has full rank, that the residual is orthogonal to every predictor, that the residual sum of squares is $\|u\|^2 - \|Pu\|^2$ — is this theorem. See projections and least squares and least-squares regression.

Conditional expectation. Grimmett and Stirzaker (Grimmett & Stirzaker (2020), §7.9, Thms (9), (14) and (17) and Def. (20), pp. 389–392) prove the projection theorem directly for a closed linear space $H$ of random variables under $\|\cdot\|_2$ , by precisely the parallelogram-law argument above, and then identify the nearest point to $Y$ as the best mean-square predictor. Taking $H$ to be the square-integrable $\mathcal{G}$ -measurable random variables gives their definition of $\mathbb{E}(Y \mid \mathcal{G})$ as the element satisfying $\mathbb{E}\bigl[(Y - \mathbb{E}(Y\mid\mathcal{G}))Z\bigr] = 0$ for all $Z \in H$ — the orthogonality relation of part 2, written out. Martingales, filtering and the Kalman recursion all descend from this reading. See conditional expectation and the Kalman filter.

Quantum mechanics. A measurement with outcome subspace $M$ sends a state $u$ to $Pu/\|Pu\|$ , and $\|Pu\|^2/\|u\|^2$ is the probability of that outcome; the complementary outcome corresponds to $1 - P$ , and $H = M \oplus M^\perp$ is the statement that the two outcomes are exhaustive and mutually exclusive. Hassani (Hassani (2013), Prop. 6.1.3, p. 170) writes this as $|a\rangle = P|a\rangle + (1-P)|a\rangle$ with both projectors hermitian. See the mathematical framework of quantum mechanics.

Variational methods for differential equations. The Rayleigh-Ritz and Galerkin methods replace a problem posed in $H$ by the same problem posed in a finite-dimensional subspace $M_n$ ; when the energy inner product is used, the Galerkin solution is the orthogonal projection of the true solution onto $M_n$ , so it is the best possible approximation from $M_n$ in the energy norm, and the error is orthogonal to the trial space. That orthogonality is the engine of finite-element error analysis. See Rayleigh-Ritz, the Lax-Milgram theorem and finite elements.

Decompositions of fields and of functions. Many physically meaningful splittings are instances of $H = M \oplus M^\perp$ . Stakgold and Holst's (Stakgold & Holst (2011), Exercise 4.5.6, p. 273, and Exercise 4.5.10, p. 274) exercises give two: the even and odd parts of a function in $L_2(-1,1)$ , and the splitting of a vector field on a bounded domain into an irrotational part $\operatorname{grad} p$ and a solenoidal part with vanishing normal component, whose existence they reduce to a Neumann boundary-value problem. The latter is the Helmholtz decomposition, the basis of the pressure-projection schemes used in computational fluid dynamics; see the Helmholtz decomposition.

Theorem

Orthonormal bases in Hilbert space

Adapted

Add to PDF

Gives Bessel's inequality for any orthonormal family, the exact criterion for a series $\sum a_k e_k$ to converge, and the equivalence of maximality, Parseval's identity and expandability that defines an orthonormal basis, with the consequence that a separable Hilbert space is $\ell^2$ in disguise.

Prerequisites: The projection theorem, Hilbert spaces, Orthonormal bases and Gram-Schmidt, Zorn's lemma and the axiom of choice

Definition(Orthonormal family, Fourier coefficients, total set, basis)AdaptedAdapted from Kreyszig (1978), §3.4, p. 157, and Def. 3.6-1 with its footnote, p. 168; Folland (1999), §5.3, p. 175; Stakgold & Holst (2011), §4.6, pp. 275–277

Let $H$ be a Hilbert space with inner product $(u,v)$ , linear in the first slot.

A family $\{e_\alpha\}_{\alpha \in A} \subseteq H$ is orthonormal if $\|e_\alpha\| = 1$ for every $\alpha$ and $(e_\alpha, e_\beta) = 0$ for $\alpha \ne \beta$ . An orthonormal family is linearly independent, and any two of its members are at distance $\|e_\alpha - e_\beta\| = \sqrt2$ .

For $u \in H$ the scalars

\hat u(\alpha) = (u, e_\alpha)

are the Fourier coefficients of $u$ with respect to the family. The name is justified by the classical case: for $H = L^2(0,2\pi)$ and $e_k(t) = (2\pi)^{-1/2}e^{ikt}$ these are the coefficients of the Fourier series of $u$ .

A subset $S \subseteq H$ is total (Kreyszig (Kreyszig (1978), Def. 3.6-1, p. 168)) or spanning (Stakgold and Holst (Stakgold & Holst (2011), §4.6, p. 275)) if the set of finite linear combinations of its members is dense in $H$ . A total orthonormal family is called an orthonormal basis. The empty family is orthonormal (vacuously), and the set of finite linear combinations of no vectors is $\{0\}$ ; so by this definition the empty family is the orthonormal basis of the zero space, which is the convention we use whenever $H = \{0\}$ is not excluded by hypothesis.

$H$ is separable if it contains a countable dense subset; equivalently, by Stakgold and Holst (Stakgold & Holst (2011), §4.6, p. 276), if it has a countable spanning set.

A warning about two words. An orthonormal basis is not a basis in the algebraic (Hamel) sense unless $H$ is finite-dimensional: its expansions are infinite series, not finite sums. And "complete" is used in this subject for two unrelated things — completeness of a metric space, and the property (a) below of an orthonormal set. Kreyszig (Kreyszig (1978), §3.6, footnote 4, p. 168) reserves the word for the metric sense and says "total" for the other; Folland (Folland (1999), Thm 5.27, p. 175) and most of the physics literature say "complete orthonormal set". We follow Kreyszig's terminology and point out the clash where it matters.

Theorem(Bessel's inequality and the convergence criterion)AdaptedAdapted from Kreyszig (1978), Thm 3.4-6, p. 157, and Thm 3.5-2, p. 164; Folland (1999), Bessel's Inequality 5.26, p. 175; Stakgold & Holst (2011), the Riesz-Fischer theorem, p. 277

Let $\{e_\alpha\}_{\alpha \in A}$ be an orthonormal family in a Hilbert space $H$ .

Bessel's inequality. For every $u \in H$ ,

\sum_{\alpha \in A} |(u,e_\alpha)|^2 \;\le\; \|u\|^2 ,

the sum being the supremum of its finite subsums. In particular at most countably many Fourier coefficients of $u$ are non-zero. 2. Convergence criterion (Riesz-Fischer). Let $(e_k)_{k\ge1}$ be an orthonormal sequence and $(\alpha_k)$ scalars. The series $\sum_{k\ge1}\alpha_k e_k$ converges in $H$ if and only if $\sum_{k\ge1}|\alpha_k|^2$ converges. When it does, its sum $u$ has $\alpha_k = (u,e_k)$ , so the series is the Fourier series of its own sum, and $\|u\|^2 = \sum_k |\alpha_k|^2$ . 3. In particular, by part 1 the series $\sum_k (u,e_k)e_k$ converges for every $u \in H$ — but, without further hypotheses, not necessarily to $u$ .

Theorem(What makes an orthonormal family a basis)AdaptedAdapted from Folland (1999), Thm 5.27, p. 175, Prop. 5.28, Prop. 5.29, p. 176, and Prop. 5.30, p. 177; Kreyszig (1978), Thm 3.6-2, p. 169, Thm 3.6-3, p. 170, Thm 3.6-4, p. 171, and Thm 3.6-5, p. 173

Let $\{e_\alpha\}_{\alpha\in A}$ be an orthonormal family in a Hilbert space $H$ . The following are equivalent.

(a) Maximality. If $(u,e_\alpha) = 0$ for every $\alpha$ , then $u = 0$ .
(b) Parseval's identity. $\displaystyle \|u\|^2 = \sum_{\alpha\in A}|(u,e_\alpha)|^2$ for every $u \in H$ .
(c) Expansion. $\displaystyle u = \sum_{\alpha\in A}(u,e_\alpha)e_\alpha$ for every $u \in H$ , the sum having only countably many non-zero terms and converging in norm regardless of the order in which they are taken.
(d) Totality. The span of $\{e_\alpha\}$ is dense in $H$ .

A family with these properties is an orthonormal basis. Further:

Existence. Every Hilbert space $H \ne \{0\}$ has an orthonormal basis.
Separability. $H$ is separable if and only if it has a countable orthonormal basis, and then every orthonormal basis of $H$ is countable.
Classification. If $\{e_\alpha\}_{\alpha\in A}$ is an orthonormal basis, then $u \mapsto \hat u$ is a unitary map of $H$ onto $\ell^2(A)$ : it is linear, onto, and preserves inner products. Consequently (Kreyszig (Kreyszig (1978), Thm 3.6-5, p. 173)) two Hilbert spaces over the same field are isomorphic if and only if their orthonormal bases have the same cardinality, the Hilbert dimension. Every separable infinite-dimensional Hilbert space is a copy of $\ell^2$ .

The polarisation of (b) holds too: for all $u,v \in H$ , $(u,v) = \sum_\alpha (u,e_\alpha)\overline{(v,e_\alpha)}$ .

Hassani (Hassani (2013), Thms 7.2.2 and 7.2.3, p. 222) states item 6 in the concrete form a physicist meets it: every complete inner product space with a countable basis is isomorphic to $L^2_w(a,b)$ , and the monomials $\{x^k\}_{k\ge0}$ are a basis of that space, which he attributes to the Stone-Weierstrass theorem.

Idea of the proofOriginal

Bessel's inequality is the Pythagorean theorem plus the projection theorem, and nothing else. For a finite subfamily $F$ , the vector $\sum_{\alpha\in F}(u,e_\alpha)e_\alpha$ is the orthogonal projection of $u$ onto the span of $\{e_\alpha\}_{\alpha\in F}$ , because the coefficients were chosen to make the error orthogonal to each $e_\alpha$ . The projection theorem then says $\|u\|^2$ splits as (projection) $^2$ + (error) $^2$ , so the projection is no longer than $u$ , which is Bessel for $F$ . Taking the supremum over finite $F$ gives the general case: the inequality is really a statement about every finite sub-sum at once, which is also why no ordering of $A$ is needed.

The convergence criterion is then almost a restatement: the partial sums of $\sum\alpha_ke_k$ are Cauchy exactly when the partial sums of $\sum|\alpha_k|^2$ are, because orthogonality makes $\|\sum_{m}^{n}\alpha_ke_k\|^2 = \sum_m^n|\alpha_k|^2$ exactly. Completeness of $H$ turns "Cauchy" into "convergent"; this is the only place it is used, and it is indispensable.

The equivalence theorem is a cycle. Given maximality, Bessel says the candidate series converges to something; subtracting it from $u$ gives a vector orthogonal to every $e_\alpha$ , which maximality kills, so the series converges to $u$ — this is (a) $\Rightarrow$ (c). Then (c) $\Rightarrow$ (b) by continuity of the norm along the partial sums, and (b) $\Rightarrow$ (a) because a vector with all coefficients zero has norm zero. The only subtlety is that $A$ may be uncountable; Bessel's countability clause reduces everything to a sequence.

Proof(Bessel's inequality and the countability of the coefficients)AdaptedAdapted from Folland (1999), Bessel's Inequality 5.26, p. 175; Kreyszig (1978), §3.4, pp. 155–157

Step 1: reduce to a finite subfamily. By definition $\sum_{\alpha\in A}|(u,e_\alpha)|^2$ is the supremum of the sums over finite $F \subseteq A$ , so it suffices to prove $\sum_{\alpha\in F}|(u,e_\alpha)|^2 \le \|u\|^2$ for each finite $F$ .

Step 2: the finite sum is a projection. Put $M_F = \operatorname{span}\{e_\alpha : \alpha\in F\}$ , a finite-dimensional, hence closed, subspace, and $v = \sum_{\alpha\in F}(u,e_\alpha)e_\alpha \in M_F$ . For each $\beta\in F$ , orthonormality gives

(u - v, e_\beta) = (u,e_\beta) - \sum_{\alpha\in F}(u,e_\alpha)(e_\alpha,e_\beta) = (u,e_\beta) - (u,e_\beta) = 0 .

So $u - v \perp M_F$ , and by the converse half of the projection theorem, $v = Pu$ is the orthogonal projection of $u$ onto $M_F$ .

Step 3: Pythagoras. Since $u = v + (u-v)$ with $v \perp u - v$ ,

\|u\|^2 = \|v\|^2 + \|u-v\|^2 \;\ge\; \|v\|^2 = \sum_{\alpha\in F}|(u,e_\alpha)|^2 ,

the last equality again by orthonormality. Taking the supremum over finite $F$ proves the inequality.

Step 4: countability. Let $A_n = \{\alpha : |(u,e_\alpha)| > 1/n\}$ , and suppose some $A_n$ were infinite. Choose an integer $N > n^2\|u\|^2$ and a subfamily $F \subseteq A_n$ with exactly $N$ members. Each of its $N$ terms exceeds $n^{-2}$ , so

\sum_{\alpha\in F}|(u,e_\alpha)|^2 \;>\; \frac{N}{n^2} \;>\; \|u\|^2 ,

contradicting step 3. So each $A_n$ is finite, and $\{\alpha : (u,e_\alpha)\ne0\} = \bigcup_{n\ge1}A_n$ is countable. $\square$

Proof(The convergence criterion)AdaptedAdapted from Kreyszig (1978), Thm 3.5-2, pp. 164–165; Stakgold & Holst (2011), the Riesz-Fischer theorem, p. 277

Let $(e_k)$ be orthonormal, $s_n = \sum_{k=1}^n \alpha_k e_k$ and $t_n = \sum_{k=1}^n|\alpha_k|^2$ .

Step 1: the two sequences have the same Cauchy behaviour. For $n > m$ , orthonormality gives exactly

\|s_n - s_m\|^2 = \Bigl\|\sum_{k=m+1}^{n}\alpha_ke_k\Bigr\|^2 = \sum_{k=m+1}^{n}|\alpha_k|^2 = t_n - t_m .

So $(s_n)$ is Cauchy in $H$ if and only if $(t_n)$ is Cauchy in $\mathbb{R}$ . This identity, not an inequality, is the whole proof.

Step 2: both sides are complete. $H$ is complete, so $(s_n)$ Cauchy means $\sum\alpha_ke_k$ converges; $\mathbb{R}$ is complete, and $(t_n)$ is increasing, so $(t_n)$ Cauchy means $\sum|\alpha_k|^2$ converges. The two conditions are therefore equivalent to each other.

Step 3: the coefficients are the Fourier coefficients. Suppose $s_n \to u$ . Fix $j$ . For $n \ge j$ , $(s_n, e_j) = \alpha_j$ by orthonormality. The inner product is continuous in its first slot, so

(u,e_j) = \lim_{n\to\infty}(s_n,e_j) = \alpha_j .

Step 4: the norm. $\|s_n\|^2 = t_n$ by orthonormality, and both sides converge, giving $\|u\|^2 = \sum_k|\alpha_k|^2$ .

Step 5: part 3 of the statement. For arbitrary $u \in H$ , Bessel's inequality says $\sum_k|(u,e_k)|^2 \le \|u\|^2 < \infty$ , so by steps 1–2 the series $\sum_k(u,e_k)e_k$ converges in $H$ . Nothing so far says its sum is $u$ : for the orthonormal sequence $(e_k)_{k\ge2}$ in a space with basis $(e_k)_{k\ge1}$ and $u = e_1$ , every coefficient vanishes and the sum is $0 \ne u$ . $\square$

Proof(The equivalence, existence, separability and classification)AdaptedAdapted from Folland (1999), Thm 5.27, pp. 175–176, Prop. 5.28 and Prop. 5.29, p. 176, and Prop. 5.30, p. 177; Kreyszig (1978), Thm 3.6-2, p. 169, Thm 3.6-3, p. 170, and Thm 3.6-4, pp. 171–172

Step 1: (a) implies (c). Let $u \in H$ . By step 4 of the Bessel proof, only countably many coefficients are non-zero; enumerate those indices as $\alpha_1,\alpha_2,\dots$ in any order. By Bessel and the convergence criterion, $\sum_j (u,e_{\alpha_j})e_{\alpha_j}$ converges; call the sum $v$ , and note that reordering changes nothing, since by the criterion the sum of the squares — a series of non-negative terms — is unconditionally convergent and step 1 of that proof then makes the partial sums Cauchy for any ordering. Now compute $(u - v, e_\beta)$ for an arbitrary $\beta \in A$ : if $\beta$ is one of the $\alpha_j$ the two terms cancel, and if not then $(u,e_\beta) = 0$ and $(v,e_\beta) = 0$ by continuity of the inner product. So $u - v$ is orthogonal to every $e_\beta$ , and (a) forces $u = v$ .

Step 2: (c) implies (b). With the notation of step 1, orthonormality gives $\bigl\|\sum_{j\le m}(u,e_{\alpha_j})e_{\alpha_j}\bigr\|^2 = \sum_{j\le m}|(u,e_{\alpha_j})|^2$ ; letting $m\to\infty$ and using continuity of the norm together with (c) gives Parseval.

Step 3: (b) implies (a). If every coefficient of $u$ vanishes, Parseval gives $\|u\|^2 = 0$ , so $u = 0$ .

Step 4: (a) is the same as (d). This is part 4 of the projection theorem: the span of a set $S$ is dense in $H$ exactly when $S^\perp = \{0\}$ , and $S^\perp = \{0\}$ is precisely (a). Completeness of $H$ is used here, inside that theorem.

Step 5: polarisation. Given (c) for both $u$ and $v$ , continuity of the inner product in both slots applied to the partial sums gives $(u,v) = \sum_\alpha (u,e_\alpha)\overline{(v,e_\alpha)}$ , the series converging absolutely by Cauchy-Schwarz in $\ell^2$ .

Step 6: existence. Order the orthonormal subsets of $H$ by inclusion. The union of a chain of orthonormal sets is orthonormal — any two of its members lie in a common member of the chain — so every chain has an upper bound, and Zorn's lemma gives a maximal orthonormal set. Maximality means no unit vector can be adjoined orthogonally to all of it, which is exactly (a): a non-zero $u$ with all coefficients zero would give the extra member $u/\|u\|$ .

Step 7: separability. If $H$ has a countable orthonormal basis $(e_k)$ , the finite linear combinations of the $e_k$ with coefficients in $\mathbb{Q}+i\mathbb{Q}$ form a countable set, and it is dense: given $u$ and $\varepsilon>0$ , choose $n$ with $\|u - \sum_{k\le n}(u,e_k)e_k\| < \varepsilon/2$ by (c), then rationalise each of the $n$ coefficients to within $\varepsilon/(2n)$ and use the triangle inequality. Conversely, if $H$ is separable and $\{e_\alpha\}$ is any orthonormal family, the balls of radius $\sqrt2/3$ about its members are pairwise disjoint, because distinct members are $\sqrt2$ apart; a countable dense set must put a point in each, so the family is countable. Applying Gram-Schmidt to a countable dense sequence (after discarding members that are in the span of their predecessors) produces a countable orthonormal family with dense span, hence a basis by step 4.

Step 8: classification. Let $\{e_\alpha\}_{\alpha\in A}$ be an orthonormal basis and $U : u \mapsto \hat u$ , $\hat u(\alpha) = (u,e_\alpha)$ . By Bessel, $\hat u \in \ell^2(A)$ , and $U$ is linear because the inner product is linear in its first slot. It preserves inner products by step 5, hence is isometric and injective. It is onto: given $f \in \ell^2(A)$ , only countably many $f(\alpha)$ are non-zero, and the convergence criterion makes $\sum_\alpha f(\alpha)e_\alpha$ converge to some $u$ with $\hat u = f$ . So $U$ is unitary. Finally, a unitary map carries orthonormal bases to orthonormal bases, so isomorphic spaces have bases of equal cardinality; and conversely a bijection $A \to B$ between index sets induces a unitary map $\ell^2(A)\to\ell^2(B)$ . $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Kreyszig (1978), Thm 3.6-2 and the remark following it, pp. 169–170, and §3.6, footnote 4, p. 168; Stakgold & Holst (2011), §4.6, pp. 276 and 288; Folland (1999), Thm 5.27, p. 175

Bessel's inequality needs no completeness and no basis. It holds in any inner product space, for any orthonormal family, and it is what makes everything else finite. Part 3 of the first statement is the sharp form of the warning: the series always converges, but it converges to the projection of $u$ onto the closed span of the family, which is $u$ only when the family is total.
Completeness of $H$ is used twice: in the convergence criterion (step 2), to turn a Cauchy sequence of partial sums into a vector, and in the equivalence (a) $\Leftrightarrow$ (d), through the projection theorem. Kreyszig (Kreyszig (1978), Thm 3.6-2(b) and the following remark, p. 169) is explicit that completeness is essential in the direction " $M^\perp = \{0\}$ implies $M$ total", and reports that J. Dixmier gave in 1953 an incomplete inner product space containing no total orthonormal set at all. We have not seen Dixmier's example and report the attribution as Kreyszig gives it.
Orthonormality, not just orthogonality, is what makes the coefficients free. For a merely orthogonal family the same results hold with $(u,e_\alpha)$ replaced by $(u,e_\alpha)/\|e_\alpha\|^2$ and $|(u,e_\alpha)|^2$ by $|(u,e_\alpha)|^2/\|e_\alpha\|^2$ ; the physics literature usually keeps the unnormalised form and carries the weights explicitly.
A spanning set is weaker than a Schauder basis, which is weaker than an orthonormal basis. Stakgold and Holst (Stakgold & Holst (2011), §4.6, p. 276) warn that a countable spanning set gives only approximation by finite combinations whose coefficients may change with the accuracy demanded: $\{1,x,x^2,\dots\}$ spans $L_2(a,b)$ , yet almost no $L_2$ function is the sum of a power series. Their Exercise 4.6.7 builds a spanning set, $h_n = e_1 + e_{n+1}/(n+1)$ , which is independent, has the same closed span as $\{e_k\}$ , and yet admits no series expansion of $e_1$ . Orthonormality is what removes this gap, by making the coefficients independent of the truncation.
The index set need not be countable in the equivalence theorem, which is why Folland (Folland (1999), Thm 5.27, p. 175) phrases (c) with an unordered sum: the countability of the non-zero terms, and hence the irrelevance of their order, is a conclusion (Bessel), not a hypothesis. Non-separable Hilbert spaces are rare in applications but do occur. Give the almost-periodic functions on $\mathbb{R}$ the inner product $(f,g) = \lim_{T\to\infty}(2T)^{-1}\int_{-T}^{T}f\bar g$ ; then $\{e^{i\lambda t}\}_{\lambda\in\mathbb{R}}$ is an uncountable orthonormal family, so no countable set can be dense. That inner product space is itself incomplete; the Hilbert space in question is its completion, the Besicovitch space $B^2$ , in which the same uncountable family sits as an orthonormal basis.
"Complete" means two different things, as the definition block says; a complete orthonormal set in an incomplete inner product space is a perfectly meaningful object, and the Dixmier example above is exactly about when one exists.

Method(Showing that an orthonormal family is a basis)Original

Prefer criterion (a). Assume $(u,e_\alpha) = 0$ for all $\alpha$ and deduce $u = 0$ . This requires no estimate and no expansion, and it is the only criterion that is a single implication rather than a statement about all $u$ at once.
For eigenfunctions of a differential operator, do not verify (a) directly: cite the theorem that produces them. Regular Sturm-Liouville problems and compact self-adjoint operators both deliver a complete orthonormal set of eigenfunctions as part of their conclusion.
For a family built from a dense class, use totality directly: if the span of $\{e_\alpha\}$ contains a set already known to be dense, it is dense. In $L^2(a,b)$ on a bounded interval, "contains all polynomials" suffices, by the Weierstrass approximation theorem and the density of $C[a,b]$ in $L^2$ .
To test a candidate numerically, take a few functions whose norms you know and compare $\sum_k|(u,e_k)|^2$ against $\|u\|^2$ . A persistent shortfall that does not shrink as more terms are added indicates a missing basis element; a shortfall that shrinks like the tail of a convergent series indicates only truncation.
To estimate how many terms are needed, use Parseval on the tail: the squared $L^2$ error of the $n$ -term truncation is exactly $\sum_{k>n}|(u,e_k)|^2$ , so the decay rate of the coefficients is the convergence rate of the expansion. Smoothness of $u$ controls that decay; see smoothness and decay.
Never drop a basis element "because its coefficient looks small" without checking the tail: removing a single $e_1$ from a basis leaves an orthonormal family that is still infinite and still independent, but is no longer total.

Example(Parseval checked exactly, and Bessel seen to be strict)Original

Work in the real Hilbert space $H = L^2(0,1)$ with $(f,g) = \int_0^1 fg\,dx$ , and take

e_n(x) = \sqrt2\,\sin(n\pi x) , \qquad n = 1,2,3,\dots

These are orthonormal: $2\int_0^1\sin(m\pi x)\sin(n\pi x)dx = \int_0^1[\cos((m-n)\pi x) - \cos((m+n)\pi x)]dx$ , which is $0$ for $m\ne n$ and $1$ for $m = n$ . They are total, being the normalised eigenfunctions of the regular Sturm-Liouville problem $-y'' = \lambda y$ , $y(0)=y(1)=0$ (Sturm-Liouville theory).

Step 1: the coefficients of a test function. Let $u(x) = x(1-x)$ . Two integrations by parts give

\int_0^1 x(1-x)\sin(n\pi x)\,dx = \frac{2\bigl(1 - (-1)^n\bigr)}{n^3\pi^3} ,

(u,e_n) = \frac{2\sqrt2\bigl(1-(-1)^n\bigr)}{n^3\pi^3} = \begin{cases} \dfrac{4\sqrt2}{n^3\pi^3}, & n \text{ odd},\\[2mm] 0, & n \text{ even}.\end{cases}

The even coefficients vanish because $u$ is symmetric about $x = \tfrac12$ while $e_n$ is antisymmetric there for even $n$ .

Step 2: the two sides of Parseval. On the left,

\|u\|^2 = \int_0^1 x^2(1-x)^2dx = \frac13 - \frac12 + \frac15 = \frac{1}{30} .

On the right, summing over odd $n = 2j-1$ ,

\sum_{n\ \mathrm{odd}} \frac{32}{n^6\pi^6} = \frac{32}{\pi^6}\sum_{n\ \mathrm{odd}}\frac{1}{n^6} = \frac{32}{\pi^6}\Bigl(1 - 2^{-6}\Bigr)\zeta(6) = \frac{32}{\pi^6}\cdot\frac{63}{64}\cdot\frac{\pi^6}{945} = \frac{2016}{60480} = \frac{1}{30} ,

using $\zeta(6) = \pi^6/945$ (the Riemann zeta function and Bernoulli numbers) and the fact that the even terms contribute $2^{-6}\zeta(6)$ . The two sides agree exactly: $\|u\|^2 = \sum_n |(u,e_n)|^2 = 1/30 = 0.0333333\ldots$ Read backwards, Parseval has just evaluated $\sum_{n\ \mathrm{odd}}n^{-6} = \pi^6/960$ .

Step 3: how fast the series converges. The first term alone is $32/\pi^6 = 0.0332852$ , which is $99.8555\%$ of $1/30$ . By item 5 of the method, the squared $L^2$ error of the one-term approximation $u \approx (u,e_1)e_1$ is the tail

\frac{1}{30} - \frac{32}{\pi^6} = 4.8166\times10^{-5} ,

so the relative error in norm is $\sqrt{4.8166\times10^{-5}\cdot30} = 0.0380$ , under $4\%$ from a single sine. The coefficients fall like $n^{-3}$ because $u$ vanishes at both endpoints but $u''$ does not.

Step 4: Bessel, strictly. Now discard $e_1$ and keep only $F = \{e_n : n \ge 2\}$ , still orthonormal. For the same $u$ ,

\sum_{n\ge2}|(u,e_n)|^2 = \frac{1}{30} - \frac{32}{\pi^6} = 4.8166\times10^{-5} \;<\; \frac{1}{30} = \|u\|^2 ,

so Bessel's inequality is strict, by a factor of about $692$ . Criterion (a) fails visibly: $e_1$ itself is a non-zero vector with every $F$ -coefficient equal to zero. The deficit $\|u\|^2 - \sum_{n\ge2}|(u,e_n)|^2 = |(u,e_1)|^2$ is exactly the squared length of the part of $u$ that $F$ cannot see — the general fact that Bessel's deficit is the squared distance to the closed span, which is step 3 of the Bessel proof read as an equality.

ApplicationOriginal

Eigenfunction expansions and the closure relation. Solving a boundary-value problem by separation of variables produces a candidate solution as an infinite series in the eigenfunctions; the series means nothing until the eigenfunctions are known to be total, and the identification of the coefficients by taking inner products is step 3 of the convergence proof. Riley, Hobson and Bence (Riley, Hobson & Bence (2006), §17.1, pp. 558–559, and §17.3.3, eq. (17.28), p. 563) present the same content in the form physicists use. They expand $f$ in an orthonormal set $\hat\phi_n$ and read off $c_n = \langle\hat\phi_n|f\rangle$ (their inner product is conjugate-linear in the first slot and carries a weight $\rho$ ); their Bessel inequality $\langle f|f\rangle \ge \sum_n|c_n|^2$ becomes an equality exactly when the sum runs over the whole basis, which is our (b); and completeness of the eigenfunctions $\hat y_n$ of a Hermitian operator is written as the closure relation

\rho(z)\sum_n \hat y_n(x)\hat y_n^*(z) = \delta(x-z) .

The weight $\rho$ there is the one in $\langle f|g\rangle = \int_a^b f^*g\rho\,dx$ , and dropping it falsifies the identity for any non-constant weight. This is Parseval's identity in the language of the Dirac delta, obtained by applying (c) to the functional "evaluate at $x$ ". See eigenfunction expansions and eigenfunction expansions of Green's functions.

Quantum mechanics. The states of a system form a separable Hilbert space, and the eigenstates of an observable with discrete spectrum form an orthonormal basis; item 6 is the reason Heisenberg's matrix mechanics on $\ell^2$ and Schrödinger's wave mechanics on $L^2$ are the same theory, a point settled historically exactly by the classification above. Parseval's identity is the statement that the probabilities $|\langle n|\psi\rangle|^2$ of the possible measurement outcomes sum to $1$ , and the closure relation is written $\sum_n |n\rangle\langle n| = 1$ . See the mathematical framework of quantum mechanics.

Signal processing and data compression. Parseval says the energy of a signal equals the energy of its coefficient sequence, so discarding small coefficients changes the signal by a controlled amount — exactly the tail estimate of item 5. This is the principle behind transform coding, and the choice of basis (Fourier, wavelet, Karhunen-Loève) is a choice of which family makes the tail decay fastest for the signals at hand. See Parseval and mean-square convergence and the discrete Fourier transform and the FFT.

Empirical orthogonal functions. In climate science and other data analysis, a covariance operator on $L^2$ of the spatial domain is diagonalised and its eigenfunctions — the EOFs — are used as the basis; the variance explained by the first $n$ modes is the $n$ -term Parseval partial sum, and the residual variance is the Bessel deficit of step 4 above. See EOF and principal component analysis.

Spectral numerical methods. A spectral method represents the unknown by its first $N$ coefficients in a total orthonormal family and converts the differential equation into equations for those coefficients. The accuracy is governed by the tail, which for smooth data decays faster than any power of $N$ ; that "spectral accuracy" is Parseval plus the decay of coefficients of smooth functions. See spectral methods.

Theorem

The Riesz representation theorem

Adapted

Add to PDF

Shows that every bounded linear functional on a Hilbert space is $u \mapsto (u,z)$ for one and only one vector $z$ , of the same norm, so that the dual space is a conjugate copy of the space itself — the statement that makes Dirac's bra a vector and underwrites weak formulations, adjoints and Green's functions.

Prerequisites: The projection theorem, Hilbert spaces, Normed and Banach spaces

Definition(Bounded linear functional, dual space)AdaptedAdapted from Stakgold & Holst (2011), §4.7, pp. 288–289; Kreyszig (1978), §3.8, p. 188

Let $H$ be a Hilbert space over $\mathbb{K} = \mathbb{R}$ or $\mathbb{C}$ , with inner product $(u,v)$ linear in the first slot and conjugate-linear in the second.

A linear functional on $H$ is a map $f : H \to \mathbb{K}$ with $f(\alpha u + \beta v) = \alpha f(u) + \beta f(v)$ . It is bounded if there is a constant $c$ with

|f(u)| \le c\|u\| \qquad\text{for all } u \in H ,

and the smallest such $c$ ,

\|f\| = \sup_{u \ne 0}\frac{|f(u)|}{\|u\|} = \sup_{\|u\| = 1}|f(u)| ,

is the norm of $f$ . The set $H^{*}$ of bounded linear functionals on $H$ , with this norm, is the dual space; it is a Banach space (see normed and Banach spaces).

Bounded means continuous. Stakgold and Holst (Stakgold & Holst (2011), §4.7, p. 289) prove the two implications separately. If $f$ is bounded and $u_n \to u$ , then $|f(u_n) - f(u)| = |f(u_n - u)| \le \|f\|\,\|u_n - u\| \to 0$ . Conversely, if $f$ were continuous but unbounded, there would be non-zero $u_n$ with $|f(u_n)| \ge n\|u_n\|$ ; the vectors $v_n = u_n/(n\|u_n\|)$ then tend to $0$ while $|f(v_n)| \ge 1$ , contradicting continuity at the origin — and continuity at the origin already implies continuity everywhere, by linearity.

A sesquilinear form on $H$ is a map $h : H\times H \to \mathbb{K}$ linear in the first argument and conjugate-linear in the second (Kreyszig (Kreyszig (1978), Def. 3.8-3, p. 191)); it is bounded if $|h(u,v)| \le c\|u\|\|v\|$ , and the least such $c$ is $\|h\|$ . The inner product itself is a bounded sesquilinear form with $\|h\| = 1$ .

Theorem(Riesz representation theorem)AdaptedAdapted from Kreyszig (1978), Thm 3.8-1, p. 188; Stakgold & Holst (2011), the Riesz representation theorem, p. 290; Folland (1999), Thm 5.25, p. 174; Reed & Simon (1980), Thm II.4, p. 43

Let $H$ be a Hilbert space.

Representation. For every bounded linear functional $f$ on $H$ there is a vector $z \in H$ with

f(u) = (u, z) \qquad\text{for every } u \in H .

Uniqueness. That $z$ is unique: if $(u,z_1) = (u,z_2)$ for all $u$ , then $z_1 = z_2$ .
Isometry. $\|z\| = \|f\|$ , and the supremum defining $\|f\|$ is attained at $u = z/\|z\|$ when $z \ne 0$ .
The dual space is a conjugate copy of $H$ . The map $J : H \to H^{*}$ , $(Jz)(u) = (u,z)$ , is a bijection which is conjugate-linear, $J(\alpha z) = \bar\alpha\,Jz$ , and isometric. On a real Hilbert space it is linear, and then $H^{*}$ and $H$ are isometrically isomorphic outright.

Conventions differ, and the statement moves with them. We write the functional as the inner product with $z$ in the second slot, because our inner product is linear in the first. Reed and Simon (Reed & Simon (1980), Thm II.4, p. 43), whose inner product is linear in the second slot, state it as $T(x) = (y_T, x)$ with the representer first; this is the same theorem with the roles of the slots exchanged, and they attribute it to F. Riesz and M. Fréchet. Evans (Evans (2010), Appendix D.3(b), Thm 2, p. 678) states it for a real Hilbert space, where the distinction disappears and the map $u^{*}\mapsto u$ is a genuine linear isomorphism. All three are the statement above.

Corollary (bounded sesquilinear forms). Reed and Simon (Reed & Simon (1980), the corollary to Thm II.4, p. 44) deduce: if $B : H\times H \to \mathbb{C}$ satisfies $|B(u,v)| \le c\|u\|\|v\|$ and is conjugate-linear in the first and linear in the second argument, there is a unique bounded linear operator $A$ with $B(u,v) = (Au, v)$ for all $u,v$ , and $\|A\|$ is the least admissible $c$ . With our convention the corollary reads: a bounded sesquilinear $h$ is $h(u,v) = (Au,v)$ for a unique bounded $A$ . This is the standard route to adjoints and to the Lax-Milgram theorem.

Idea of the proofAdaptedAdapted from Kreyszig (1978), proof of Thm 3.8-1, part (a), p. 189

A linear functional is determined by two pieces of data: the hyperplane it kills, and how fast it grows off that hyperplane. If $f = (\cdot\,,z)$ then $z$ must be orthogonal to the null space $N = f^{-1}(0)$ , because $f$ vanishes there; so the representer, if it exists at all, must live in $N^{\perp}$ (this is how Kreyszig (Kreyszig (1978), proof of Thm 3.8-1, p. 189) motivates the proof). Boundedness makes $N$ closed, and then the projection theorem says $N^\perp$ is not $\{0\}$ unless $N$ is everything. That already pins $z$ down to a line: $N^\perp$ turns out to be one-dimensional, because two independent vectors in it could be combined to make a non-zero vector of $N^\perp$ on which $f$ vanishes, which would then lie in $N \cap N^\perp = \{0\}$ .

The proof is just this observation made constructive. Take any non-zero $z_0 \in N^\perp$ . For an arbitrary $u$ , the combination $f(u)z_0 - f(z_0)u$ is annihilated by $f$ , hence lies in $N$ , hence is orthogonal to $z_0$ . Writing that orthogonality out and solving for $f(u)$ produces the formula, with $z$ a multiple of $z_0$ . The norm equality then costs two lines: Cauchy-Schwarz gives $\|f\| \le \|z\|$ , and testing $f$ on $z$ itself gives the reverse.

Proof(Representation, uniqueness and the norm)AdaptedAdapted from Kreyszig (1978), Thm 3.8-1, pp. 189–190; Folland (1999), Thm 5.25, p. 174; Stakgold & Holst (2011), the Riesz representation theorem, p. 290

Step 0: the trivial case. If $f = 0$ , take $z = 0$ ; then $f(u) = (u,0) = 0$ and $\|z\| = 0 = \|f\|$ . Assume from now on that $f \ne 0$ .

Step 1: the null space is a proper closed subspace. $N = \{u : f(u) = 0\}$ is a subspace by linearity, and it is closed because $f$ is bounded, hence continuous, so $N$ is the preimage of the closed set $\{0\}$ . Since $f \ne 0$ , $N \ne H$ .

Step 2: produce a direction off the null space. By part 4 of the projection theorem — equivalently the corollary Stakgold and Holst (Stakgold & Holst (2011), §4.5, p. 268) draw from it — a proper closed subspace has a non-zero orthogonal complement. Pick $z_0 \in N^{\perp}$ with $z_0 \ne 0$ . Note $f(z_0)\ne0$ : otherwise $z_0 \in N\cap N^\perp = \{0\}$ .

Step 3: the key combination. Fix $u \in H$ and set

v = f(u)\,z_0 - f(z_0)\,u .

Applying $f$ and using linearity, $f(v) = f(u)f(z_0) - f(z_0)f(u) = 0$ , so $v \in N$ . Since $z_0 \perp N$ ,

0 = (v, z_0) = f(u)\,(z_0,z_0) - f(z_0)\,(u,z_0) = f(u)\|z_0\|^2 - f(z_0)(u,z_0) .

Step 4: solve for $f(u)$ . As $\|z_0\|^2 \ne 0$ ,

f(u) = \frac{f(z_0)}{\|z_0\|^2}\,(u,z_0) = \Bigl(u,\ \frac{\overline{f(z_0)}}{\|z_0\|^2}z_0\Bigr) ,

the conjugate appearing because the inner product is conjugate-linear in its second slot. So part 1 holds with

z = \frac{\overline{f(z_0)}}{\|z_0\|^{2}}\,z_0 ,

and $u$ was arbitrary.

Step 5: uniqueness. If $(u,z_1) = (u,z_2)$ for all $u$ then $(u, z_1 - z_2) = 0$ for all $u$ ; taking $u = z_1 - z_2$ gives $\|z_1-z_2\|^2 = 0$ , hence $z_1 = z_2$ . (Kreyszig (Kreyszig (1978), Lemma 3.8-2, p. 190) isolates this as a lemma, because the same two lines are needed repeatedly: a vector orthogonal to everything is zero.)

Step 6: the norm. Cauchy-Schwarz gives $|f(u)| = |(u,z)| \le \|z\|\,\|u\|$ , so $\|f\| \le \|z\|$ . For the reverse, test on $z$ itself: $f(z) = (z,z) = \|z\|^2$ , while $|f(z)| \le \|f\|\,\|z\|$ . If $z \ne 0$ , dividing by $\|z\|$ gives $\|z\| \le \|f\|$ ; if $z = 0$ then $f = 0$ by part 1 and both norms vanish. Hence $\|z\| = \|f\|$ , and $f(z/\|z\|) = \|z\| = \|f\|$ shows the supremum is attained.

Step 7: part 4. $J$ is onto by step 1–4 and injective by step 5, and isometric by step 6. It is conjugate-linear because $(u,\alpha z) = \bar\alpha(u,z)$ . $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Stakgold & Holst (2011), §4.7, p. 289 and Exercises 4.7.1–4.7.3, pp. 291–292; Folland (1999), §5.5, p. 175; Kreyszig (1978), §3.8, p. 188

Boundedness is used in step 1, to make $N$ closed, and it cannot be dropped. Stakgold and Holst (Stakgold & Holst (2011), §4.7, p. 289) give the standard failure: on $H = L_2(-1,1)$ let $M$ be the dense subspace of continuous functions and $l(u) = u(0)$ . Taking a fixed continuous $\varphi \ge 0$ with $\varphi(\pm1)=0$ and $\varphi > 0$ inside, the functions $u_k(x) = \varphi(kx)$ for $|x| \le k^{-1}$ and $0$ elsewhere satisfy $\int_{-1}^{1}u_k^2 = k^{-1}\int_{-1}^{1}\varphi^2 \to 0$ , so $u_k \to 0$ in $L_2$ , while $u_k(0) = \varphi(0)$ is constant. So $l$ is linear and unbounded on $M$ , and no $z$ can represent it: point evaluation on $L^2$ is not an inner product with anything. Note that $l$ is defined only on the dense subspace $M$ , not on all of $H$ ; what the example shows is that point evaluation has no bounded extension to $H$ . Unbounded linear functionals defined on the whole of an infinite-dimensional $H$ do exist, but only non-constructively (through a Hamel basis), and the theorem says nothing about them except that they are not of the form $(u,z)$ . (Making it one is exactly what a reproducing-kernel space, or the Dirac delta as a distribution, is for.)
Completeness of $H$ is used inside the projection theorem at step 2, and it too is essential. Let $X \subset \ell^2$ be the incomplete inner product space of sequences with only finitely many non-zero entries, and $f(u) = \sum_{n\ge1}u_n/n$ . Then $|f(u)| \le \|u\|\bigl(\sum n^{-2}\bigr)^{1/2} = \pi\|u\|/\sqrt6$ , so $f$ is bounded on $X$ ; but a representer $z \in X$ would have to satisfy $z_n = 1/n$ for every $n$ , which is not finitely supported. The representer exists in the completion, not in the space.
Linearity of $f$ is used in step 3, where $f(u)z_0 - f(z_0)u$ is formed with scalar coefficients and $f$ is applied to it. Stakgold and Holst's (Stakgold & Holst (2011), §4.7, Examples 1 and 3, p. 288) examples on a complex $H$ , written here with a fixed non-zero vector $g$ , show the ways this can fail. $l(u) = \|u\|$ and $l(u) = (u,g)+a$ with a constant $a \ne 0$ are not even additive. $l(u) = (g,u)$ is additive but conjugate-linear, $l(\alpha u) = \bar\alpha\, l(u)$ , so it fails only homogeneity under complex scalars; additivity alone is therefore not enough. None of the three is representable as $(u,z)$ .
The slot matters, and so does the conjugate. Over $\mathbb{C}$ the map $J$ is conjugate-linear, so $H^{*}$ is anti-isomorphic to $H$ rather than isomorphic; Folland (Folland (1999), §5.5, p. 175) puts it as “Hilbert spaces are reflexive in a very strong sense”: $H$ is naturally isomorphic to its second dual $H^{**}$ , and to $H^*$ itself as well, through a conjugate-linear map. Stakgold and Holst's (Stakgold & Holst (2011), Exercise 4.7.1, p. 291) Exercise 4.7.1 asks the reader to notice the difference by looking at $l(\alpha u)$ .
Nothing here needs separability or a basis, although Stakgold and Holst (Stakgold & Holst (2011), Exercise 4.7.3, pp. 291–292) sketch an alternative proof that does: expand in an orthonormal basis and set $z = \sum_i \overline{l(e_i)}e_i$ , the work then being to show that $\sum_i|l(e_i)|^2$ converges. A third proof, their Exercise 4.7.2, maximises $|l(u)|/\|u\|$ directly using the parallelogram law. We followed the null-space route because it uses the projection theorem as a black box and gives the norm equality for free.
The theorem is false in general Banach spaces, where there is no inner product to represent anything with; what survives is the Hahn-Banach theorem, which guarantees that enough functionals exist without describing them. See the Hahn-Banach theorem.

Method(Finding the representer)Original

Check boundedness first, with an explicit constant: estimate $|f(u)|$ by $c\|u\|$ , usually with Cauchy-Schwarz or Hölder. If no such $c$ exists, stop — there is no representer, and the object you have is a distribution or an unbounded functional, not an element of $H^{*}$ .
If $f$ is already written as an integral against something, say $f(u) = \int u\,\bar g$ , then $g$ is the representer, provided $g \in H$ . Most functionals met in practice arrive in this form and need no theorem; the theorem is what tells you the remaining ones are of this form too.
Otherwise expand in an orthonormal basis. Set $z = \sum_k \overline{f(e_k)}\,e_k$ ; this converges precisely when $\sum_k|f(e_k)|^2 < \infty$ , which boundedness guarantees: for each $N$ , linearity gives $\sum_{k\le N}|f(e_k)|^2 = f\bigl(\sum_{k\le N}\overline{f(e_k)}\,e_k\bigr) \le \|f\|\bigl(\sum_{k\le N}|f(e_k)|^2\bigr)^{1/2}$ , so every partial sum is at most $\|f\|^2$ . Then $(u,z) = \sum_k (u,e_k)\overline{\overline{f(e_k)}} = f(u)$ by continuity. See orthonormal bases.
Read off the norm: $\|f\| = \|z\|$ . This is often the easiest way to compute the norm of a functional, and it identifies the maximiser as $z/\|z\|$ .
To invert a non-orthogonal expansion, build the reciprocal basis. Given independent $h_1,\dots,h_n$ , the functionals " $j$ -th coordinate" are bounded on their span; their representers $h_j^{*}$ satisfy $(h_i,h_j^{*}) = \delta_{ij}$ , and then the coordinates of any $u$ in the span are $\alpha_j = (u,h_j^{*})$ . Concretely $h_j^{*} = \sum_k (G^{-1})_{jk}h_k$ with $G_{ik} = (h_i,h_k)$ the Gram matrix: then $(h_i,h_j^{*}) = \sum_k G_{ik}\,\overline{(G^{-1})_{jk}} = (G\,G^{-1})_{ij} = \delta_{ij}$ , using that $G$ , and hence $G^{-1}$ , is Hermitian. The order of the slots in $G$ matters only over $\mathbb{C}$ . This is Stakgold and Holst's (Stakgold & Holst (2011), §4.7, (4.7.4)–(4.7.5), p. 291) construction.
For a bilinear or sesquilinear expression, apply the theorem one slot at a time to get an operator rather than a vector, as in the corollary above.

Example(A representer, its norm, and a reciprocal basis)Original

Work in the real Hilbert space $H = L^2(0,1)$ , $(u,v) = \int_0^1 uv\,dx$ .

Part A: a functional and its representer. Define

f(u) = \int_0^{1/2}u(x)\,dx \;-\; 2\int_0^1 x\,u(x)\,dx .

It is linear, and bounded because each term is an inner product: indeed

f(u) = (u, z), \qquad z(x) = \chi_{[0,1/2]}(x) - 2x = \begin{cases}1 - 2x, & 0 < x \le \tfrac12,\\ -2x, & \tfrac12 < x < 1.\end{cases}

By part 3 of the theorem, $\|f\| = \|z\|$ , and

\|z\|^2 = \int_0^{1/2}(1-2x)^2dx + \int_{1/2}^{1}4x^2dx = \frac16 + \frac{7}{6} = \frac{4}{3} ,

using $\int_0^{1/2}(1-2x)^2dx = \tfrac12\int_0^1t^2dt = \tfrac16$ (substitute $t = 1-2x$ ) and $\int_{1/2}^1 4x^2dx = \tfrac43(1 - \tfrac18) = \tfrac76$ . Hence

\|f\| = \frac{2}{\sqrt3} = 1.154700538\ldots

Check that the supremum is attained: with $u = z/\|z\|$ ,

f(u) = \frac{(z,z)}{\|z\|} = \|z\| = \frac{2}{\sqrt3} ,

as part 3 predicts. A competitor confirms the bound is not beaten: the unit vector $u(x) = \sqrt3\,(1-2x)$ gives $f(u) = \sqrt3\int_0^1(1-2x)z(x)\,dx = \sqrt3\bigl(\tfrac16 + \tfrac5{12}\bigr) = \tfrac{7\sqrt3}{12} = 1.010363 < 1.154701$ .

Part B: a reciprocal basis. Take the oblique (non-orthogonal) pair $h_1(x) = 1$ , $h_2(x) = x$ , spanning the degree- $\le1$ polynomials. The Gram matrix and its inverse are

G = \begin{pmatrix}1 & \tfrac12\\[1mm] \tfrac12 & \tfrac13\end{pmatrix} , \qquad G^{-1} = \begin{pmatrix}4 & -6\\ -6 & 12\end{pmatrix} ,

since $\det G = \tfrac13 - \tfrac14 = \tfrac1{12}$ . By item 5 of the method,

h_1^{*}(x) = 4 - 6x , \qquad h_2^{*}(x) = -6 + 12x .

Verify the defining relations: $(h_1,h_1^{*}) = \int_0^1(4-6x)dx = 4-3 = 1$ ; $(h_2,h_1^{*}) = \int_0^1 x(4-6x)dx = 2 - 2 = 0$ ; $(h_1,h_2^{*}) = -6+6 = 0$ ; $(h_2,h_2^{*}) = -3 + 4 = 1$ . All four are correct.

Now use the coordinate formula on a function outside the span, $u(x) = x^3$ :

(u,h_1^{*}) = \int_0^1 x^3(4-6x)dx = 1 - \tfrac65 = -\tfrac15 , \qquad (u,h_2^{*}) = \int_0^1 x^3(-6+12x)dx = -\tfrac32 + \tfrac{12}{5} = \tfrac{9}{10} .

These are exactly the coefficients of the orthogonal projection of $x^3$ onto $\operatorname{span}\{1,x\}$ computed in the projection theorem, namely $-\tfrac15 + \tfrac9{10}x$ — which is the correct reading of the formula: $\alpha_j = (u,h_j^{*})$ returns the coordinates of $u$ when $u$ is in the span, and the coordinates of its projection when it is not, because $h_j^{*}$ lies in the span and so does not see the part of $u$ orthogonal to it.

ApplicationOriginal

Dirac notation. In quantum mechanics the Riesz theorem is what licenses the bracket. A bra $\langle\phi|$ is by definition a bounded linear functional on the state space, and the theorem says every such functional is $|\psi\rangle \mapsto \langle\phi|\psi\rangle$ for exactly one ket $|\phi\rangle$ — so bras and kets are in one-to-one correspondence, the correspondence being conjugate-linear, which is precisely why $\langle\alpha\phi| = \bar\alpha\langle\phi|$ . Hassani (Hassani (2013), §2.5, p. 49 and Thm 2.5.2, pp. 50–51) sets this up in finite dimensions, where $\gamma_a(|b\rangle) = \langle a|b\rangle$ and the dual basis $\phi_i(|a_j\rangle) = \delta_{ij}$ is the reciprocal basis of the example; the infinite-dimensional statement needs the theorem. Dirac's $\delta$ is the celebrated exception: point evaluation is not bounded on $L^2$ , as the remark shows, which is why the delta lives in a distribution space and not in the Hilbert space. See the mathematical framework of quantum mechanics.

Weak formulations of boundary-value problems. A weak formulation reads: find $u \in H$ with $a(u,v) = f(v)$ for all $v \in H$ , where $a$ is a bounded sesquilinear form and $f$ a bounded functional of the same type in $v$ as $a(u,\cdot)$ : on a real Hilbert space, as in Evans, simply a bounded linear functional; on a complex one, a bounded conjugate-linear functional. The corollary turns $a$ into an operator, $a(u,v) = (Au,v)$ , and the theorem, applied to the linear functional $v \mapsto \overline{f(v)}$ , turns $f$ into a vector, $f(v) = (z,v)$ , reducing the problem to $Au = z$ in $H$ ; adding coercivity of $a$ gives the Lax-Milgram theorem and with it existence, uniqueness and a stability bound in one stroke. This is the standard existence theory for elliptic equations and the foundation of the finite-element method. See the Lax-Milgram theorem, weak solutions and Sobolev spaces and finite elements.

Green's functions as representers. When the solution operator of a boundary-value problem is bounded, the map "solution evaluated against a test function" is a bounded functional of the data, and its representer is the Green's function. The symmetry of the Green's function for a self-adjoint problem is the statement that the corresponding operator equals its adjoint, which is itself built from the theorem. See Green's functions for boundary-value problems and Hilbert-Schmidt kernels.

Adjoints and the whole operator theory. Folland's (Folland (1999), Exercise 57a, p. 177) Exercise 57a defines the adjoint $T^{*}$ by applying the theorem to the functional $x \mapsto (Tx,y)$ for each fixed $y$ : the representer is $T^{*}y$ . Everything about self-adjointness, unitarity and the spectral theorem rests on that construction. Evans (Evans (2010), Appendix D.3(b), p. 678) introduces the adjoint in exactly this position, immediately after the representation theorem. See adjoints, self-adjoint and unitary operators.

Estimation and filtering. In statistics the best linear unbiased estimator of a linear functional of an unknown signal is found by representing that functional in the Hilbert space generated by the observations; the representer is the optimal weight vector, and its norm is the standard error. This "representer method" is used directly in data assimilation, where the analysis increment is a combination of representers of the observation functionals. See data assimilation and the Kalman filter.

Definition · advanced

Weak convergence

Adapted

Add to PDF

Defines weak convergence as convergence of every bounded linear functional along the sequence, collects what survives from the norm theory (uniqueness of the limit, boundedness, lower semicontinuity of the norm) and what does not (the norm itself is not continuous), and gives the working test in $\ell^2$ and $\ell^p$ — bounded plus coordinatewise.

Prerequisites: The Riesz representation theorem, Normed and Banach spaces, Hilbert spaces, Orthonormal bases in Hilbert space, Uniform boundedness and open mapping theorems, The Hahn-Banach theorem

Definition(Strong, weak and weak-

*

convergence)AdaptedAdapted from Kreyszig (1978), §4.8, Def. 4.8-1, p. 256, and Def. 4.8-2, p. 257; Stakgold & Holst (2011), §4.8, the three notions listed on p. 295; Evans (2010), Appendix D.4, p. 678; Folland (1999), §5.4, pp. 168–169

Let $X$ be a normed space over $\mathbb{K} = \mathbb{R}$ or $\mathbb{C}$ and let $X'$ be its dual: the space whose elements are the bounded linear functionals on $X$ , normed by $\lVert f\rVert = \sup_{\lVert u\rVert\le1}|f(u)|$ . The space $X'$ is always complete in that norm, whatever $X$ is, because the scalars are (Stakgold and Holst (Stakgold & Holst (2011), Thm 4.8.4, p. 294)); the proof of part 4 below turns on this.

A sequence $(x_n)$ in $X$ converges strongly to $x \in X$ if $\lVert x_n - x\rVert \to 0$ . This is ordinary convergence in the norm; the adjective is used only to mark the contrast with what follows. We write $x_n \to x$ .

The sequence converges weakly to $x \in X$ if the numbers $f(x_n)$ converge to $f(x)$ for every single $f \in X'$ :

x_n \rightharpoonup x \quad\Longleftrightarrow\quad f(x_n) \to f(x) \ \text{ in } \mathbb{K}, \quad \text{for every } f \in X' .

The vector $x$ is the weak limit. Two things in this definition deserve to be said aloud. First, it is a statement about a whole family of numerical sequences at once, one for each $f$ , and nothing is asked of them uniformly in $f$ . Second, the limit $x$ is required to be an element of $X$ : weak convergence is not a device for manufacturing limits that the space does not contain.

If the space is a Hilbert space $H$ , every bounded functional is an inner product by the Riesz representation theorem, so the definition collapses to a statement about inner products:

x_n \rightharpoonup x \quad\Longleftrightarrow\quad (x_n , z) \to (x , z) \quad\text{for every } z \in H ,

which is Kreyszig's (Kreyszig (1978), Example 4.8-5, p. 260) Example 4.8-5. Our inner product is linear in the first slot, as throughout this chapter; the slot in which $z$ sits is immaterial here, since $\overline{a_n} \to \overline{a}$ exactly when $a_n \to a$ .

A third notion lives on a dual space. If $X$ is a normed space, a sequence $(f_n)$ in $X'$ converges weak- $*$ to $f \in X'$ , written $f_n \overset{*}{\rightharpoonup} f$ , if $f_n(u) \to f(u)$ for every $u \in X$ : pointwise convergence of functionals. Stakgold and Holst (Stakgold & Holst (2011), §4.8, p. 295) list the three side by side. Weak- $*$ convergence on $X'$ asks less than weak convergence on $X'$ , which would test against all of $X''$ ; Folland (Folland (1999), §5.4, p. 169) records the comparison at the level of topologies: the weak and the weak- $*$ topology on $X'$ coincide precisely when $X$ is reflexive, and where the topologies coincide so do the two notions of sequential convergence. Every Hilbert space is reflexive (Kreyszig (Kreyszig (1978), Thm 4.6-6, p. 242)), so in Hilbert space there is nothing new here.

Weak convergence is convergence in a topology. Reed and Simon (Reed & Simon (1980), §IV.5, p. 111) and Folland (Folland (1999), §5.4, pp. 168–169) define the weak topology on $X$ to be the coarsest topology for which every $f \in X'$ is still continuous; weak convergence of a sequence is convergence in that topology. This handbook treats only sequences, which is all the applications need, but the distinction is not idle: several of the statements below are true for sequences and false for nets, and the point is flagged in the remark.

Two theorems are quoted ahead of their place in this part. Part 4 below uses the uniform boundedness theorem and part 2 uses the separation corollary of the Hahn-Banach theorem. Both are genuine prerequisites and both are listed as such, but both are treated later in this part than this unit; they are used here as quoted results, and nothing in this unit enters their proofs, so the dependence is one-way.

Notation. The half-arrow $\rightharpoonup$ is used by Kreyszig (Kreyszig (1978), §4.8, p. 257), Stakgold and Holst (Stakgold & Holst (2011), §4.8, p. 295) and Evans (Evans (2010), Appendix D.4, p. 678) alike. Some texts, Reed and Simon (Reed & Simon (1980), §IV.5, p. 112) among them, write $x_n \xrightarrow{\ w\ } x$ instead.

Definition(What weak convergence does and does not inherit)AdaptedAdapted from Kreyszig (1978), Lemma 4.8-3, p. 258, Thm 4.8-4, p. 259, Example 4.8-6, p. 261, and Lemma 4.8-7, p. 261; Reed & Simon (1980), the Proposition and Example 1, §IV.5, p. 112; Stakgold & Holst (2011), Thms 4.8.5–4.8.7, pp. 295–296, and Thm 4.8.9, p. 297; Evans (2010), Appendix D.4, p. 678

Let $X$ be a normed space and $(x_n)$ a sequence in $X$ .

Weak convergence deserves its name. Norm convergence forces it: if $x_n \to x$ then $x_n \rightharpoonup x$ , and the two limits are the same vector.
The weak limit is unique. If $x_n \rightharpoonup x$ and $x_n \rightharpoonup y$ then $x = y$ .
Subsequences inherit it. If $x_n \rightharpoonup x$ , every subsequence of $(x_n)$ converges weakly to $x$ .
Weakly convergent sequences are norm bounded: $\sup_n \lVert x_n\rVert < \infty$ . No completeness of $X$ is needed.
The norm is weakly lower semicontinuous, not weakly continuous: if $x_n \rightharpoonup x$ then

\lVert x \rVert \;\le\; \liminf_{n\to\infty}\lVert x_n\rVert ,

and the inequality is often strict. 6. The working criterion. Let $M \subset X'$ be a total set, meaning that the finite linear combinations of members of $M$ are dense in $X'$ . Then $x_n \rightharpoonup x$ if and only if

(A) $(\lVert x_n\rVert)$ is bounded, and
(B) $f(x_n) \to f(x)$ for every $f \in M$ .

Two specialisations.
- In a Hilbert space $H$ with orthonormal basis $(e_\alpha)_{\alpha\in A}$ , the index set $A$ arbitrary: $x_n \rightharpoonup x$ if and only if $(\lVert x_n\rVert)$ is bounded and $(x_n,e_\alpha) \to (x,e_\alpha)$ for each fixed $\alpha$ . No countability is asked of $A$ , so $H$ need not be separable; Reed and Simon (Reed & Simon (1980), §IV.5, Example 1, p. 112) likewise state their Example 1 for an arbitrary index set.
- In $\ell^p$ with $1 < p < \infty$ : $x_n \rightharpoonup x$ if and only if $(\lVert x_n\rVert_p)$ is bounded and each coordinate converges, $\xi^{(n)}_j \to \xi_j$ for every fixed $j$ . This is Kreyszig's (Kreyszig (1978), Example 4.8-6, p. 261) Example 4.8-6, obtained from part 6 with $M$ the coordinate functionals, which form a Schauder basis of the dual $\ell^q$ .
Finite dimensions, and only finite dimensions. If $\dim X < \infty$ then weak convergence implies strong convergence, so the two notions coincide. In every infinite-dimensional Hilbert space they differ: an orthonormal sequence $(e_n)$ satisfies $e_n \rightharpoonup 0$ while $\lVert e_n - e_m\rVert = \sqrt2$ for $n \ne m$ , so $(e_n)$ has no strong limit at all.
Why one puts up with it (quoted, not proved here). In a reflexive Banach space — in particular in any Hilbert space, and in $L^p$ for $1<p<\infty$ — every bounded sequence has a weakly convergent subsequence. Evans (Evans (2010), Appendix D.4, Thm 3, p. 678) states it as a weak compactness theorem; Stakgold and Holst (Stakgold & Holst (2011), Thms 4.8.8–4.8.9, p. 297) state it as the Eberlein–Shmulian theorem and in the equivalent form that reflexivity is characterised by this property. Both refer the proof elsewhere, and so do we: it is the one statement in this unit we use without proving.

Kreyszig (Kreyszig (1978), §4.8, p. 260) adds a warning against reading part 8 as "infinite dimensions always separate the two": he reports that in $\ell^1$ weak and strong convergence are again equivalent, a theorem of I. Schur from 1921. So the right dividing line is not dimension but the geometry of the particular space.

Idea of the proofOriginal

A vector in a normed space can be probed in two ways: by its own size, or by the numbers a measuring instrument returns when applied to it. Weak convergence says every instrument eventually reads the same as it would on the limit, and says nothing about size. The two differ because in infinite dimensions a sequence can run off in ever-new directions: each fixed instrument is sensitive to only "finitely much" of the space in the relevant sense, so it stops noticing, while the vectors themselves stay a fixed distance apart. An orthonormal sequence is the model — Bessel's inequality says every fixed $z$ has square-summable coefficients, so $(e_n,z)\to0$ , while the $e_n$ sit at mutual distance $\sqrt2$ forever.

The three good properties all come from the same move: turn the vectors into functionals. The map $x \mapsto g_x$ , $g_x(f) = f(x)$ , sends $X$ isometrically into the dual of $X'$ ; weak convergence of $(x_n)$ says exactly that the $g_{x_n}$ converge pointwise on $X'$ . Pointwise convergent means pointwise bounded, $X'$ is complete whatever $X$ is, and the uniform boundedness theorem then converts pointwise boundedness into bounded norms — which are the norms $\lVert x_n\rVert$ . Uniqueness of the limit comes from the other Hahn–Banach corollary, that the functionals separate points. Lower semicontinuity is a single application of Cauchy–Schwarz (or, in a general normed space, of the norming functional, which needs no completeness either): you can test the limit against itself, and the test cannot read more than the norms it is applied to.

The criterion in part 6 is an $\varepsilon/3$ argument and nothing more. Boundedness is what makes the approximation of an arbitrary $f$ by elements of $\operatorname{span} M$ uniform along the sequence; without it the error $\lVert f - h\rVert\,\lVert x_n\rVert$ is uncontrolled, and the criterion is false.

Proof(Parts 1 to 5)AdaptedAdapted from Kreyszig (1978), Lemma 4.8-3 and its proof, p. 258, Thm 4.8-4(a), p. 259, Lemma 4.6-1, p. 240, and Cor. 4.3-4, p. 223; Stakgold & Holst (2011), Thm 4.8.5, pp. 295–296, Thm 4.8.3 and Thm 4.8.4, p. 294

Part 1. Let $x_n \to x$ and fix $f \in X'$ . Then

|f(x_n) - f(x)| = |f(x_n - x)| \le \lVert f\rVert\,\lVert x_n - x\rVert \longrightarrow 0 ,

the inequality being the boundedness of $f$ and nothing else. As $f$ was arbitrary, $x_n \rightharpoonup x$ .

Part 2. Suppose $x_n \rightharpoonup x$ and $x_n \rightharpoonup y$ . For each $f \in X'$ the numerical sequence $(f(x_n))$ has the two limits $f(x)$ and $f(y)$ ; limits of sequences of scalars are unique, so $f(x) = f(y)$ , that is $f(x-y) = 0$ . This holds for every bounded functional, and a vector annihilated by all of them is zero — Kreyszig's (Kreyszig (1978), Cor. 4.3-4, p. 223) Corollary 4.3-4, a consequence of the Hahn-Banach theorem, or in a Hilbert space the one-line argument of taking $f = (\,\cdot\,, x-y)$ and reading off $\lVert x-y\rVert^2 = 0$ . Hence $x = y$ .

Part 3. For fixed $f$ the sequence $(f(x_n))$ converges in $\mathbb{K}$ , so each of its subsequences converges to the same number. A subsequence of $(x_n)$ produces exactly such a subsequence of numbers, for every $f$ .

Part 4. Define, for each $n$ , a functional on the dual space by

g_n : X' \to \mathbb{K}, \qquad g_n(f) = f(x_n) .

Each $g_n$ is linear in $f$ and bounded, and in fact $\lVert g_n\rVert = \lVert x_n\rVert$ : this is the canonical embedding of $X$ into its bidual, Kreyszig's (Kreyszig (1978), Lemma 4.6-1, p. 240) Lemma 4.6-1, whose proof is again Corollary 4.3-4. So the quantity we want to bound, $\lVert x_n\rVert$ , has been rewritten as an operator norm.

Now fix $f \in X'$ . By hypothesis $(f(x_n))$ converges, hence is a bounded sequence of scalars: there is $c_f < \infty$ , depending on $f$ but not on $n$ , with

|g_n(f)| = |f(x_n)| \le c_f \qquad \text{for all } n .

Thus $\{g_n\}$ is a family of bounded linear functionals on $X'$ that is bounded at each point of $X'$ . The domain $X'$ is a Banach space for any normed $X$ whatsoever, because the scalars are complete (Stakgold and Holst (Stakgold & Holst (2011), Thm 4.8.4, p. 294)), so the uniform boundedness theorem applies and gives $\sup_n\lVert g_n\rVert < \infty$ . Since $\lVert g_n\rVert = \lVert x_n\rVert$ , part 4 is proved — and note where completeness was needed: of $X'$ , never of $X$ .

Part 5. If $x = 0$ the assertion reads $0 \le \liminf_n\lVert x_n\rVert$ and there is nothing to prove, so assume $x \ne 0$ throughout; this is also the hypothesis the norming functional will need below.

Take first a Hilbert space, where the computation is transparent. Testing against the limit itself, weak convergence gives $(x_n,x) \to (x,x) = \lVert x\rVert^2$ . The limit is real and non-negative, so the moduli converge to it as well, $|(x_n,x)| \to \lVert x\rVert^2$ , and each modulus is bounded by Cauchy–Schwarz:

\lVert x\rVert^2 = \lim_{n\to\infty}\bigl|(x_n , x)\bigr| \le \liminf_{n\to\infty}\lVert x_n\rVert\,\lVert x\rVert .

Divide by $\lVert x\rVert > 0$ . In a general normed space, replace "testing against $x$ " by the norming functional of Kreyszig's (Kreyszig (1978), Thm 4.3-3, p. 223) Theorem 4.3-3, which is stated for a non-zero vector and so applies to our $x$ : there is $f \in X'$ with $\lVert f\rVert = 1$ and $f(x) = \lVert x\rVert$ . Then $f(x_n) \to f(x) = \lVert x\rVert$ , a real non-negative limit, so again $|f(x_n)| \to \lVert x\rVert$ , and $|f(x_n)| \le \lVert f\rVert\,\lVert x_n\rVert = \lVert x_n\rVert$ gives

\lVert x \rVert = \lim_{n\to\infty}|f(x_n)| \le \liminf_{n\to\infty}\lVert x_n\rVert .

Evans (Evans (2010), Appendix D.4, p. 678) records the conclusion in this form. $\square$

Proof(The criterion, its specialisations, and finite dimensions)AdaptedAdapted from Kreyszig (1978), Lemma 4.8-7 with its proof, pp. 261–262, Example 4.8-6, p. 261, and Thm 4.8-4(b),(c) with its proof, pp. 259–260; Reed & Simon (1980), §IV.5, Example 1, p. 112

Part 6, the easy direction. If $x_n \rightharpoonup x$ then (A) is part 4 and (B) is the definition restricted to $M$ .

Part 6, the substantial direction. Assume (A) and (B). Put

c = \max\Bigl\{\ \sup_n\lVert x_n\rVert ,\ \lVert x\rVert\ \Bigr\} ,

finite by (A). Let $f \in X'$ and $\varepsilon > 0$ be given.

Approximate the test functional. Since $M$ is total, $\operatorname{span}M$ is dense in $X'$ , so there is $h \in \operatorname{span}M$ with

\lVert f - h\rVert < \frac{\varepsilon}{3(1+c)} .

Use the hypothesis on $h$ . Write $h = \sum_{i=1}^{m}\lambda_i f_i$ with $f_i \in M$ . Each $f_i(x_n) \to f_i(x)$ by (B), and the sum is finite, so $h(x_n) \to h(x)$ . Choose $N$ with $|h(x_n) - h(x)| < \varepsilon/3$ for all $n > N$ .
Split the difference three ways. For $n > N$ ,

|f(x_n) - f(x)| \le \underbrace{|(f-h)(x_n)|}_{\le\,\lVert f-h\rVert\,\lVert x_n\rVert} + \underbrace{|h(x_n) - h(x)|}_{<\,\varepsilon/3} + \underbrace{|(h-f)(x)|}_{\le\,\lVert f-h\rVert\,\lVert x\rVert} .

Collect. The first and third terms are each at most $\lVert f-h\rVert\,c < \varepsilon c/(3(1+c)) < \varepsilon/3$ . Hence $|f(x_n) - f(x)| < \varepsilon$ for $n > N$ .

Since $f \in X'$ and $\varepsilon$ were arbitrary, $x_n \rightharpoonup x$ .

Part 7, the Hilbert case. Let $(e_\alpha)_{\alpha\in A}$ be an orthonormal basis of $H$ , with $A$ any index set, and let $M = \{(\,\cdot\,,e_\alpha) : \alpha \in A\} \subset H'$ . The conjugate-linear isometric bijection $J : H \to H'$ , $(Jz)(u) = (u,z)$ , of the Riesz representation theorem carries $\operatorname{span}\{e_\alpha : \alpha\in A\}$ onto $\operatorname{span}M$ and, being a surjective isometry, carries dense sets to dense sets. The span of an orthonormal basis is dense in $H$ by the characterisation of orthonormal bases; hence $\operatorname{span}M$ is dense in $H'$ , that is $M$ is total, and part 6 applies verbatim. Spans are finite linear combinations throughout, so the argument never counts the index set; separability of $H$ is nowhere used. This is the content of Reed and Simon's (Reed & Simon (1980), §IV.5, Example 1, p. 112) Example 1.

Part 7, the $\ell^p$ case. For $1<p<\infty$ the dual of $\ell^p$ is $\ell^q$ with $p^{-1}+q^{-1}=1$ , and the coordinate sequences $e_j = (\delta_{ji})_i$ form a Schauder basis of $\ell^q$ , so their span is dense; the functional attached to $e_j$ is "take the $j$ -th coordinate". Kreyszig (Kreyszig (1978), Example 4.8-6, p. 261) draws exactly this conclusion from the lemma, and we follow him.

Part 8, the counterexample. Let $(e_n)$ be orthonormal in a Hilbert space $H$ and let $f \in H'$ . By Riesz, $f = (\,\cdot\,,z)$ for some $z$ , and Bessel's inequality gives $\sum_{n\ge1}|(e_n,z)|^2 \le \lVert z\rVert^2 < \infty$ . A convergent series has terms tending to zero, so $f(e_n) = (e_n,z) \to 0 = f(0)$ . As $f$ was arbitrary, $e_n \rightharpoonup 0$ . Strong convergence fails because orthonormality and Pythagoras give $\lVert e_n - e_m\rVert^2 = \lVert e_n\rVert^2 + \lVert e_m\rVert^2 = 2$ for $n \ne m$ , so $(e_n)$ is not Cauchy.

Part 8, the finite-dimensional theorem. Let $\dim X = k$ , let $\{v_1,\dots,v_k\}$ be a basis and write $x_n = \sum_{j} a_j^{(n)}v_j$ , $x = \sum_j a_j v_j$ . Let $f_1,\dots,f_k$ be the dual basis, $f_i(v_j) = \delta_{ij}$ . Each $f_i$ is bounded, and the one line that proves it is worth writing out, since it is the step that fails in infinite dimensions. The coordinate sum $\lVert\sum_j a_jv_j\rVert_1 = \sum_j|a_j|$ is itself a norm on $X$ , and on a finite-dimensional space all norms are equivalent (normed and Banach spaces), so there is a constant $c>0$ with $\sum_j|a_j| \le c\lVert x\rVert$ for every $x = \sum_j a_jv_j$ . Hence $|f_i(x)| = |a_i| \le c\lVert x\rVert$ . So $f_i \in X'$ , and weak convergence gives $a_i^{(n)} = f_i(x_n) \to f_i(x) = a_i$ for each $i$ . Then

\lVert x_n - x\rVert = \Bigl\lVert \sum_{j=1}^{k}\bigl(a_j^{(n)} - a_j\bigr)v_j \Bigr\rVert \le \sum_{j=1}^{k}\bigl|a_j^{(n)} - a_j\bigr|\;\lVert v_j\rVert \longrightarrow 0 ,

a finite sum of null sequences. Hence $x_n \to x$ strongly. $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Kreyszig (1978), §4.8, Lemma 4.8-7 and Problems 6, 7 and 10, pp. 261–263; Reed & Simon (1980), §IV.5, the Proposition and Problems 39 and 40, pp. 112–113; Folland (1999), §5.4, Exercises 47b, 48a and 49a, p. 170, and Exercise 51, p. 171; Evans (2010), Appendix D.4, pp. 678–679

Boundedness, hypothesis (A), cannot be dropped from the criterion of part 6. In $\ell^2$ take $w_n = n\,e_n$ . Every coordinate functional sees $w_n \to 0$ , since the $k$ -th coordinate of $w_n$ vanishes as soon as $n \ne k$ . But with $z = (1,\tfrac12,\tfrac13,\dots) \in \ell^2$ the functional $(\,\cdot\,,z)$ returns $(w_n,z) = n\cdot\frac1n = 1$ for every $n$ , which does not tend to $(0,z)=0$ . So $(w_n)$ converges on a total set and is not weakly convergent — consistent with part 4, since $\lVert w_n\rVert = n$ is unbounded.
Totality of $M$ cannot be dropped either. In $\ell^2$ let $M = \{(\,\cdot\,,e_1)\}$ , whose span is one-dimensional and hence far from dense, and let $x_n = (-1)^n e_2$ . Then $(\lVert x_n\rVert)$ is bounded and $(x_n,e_1) = 0$ converges, yet $(x_n,e_2) = (-1)^n$ has no limit, so there is no weak limit.
Completeness of $X$ is nowhere used. Part 4 needs the uniform boundedness theorem applied on the domain $X'$ , and $X'$ is complete even when $X$ is not. Weak convergence is therefore a notion for normed spaces, not only Banach spaces; what completeness (indeed reflexivity) buys is the compactness of part 9, not the basic properties.
Part 4 is a statement about sequences, not about nets. Reed and Simon (Reed & Simon (1980), §IV.5, p. 112) emphasise this and set the construction of a weakly convergent unbounded net as an exercise. The reason is visible in the proof: a net of scalars that converges is bounded eventually, but "eventually" is not a uniform statement over the index set in the way the uniform boundedness theorem needs.
The norm is genuinely only semicontinuous. Part 5 gives $\lVert x\rVert \le \liminf\lVert x_n\rVert$ and no more: for an orthonormal sequence the left side is $0$ and the right side is $1$ . The quantity $\lVert x_n\rVert^2 - \lVert x\rVert^2$ that survives in the limit measures how much "energy" escapes to infinity. In a Hilbert space there is an exact accounting: if $x_n \rightharpoonup x$ then

\lVert x_n - x\rVert^2 = \lVert x_n\rVert^2 - 2\,\operatorname{Re}(x_n , x) + \lVert x\rVert^2 \longrightarrow \Bigl(\lim_n \lVert x_n\rVert^2\Bigr) - \lVert x\rVert^2

whenever the limit on the right exists, because $(x_n,x) \to (x,x) = \lVert x\rVert^2$ by weak convergence. Two consequences: weak convergence together with $\lVert x_n\rVert \to \lVert x\rVert$ forces strong convergence, and that is the only way to recover it. The computation is ours; we have not taken it from a cited source.

Weak limits are not pointwise limits. Evans (Evans (2010), Appendix D.4, p. 679) warns that $f_k \rightharpoonup f$ in $L^p$ says nothing about $f_k(x)$ at any point, even along a subsequence: the functions may oscillate ever faster, and the weak limit records only their local averages. Part D of the example below shows this concretely.
Weakly closed is strictly stronger than closed — except for convex sets. Folland (Folland (1999), Exercise 48a, p. 170) has the closed unit ball weakly closed, and Folland (Folland (1999), Exercise 51, p. 171) has a subspace norm closed exactly when it is weakly closed. The unit sphere is not weakly closed: in infinite dimensions its weak closure is the whole ball, which is recorded by Reed and Simon (Reed & Simon (1980), §IV.5, p. 113). The flavour of that collapse is also in Folland's (Folland (1999), Exercise 49a, p. 170) Exercise 49a, which says that once $X$ is a Banach space of infinite dimension, no nonempty weakly open subset of it is bounded in norm — so no weak neighbourhood can be squeezed inside a shell around the sphere. The general principle is Mazur's theorem, quoted by Evans (Evans (2010), Appendix D.4, p. 678): a convex norm-closed set is weakly closed. Convexity is exactly the hypothesis that fails for the sphere.
Weak Cauchy does not imply weakly convergent in general. Call $(x_n)$ weak Cauchy if $(f(x_n))$ is a Cauchy sequence of scalars for every $f \in X'$ , and call $X$ weakly complete if every weak Cauchy sequence in $X$ converges weakly in $X$ . Kreyszig (Kreyszig (1978), §4.8, Problem 10, p. 263) sets as a problem one implication only: if $X$ is reflexive then $X$ is weakly complete. Nothing here says the converse, and part 9 must not be read backwards into it — the Eberlein–Shmulian characterisation quoted there is an equivalence between reflexivity and weak sequential compactness of bounded sets, which is a different property from weak completeness. Reflexivity is a sufficient condition for weak completeness, not an announced necessary one.

Method(Deciding whether a sequence converges weakly, and to what)Original

Check boundedness first. If $\lVert x_n\rVert \to \infty$ along any subsequence, stop: by part 4 the sequence is not weakly convergent. This test is cheap and kills most candidates.
Guess the limit coordinatewise. Fix an orthonormal basis (or, in $\ell^p$ , the coordinates) and compute $\lim_n (x_n , e_k)$ for each fixed $k$ . Call the resulting numbers $c_k$ .
Check that the guess lies in the space. The candidate limit is $x = \sum_k c_k e_k$ , which exists in $H$ only if $\sum_k |c_k|^2 < \infty$ . (Steps 1 and 2 guarantee this, by truncation: for each fixed $K$ , $\sum_{k\le K}|c_k|^2 = \lim_n\sum_{k\le K}|(x_n,e_k)|^2 \le \liminf_n\lVert x_n\rVert^2$ , the last step being Bessel's inequality applied to each $x_n$ ; now let $K \to \infty$ on the left.) If the coordinate limits are not square-summable, the sequence was not bounded and step 1 should have caught it.
Invoke the criterion. Boundedness from step 1 plus coordinatewise convergence from step 2 give $x_n \rightharpoonup x$ by part 7. Do not try to verify the definition against a general $f$ ; that is what the criterion is for.
Then ask whether the convergence is strong. Only $\liminf_n\lVert x_n\rVert$ is guaranteed to be meaningful here, by part 5, so start with it: if $\liminf_n\lVert x_n\rVert > \lVert x\rVert$ the convergence is certainly not strong. Otherwise examine $\lVert x_n\rVert$ itself. If $\lVert x_n\rVert \to \lVert x\rVert$ , the convergence is strong, by the identity in the remark; if $\lVert x_n\rVert$ converges to something larger, the convergence is weak and not strong and the difference $\lim_n\lVert x_n\rVert^2 - \lVert x\rVert^2$ is exactly $\lim_n\lVert x_n - x\rVert^2$ . If $\lVert x_n\rVert$ has no limit at all — which part 4 permits, since boundedness is all that is guaranteed — pass to a subsequence along which it does and read the verdict there.
To get strong convergence anyway, apply a compact operator. If $K$ is compact and $x_n \rightharpoonup x$ , then $Kx_n \to Kx$ in norm (see compact operators). In practice this is how weak limits are made useful: extract a weakly convergent subsequence by part 9, then push it through a compact embedding.
For a functional on a dual space, test weak- $*$ instead, which asks only pointwise convergence on $X$ and is therefore easier to verify; the two coincide if the underlying space is reflexive.

Example(Four sequences in

\ell^2

and one in

L^2(0,1)

)Original

Throughout, $(e_k)_{k\ge1}$ is the standard orthonormal basis of $\ell^2$ and $(x,y) = \sum_k x_k\overline{y_k}$ .

Part A: weak but not strong, with the energy gap measured. For $n \ge 1$ set

u_n = \bigl(1 + 2^{-n}\bigr)e_1 + 3\,e_{n+1} .

Step 1 (boundedness). By Pythagoras, $\lVert u_n\rVert^2 = (1+2^{-n})^2 + 9$ . This increases as $n$ decreases, so the largest value is at $n=1$ : $\lVert u_1\rVert^2 = (1.5)^2+9 = 11.25$ , and $\lVert u_n\rVert \le \sqrt{11.25} = 3.354102$ for all $n$ . Bounded.

Step 2 (coordinates). The first coordinate is $1+2^{-n} \to 1$ . For fixed $k \ge 2$ , the $k$ -th coordinate of $u_n$ equals $3$ when $n = k-1$ and $0$ for every other $n$ , so it tends to $0$ . The coordinate limits are $(1,0,0,\dots)$ .

Step 3 (the limit is in the space and the criterion applies). The candidate is $u = e_1 \in \ell^2$ . Steps 1 and 2 are hypotheses (A) and (B) of part 6 with the total set $M = \{(\,\cdot\,,e_k)\}$ , so

u_n \rightharpoonup e_1 .

Step 4 (a concrete functional, checked numerically). Take $z = (1,\tfrac12,\tfrac13,\dots)$ , so $\lVert z\rVert^2 = \pi^2/6$ and $z \in \ell^2$ . Then

(u_n , z) = \bigl(1+2^{-n}\bigr)\cdot 1 + 3\cdot\frac{1}{n+1} .

At $n = 4$ this is $1.0625 + 0.6 = 1.6625$ ; at $n = 100$ it is $1.0000000 + 0.0297030 = 1.0297030$ ; and the limit is $1 = (e_1 , z)$ , as it must be.

Step 5 (not strong, and the gap). $\lVert u_n\rVert^2 \to 1 + 9 = 10$ , so $\lVert u_n\rVert \to \sqrt{10} = 3.162278$ , whereas $\lVert u\rVert = 1$ . The inequality of part 5 holds strictly: $1 < 3.162278$ . The identity in the remark predicts $\lVert u_n - u\rVert^2 \to 10 - 1 = 9$ , and directly

\lVert u_n - e_1\rVert^2 = 4^{-n} + 9 \longrightarrow 9 , \qquad \lVert u_n - e_1\rVert \to 3 .

The sequence converges weakly to $e_1$ and stays a distance $3$ away from it.

Part B: weak plus convergent norms gives strong. Set $v_n = e_1 + n^{-1}e_{n+1}$ . The same three steps give $v_n \rightharpoonup e_1$ . Here $\lVert v_n\rVert^2 = 1 + n^{-2} \to 1 = \lVert e_1\rVert^2$ , and indeed $\lVert v_n - e_1\rVert = 1/n \to 0$ : the convergence is strong, exactly as step 5 of the method predicts.

Part C: boundedness is not optional. Set $w_n = n\,e_n$ . Each coordinate tends to $0$ , so hypothesis (B) holds with $x = 0$ for the total set of coordinate functionals. But $(w_n , z) = n\cdot(1/n) = 1$ for the same $z$ as in Part A, so $(w_n,z) \not\to 0$ and $(w_n)$ does not converge weakly. The failure is of hypothesis (A): $\lVert w_n\rVert = n$ .

Part D: oscillation in $L^2(0,1)$ , and why weak limits do not commute with squaring. Let $f_n(t) = \sqrt2\,\sin(n\pi t)$ on $(0,1)$ , which is orthonormal there, with $\lVert f_n\rVert = 1$ . By part 8, $f_n \rightharpoonup 0$ . Tested against $g(t) = t$ :

(f_n , g) = \sqrt2\int_0^1 t\,\sin(n\pi t)\,dt = \sqrt2\left[\frac{-t\cos(n\pi t)}{n\pi}\right]_0^1 + \frac{\sqrt2}{n\pi}\int_0^1\cos(n\pi t)\,dt = \frac{\sqrt2\,(-1)^{n+1}}{n\pi} ,

the last integral vanishing for every $n\ge1$ . At $n = 5$ this is $\sqrt2/(5\pi) = 0.0900316$ , and at $n = 50$ it is $-0.0090032$ : the inner product decays like $1/n$ while $\lVert f_n\rVert$ stays at $1$ . Now square: $f_n^2 = 2\sin^2(n\pi t) = 1 - \cos(2n\pi t)$ , and the functions $c_n(t) = \sqrt2\cos(2n\pi t)$ are themselves orthonormal in $L^2(0,1)$ , since $\int_0^1 2\cos^2(2n\pi t)\,dt = 1$ and $\int_0^1 2\cos(2n\pi t)\cos(2m\pi t)\,dt = 0$ for $n\ne m$ ; so part 8 applied to $(c_n)$ gives $c_n \rightharpoonup 0$ , hence $\cos(2n\pi\,\cdot\,) \rightharpoonup 0$ and

f_n^2 \rightharpoonup 1 \quad\text{while}\quad \bigl(\text{weak-}\lim f_n\bigr)^2 = 0 .

The weak limit of the square is $1$ , not $0$ . Weak convergence is compatible with linear operations and with nothing else.

ApplicationOriginal

The direct method in the calculus of variations. This is the reason weak convergence was invented, and Evans (Evans (2010), §8.2, pp. 439–442) lays it out in three moves. To minimise $I[w] = \int_U L(Dw,w,x)\,dx$ over an admissible class, take a minimising sequence $(u_k)$ . A coercivity bound $L \ge \alpha|p|^q$ makes $(u_k)$ bounded in the Sobolev space $W^{1,q}(U)$ but gives no strongly convergent subsequence, because the space is infinite-dimensional. Part 9 supplies a weakly convergent subsequence, $u_{k_j} \rightharpoonup u$ . The functional $I$ is then typically not weakly continuous — and Evans (Evans (2010), §8.2, p. 439) hedges that word deliberately, saying it holds "in essentially all cases of interest" rather than always. Part D of the example shows the mechanism in miniature, a quadratic whose weak limit is not the quadratic of the weak limit. The hedge is needed, because nonlinearity by itself is not the obstruction: on $L^2(0,1)$ the nonlinear functional $u \mapsto \bigl(\int_0^1 u\,dt\bigr)^2$ is weakly continuous, being a continuous function of the single weakly continuous reading $u \mapsto \int_0^1 u\,dt$ . What fails is a functional that integrates a nonlinear expression pointwise, since there the oscillation is not averaged away before the nonlinearity acts. Convexity of $L$ in the gradient nevertheless makes $I$ weakly lower semicontinuous, $I[u]\le\liminf I[u_{k_j}]$ , and that one-sided inequality is all a minimisation needs. Evans's (Evans (2010), §8.2, p. 442) remark puts the mechanism precisely: the gradient enters the convexity inequality linearly, and weak convergence respects linear expressions. See functionals and the first variation and weak solutions and Sobolev spaces.

Existence theory for elliptic equations, and the finite element method. The same pattern underlies the Lax-Milgram theorem and the convergence proofs for finite elements and the Rayleigh-Ritz method: a Galerkin sequence is bounded by an energy estimate, hence has a weak limit, and the weak limit is shown to solve the weak formulation because the formulation is linear in the test function. Note the contrast Stakgold and Holst (Stakgold & Holst (2011), §10.7, p. 812) draw: once the problem is discretised the spaces are finite-dimensional, where by part 8 weak and strong convergence coincide, so the practising numerical analyst measures only norms — the weak topology does its work in the existence proof, not in the computation.

Compactness as the bridge back to strong convergence. A compact operator turns weak convergence into strong: if $K$ is compact and $u_k \rightharpoonup u$ then $Ku_k \to Ku$ , which Evans (Evans (2010), Appendix D.5, p. 679) uses to prove that adjoints of compact operators are compact and to run the Fredholm alternative. In partial differential equations the compact object is usually a Sobolev embedding, and the two-step pattern "extract weakly, upgrade by compactness" is the standard existence argument. See compact operators and the spectral theorem for compact self-adjoint operators.

Homogenization, turbulence and effective media. Part D is a physical statement in disguise, but the bookkeeping has to be done with care, because the energy density $a|e|^2$ is quadratic in the field $e$ and only linear in the conductivity $a$ ; what refuses to pass to the weak limit is their product. Take the one-dimensional laminate. On $(0,1)$ let $a_n$ alternate between the values $1$ and $9$ on $2n$ equal intervals, and let $u_n$ solve the steady conduction problem $-(a_nu_n')'=0$ with $u_n(0)=0$ , $u_n(1)=1$ . The flux $j_n = a_nu_n'$ is then a constant, fixed by $\int_0^1u_n' = 1$ to be

j_n = \Bigl(\int_0^1 a_n^{-1}\Bigr)^{-1} = \frac{2}{1 + \tfrac19} = 1.8 \qquad\text{for every } n ,

the harmonic mean of $1$ and $9$ . Each of the three fields below is a function taking two values alternately on the same $2n$ intervals, so one lemma settles all of them. If $b_n$ takes two values alternately on the $2n$ equal intervals and $\bar b$ is its mean, then $b_n \rightharpoonup \bar b$ in $L^2(0,1)$ . Indeed the primitive $B_n(x) = \int_0^x(b_n - \bar b)$ vanishes at every multiple of the period $1/n$ , in particular at $x=1$ , and satisfies $\lVert B_n\rVert_\infty \le \delta/(2n)$ with $\delta$ the gap between the two values; so for $g \in C^1[0,1]$ an integration by parts gives $\bigl|\int_0^1(b_n-\bar b)g\bigr| = \bigl|\int_0^1 B_ng'\bigr| \le \delta\lVert g'\rVert_{L^1}/(2n) \to 0$ . The functions $b_n$ are bounded in $L^2(0,1)$ and $C^1[0,1]$ is dense in $L^2(0,1) \cong L^2(0,1)'$ , hence total, so part 6 applies. Now read off three weak limits on this one sequence:

the conductivity $a_n \rightharpoonup \bar a = 5$ , the arithmetic mean;
the field $e_n = u_n' = j_n/a_n$ , which takes the values $1.8$ and $0.2$ , satisfies $e_n \rightharpoonup \bar e = 1$ ;
the energy density $a_ne_n^2 = j_ne_n$ , which takes the values $3.24$ and $0.36$ , satisfies $a_ne_n^2 \rightharpoonup 1.8$ .

So the limiting energy density is $1.8$ and not $\bar a\,\bar e^{\,2} = 5$ : it is strictly smaller than the energy the average field would carry in a conductor of the average conductivity. Equivalently the effective conductivity is $a^\ast = 1.8$ , the harmonic mean of the constituents and not their arithmetic mean, which is the extreme case of the Reuss and Voigt bounds $a_{\mathrm{harm}} \le a^\ast \le a_{\mathrm{arith}}$ ; a laminate measured across its layers attains the lower one. The mechanism is not the semicontinuity of part 5, which compares a vector with its weak limit in a fixed norm, whereas here the quadratic form oscillates along with the field. It is the failure Part D exhibits, in the form "the weak limit of a product is not the product of the weak limits": $a_n \rightharpoonup 5$ and $e_n \rightharpoonup 1$ , yet $a_ne_n \equiv 1.8$ , so $a_ne_n \rightharpoonup 1.8 \ne 5$ . That this particular product has a limit at all is no accident — the flux is constant — and it is the elementary case of the compensated-compactness phenomenon that makes homogenization work. The same failure is the closure problem for the Reynolds-averaged equations of turbulence: the average of $u^2$ is not the square of the average, and the Reynolds stress is the difference.

Quantum mechanics: probability that escapes, and probability that only seems to. Let $(\psi_n)$ be normalised states in $L^2(\mathbb{R})$ with $\psi_n \rightharpoonup 0$ . The semicontinuity gap of part 5 is then $\lim\lVert\psi_n\rVert^2 - \lVert 0\rVert^2 = 1$ : none of the probability survives into the limit. It is tempting to read that $1$ as probability that has run off to infinity, so that every fixed observation region eventually sees nothing. Weak convergence does not license that reading, and the reason is exactly the one Part D of the example makes.

What weak nullity says. Every fixed linear reading dies: $(\psi_n,\chi)\to0$ for each fixed state $\chi$ , so the overlap with any fixed state tends to $0$ and no fixed state is ever approached.

What it does not say. What a fixed observation region $B$ sees is $\int_B\lvert\psi_n\rvert^2 = \lVert\chi_B\psi_n\rVert^2$ , which is quadratic in $\psi_n$ , and the whole content of Part D is that quadratic readings need not follow the weak limit. Part D is in fact a counterexample in disguise. The functions $f_n(t)=\sqrt2\,\sin(n\pi t)$ are the normalised energy eigenstates of a particle in an infinite square well on $(0,1)$ ; they converge weakly to $0$ with $\lVert f_n\rVert = 1$ , and yet

\int_0^1 \lvert f_n(t)\rvert^2\,dt = 1 \qquad\text{for every } n .

Extended by zero these are states of $L^2(\mathbb{R})$ and are weakly null there too, since the restriction to $(0,1)$ of a test function in $L^2(\mathbb{R})$ is a test function in $L^2(0,1)$ . The whole probability stays inside the box forever; no region sees less of the particle as $n$ grows, and nothing escapes in position at all. What runs off to infinity here is the momentum, the energy being $E_n \propto n^2$ . So weak nullity leaves three mechanisms open — the packet spreads, the packet translates away, or the packet merely oscillates faster and faster — and only the first two move probability out of a fixed region.

The statement that is strictly stronger. "Every fixed observation region eventually sees nothing" is $\lVert\chi_B\psi_n\rVert\to0$ for every bounded $B$ , that is, $\psi_n \to 0$ in $L^2_{\mathrm{loc}}$ . Weak convergence cannot give it: if it did, multiplication by $\chi_B$ would turn every weakly convergent sequence into a strongly convergent one and so would be a compact operator (compact operators), which it is not — it is an orthogonal projection with infinite-dimensional range, acting as the identity on that range.

When the escape picture is right. Assume $\psi_n \to 0$ in $L^2_{\mathrm{loc}}$ and the reading becomes correct: the probability really does leave every laboratory, and the gap of $1$ is exactly what left. A travelling packet does this. Fix $\varphi$ normalised and put $\psi_n(x) = \varphi(x-n)$ . For $g$ continuous with support in $[-R,R]$ , Cauchy-Schwarz gives $\lvert(\psi_n,g)\rvert \le \lVert g\rVert\bigl(\int_{-R-n}^{R-n}\lvert\varphi\rvert^2\bigr)^{1/2} \to 0$ , since the tail of a fixed square-integrable function past distance $n$ vanishes; such $g$ are total in $L^2(\mathbb{R})$ and $\lVert\psi_n\rVert=1$ , so part 6 gives $\psi_n \rightharpoonup 0$ . And for bounded $B \subseteq [-R,R]$ , $\int_B\lvert\psi_n\rvert^2 = \int_{B-n}\lvert\varphi\rvert^2 \to 0$ by the same tail estimate. Scattering theory is built on separating this behaviour from the bound states, where the weak limit is attained and the norm is not lost; see the mathematical framework of quantum mechanics.

Statistics and signal processing. Weak convergence in $L^2$ is convergence of all second-order statistics against fixed test functions, which is what an instrument with a fixed response actually measures. A signal that oscillates faster than any filter can follow converges weakly to its mean while retaining its variance — the reason a low-pass measurement cannot distinguish a smooth field from a rapidly fluctuating one of the same average, and the reason variance must be tracked separately. See linear filters and stationary processes.