Eigenvalues and matrix decompositions

Spectral theory of matrices and its uses.

In this chapter 10

Tick a unit to add it to your PDF basket.

Eigenvalues and diagonalization
Quadratic forms, SVD and matrix functions

Eigenvalues and diagonalization

Definition

Eigenvalues and eigenvectors

Adapted

Add to PDF

Defines eigenvalues, eigenvectors, eigenspaces and the characteristic polynomial, distinguishes algebraic from geometric multiplicity, and gives the diagonalizability criterion together with trace and determinant as the sum and product of the eigenvalues.

Prerequisites: Determinants, Vector spaces and bases

Definition(Eigenvalue, eigenvector, eigenspace)AdaptedAdapted from Hassani (2013), §6.2, Defs 6.2.1 and 6.2.3, Prop 6.2.2, pp. 172-173; Riley, Hobson & Bence (2006), §8.13, Eqs (8.67)-(8.68), pp. 272-273, and §8.13.1 (the word 'degenerate'), p. 275

Let $A$ be an $n \times n$ matrix with complex entries, $n \ge 1$ , equivalently a linear operator on an $n$ -dimensional complex vector space with a basis fixed. A number $\lambda \in \mathbb{C}$ is an eigenvalue of $A$ if

A x = \lambda x \quad \text{for some } x \in \mathbb{C}^n,\ x \neq 0,

and any such $x$ is an eigenvector of $A$ for $\lambda$ . The zero vector is excluded; without that exclusion every $\lambda$ would qualify. Any non-zero scalar multiple of an eigenvector for $\lambda$ is again an eigenvector for $\lambda$ , so a single eigenvector never comes with a preferred length; for that reason eigenvectors are usually normalised to $x^\dagger x = 1$ . (When $\lambda$ has more than one independent eigenvector, as happens below, normalisation leaves far more freedom than a scalar: the choice is then a choice of basis in the eigenspace.)

The set

E_\lambda = \ker(A - \lambda I) = \{x \in \mathbb{C}^n : A x = \lambda x\}

is a subspace of $\mathbb{C}^n$ , called the eigenspace of $\lambda$ ; its non-zero vectors are exactly the eigenvectors for $\lambda$ . Its dimension $g(\lambda) = \dim E_\lambda \ge 1$ is the geometric multiplicity of $\lambda$ , and $\lambda$ is simple when $g(\lambda) = 1$ and degenerate when $g(\lambda) \ge 2$ . The set of all eigenvalues is the spectrum $\sigma(A)$ .

Definition(Characteristic polynomial and algebraic multiplicity)AdaptedAdapted from Hassani (2013), §6.2, Eqs (6.7)-(6.8) and Def 6.2.9, pp. 173-174; Riley, Hobson & Bence (2006), §8.14, Eqs (8.85)-(8.86), p. 280; Arfken, Weber & Harris (2013), §6.5, 'Defective Matrices', p. 324

$A x = \lambda x$ has a non-zero solution exactly when $A - \lambda I$ fails to be invertible. Since that happens precisely when its determinant vanishes, $\sigma(A)$ is the zero set of the characteristic polynomial

\chi_A(\lambda) = \det(A - \lambda I),

a polynomial of degree exactly $n$ in $\lambda$ with leading term $(-1)^n \lambda^n$ . The equation $\chi_A(\lambda) = 0$ is the characteristic (or secular) equation. Over $\mathbb{C}$ , if $\lambda_1, \dots, \lambda_p$ are the distinct roots then

\chi_A(\lambda) = \prod_{j=1}^{p} (\lambda_j - \lambda)^{m_j}, \qquad \sum_{j=1}^{p} m_j = n,

and the exponent $m_j = m(\lambda_j) \ge 1$ is the algebraic multiplicity of $\lambda_j$ .

$A$ is diagonalizable if $\mathbb{C}^n$ has a basis consisting of eigenvectors of $A$ ; collecting such a basis $v_1,\dots,v_n$ as the columns of an invertible $S$ gives $S^{-1} A S = \operatorname{diag}(\mu_1, \dots, \mu_n)$ , where $\mu_k$ is the eigenvalue of $v_k$ . The $\mu_k$ are the eigenvalues listed with repetitions, each $\lambda_j$ occurring $g(\lambda_j)$ times; they are not the distinct roots $\lambda_1,\dots,\lambda_p$ above unless $p = n$ . A matrix that is not diagonalizable is called defective.

Definition(Spectrum, trace and determinant)AdaptedAdapted from Hassani (2013), Thm 6.2.5, Props 6.2.4 and 6.2.6, Eq. (6.9), pp. 173-174; Riley, Hobson & Bence (2006), Eq. (8.87), p. 280; Arfken, Weber & Harris (2013), §6.4, p. 313, which states part 3 for a Hermitian matrix

Let $A$ be an $n \times n$ complex matrix, $n \ge 1$ , with distinct eigenvalues $\lambda_1, \dots, \lambda_p$ of algebraic multiplicities $m_1, \dots, m_p$ . Then:

$A$ has at least one eigenvalue, and $\sum_j m_j = n$ .
Eigenvectors belonging to distinct eigenvalues are linearly independent, and $E_{\lambda_i} \cap E_{\lambda_j} = \{0\}$ for $i \neq j$ .
$\displaystyle \det A = \prod_{j=1}^{p} \lambda_j^{m_j}$ and $\displaystyle \operatorname{tr} A = \sum_{j=1}^{p} m_j \lambda_j$ : the determinant is the product and the trace the sum of the $n$ eigenvalues counted with algebraic multiplicity.
$A$ is invertible if and only if $0 \notin \sigma(A)$ .

Statement 1 uses that a non-constant polynomial over $\mathbb{C}$ has a root; over $\mathbb{R}$ it can fail. Part 3 is asserted for every complex $A$ , which is the generality of Hassani's Eq. (6.9) for the determinant and of Riley, Hobson and Bence (Riley, Hobson & Bence (2006), Eq. (8.87), p. 280) for the trace; Arfken's §6.4 gives the pair only for a Hermitian matrix, as invariants of a unitary transformation.

Idea of the proofOriginal

The characteristic polynomial carries three of the four claims. Its degree is exactly $n$ because, in the sum over permutations that defines the determinant, only the identity permutation keeps all $n$ diagonal entries and so reaches that degree; the fundamental theorem of algebra then hands over a root, and a root is an eigenvalue because a vanishing determinant means a non-trivial kernel. The determinant and the trace are two of the polynomial's coefficients, each computable in two ways — from the matrix entries, and from the factorization into roots — and equating the two readings is exactly the assertion that they are the product and the sum of the eigenvalues. Independence of eigenvectors for distinct eigenvalues is a separate and shorter argument: applying $A-\lambda_p I$ to a relation annihilates its last term and leaves a shorter relation.

ProofAdaptedAdapted from Hassani (2013), §6.2, Thm 6.2.5, Prop 6.2.4, Prop 6.2.6 and eqs. (6.7)-(6.9), pp. 173-174; Riley, Hobson & Bence (2006), §8.14, the coefficient of lambda^(N-1) and Eq. (8.87), p. 280

Part 1. By the permutation formula for the determinant (Determinants),

\chi_A(\lambda) = \det(A-\lambda I) = \sum_{\sigma} \operatorname{sgn}(\sigma)\prod_{i=1}^{n}(A-\lambda I)_{i\sigma(i)} .

Each factor is a polynomial in $\lambda$ of degree at most $1$ , and a factor has degree $1$ only when $\sigma(i) = i$ . A permutation other than the identity moves at least two indices, so its term has degree at most $n-2$ ; the identity contributes $\prod_{i}(A_{ii}-\lambda)$ , of degree $n$ with leading term $(-1)^n\lambda^n$ . Hence $\chi_A$ has degree exactly $n$ with leading coefficient $(-1)^n$ , and is non-constant because $n\ge1$ . The fundamental theorem of algebra gives a root $\lambda_1\in\mathbb{C}$ ; then $\det(A-\lambda_1I) = 0$ , so $A-\lambda_1I$ is not invertible and its kernel contains some $x\neq0$ with $Ax = \lambda_1x$ . So $A$ has an eigenvalue. Its eigenvalues are exactly the roots of $\chi_A$ , by the same equivalence read in both directions. Factoring $\chi_A$ completely over $\mathbb{C}$ and collecting equal roots gives $\chi_A(\lambda) = c\prod_{j}(\lambda_j-\lambda)^{m_j}$ ; comparing degrees gives $\sum_jm_j = n$ , and comparing leading coefficients gives $c(-1)^n = (-1)^n$ , so $c=1$ .

Part 2. Let $x_1,\dots,x_p$ be eigenvectors for the distinct eigenvalues $\lambda_1,\dots,\lambda_p$ and suppose $\sum_{j=1}^{p}c_jx_j = 0$ . We argue by induction on $p$ . For $p=1$ , $c_1x_1 = 0$ with $x_1\neq0$ gives $c_1 = 0$ . For $p\ge2$ , apply $A-\lambda_pI$ to the relation; since $(A-\lambda_pI)x_j = (\lambda_j-\lambda_p)x_j$ and the $x_p$ term dies,

\sum_{j=1}^{p-1}c_j(\lambda_j-\lambda_p)\,x_j = 0 .

By the inductive hypothesis $c_j(\lambda_j-\lambda_p) = 0$ for $j<p$ , and $\lambda_j\neq\lambda_p$ , so $c_j = 0$ for $j<p$ . The original relation then reads $c_px_p = 0$ , giving $c_p = 0$ as well. For the intersection: if $v\in E_{\lambda_i}\cap E_{\lambda_j}$ with $i\neq j$ then $\lambda_iv = Av = \lambda_jv$ , so $(\lambda_i-\lambda_j)v = 0$ and $v = 0$ .

Part 3. For the determinant, evaluate both expressions for $\chi_A$ at $\lambda = 0$ : the definition gives $\chi_A(0) = \det A$ , and the factorization gives $\chi_A(0) = \prod_j\lambda_j^{m_j}$ .

For the trace, compare the coefficients of $\lambda^{n-1}$ . On the matrix side, only the identity permutation contributes in that degree, as observed in part 1, and

\prod_{i=1}^{n}(A_{ii}-\lambda) = (-1)^n\lambda^n+(-1)^{n-1}\Bigl(\sum_{i=1}^{n}A_{ii}\Bigr)\lambda^{n-1}+\cdots .

On the factorized side, $\chi_A(\lambda) = \prod_j(\lambda_j-\lambda)^{m_j} = (-1)^n\prod_j(\lambda-\lambda_j)^{m_j}$ , a product of $n$ linear factors $\lambda-\lambda_j$ in which $\lambda_j$ appears $m_j$ times; expanding it, the coefficient of $\lambda^{n-1}$ is minus the sum of those $n$ roots, so the coefficient of $\lambda^{n-1}$ in $\chi_A$ is $(-1)^{n}\bigl(-\sum_jm_j\lambda_j\bigr) = (-1)^{n-1}\sum_jm_j\lambda_j$ . Equating the two gives $\operatorname{tr}A = \sum_jm_j\lambda_j$ .

Part 4. By part 3, $\det A = \prod_j\lambda_j^{m_j}$ , which is non-zero exactly when no $\lambda_j$ equals $0$ ; and $A$ is invertible exactly when $\det A\neq0$ (Determinants). $\square$

Definition(The two multiplicities and diagonalizability)Original

With the notation above:

$1 \le g(\lambda_j) \le m(\lambda_j)$ for every $j$ .
$A$ is diagonalizable if and only if $g(\lambda_j) = m(\lambda_j)$ for every $j$ , equivalently if and only if $\sum_j g(\lambda_j) = n$ .
If $A$ has $n$ distinct eigenvalues, then every $m_j = 1$ , so $A$ is diagonalizable. This is sufficient but not necessary: the identity matrix has one eigenvalue and is diagonal.
Similar matrices have the same characteristic polynomial, hence the same eigenvalues with the same algebraic multiplicities, the same trace and the same determinant; corresponding eigenspaces have the same dimension. If $B = S^{-1} A S$ and $A x = \lambda x$ , then $B (S^{-1} x) = \lambda (S^{-1} x)$ .

Idea of the proofOriginal

Take part 4 first, since the rest leans on it: a similarity leaves $\det(A-\lambda I)$ alone, because the two copies of $S$ cancel inside the determinant. Then the bound $g\le m$ becomes visible by choosing coordinates well: put a basis of the eigenspace first, and in those coordinates the matrix begins with $g$ columns that are $\lambda$ times the first $g$ standard basis vectors, which forces $g$ factors of $(\lambda-\mu)$ out of the characteristic polynomial. The diagonalizability criterion is then a counting argument: the eigenspaces are independent of one another, so the only way to accumulate $n$ independent eigenvectors is for each eigenspace to be as large as its algebraic multiplicity allows.

ProofOriginal

Part 4 comes first, because the other parts use it. Let $B = S^{-1}AS$ with $S$ invertible. Then $B-\lambda I = S^{-1}(A-\lambda I)S$ , so similarity invariance of the determinant (Determinants) gives

\chi_B(\lambda) = \det\bigl(S^{-1}(A-\lambda I)S\bigr) = \det(A-\lambda I) = \chi_A(\lambda)

for every $\lambda$ . Equal characteristic polynomials have the same roots with the same exponents, which is the same spectrum with the same algebraic multiplicities; the trace and determinant then agree by part 3 of the previous statement. For the eigenspaces, the map $x\mapsto S^{-1}x$ is linear and injective, and it carries $E_\lambda(A)$ into $E_\lambda(B)$ : if $Ax = \lambda x$ then

B(S^{-1}x) = S^{-1}AS\,S^{-1}x = S^{-1}Ax = \lambda\,(S^{-1}x) .

The same computation with $A$ and $B$ exchanged, and $S^{-1}$ in place of $S$ , shows $y\mapsto Sy$ carries $E_\lambda(B)$ into $E_\lambda(A)$ . The two maps are mutually inverse, so $E_\lambda(A)$ and $E_\lambda(B)$ are isomorphic and have the same dimension.

Part 1. $g(\lambda_j)\ge1$ because an eigenspace contains an eigenvector, which is non-zero by definition.

For the upper bound, write $\lambda = \lambda_j$ and $g = g(\lambda)$ , choose a basis $v_1,\dots,v_g$ of $E_\lambda$ , and extend it to a basis $v_1,\dots,v_n$ of $\mathbb{C}^n$ (Vector spaces and bases). Let $S$ be the invertible matrix with these columns, so $Se_i = v_i$ , and put $B = S^{-1}AS$ . For $i\le g$ the $i$ th column of $B$ is

Be_i = S^{-1}Av_i = S^{-1}(\lambda v_i) = \lambda e_i ,

so the first $g$ columns of $B-\mu I$ are $(\lambda-\mu)e_1,\dots,(\lambda-\mu)e_g$ . Expand $\det(B-\mu I)$ along its first column by cofactors (Determinants): only the $(1,1)$ entry is non-zero there, so $\det(B-\mu I) = (\lambda-\mu)\det M$ , where $M$ is $B-\mu I$ with its first row and column deleted. That deletion turns the old columns $2,\dots,g$ into $(\lambda-\mu)e_1,\dots,(\lambda-\mu)e_{g-1}$ , so $M$ has the same shape with $g-1$ in place of $g$ . Iterating the expansion $g$ times,

\chi_B(\mu) = (\lambda-\mu)^g\,q(\mu)

for some polynomial $q$ . By part 4, $\chi_A = \chi_B$ , so $(\lambda-\mu)^g$ divides $\chi_A(\mu)$ , and therefore the exponent $m(\lambda)$ in the factorization of $\chi_A$ is at least $g$ .

Part 2. Suppose $A$ is diagonalizable and fix a basis of $\mathbb{C}^n$ of eigenvectors. Sort its members by eigenvalue, say $k_j$ of them lie in $E_{\lambda_j}$ ; then $\sum_jk_j = n$ , and those $k_j$ members are linearly independent elements of $E_{\lambda_j}$ , so $k_j\le g(\lambda_j)$ . With part 1 and $\sum_jm_j = n$ from the previous statement,

n = \sum_j k_j \ \le\ \sum_j g(\lambda_j) \ \le\ \sum_j m(\lambda_j) = n ,

so both inequalities are equalities. Since $g(\lambda_j)\le m(\lambda_j)$ term by term, equality of the sums forces $g(\lambda_j) = m(\lambda_j)$ for every $j$ .

Conversely, suppose $\sum_jg(\lambda_j) = n$ ; by part 1 and $\sum_jm_j = n$ this is the same condition as $g(\lambda_j) = m(\lambda_j)$ for all $j$ . Choose a basis of each $E_{\lambda_j}$ and concatenate them into a list of $n$ vectors. The list is linearly independent: group a vanishing linear combination by eigenvalue as $\sum_ju_j = 0$ with $u_j\in E_{\lambda_j}$ . If some $u_j$ were non-zero, the non-zero ones would be eigenvectors for distinct eigenvalues obeying a non-trivial relation, which part 2 of the previous statement forbids. So every $u_j = 0$ , and then every coefficient vanishes because each chosen basis is independent. A list of $n$ independent vectors in $\mathbb{C}^n$ is a basis, and this one consists of eigenvectors, so $A$ is diagonalizable.

Part 3. If $A$ has $n$ distinct eigenvalues then $p = n$ , and $m_j\ge1$ with $\sum_{j=1}^{n}m_j = n$ forces $m_j = 1$ for each $j$ . By part 1, $1\le g(\lambda_j)\le m_j = 1$ , so the two multiplicities agree and part 2 makes $A$ diagonalizable. The converse fails: $I_n$ has the single eigenvalue $1$ with $m = g = n$ , and is already diagonal. $\square$

NoteOriginal

An eigenvector is a direction that $A$ only stretches, by the factor $\lambda$ . If there are enough such directions to make a basis, $A$ acts as $n$ independent one-dimensional stretches and every question about $A$ — powers, inverses, exponentials, long-time behaviour of $\dot x = Ax$ — becomes $n$ scalar questions. The characteristic polynomial is how you find the stretch factors; the gap between algebraic and geometric multiplicity is what can stop the basis from existing.

Remark(Where the hypotheses matter)AdaptedAdapted from Hassani (2013), §6.2, the rotation R_z(theta), p. 172, and Example 6.2.8, p. 174; Arfken, Weber & Harris (2013), §6.5, 'Defective Matrices', p. 324

The field matters. A rotation of the plane through an angle that is not a multiple of $\pi$ moves every non-zero vector off its own line, as Hassani (Hassani (2013), §6.2, p. 172) observes of $R_z(\theta)$ ; concretely, $\begin{pmatrix} 0 & 1 \\ -1 & 0 \end{pmatrix}$ has $\chi(\lambda) = \lambda^2 + 1$ and so no real eigenvalue and no real eigenvector, although as a complex matrix it has the eigenvalues $\pm i$ . All statements above are for $\mathbb{C}$ ; a real matrix is handled by regarding it as complex, and its non-real eigenvalues then come in conjugate pairs.

Algebraic multiplicity does not bound geometric multiplicity from below. For $\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ the only eigenvalue is $\lambda = 1$ with $m = 2$ , but $(A - I)x = 0$ forces the second component of $x$ to vanish, so $g = 1$ . The matrix is defective: it has no eigenbasis, and the shortfall $m - g$ is what the Jordan form of a defective matrix records.

Conventions. Many texts use $\det(\lambda I - A)$ , which is monic, instead of $\det(A - \lambda I)$ ; the two differ by $(-1)^n$ and have the same roots. "Degenerate" (physics) and "repeated" or "multiple" (mathematics) both signal that an eigenvalue occurs more than once, but they count different things: "repeated" refers to algebraic multiplicity, $m(\lambda)\ge2$ , and "degenerate" to geometric multiplicity, $g(\lambda)\ge2$ . On a defective matrix the two conditions come apart, so the words are not interchangeable there; the second example below is repeated but not degenerate. The spectrum is sometimes defined as a set and sometimes as a list with multiplicities; we take it as a set.

Example(A diagonalizable 3 by 3 matrix, worked through)Original

Let

A = \begin{pmatrix}4&1&-1\\2&5&-2\\1&1&2\end{pmatrix} .

The characteristic polynomial. Expanding $\det(A-\lambda I)$ along the first row,

\chi_A(\lambda) = (4-\lambda)\bigl[(5-\lambda)(2-\lambda)+2\bigr]-\bigl[2(2-\lambda)+2\bigr]-\bigl[2-(5-\lambda)\bigr] .

Since $(5-\lambda)(2-\lambda)+2 = \lambda^2-7\lambda+12$ , the three brackets give $-\lambda^3+11\lambda^2-40\lambda+48$ , then $2\lambda-6$ , then $3-\lambda$ , so

\chi_A(\lambda) = -\lambda^3+11\lambda^2-39\lambda+45 = -(\lambda-5)(\lambda-3)^2 .

The factorization is checked by expanding: $(\lambda-5)(\lambda^2-6\lambda+9) = \lambda^3-11\lambda^2+39\lambda-45$ . So $\sigma(A) = \{5,3\}$ with $m(5) = 1$ and $m(3) = 2$ .

First check. $\operatorname{tr}A = 4+5+2 = 11$ and $5+3+3 = 11$ ; $\det A = \chi_A(0) = 45$ and $5\cdot3\cdot3 = 45$ . Computing $\det A$ from the matrix independently, $4(10+2)-1(4+2)-1(2-5) = 48-6+3 = 45$ .

The eigenspaces. For $\lambda = 5$ ,

A-5I = \begin{pmatrix}-1&1&-1\\2&0&-2\\1&1&-3\end{pmatrix} ,

whose second row gives $x_1 = x_3$ and whose first row then gives $x_2 = x_1+x_3 = 2x_1$ ; the third row is satisfied automatically, since $x_1+2x_1-3x_1 = 0$ . So $E_5 = \operatorname{span}\{(1,2,1)\}$ and $g(5) = 1 = m(5)$ .

For $\lambda = 3$ ,

A-3I = \begin{pmatrix}1&1&-1\\2&2&-2\\1&1&-1\end{pmatrix} ,

all of whose rows are multiples of $(1,1,-1)$ , so the single condition is $x_1+x_2-x_3 = 0$ . Hence $E_3$ is a plane, $g(3) = 2 = m(3)$ , with basis $(-1,1,0)$ and $(1,0,1)$ .

Diagonalization. Since $\sum_jg(\lambda_j) = 1+2 = 3$ , part 2 of the second statement says $A$ is diagonalizable, and the three eigenvectors found are a basis. Collecting them as columns,

S = \begin{pmatrix}1&-1&1\\2&1&0\\1&0&1\end{pmatrix}, \qquad \det S = 1(1)-(-1)(2)+1(-1) = 2\neq0 ,

and $S^{-1}AS = \operatorname{diag}(5,3,3)$ . Rather than invert $S$ , verify the equivalent form $AS = S\operatorname{diag}(5,3,3)$ column by column:

A\begin{pmatrix}1\\2\\1\end{pmatrix} = \begin{pmatrix}5\\10\\5\end{pmatrix}, \qquad A\begin{pmatrix}-1\\1\\0\end{pmatrix} = \begin{pmatrix}-3\\3\\0\end{pmatrix}, \qquad A\begin{pmatrix}1\\0\\1\end{pmatrix} = \begin{pmatrix}3\\0\\3\end{pmatrix} ,

which are $5$ , $3$ and $3$ times the respective columns.

Example(One entry away: a defective matrix)Original

Change the $(2,2)$ entry of the previous matrix from $5$ to $6$ :

B = \begin{pmatrix}4&1&-1\\2&6&-2\\1&1&2\end{pmatrix} .

The same expansion along the first row changes in two places: the first bracket becomes $(6-\lambda)(2-\lambda)+2 = \lambda^2-8\lambda+14$ , and the third cofactor $\det\begin{pmatrix}2&6-\lambda\\1&1\end{pmatrix} = \lambda-4$ contributes $4-\lambda$ where it contributed $3-\lambda$ before. The middle bracket, $2(2-\lambda)+2$ , is unchanged. Hence

\chi_B(\lambda) = -\lambda^3+12\lambda^2-45\lambda+54 = -(\lambda-6)(\lambda-3)^2 ,

so $m(6) = 1$ and $m(3) = 2$ , and the two checks hold again: $\operatorname{tr}B = 12 = 6+3+3$ and $\det B = 54 = 6\cdot9$ . The spectrum looks much like before. The eigenspaces do not.

For $\lambda = 6$ , the rows of $B-6I = \begin{pmatrix}-2&1&-1\\2&0&-2\\1&1&-4\end{pmatrix}$ give $x_1 = x_3$ from the second and then $x_2 = 2x_1+x_3 = 3x_1$ from the first, so $E_6 = \operatorname{span}\{(1,3,1)\}$ and $g(6) = 1$ .

For $\lambda = 3$ ,

B-3I = \begin{pmatrix}1&1&-1\\2&3&-2\\1&1&-1\end{pmatrix} .

Rows one and three are identical and give $x_1+x_2-x_3 = 0$ ; subtracting twice that from the second row leaves $x_2 = 0$ , and then $x_1 = x_3$ . So $E_3 = \operatorname{span}\{(1,0,1)\}$ is only a line: $g(3) = 1 < 2 = m(3)$ .

Hence $\sum_jg(\lambda_j) = 1+1 = 2 < 3$ , and by part 2 of the second statement $B$ has no eigenbasis: it is defective. The two multiplicities differ by exactly one, which is the amount of Jordan structure $B$ carries (Jordan normal form). Note how little the spectrum revealed: the trace and determinant tests passed, the eigenvalues were real and simple to find, and nothing but computing $\dim E_3$ exposed the defect.

ApplicationOriginal

Decoupling a linear system. The reason to want an eigenbasis is that it reduces one $n$ -dimensional problem to $n$ one-dimensional ones. For $\dot x = Ax$ with $A$ diagonalizable, substituting $x = Sy$ turns the system into $\dot y = \operatorname{diag}(\lambda_k)y$ , whose solutions are $y_k(t) = y_k(0)e^{\lambda_kt}$ ; the general solution is then a superposition $x(t) = \sum_kc_ke^{\lambda_kt}v_k$ over the eigenvectors. Every qualitative question becomes a question about where the $\lambda_k$ sit: the equilibrium is stable when all $\operatorname{Re}\lambda_k<0$ , the slowest decay rate is $\max_k\operatorname{Re}\lambda_k$ , and a complex conjugate pair $\alpha\pm i\beta$ is an oscillation of frequency $\beta$ with envelope $e^{\alpha t}$ . The same substitution in $x_{t+1} = Ax_t$ turns a discrete model — a population projection matrix, a Markov chain, a numerical iteration — into independent geometric sequences, and the eigenvalue of largest modulus governs the long-run behaviour.

Where the defect shows up physically. The gap $m-g$ of the second example is not a curiosity. A damped oscillator $\ddot u+2\gamma\dot u+\omega_0^2u = 0$ written as a first-order system has a defective coefficient matrix exactly at critical damping $\gamma = \omega_0$ , where the two roots coincide but only one eigenvector survives; the solution there is $(c_1+c_2t)e^{-\gamma t}$ , and the factor of $t$ is what stands in for the eigenvector that is missing. The same thing produces secular growth in perturbation theory at resonance and in orbital mechanics: a repeated root with a shortfall in eigenvectors always contributes polynomial-times-exponential behaviour, which is why engineers avoid designing near such a coincidence.

Observables and degeneracy. In quantum mechanics the measurable values of a quantity are the eigenvalues of its operator and the states with a definite value are the eigenvectors, so the spectrum is literally the set of possible outcomes of a measurement. The geometric multiplicity is what physics calls the degree of degeneracy — the $2\ell+1$ states of a given angular momentum, the two spin states of an electron — and it is almost always forced by a symmetry of the Hamiltonian. Splitting a degeneracy by an applied field (the Zeeman and Stark effects) is the statement that a perturbation has reduced some $g(\lambda)$ by breaking the symmetry that protected it. For such operators the eigenbasis is guaranteed and can even be chosen orthonormal; that is the subject of the spectral theorem for Hermitian matrices.

Theorem

Spectral theorem for Hermitian matrices

Adapted

Add to PDF

A Hermitian matrix has real eigenvalues, orthogonal eigenvectors for distinct eigenvalues, and an orthonormal eigenbasis, so some unitary matrix of eigenvector columns diagonalizes it; a real symmetric matrix can be diagonalized by a real orthogonal matrix.

Prerequisites: Eigenvalues and eigenvectors, Orthonormal bases and Gram-Schmidt, Inner products and norms, Determinants

Definition(Hermitian matrix)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.12.5, p. 271; Arfken, Weber & Harris (2013), §6.3, p. 310

For a complex $m \times n$ matrix $A$ , the Hermitian conjugate (adjoint) $A^\dagger$ is the conjugate transpose, $(A^\dagger)_{ij} = \overline{A_{ji}}$ . On $\mathbb{C}^n$ we use the inner product

(x, y) = \sum_{i=1}^{n} x_i \overline{y_i} = y^\dagger x ,

which is linear in the first slot and conjugate-linear in the second, with $\|x\|^2 = (x, x)$ .

A square matrix $A$ is Hermitian if $A = A^\dagger$ , equivalently if $(A x, y) = (x, A y)$ for all $x, y \in \mathbb{C}^n$ . For a matrix with real entries this is the condition $A = A^{\mathsf T}$ , i.e. that $A$ is symmetric. A matrix $U$ is unitary if $U^\dagger U = I$ , and a real matrix $Q$ is orthogonal if $Q^{\mathsf T} Q = I$ ; see unitary and orthogonal matrices.

Theorem(Spectral theorem)AdaptedAdapted from Arfken, Weber & Harris (2013), §6.3, pp. 310-311 and §6.4, pp. 311-313; Riley, Hobson & Bence (2006), §8.13.1-§8.13.2, pp. 274-277; Hassani (2013), Cor 6.4.9, p. 181 and Cor 6.4.13, p. 182

Let $A$ be an $n \times n$ complex matrix with $A = A^\dagger$ . Then:

Reality. Every eigenvalue of $A$ is real.
Orthogonality. If $A x = \lambda x$ , $A y = \mu y$ and $\lambda \neq \mu$ , then $(x, y) = 0$ .
Orthonormal eigenbasis. There is an orthonormal basis $u_1, \dots, u_n$ of $\mathbb{C}^n$ with $A u_k = \lambda_k u_k$ , the $\lambda_k$ real and listed with repetitions. Within a degenerate eigenspace the basis is not forced on you: any linearly independent set spanning the eigenspace can be orthonormalised by the Gram-Schmidt process, and the result is still an eigenbasis of that eigenspace.
Unitary diagonalization. Equivalently, the matrix $U = (u_1 \ \cdots \ u_n)$ whose columns are such a basis is unitary and

U^\dagger A U = \Lambda = \operatorname{diag}(\lambda_1, \dots, \lambda_n), \qquad \text{so} \qquad A = U \Lambda U^\dagger .

Spectral resolution. Writing this out columnwise, $\displaystyle A = \sum_{k=1}^{n} \lambda_k\, u_k u_k^\dagger$ , and $\operatorname{tr} A = \sum_k \lambda_k$ , $\det A = \prod_k \lambda_k$ .
Real symmetric case. If in addition $A$ is real, then $A = A^{\mathsf T}$ , the eigenvalues are real, and the $u_k$ may be chosen real, so that $S = (u_1 \ \cdots \ u_n)$ is a real orthogonal matrix with $S^{\mathsf T} A S = \Lambda$ .

Idea of the proofOriginal

Parts 1 and 2 are one manoeuvre done twice: move $A$ from one side of the inner product to the other, which is free because $A = A^\dagger$ , and compare the two readings of the same number. A conjugate appears on one side and not on the other, and that mismatch is what makes the eigenvalue real and makes eigenvectors belonging to different eigenvalues perpendicular. Part 3 is where the work is. It is an induction that peels off one direction at a time: over $\mathbb{C}$ there is always at least one eigenvalue, and the hyperplane perpendicular to its eigenvector is again carried into itself by $A$ — this is the third use of $A = A^\dagger$ — so the same argument runs in one dimension lower. Parts 4 and 5 then only restate part 3: an orthonormal basis written as the columns of a matrix is a unitary matrix, and $A = U\Lambda U^\dagger$ read entry by entry is the spectral resolution.

ProofAdaptedAdapted from Arfken, Weber & Harris (2013), §6.3, Eqs. (6.17)-(6.21), pp. 310-311, and §6.4, Eqs. (6.22)-(6.24), pp. 311-312; Riley, Hobson & Bence (2006), §8.13.2, Eqs. (8.81)-(8.84), pp. 276-277; Hassani (2013), §6.4, Thm 6.4.3, p. 177, Cor 6.4.9, p. 181, and Cor 6.4.13, p. 182

Throughout we use hermiticity in the form $(Ax,y) = (x,Ay)$ recorded in the definition, and conjugate-linearity of $(\cdot,\cdot)$ in its second slot.

Part 1. Let $A x = \lambda x$ with $x\neq0$ . Then

\lambda\,(x,x) = (\lambda x, x) = (Ax,x) = (x,Ax) = (x,\lambda x) = \overline{\lambda}\,(x,x) ,

the second-to-last step using $A = A^\dagger$ and the last conjugate-linearity in the second slot. Since $x\neq0$ gives $(x,x) = \|x\|^2>0$ , we may cancel it and conclude $\lambda = \overline\lambda$ , that is $\lambda\in\mathbb{R}$ .

Part 2. Let $Ax = \lambda x$ , $Ay = \mu y$ with $\lambda\neq\mu$ ; both are real by part 1. Then

\lambda\,(x,y) = (Ax,y) = (x,Ay) = (x,\mu y) = \overline{\mu}\,(x,y) = \mu\,(x,y) ,

where the last equality uses that $\mu$ is real. Hence $(\lambda-\mu)(x,y) = 0$ , and $\lambda\neq\mu$ forces $(x,y) = 0$ .

Part 3. We prove, by induction on $d$ , the following statement: if $W$ is a complex inner product space with $\dim_{\mathbb{C}}W = d\ge1$ and $T\colon W\to W$ is linear with $(Tx,y) = (x,Ty)$ for all $x,y\in W$ , then $W$ has an orthonormal basis of eigenvectors of $T$ , and the corresponding eigenvalues are real. Taking $W = \mathbb{C}^n$ and $T = A$ gives part 3.

For $d=1$ , any unit vector $u$ of $W$ spans it, so $Tu$ is a multiple of $u$ and $(u)$ is such a basis; the eigenvalue is real by the computation of part 1, which used nothing but $(Tx,y) = (x,Ty)$ .

Let $d\ge2$ and assume the statement in dimension $d-1$ . Fixing any basis of $W$ represents $T$ by a $d\times d$ complex matrix, which has an eigenvalue by part 1 of Eigenvalues and eigenvectors; so there are $\lambda_1\in\mathbb{C}$ and $u_1\in W$ with $u_1\neq0$ and $Tu_1 = \lambda_1u_1$ , and we may scale $u_1$ to $\|u_1\| = 1$ . By part 1, $\lambda_1$ is real. Put

W_1 = \{w\in W : (w,u_1) = 0\} .

$W_1$ has dimension $d-1$ . Extend $(u_1)$ to an orthonormal basis $(u_1,f_2,\dots,f_d)$ of $W$ , which is possible by part 3 of the Gram-Schmidt statement in Orthonormal bases and Gram-Schmidt. By the expansion statement there, $w = (w,u_1)u_1+\sum_{i\ge2}(w,f_i)f_i$ for every $w\in W$ , so $(w,u_1) = 0$ holds exactly when $w\in\operatorname{span}\{f_2,\dots,f_d\}$ . Hence $W_1 = \operatorname{span}\{f_2,\dots,f_d\}$ , an inner product space of dimension $d-1$ under the restricted inner product.

$T$ maps $W_1$ into itself. For $w\in W_1$ ,

(Tw,u_1) = (w,Tu_1) = (w,\lambda_1u_1) = \overline{\lambda_1}\,(w,u_1) = 0 ,

the first equality being the hypothesis on $T$ . So $Tw\in W_1$ .

The restriction of $T$ to $W_1$ is linear and still satisfies $(Tx,y) = (x,Ty)$ for $x,y\in W_1$ , so the induction hypothesis supplies an orthonormal basis $u_2,\dots,u_d$ of $W_1$ consisting of eigenvectors of $T$ with real eigenvalues. Each $u_i$ with $i\ge2$ lies in $W_1$ and is therefore orthogonal to $u_1$ , so $(u_1,\dots,u_d)$ is orthonormal; it has $d$ members in a space of dimension $d$ , hence is a basis by part 1 of the expansion statement. This completes the induction.

The freedom inside a degenerate eigenspace. Let $\lambda$ be an eigenvalue of $A$ and let $(w_1,\dots,w_d)$ be any linearly independent list spanning $E_\lambda$ . Gram-Schmidt applied inside the inner product space $E_\lambda$ returns an orthonormal basis of $E_\lambda$ , and each output is a linear combination of $w_1,\dots,w_d$ , hence still an element of the subspace $E_\lambda$ , hence still an eigenvector for $\lambda$ . Doing this in each eigenspace and taking the union gives an orthonormal set, because vectors drawn from eigenspaces of different eigenvalues are orthogonal by part 2 and those drawn from one eigenspace are orthonormal by construction.

It remains to see that this set is a basis of $\mathbb{C}^n$ , that is that $\sum_\lambda\dim E_\lambda = n$ , the sum running over the distinct eigenvalues. One inequality comes from the eigenbasis $u_1,\dots,u_n$ just constructed: sorting its members by eigenvalue, the $k_\lambda$ of them lying in $E_\lambda$ are linearly independent elements of $E_\lambda$ , so $k_\lambda\le\dim E_\lambda$ , and summing over $\lambda$ gives $n = \sum_\lambda k_\lambda\le\sum_\lambda\dim E_\lambda$ . The other comes from the union itself: being orthonormal it is linearly independent, by part 1 of the expansion statement in Orthonormal bases and Gram-Schmidt, and it has $\sum_\lambda\dim E_\lambda$ members in $\mathbb{C}^n$ , so $\sum_\lambda\dim E_\lambda\le n$ . Hence the two are equal to $n$ , and an orthonormal list of $n$ vectors in $\mathbb{C}^n$ is an orthonormal basis.

Part 4. Let $U = (u_1\ \cdots\ u_n)$ for such a basis. Then

(U^\dagger U)_{ij} = \sum_{l=1}^{n}\overline{U_{li}}\,U_{lj} = (u_j,u_i) = \delta_{ij} ,

so $U^\dagger U = I$ : $U$ is unitary and $U^{-1} = U^\dagger$ . Column $j$ of $AU$ is $Au_j = \lambda_ju_j$ , which is also column $j$ of $U\Lambda$ ; so $AU = U\Lambda$ , and multiplying on the left by $U^\dagger$ gives $U^\dagger AU = \Lambda$ , on the right by $U^\dagger$ gives $A = U\Lambda U^\dagger$ .

Part 5. Read $A = U\Lambda U^\dagger$ entry by entry:

A_{ij} = \sum_{k=1}^{n}U_{ik}\lambda_k\overline{U_{jk}} = \sum_{k=1}^{n}\lambda_k\,(u_k)_i\,\overline{(u_k)_j} = \Bigl(\sum_{k=1}^{n}\lambda_k\,u_ku_k^\dagger\Bigr)_{ij} .

For the last two identities, $\Lambda = U^{-1}AU$ is similar to $A$ , so it has the same trace and the same determinant (Eigenvalues and eigenvectors, second statement, part 4); and for a diagonal matrix those are $\sum_k\lambda_k$ and $\prod_k\lambda_k$ read off the diagonal.

Part 6. Let $A$ be real with $A = A^{\mathsf T}$ ; then $A = A^\dagger$ , so every eigenvalue of $A$ is real by part 1. Give $\mathbb{R}^n$ the real inner product $x\cdot y = \sum_ix_iy_i$ , which is symmetric, so that $A = A^{\mathsf T}$ reads $(Aw)\cdot z = w\cdot(Az)$ .

We prove by induction on $d$ : if $W\subseteq\mathbb{R}^n$ is a subspace with $\dim_{\mathbb{R}}W = d\ge1$ and $AW\subseteq W$ , then $W$ has an orthonormal basis of real eigenvectors of $A$ . Part 6 is the case $W = \mathbb{R}^n$ , which satisfies $A\mathbb{R}^n\subseteq\mathbb{R}^n$ because $A$ is real.

One real eigenvector in $W$ . The point of the induction is that the restriction of $A$ to $W$ must be made to look symmetric again, and for that the basis of $W$ has to be orthonormal. So take an orthonormal basis $(f_1,\dots,f_d)$ of $W$ , which exists by part 3 of the Gram-Schmidt statement in Orthonormal bases and Gram-Schmidt applied to the real inner product space $W$ (the statement is proved there over $\mathbb{R}$ as well as over $\mathbb{C}$ ). Let $B\in M_{d\times d}(\mathbb{R})$ be the matrix of $A|_W$ in that basis, so $Af_j = \sum_iB_{ij}f_i$ ; pairing with $f_i$ and using orthonormality gives $B_{ij} = (Af_j)\cdot f_i$ . Then

B_{ji} = (Af_i)\cdot f_j = f_i\cdot(Af_j) = (Af_j)\cdot f_i = B_{ij} ,

the middle equality being $A = A^{\mathsf T}$ and the last the symmetry of the real inner product. So $B$ is real symmetric, hence Hermitian as a complex matrix. Read as a complex matrix it has an eigenvalue (Eigenvalues and eigenvectors, first statement, part 1, which applies because $d\ge1$ ), and that eigenvalue $\lambda$ is real by part 1 above. Therefore $B-\lambda I_d$ is a real matrix with $\det(B-\lambda I_d) = 0$ , so its columns are linearly dependent over $\mathbb{R}$ (Determinants): there is a real $c\neq0$ in $\mathbb{R}^d$ with $(B-\lambda I_d)c = 0$ . Setting $u = \sum_jc_jf_j$ , a non-zero real vector of $W$ ,

Au = \sum_jc_jAf_j = \sum_i\Bigl(\sum_jB_{ij}c_j\Bigr)f_i = \lambda\sum_ic_if_i = \lambda u .

Scale $u$ to $\lVert u\rVert = 1$ . If $d = 1$ then $(u)$ is the required basis and we are done.

Peeling it off. Let $d\ge2$ and put $W' = \{w\in W : w\cdot u = 0\}$ . Expanding any $w\in W$ in an orthonormal basis of $W$ whose first member is $u$ — which Gram-Schmidt supplies, by part 3 of the Gram-Schmidt statement applied inside $W$ — shows $\dim_{\mathbb{R}}W' = d-1$ , exactly as in part 3. And $AW'\subseteq W'$ : for $w\in W'$ we have $Aw\in W$ because $AW\subseteq W$ , and

(Aw)\cdot u = w\cdot(Au) = \lambda\,(w\cdot u) = 0 .

By the induction hypothesis $W'$ has an orthonormal basis $u_2,\dots,u_d$ of real eigenvectors of $A$ ; each lies in $W'$ and so is orthogonal to $u$ , so $(u,u_2,\dots,u_d)$ is an orthonormal list of $d$ vectors in $W$ , hence a basis of $W$ , and all of its members are real eigenvectors of $A$ . This closes the induction.

Applying it to $W = \mathbb{R}^n$ gives a real orthonormal basis $u_1,\dots,u_n$ of $\mathbb{R}^n$ of eigenvectors of $A$ , with real eigenvalues $\lambda_1,\dots,\lambda_n$ . Then $S = (u_1\ \cdots\ u_n)$ satisfies $S^{\mathsf T}S = I$ and $S^{\mathsf T}AS = \Lambda$ by the computation of part 4 with every conjugation removed. $\square$

NoteOriginal

A Hermitian matrix is, after a rotation of the coordinate axes, just a list of real numbers: it stretches $n$ mutually perpendicular directions by $n$ real factors and does nothing else. This is why Hermitian operators model observables (real measured values, orthogonal states) and why a real symmetric matrix always has principal axes. Reach for it whenever a matrix built as a Gram matrix, a Hessian, an energy or an adjacency matrix turns out to equal its own adjoint.

Remark(Where the hypotheses are used, and conventions)Original

Real eigenvalues are not enough. $\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ has the single real eigenvalue $1$ but a one-dimensional eigenspace, so no eigenbasis exists; it is not symmetric. Part 3 is a statement about $A = A^\dagger$ , not about where the spectrum lies.

Diagonalizability is not enough; orthogonality is exactly enough. $\begin{pmatrix} 1 & 0 \\ 0 & 2 \end{pmatrix}$ and $\begin{pmatrix} 1 & 1 \\ 0 & 2 \end{pmatrix}$ have the same eigenvalues, and the second is diagonalizable, but only the first is symmetric and only the first has orthogonal eigenvectors. The converse of parts 3-4 holds in the sharp form: if $U^\dagger A U$ is diagonal with real entries for some unitary $U$ , then $A = U \Lambda U^\dagger$ satisfies $A^\dagger = A$ . Dropping "real entries" gives exactly the normal matrices; see normal matrices.

No Hermitian matrix is defective. Part 3 supplies $n$ independent eigenvectors, so geometric and algebraic multiplicity agree for every eigenvalue of a Hermitian matrix.

Conventions. Some books state part 4 as " $U A U^{-1}$ is diagonal", with the columns of $U^{-1}$ , not of $U$ , being the eigenvectors; that is the same statement with $U$ replaced by $U^\dagger$ . Mathematical texts write $A^*$ for $A^\dagger$ and say "self-adjoint" where physics texts say "Hermitian". Our inner product is linear in the first argument; texts that take it linear in the second obtain $(x, Ay) = (A x, y)$ in the same form, because $A$ is Hermitian, so no statement above changes.

Example(A real symmetric matrix with a degenerate eigenvalue)Original

Let

H = \begin{pmatrix}6&2&-4\\2&3&-2\\-4&-2&6\end{pmatrix} = H^{\mathsf T} .

Eigenvalues. Expanding $\det(H-\lambda I)$ along the first row, with $(3-\lambda)(6-\lambda)-4 = \lambda^2-9\lambda+14$ ,

\det(H-\lambda I) = (6-\lambda)(\lambda^2-9\lambda+14)-2(4-2\lambda)-4(8-4\lambda) = -\lambda^3+15\lambda^2-48\lambda+44 ,

which factors as $-(\lambda-11)(\lambda-2)^2$ . The eigenvalues $11,2,2$ are real, as part 1 requires, and the two checks hold: $\operatorname{tr}H = 6+3+6 = 15 = 11+2+2$ and $\det H = 44 = 11\cdot2\cdot2$ .

The simple eigenvalue. The rows of $H-11I = \begin{pmatrix}-5&2&-4\\2&-8&-2\\-4&-2&-5\end{pmatrix}$ reduce to $x_1 = -x_3$ and $2x_2 = -x_3$ , so $E_{11} = \operatorname{span}\{(2,1,-2)\}$ . Since $\|(2,1,-2)\| = 3$ , take

u_1 = \tfrac13(2,1,-2), \qquad Hu_1 = \tfrac13(22,11,-22) = 11u_1 .

The degenerate eigenvalue. Every row of $H-2I = \begin{pmatrix}4&2&-4\\2&1&-2\\-4&-2&4\end{pmatrix}$ is a multiple of $(2,1,-2)$ , so $E_2$ is the plane $2x_1+x_2-2x_3 = 0$ , of dimension $2$ : here $g = m = 2$ , as part 3 requires, since an eigenbasis of $\mathbb{R}^3$ has to take two of its three members from $E_2$ . A convenient spanning pair is

v_1 = (1,0,1), \qquad v_2 = (0,2,1) ,

and these are not orthogonal: $v_1\cdot v_2 = 1$ . Part 3 says Gram-Schmidt inside $E_2$ fixes that. First $\|v_1\| = \sqrt2$ , so $q_1 = \tfrac{1}{\sqrt2}(1,0,1)$ . Then $v_2\cdot q_1 = 1/\sqrt2$ and

v_2-\tfrac{1}{\sqrt2}q_1 = (0,2,1)-\tfrac12(1,0,1) = \tfrac12(-1,4,1) ,

of norm $\tfrac12\sqrt{18} = \tfrac{3}{\sqrt2}$ , so $q_2 = \tfrac{1}{3\sqrt2}(-1,4,1)$ . Both are still in $E_2$ — check $2(-1)+4-2(1) = 0$ — hence still eigenvectors for $\lambda = 2$ .

Orthogonality across eigenspaces is automatic by part 2, and visibly so: $u_1\cdot q_1 = \frac{2-2}{3\sqrt2} = 0$ and $u_1\cdot q_2 = \frac{-2+4-2}{9\sqrt2} = 0$ .

The diagonalizing matrix. Put

S = \begin{pmatrix} 2/3 & 1/\sqrt2 & -1/(3\sqrt2)\\ 1/3 & 0 & 4/(3\sqrt2)\\ -2/3 & 1/\sqrt2 & 1/(3\sqrt2) \end{pmatrix} .

Its columns are orthonormal: each was normalised to length $1$ above, the two pairings $u_1\cdot q_1$ and $u_1\cdot q_2$ were just computed, and $q_1\cdot q_2 = 0$ holds by the Gram-Schmidt construction of $q_2$ — which one can also read off directly, $q_1\cdot q_2 = \frac{-1+0+1}{6} = 0$ . So $S^{\mathsf T}S = I_3$ and $S^{\mathsf T}HS = \operatorname{diag}(11,2,2)$ , which is part 6.

The spectral resolution, checked. Part 5 reads $H = 11\,u_1u_1^{\mathsf T}+2\,q_1q_1^{\mathsf T}+2\,q_2q_2^{\mathsf T}$ . Because the three projectors sum to $I$ , this is the same as $H = 2I+9\,u_1u_1^{\mathsf T}$ , and

9\,u_1u_1^{\mathsf T} = 9\cdot\tfrac19\begin{pmatrix}4&2&-4\\2&1&-2\\-4&-2&4\end{pmatrix} = H-2I ,

which is exactly the matrix $H-2I$ written out above.

Example(A complex Hermitian matrix of size two)Original

Let

H = \begin{pmatrix}1&1+i\\1-i&0\end{pmatrix} .

It is Hermitian: the diagonal is real and the off-diagonal entries are conjugates. Since $(1+i)(1-i) = 2$ ,

\det(H-\lambda I) = (1-\lambda)(-\lambda)-2 = \lambda^2-\lambda-2 = (\lambda-2)(\lambda+1) ,

so the eigenvalues are $2$ and $-1$ : real, and summing to $\operatorname{tr}H = 1$ with product $\det H = -2$ .

For $\lambda = 2$ , the first row of $H-2I$ gives $-x_1+(1+i)x_2 = 0$ , so $x = (1+i,1)$ spans $E_2$ ; its squared norm is $2+1 = 3$ . For $\lambda = -1$ , the first row of $H+I$ gives $2x_1+(1+i)x_2 = 0$ , hence $x_2 = -\tfrac{2}{1+i}x_1 = -(1-i)x_1$ , so $x = (1,-1+i)$ spans $E_{-1}$ , again of squared norm $3$ . Normalising,

u_1 = \tfrac{1}{\sqrt3}(1+i,\,1), \qquad u_2 = \tfrac{1}{\sqrt3}(1,\,-1+i) .

Part 2 is confirmed directly: $(u_1,u_2) = \tfrac13\bigl[(1+i)\overline{1}+1\cdot\overline{(-1+i)}\bigr] = \tfrac13\bigl[(1+i)+(-1-i)\bigr] = 0$ . Note where the conjugate sits — without it the two vectors would not come out orthogonal.

Spectral resolution. With

u_1u_1^\dagger = \tfrac13\begin{pmatrix}2&1+i\\1-i&1\end{pmatrix}, \qquad u_2u_2^\dagger = \tfrac13\begin{pmatrix}1&-1-i\\-1+i&2\end{pmatrix} ,

part 5 predicts

2u_1u_1^\dagger-u_2u_2^\dagger = \tfrac13\begin{pmatrix}4-1&2+2i+1+i\\2-2i+1-i&2-2\end{pmatrix} = \tfrac13\begin{pmatrix}3&3+3i\\3-3i&0\end{pmatrix} = \begin{pmatrix}1&1+i\\1-i&0\end{pmatrix} = H .

Inside the bracket each slot is $2(u_1u_1^\dagger)_{ij}-(u_2u_2^\dagger)_{ij}$ with the common factor $\tfrac13$ pulled out; in the $(2,1)$ slot that is $2(1-i)-(-1+i) = 2-2i+1-i = 3-3i$ , where the minus sign in front of $u_2u_2^\dagger$ and the minus signs inside it combine, and in the $(1,2)$ slot it is the conjugate $3+3i$ .

Equivalently $U = \tfrac{1}{\sqrt3}\begin{pmatrix}1+i&1\\1&-1+i\end{pmatrix}$ satisfies $U^\dagger U = I_2$ and $U^\dagger HU = \operatorname{diag}(2,-1)$ .

ApplicationOriginal

Observables and measurement. Quantum mechanics represents a measurable quantity by a Hermitian operator, and the theorem is what makes that representation coherent: part 1 says the possible measured values are real, part 2 says states with different values are distinguishable (orthogonal), and part 3 says every state can be expanded in states of definite value, with $\lvert(\psi,u_k)\rvert^2$ the probability of the outcome $\lambda_k$ . Part 5 is the resolution of the identity that textbook derivations insert between operators, and the geometric multiplicity of an eigenvalue is the degeneracy of that energy level. For $2\times2$ problems the matrix of the second example is the generic case: any Hermitian $2\times2$ matrix is $aI+\mathbf{b}\cdot\boldsymbol{\sigma}$ with $a$ and $\mathbf{b}$ real, so its eigenvalues are $a\pm\lvert\mathbf{b}\rvert$ and its eigenvectors are the spin states along $\mathbf{b}$ .

Principal axes. A real symmetric matrix met as a physical tensor acquires, through part 6, a frame in which it is diagonal and which is still a genuine rotation or reflection of space. For the inertia tensor these are the principal axes and the eigenvalues are the principal moments, which is why a rigid body has three special axes about which it can spin without wobble; for the stress tensor they are the principal stresses and the directions of zero shear; for the strain tensor they give the directions of pure extension; for the Hessian of a potential they give the directions of pure curvature, with the signs of the eigenvalues classifying the critical point. In each case the content of part 6 is that the diagonalizing change of coordinates is orthogonal, so lengths and angles — and therefore the physics — are unchanged.

Small oscillations and data. A system near a stable equilibrium has a symmetric stiffness matrix and a symmetric positive-definite mass matrix, and the simultaneous diagonalization of the two is what produces normal modes with definite frequencies, each oscillating independently (Generalized eigenproblems and normal modes). The same theorem underlies principal component analysis: a covariance matrix is symmetric by construction, its orthonormal eigenvectors are the principal directions of a cloud of data, and the eigenvalues are the variances along them, so part 5 decomposes the covariance into uncorrelated contributions ranked by size. Because the eigenvalues are real they can be ordered, which is what makes the Rayleigh quotient a usable bound on them (Quadratic forms and the Rayleigh quotient).

Definition

Unitary and orthogonal matrices

Adapted

Add to PDF

Defines unitary and orthogonal matrices, shows they are exactly the matrices that preserve the inner product and have orthonormal columns, and records that their eigenvalues have modulus one and that the real ones reduce to rotations and reflections.

Prerequisites: Inner products and norms, Eigenvalues and eigenvectors, Determinants

Definition(Unitary and orthogonal matrices)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.12.4 and §8.12.6, pp. 270-272

Let $U$ be an $n \times n$ complex matrix. $U$ is unitary if it is invertible with

U^\dagger = U^{-1}, \qquad \text{equivalently} \qquad U^\dagger U = I ,

and for a square matrix $U^\dagger U = I$ already implies $U U^\dagger = I$ . A real matrix $Q$ with $Q^{\mathsf T} = Q^{-1}$ is orthogonal; an orthogonal matrix is exactly a unitary matrix with real entries. The real orthogonal $n \times n$ matrices form the group $\mathrm{O}(n)$ , those with $\det Q = +1$ the subgroup $\mathrm{SO}(n)$ , and the unitary $n \times n$ matrices the group $\mathrm{U}(n)$ : in each case the identity belongs to the set, and products and inverses stay in it.

As in the spectral theorem we use $(x, y) = y^\dagger x$ on $\mathbb{C}^n$ , with $\|x\|^2 = (x,x)$ .

Definition(Characterisation)Original

For an $n \times n$ complex matrix $U$ the following are equivalent:

$U^\dagger U = I$ ;
the columns of $U$ are orthonormal in $\mathbb{C}^n$ , i.e. $(U e_j, U e_k) = \delta_{jk}$ — and likewise for the rows;
$(U x, U y) = (x, y)$ for all $x, y \in \mathbb{C}^n$ ;
$\|U x\| = \|x\|$ for all $x \in \mathbb{C}^n$ .

The same four statements are equivalent for a real matrix $Q$ with $Q^{\mathsf T} Q = I$ , real vectors and the real inner product $x^{\mathsf T} y$ . That 4 implies 3 is the polarization identity. Over $\mathbb{R}$ one identity suffices,

2 x^{\mathsf T} y = \|x+y\|^2 - \|x\|^2 - \|y\|^2 ,

and over $\mathbb{C}$ the real and imaginary parts are recovered separately,

2\operatorname{Re}(x, y) = \|x+y\|^2 - \|x\|^2 - \|y\|^2 , \qquad 2\operatorname{Im}(x, y) = \|x+iy\|^2 - \|x\|^2 - \|y\|^2 ,

the second being the first applied with $y$ replaced by $iy$ ; each right-hand side is built only from norms, so hypothesis 4 — used at $x$ , at $y$ and at $x+y$ and $x+iy$ — reproduces $(Ux, Uy) = (x,y)$ in full.

Idea of the proofOriginal

The single identity $(Ux, Uy) = y^\dagger U^\dagger U x$ turns the matrix condition $U^\dagger U = I$ into a statement about inner products, and reading it at the basis vectors $x = e_j$ , $y = e_k$ turns it back again. So conditions 1, 2 and 3 are the same equation tested on all vectors, on basis vectors, and as a matrix. Condition 4 looks weaker, because it tests only $x = y$ ; the polarization identities say it is not, because an inner product on $\mathbb{C}^n$ is already determined by the norms of $x$ , $y$ , $x+y$ and $x+iy$ .

ProofOriginal

Throughout, $(x,y) = y^\dagger x$ , which is linear in the first slot and conjugate-linear in the second, since $(x, cy) = (cy)^\dagger x = \overline{c}\,(x,y)$ .

1 implies 3. For any $x, y$ , the associativity of matrix multiplication gives

(U x, U y) = (Uy)^\dagger (Ux) = y^\dagger U^\dagger U x = y^\dagger I x = (x, y),

where hypothesis 1 was used at the third equality.

3 implies 2. Take $x = e_j$ and $y = e_k$ : then $(U e_j, U e_k) = (e_j, e_k) = e_k^\dagger e_j = \delta_{jk}$ , which is condition 2 for the columns, since $U e_j$ is the $j$ -th column of $U$ .

2 implies 1. The $(k,j)$ entry of $U^\dagger U$ is $e_k^\dagger U^\dagger U e_j = (U e_j, U e_k)$ , which is $\delta_{kj}$ by hypothesis 2. Every entry of $U^\dagger U$ therefore agrees with the corresponding entry of $I$ .

The rows. Condition 1 gives $\det(U^\dagger U) = \det I = 1$ , so $\det U \neq 0$ by the multiplicativity of the determinant (Determinants) and $U$ is invertible with $U^{-1} = U^\dagger$ ; substituting that identification into $U U^{-1} = I$ gives $U U^\dagger = I$ . Its $(j,k)$ entry is $\sum_m U_{jm}\overline{U_{km}}$ , which is exactly the inner product of the $j$ -th row with the $k$ -th row, so the rows are orthonormal as well. Conversely, orthonormal rows give $U U^\dagger = I$ and hence, by the same argument with $U$ and $U^\dagger$ exchanged, $U^\dagger U = I$ . This is the step that uses squareness: for a non-square matrix $U^\dagger U = I$ and $U U^\dagger = I$ are not equivalent, and only one of them can hold.

3 implies 4. Put $y = x$ : then $\|Ux\|^2 = (Ux, Ux) = (x,x) = \|x\|^2$ , and both norms are non-negative, so $\|Ux\| = \|x\|$ .

4 implies 3. Fix $x, y$ . Hypothesis 4 applied at the four vectors $x$ , $y$ , $x+y$ and $x+iy$ gives

\|Ux\| = \|x\|, \quad \|Uy\| = \|y\|, \quad \|U(x+y)\| = \|x+y\|, \quad \|U(x+iy)\| = \|x+iy\| ,

and $U$ is linear, so $U(x+y) = Ux + Uy$ and $U(x + iy) = Ux + i\,Uy$ . Substituting these four equalities into the two polarization identities displayed in the statement — once for the pair $(Ux, Uy)$ and once for the pair $(x,y)$ — makes their right-hand sides identical, so $\operatorname{Re}(Ux, Uy) = \operatorname{Re}(x,y)$ and $\operatorname{Im}(Ux,Uy) = \operatorname{Im}(x,y)$ . A complex number is fixed by its real and imaginary parts, so $(Ux, Uy) = (x,y)$ .

The four conditions are therefore equivalent: 1, 2 and 3 are equivalent by the first three paragraphs, and 3 and 4 by the last two.

The real case. Replace $\dagger$ by $\mathsf{T}$ and $\mathbb{C}^n$ by $\mathbb{R}^n$ throughout. Every step above is unchanged except the last, where only the first polarization identity is available and is all that is needed, because a real inner product has no imaginary part to recover. $\square$

Definition(Determinant and spectrum)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.12.4, p. 270, §8.12.6, p. 271 and §8.13.3, p. 278

Let $U$ be unitary and $Q$ real orthogonal, both $n \times n$ . Then:

$|\det U| = 1$ , and $\det Q = \pm 1$ .
Every eigenvalue $\lambda$ of $U$ satisfies $|\lambda| = 1$ , so $\lambda = e^{i\theta}$ for some real $\theta$ . In particular the only possible real eigenvalues are $+1$ and $-1$ .
$U^{-1} = U^\dagger$ is unitary, and $Q^{-1} = Q^{\mathsf T}$ is orthogonal.

Part 1 does not characterise unitarity: $\begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ has determinant $1$ and is not orthogonal.

Idea of the proofOriginal

Take determinants of $U^\dagger U = I$ and the modulus of $\det U$ falls out, because conjugating a matrix conjugates its determinant. The eigenvalues are controlled by the characterisation rather than by any new computation: an eigenvector is stretched by $|\lambda|$ and not stretched at all, so $|\lambda| = 1$ . The third part is the observation that the defining relation is symmetric in $U$ and $U^\dagger$ .

ProofAdaptedAdapted from Riley, Hobson & Bence (2006), §8.12.4, p. 270, §8.12.6, p. 271 and §8.13.3, p. 278

Part 1. Taking determinants in $U^\dagger U = I$ and using multiplicativity (Determinants),

1 = \det I = \det(U^\dagger)\,\det U .

Now $U^\dagger = \overline{U}^{\mathsf T}$ , and the determinant is unchanged by transposition and conjugated by entrywise conjugation, so $\det(U^\dagger) = \overline{\det U}$ . Hence $1 = \overline{\det U}\,\det U = |\det U|^2$ , that is $|\det U| = 1$ . For a real orthogonal $Q$ the same computation reads $1 = \det(Q^{\mathsf T})\det Q = (\det Q)^2$ , and $\det Q$ is real, so $\det Q = \pm 1$ .

Part 2. Let $U x = \lambda x$ with $x \neq 0$ . Part 4 of the characterisation, applied to this $x$ , gives $\|U x\| = \|x\|$ ; on the other hand $\|U x\| = \|\lambda x\| = |\lambda|\,\|x\|$ by homogeneity of the norm. Since $x \neq 0$ we may divide by $\|x\| > 0$ to get $|\lambda| = 1$ , and every complex number of modulus one is $e^{i\theta}$ with $\theta$ real. If moreover $\lambda$ is real then $\lambda = \pm 1$ , those being the only real numbers of modulus one. (Riley, Hobson and Bence reach the same conclusion by sandwiching: $x^\dagger x = x^\dagger U^\dagger U x = \overline{\lambda}\lambda\, x^\dagger x$ , so $|\lambda|^2 = 1$ .)

Part 3. By part 1, $\det U \neq 0$ , so $U$ is invertible, and $U^\dagger U = I$ identifies $U^{-1} = U^\dagger$ ; substituting that into $UU^{-1} = I$ gives $U U^\dagger = I$ . Writing $V = U^\dagger$ , this says $V^\dagger V = U U^\dagger = I$ , so $V$ is unitary. The real case is the same computation with transposes. $\square$

Definition(Rotations and reflections)AdaptedAdapted from Arfken, Weber & Harris (2013), §3.3, Eqs (3.23)-(3.28) and 'Reflections', pp. 134-136; Hassani (2013), Thm 6.6.12 and Eq. (6.26), p. 201

Plane rotations. The rotation of $\mathbb{R}^2$ about the origin by an angle $\theta$ has matrix

R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}, \qquad \det R(\theta) = +1 .

Every $2 \times 2$ real orthogonal matrix is either $R(\theta)$ , with determinant $+1$ , or

F(\theta) = \begin{pmatrix} \cos\theta & \sin\theta \\ \sin\theta & -\cos\theta \end{pmatrix}, \qquad \det F(\theta) = -1 ,

for some $\theta \in [0, 2\pi)$ . $F(\theta)$ fixes the line through the origin at angle $\theta/2$ and negates the perpendicular direction, so it is the reflection in that line. 3. Real canonical form. For any real orthogonal $Q$ there is a real orthogonal $P$ with

P^{\mathsf T} Q P = \operatorname{diag}\bigl(\underbrace{1, \dots, 1}_{N_+},\ \underbrace{-1, \dots, -1}_{N_-},\ R(\theta_1), \dots, R(\theta_m)\bigr),

a block-diagonal matrix with $N_+ + N_- + 2m = n$ . A real orthogonal matrix therefore cannot in general be diagonalized over $\mathbb{R}$ ; this canonical form is as close as one can get.

Idea of the proofOriginal

In two dimensions orthogonality is almost no freedom at all. The first column is a unit vector of the plane, so it is $(\cos\theta, \sin\theta)$ for one angle $\theta$ ; the second must be a unit vector on the line perpendicular to it, and such a line contains exactly two unit vectors. One choice gives $R(\theta)$ and the other $F(\theta)$ , which is the whole of part 2. That $F(\theta)$ is a reflection is then read off from the angle-addition formulas, which show at once that it fixes the direction at angle $\theta/2$ and reverses the direction perpendicular to it.

Proof(Parts 1 and 2)AdaptedAdapted from Arfken, Weber & Harris (2013), §3.3, Eqs (3.23)-(3.28), pp. 134-135; Hassani (2013), §6.6, Eq. (6.26), p. 201

Part 1. Write a point of $\mathbb{R}^2$ in polar form as $(r\cos\varphi, r\sin\varphi)$ . By definition the rotation by $\theta$ about the origin leaves $r$ alone and sends $\varphi$ to $\varphi + \theta$ , so it sends this point to $(r\cos(\varphi+\theta),\, r\sin(\varphi+\theta))$ . Expanding with the angle-addition formulas,

r\cos(\varphi+\theta) = x\cos\theta - y\sin\theta , \qquad r\sin(\varphi+\theta) = x\sin\theta + y\cos\theta ,

where $x = r\cos\varphi$ , $y = r\sin\varphi$ . Both are linear in $(x,y)$ , and reading off the coefficients gives exactly $R(\theta)\,(x,y)^{\mathsf T}$ . Its columns $(\cos\theta, \sin\theta)$ and $(-\sin\theta, \cos\theta)$ each have norm $\cos^2\theta + \sin^2\theta = 1$ and inner product $-\cos\theta\sin\theta + \sin\theta\cos\theta = 0$ , so $R(\theta)$ is orthogonal by part 2 of the characterisation, and $\det R(\theta) = \cos^2\theta + \sin^2\theta = +1$ .

Part 2, the two forms. Let $Q$ be a $2 \times 2$ real orthogonal matrix with columns $c_1, c_2$ . By the characterisation these are orthonormal. Since $\|c_1\| = 1$ , there is exactly one $\theta \in [0, 2\pi)$ with $c_1 = (\cos\theta, \sin\theta)^{\mathsf T}$ : this is the polar form of a unit vector of the plane. The vectors orthogonal to $c_1$ in $\mathbb{R}^2$ form the solution set of the single non-trivial linear equation $c_1^{\mathsf T} v = 0$ , hence a line, and $(-\sin\theta, \cos\theta)^{\mathsf T}$ is a unit vector on it; a line through the origin contains exactly two unit vectors, so

c_2 = \pm(-\sin\theta, \cos\theta)^{\mathsf T} .

With the upper sign $Q = R(\theta)$ . With the lower sign $c_2 = (\sin\theta, -\cos\theta)^{\mathsf T}$ and $Q = F(\theta)$ , whose determinant is $\cos\theta\,(-\cos\theta) - \sin\theta\,\sin\theta = -1$ . These are the only possibilities, so every $2\times 2$ real orthogonal matrix is one or the other, and the determinant says which.

Part 2, why $F(\theta)$ is a reflection. Set $v = (\cos(\theta/2), \sin(\theta/2))^{\mathsf T}$ , the unit vector at angle $\theta/2$ , and $w = (-\sin(\theta/2), \cos(\theta/2))^{\mathsf T}$ , the unit vector perpendicular to it. Multiplying out and using the angle-subtraction formulas with $\theta - \theta/2 = \theta/2$ ,

F(\theta)\,v = \begin{pmatrix} \cos\theta\cos\tfrac\theta2 + \sin\theta\sin\tfrac\theta2 \\[2pt] \sin\theta\cos\tfrac\theta2 - \cos\theta\sin\tfrac\theta2 \end{pmatrix} = \begin{pmatrix} \cos\tfrac\theta2 \\[2pt] \sin\tfrac\theta2\end{pmatrix} = v ,

F(\theta)\,w = \begin{pmatrix} -\cos\theta\sin\tfrac\theta2 + \sin\theta\cos\tfrac\theta2 \\[2pt] -\sin\theta\sin\tfrac\theta2 - \cos\theta\cos\tfrac\theta2\end{pmatrix} = \begin{pmatrix} \sin\tfrac\theta2 \\[2pt] -\cos\tfrac\theta2\end{pmatrix} = -w .

So $F(\theta)$ fixes every vector of the line $\mathbb{R}v$ , which makes angle $\theta/2$ with the first axis, and negates every vector of the perpendicular line $\mathbb{R}w$ . Since $\{v, w\}$ is a basis of $\mathbb{R}^2$ , these two actions determine $F(\theta)$ , and a linear map that fixes a line through the origin and negates its orthogonal complement is by definition the reflection in that line. $\square$

Note(Part 3 is not proved here)AdaptedAdapted from Hassani (2013), §6.6, Thm 6.6.11, pp. 200-201, Thm 6.6.12 and Eq. (6.26), pp. 201-202

The real canonical form, part 3 of the statement above, is quoted without proof: it rests on the real spectral decomposition of a normal operator, which decomposes a real inner product space into one- and two-dimensional subspaces invariant under the operator, and that machinery is outside this unit. (Hassani (2013), Thm 6.6.11, pp. 200-201) proves the decomposition and then deduces the canonical form, with the orthogonality of the transforming matrix, in (Hassani (2013), Thm 6.6.12, pp. 201-202). Hassani also gives, in Example 6.6.13 on p. 202, a short non-rigorous route through the complexification: regard $Q$ as unitary, write it as $e^{A}$ with $A$ real antisymmetric, note that the eigenvalues of $A$ are $0$ and pure imaginary pairs $\pm i\theta_j$ , and exponentiate. Our worked example below carries out the construction by hand in a single case, which is a check on the form but not a proof of it.

NoteOriginal

A unitary matrix is a rigid motion of $\mathbb{C}^n$ fixing the origin: it moves vectors around but changes no length and no angle. That is why unitary matrices are the allowed changes of orthonormal basis, the matrices that diagonalize Hermitian and normal matrices, the time evolutions of quantum mechanics, and — in the real case — the rotations and reflections of rigid-body kinematics.

Remark(Conventions and traps)Original

Active and passive. Expressing the same vector in axes rotated by $+\theta$ multiplies its component column by $R(\theta)^{\mathsf T} = R(-\theta)$ , whereas rotating the vector by $+\theta$ in fixed axes multiplies it by $R(\theta)$ . Books that set up rotations by rotating the coordinate system therefore display the transpose of the matrix above. Always say which of the two you mean.

$\mathrm{O}(n)$ is not the same as $\mathrm{SO}(n)$ . Orthogonality forces only $\det Q = \pm 1$ ; proper rotations are the $\det Q = +1$ half. The two halves are not connected: the determinant is continuous and takes no value between $-1$ and $+1$ on $\mathrm{O}(n)$ , so no continuous family of rotations starting at the identity ever reaches a reflection.

"Orthogonal matrix" is not "matrix with orthogonal columns". The columns must also be unit vectors. A matrix $Q$ whose columns are pairwise orthogonal and all non-zero satisfies $Q^{\mathsf T} Q = D$ with $D$ diagonal and every diagonal entry positive; such a $Q$ stretches the $j$ -th coordinate direction by $\sqrt{D_{jj}}$ , so unless all the $D_{jj}$ are equal it changes lengths by different factors in different directions, and unless they all equal $1$ it changes lengths at all. Dropping "non-zero" loses even this: the zero vector is orthogonal to everything, so a matrix with a zero column has pairwise orthogonal columns while being singular.

Preserving the norm requires all $x$ . $\begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$ sends $e_2$ to $e_1$ , so it preserves the length of $e_2$ , yet it kills $e_1$ ; part 4 of the characterisation is a statement about every vector, not about one.

The real block form is not unique as displayed. Part 3 of "Rotations and reflections" leaves each $\theta_j$ free in $[0, 2\pi)$ , and $R(0) = I_2$ while $R(\pi) = \operatorname{diag}(-1, -1)$ : a block $R(0)$ is two $+1$ entries and a block $R(\pi)$ two $-1$ entries, so the same $Q$ admits displays with different $N_\pm$ and $m$ . Uniqueness, up to the order of the blocks, needs $\theta_j \in (0, \pi)$ , which forces $\sin\theta_j \neq 0$ ; then the $2 \times 2$ blocks carry exactly the non-real eigenvalue pairs $e^{\pm i \theta_j}$ and $N_\pm$ are the multiplicities of $\pm 1$ . (Hassani (2013), Thm 6.6.12, p. 201) states the form without that restriction, and part 3 follows the source.

Real versus complex diagonalization. A unitary matrix is normal, hence unitarily diagonalizable over $\mathbb{C}$ . $R(\theta)$ with $\sin\theta \neq 0$ illustrates the difference: its eigenvalues $e^{\pm i\theta}$ are not real, so it is diagonalizable over $\mathbb{C}$ but not over $\mathbb{R}$ , which is what part 3 of the previous statement expresses.

Example(Classifying two orthogonal matrices)Original

Part A: a $2 \times 2$ matrix. Let

Q_1 = \frac15 \begin{pmatrix} 3 & 4 \\ 4 & -3 \end{pmatrix} .

Its columns are $(3,4)/5$ and $(4,-3)/5$ , each of norm $\sqrt{9+16}/5 = 1$ , with inner product $(12 - 12)/25 = 0$ ; so $Q_1$ is orthogonal by part 2 of the characterisation. Its determinant is $(-9 - 16)/25 = -1$ , so by part 2 of "Rotations and reflections" it is $F(\theta)$ , not a rotation, with $\cos\theta = 3/5$ and $\sin\theta = 4/5$ . The mirror line is the one at angle $\theta/2$ , and

\tan\frac\theta2 = \frac{\sin\theta}{1+\cos\theta} = \frac{4/5}{1 + 3/5} = \frac{4/5}{8/5} = \frac12 ,

so the mirror is the line $y = x/2$ , spanned by $(2,1)$ . Checking directly: $Q_1 (2,1)^{\mathsf T} = \tfrac15(6+4,\, 8-3)^{\mathsf T} = (2,1)^{\mathsf T}$ , fixed; and $Q_1(-1,2)^{\mathsf T} = \tfrac15(-3+8,\, -4-6)^{\mathsf T} = (1,-2)^{\mathsf T} = -(-1,2)^{\mathsf T}$ , negated. The eigenvalues are $+1$ and $-1$ , as part 2 of "Determinant and spectrum" allows for a real matrix.

Part B: a $3 \times 3$ matrix, reduced to the canonical form. Let

Q = \frac13 \begin{pmatrix} 2 & -2 & 1 \\ 1 & 2 & 2 \\ -2 & -1 & 2 \end{pmatrix} .

Step 1: it is orthogonal. The columns are $c_1 = (2,1,-2)/3$ , $c_2 = (-2,2,-1)/3$ , $c_3 = (1,2,2)/3$ . Each has squared norm $(4+1+4)/9 = (4+4+1)/9 = (1+4+4)/9 = 1$ , and

c_1^{\mathsf T} c_2 = \tfrac19(-4+2+2) = 0, \quad c_1^{\mathsf T} c_3 = \tfrac19(2+2-4) = 0, \quad c_2^{\mathsf T} c_3 = \tfrac19(-2+4-2) = 0 .

So $Q^{\mathsf T} Q = I$ .

Step 2: norms are preserved. As a check on part 4 of the characterisation, take $x = (1,2,3)^{\mathsf T}$ . Then $Qx = \tfrac13(2-4+3,\ 1+4+6,\ -2-2+6)^{\mathsf T} = \tfrac13(1,11,2)^{\mathsf T}$ , and

\|Qx\|^2 = \frac{1 + 121 + 4}{9} = \frac{126}{9} = 14 = 1 + 4 + 9 = \|x\|^2 .

Step 3: the determinant. Expanding along the first row of the integer matrix $3Q$ ,

\det(3Q) = 2(4+2) + 2(2+4) + 1(-1+4) = 12 + 12 + 3 = 27 ,

so $\det Q = 27/3^3 = +1$ and $Q \in \mathrm{SO}(3)$ : a rotation, not a reflection.

Step 4: the spectrum. The trace is $(2+2+2)/3 = 2$ , the sum of the three $2\times 2$ principal minors of $Q$ is $2$ , and $\det Q = 1$ , so the characteristic polynomial is

\lambda^3 - 2\lambda^2 + 2\lambda - 1 = (\lambda - 1)(\lambda^2 - \lambda + 1) ,

with roots $1$ and $\tfrac12 \pm \tfrac{\sqrt3}{2} i = e^{\pm i\pi/3}$ . All three have modulus one, as part 2 of "Determinant and spectrum" requires. Comparing the trace with the canonical form $\operatorname{diag}(1, R(\theta))$ , whose trace is $1 + 2\cos\theta$ , gives $\cos\theta = \tfrac12$ : the rotation angle is $\theta = \pi/3$ , or $60^\circ$ .

Step 5: the axis. Solving $(Q - I)x = 0$ , i.e. $\tfrac13(-x_1 - 2x_2 + x_3,\ x_1 - x_2 + 2x_3,\ -2x_1 - x_2 - x_3) = 0$ , gives the one-dimensional solution space spanned by $n = (-1,1,1)^{\mathsf T}$ ; indeed $Qn = \tfrac13(-2-2+1,\ -1+2+2,\ 2-1+2)^{\mathsf T} = \tfrac13(-3,3,3)^{\mathsf T} = n$ . Normalising, $\hat n = (-1,1,1)^{\mathsf T}/\sqrt3$ .

Step 6: the canonical form. Pick any unit vector perpendicular to $\hat n$ , say $\hat e = (1,1,0)^{\mathsf T}/\sqrt2$ (note $\hat n^{\mathsf T}\hat e = (-1+1+0)/\sqrt6 = 0$ ), and complete to a right-handed frame with

\hat f = \hat n \times \hat e = \frac{1}{\sqrt6}\,(-1, 1, -2)^{\mathsf T} .

Now $Q\hat e = \tfrac13(2-2,\ 1+2,\ -2-1)^{\mathsf T}/\sqrt2 = (0,1,-1)^{\mathsf T}/\sqrt2$ , and

\hat e^{\mathsf T} Q \hat e = \tfrac12(0 + 1 + 0) = \tfrac12 = \cos\tfrac\pi3 , \qquad \hat f^{\mathsf T} Q \hat e = \tfrac{1}{\sqrt{12}}(0 + 1 + 2) = \tfrac{3}{2\sqrt3} = \tfrac{\sqrt3}{2} = \sin\tfrac\pi3 .

With $P = (\hat n\ \ \hat e\ \ \hat f)$ , which is orthogonal because its columns are orthonormal by construction and has $\det P = +1$ ,

P^{\mathsf T} Q P = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1/2 & -\sqrt3/2 \\ 0 & \sqrt3/2 & 1/2\end{pmatrix} = \operatorname{diag}\bigl(1,\, R(\pi/3)\bigr),

the canonical form of part 3 with $N_+ = 1$ , $N_- = 0$ , $m = 1$ and $\theta_1 = \pi/3$ . Since $\sin(\pi/3) \neq 0$ this display is the unique one up to the order of the blocks, as the remark above requires. The upper-left $1$ is the axis and the $2\times 2$ block the rotation by $60^\circ$ in the plane perpendicular to it.

ApplicationOriginal

Unitarity is conservation of probability. In quantum mechanics a state is a unit vector $\psi$ of a complex inner product space, and $|(\psi, \phi)|^2$ is the probability of finding the state $\psi$ in the state $\phi$ ; the total probability is $\|\psi\|^2 = 1$ . A time evolution is therefore allowed only if it preserves the norm of every state — part 4 of the characterisation — which by that characterisation means exactly that it is unitary. For a finite-dimensional system with Hermitian Hamiltonian $H$ , the evolution $U(t) = e^{-iHt/\hbar}$ satisfies $U(t)^\dagger = e^{+iH^\dagger t/\hbar} = e^{iHt/\hbar} = U(t)^{-1}$ , so it is unitary, and part 2 above says its eigenvalues are the phases $e^{-iE_kt/\hbar}$ attached to the energy levels: time evolution rotates phases and changes no amplitude. The same argument constrains a scattering matrix $S$ : unitarity of $S$ is the statement that no probability is lost between the incoming and outgoing channels, and it is the hypothesis behind the optical theorem. Changing between two orthonormal bases is likewise a unitary matrix, which is why the predictions of the theory do not depend on the basis chosen.

Orthogonality turns a rigid rotation into an antisymmetric angular velocity. Let $Q(t) \in \mathrm{SO}(3)$ be the matrix carrying body-fixed axes to laboratory axes at time $t$ , a differentiable function of $t$ . Differentiating the constraint $Q^{\mathsf T} Q = I$ gives

\dot Q^{\mathsf T} Q + Q^{\mathsf T}\dot Q = 0, \qquad \text{so} \qquad \Omega = Q^{\mathsf T}\dot Q \text{ satisfies } \Omega^{\mathsf T} = -\Omega .

A $3\times 3$ antisymmetric matrix has only three independent entries, and those three numbers are the components of the angular velocity vector: a rigid body in three dimensions has three rotational degrees of freedom, not nine, and this is why. The determinant is what forbids the body from passing continuously into a mirror image of itself, as the remark on $\mathrm{O}(n)$ versus $\mathrm{SO}(n)$ explains: $\det Q(t)$ is continuous in $t$ and confined to $\{\pm 1\}$ , so it cannot change during a motion.

Orthogonal transformations are the numerically safe ones. In numerical linear algebra the standard algorithms — $QR$ factorisation, the reduction to Hessenberg or tridiagonal form, the $QR$ eigenvalue iteration, the singular value decomposition — act by multiplying on the left by orthogonal matrices, built either as Givens plane rotations $R(\theta)$ embedded in a pair of coordinates or as Householder reflections, which are the $n$ -dimensional analogues of $F(\theta)$ . The reason is part 4 of the characterisation: because $\|Qx\| = \|x\|$ exactly, an orthogonal step neither magnifies the vector nor magnifies the error already in it, so the rounding errors of the previous steps are carried along rather than amplified. A general invertible step, Gaussian elimination for instance, can multiply an error by a large factor. In the language of conditioning, an orthogonal matrix has condition number $\|Q\|\,\|Q^{-1}\| = 1$ in the Euclidean norm, the smallest value possible, which is the quantitative form of the same statement.

Theorem

Normal matrices and simultaneous diagonalization

Adapted

Add to PDF

A complex matrix has an orthonormal eigenbasis exactly when it is normal, and two normal matrices can be given a common orthonormal eigenbasis exactly when they commute.

Prerequisites: Spectral theorem for Hermitian matrices, Unitary and orthogonal matrices, Inner products and norms, Orthonormal bases and Gram-Schmidt

Definition(Normal matrix)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.12.7, p. 272

An $n \times n$ complex matrix $A$ is normal if it commutes with its Hermitian conjugate:

A A^\dagger = A^\dagger A .

Hermitian matrices ( $A = A^\dagger$ ), anti-Hermitian matrices ( $A^\dagger = -A$ ) and unitary matrices ( $A^\dagger = A^{-1}$ ) are normal, since in each case $A^\dagger$ is a function of $A$ that commutes with it; in the real case this covers symmetric, antisymmetric and orthogonal matrices. If $A$ is normal and invertible then $A^{-1}$ is normal.

Theorem(Spectral theorem for normal matrices)AdaptedAdapted from Hassani (2013), Thm 6.4.8 and Cor 6.4.9, pp. 180-181; Riley, Hobson & Bence (2006), §8.13.1, pp. 274-276

Let $A$ be a normal $n \times n$ complex matrix, with distinct eigenvalues $\lambda_1, \dots, \lambda_p$ and eigenspaces $E_{\lambda_j}$ . Then:

$\mathbb{C}^n = E_{\lambda_1} \oplus \cdots \oplus E_{\lambda_p}$ , the summands being mutually orthogonal, and $\mathbb{C}^n$ has an orthonormal basis $u_1, \dots, u_n$ of eigenvectors of $A$ . Equivalently, the matrix $U$ with these columns is unitary and $U^\dagger A U = \Lambda = \operatorname{diag}(\lambda_1', \dots, \lambda_n')$ , where the $\lambda_k'$ are the eigenvalues listed with multiplicity.
Spectral resolution. $\displaystyle A = \sum_{j=1}^{p} \lambda_j P_j = \sum_{k=1}^{n} \lambda_k'\, u_k u_k^\dagger$ , where $P_j$ is the orthogonal projection onto $E_{\lambda_j}$ , so that $P_j = P_j^\dagger = P_j^2$ , $P_i P_j = 0$ for $i \neq j$ , and $\sum_j P_j = I$ .
The adjoint shares the eigenvectors. If $A x = \lambda x$ then $A^\dagger x = \overline{\lambda} x$ ; the same $x$ serves both matrices, and $\sigma(A^\dagger) = \{\overline{\lambda} : \lambda \in \sigma(A)\}$ .
Eigenvectors belonging to distinct eigenvalues are orthogonal, and within one eigenspace any spanning independent set may be replaced by an orthonormal one by the Gram-Schmidt process.

Idea of the proofOriginal

A normal matrix is a complex combination of two commuting Hermitian ones: put $H = \tfrac12(A + A^\dagger)$ and $K = \tfrac{1}{2i}(A - A^\dagger)$ , so that $A = H + iK$ , and then normality of $A$ is exactly the statement $HK = KH$ . So the whole theorem reduces to one question about Hermitian matrices, which the spectral theorem already answers one matrix at a time: can two commuting Hermitian matrices be diagonalized in the same orthonormal basis? They can, because each eigenspace of $H$ is carried into itself by $K$ , so the Hermitian spectral theorem can be applied a second time inside each eigenspace, where $H$ is already a scalar. A common orthonormal eigenbasis of $H$ and $K$ is then an eigenbasis of $A = H + iK$ , with eigenvalues $\alpha_k + i\beta_k$ . The remaining parts — that $A^\dagger$ has the same eigenvectors with conjugated eigenvalues, and that distinct eigenvalues give orthogonal eigenvectors — come from the single observation that $\|Bx\| = \|B^\dagger x\|$ whenever $B$ is normal, applied to $B = A - \lambda I$ .

ProofAdaptedAdapted from Riley, Hobson & Bence (2006), §8.13.1, pp. 274-275; Hassani (2013), Thm 6.4.8 and Cor 6.4.9, pp. 180-181

A restriction lemma. Let $W \subseteq \mathbb{C}^n$ be a subspace of dimension $d \ge 1$ that is carried into itself by a matrix $T$ and by $T^\dagger$ . Choose an orthonormal basis $w_1, \dots, w_d$ of $W$ , which exists by Gram-Schmidt, and set $S_{jk} = (T w_k, w_j)$ . Since $T w_k$ lies in $W$ , expanding it in that orthonormal basis gives $T w_k = \sum_j S_{jk} w_j$ : the matrix $S$ represents $T$ on $W$ . The matrix representing $T^\dagger$ on $W$ in the same basis has entries

(T^\dagger w_k, w_j) = (w_k, T w_j) = \overline{(T w_j, w_k)} = \overline{S_{kj}} = (S^\dagger)_{jk},

where the first equality is the definition of the adjoint and the second the conjugate symmetry of the inner product (Inner products and norms). So $T^\dagger$ is represented by $S^\dagger$ . Because $W$ is invariant under both, the restriction of a product is the product of the restrictions, so $TT^\dagger = T^\dagger T$ on $\mathbb{C}^n$ forces $SS^\dagger = S^\dagger S$ , and $T = T^\dagger$ forces $S = S^\dagger$ : a Hermitian matrix restricts to a Hermitian matrix, and a normal one to a normal one.

Eigenspaces from an eigenbasis. If $T$ has an orthonormal eigenbasis $u_1, \dots, u_n$ with $Tu_k = \tau_k u_k$ , then for each distinct eigenvalue $\tau$ the span $F_\tau$ of those $u_k$ with $\tau_k = \tau$ is the whole eigenspace $E_\tau(T)$ . Indeed $F_\tau \subseteq E_\tau(T)$ ; and if $Tx = \tau x$ , then writing $x = \sum_k (x, u_k)u_k$ and applying $T$ gives $\sum_k \tau_k (x,u_k) u_k = \sum_k \tau (x,u_k)u_k$ , so $(\tau_k - \tau)(x, u_k) = 0$ for every $k$ and the coefficients with $\tau_k \neq \tau$ vanish, putting $x$ in $F_\tau$ . The distinct $F_\tau$ are mutually orthogonal and together span $\mathbb{C}^n$ .

Lemma: two commuting Hermitian matrices have a common orthonormal eigenbasis. Let $H$ and $K$ be Hermitian with $HK = KH$ . By the Hermitian spectral theorem (Spectral theorem for Hermitian matrices) and the previous paragraph, $\mathbb{C}^n$ is the orthogonal direct sum of the eigenspaces $E_\mu(H)$ over the distinct eigenvalues $\mu$ of $H$ . Each is invariant under $K$ : if $Hx = \mu x$ then

H(Kx) = (HK)x = (KH)x = K(Hx) = \mu\,(Kx),

which is where the commutation hypothesis enters, and it is invariant under $K^\dagger = K$ for the same reason. By the restriction lemma, $K$ is represented on $E_\mu(H)$ by a Hermitian matrix, so the Hermitian spectral theorem applied a second time — now inside $E_\mu(H)$ — produces an orthonormal basis of $E_\mu(H)$ made of eigenvectors of $K$ . Every vector of $E_\mu(H)$ is an eigenvector of $H$ , so these are eigenvectors of both. Collecting them over all $\mu$ gives $n$ vectors, orthonormal because they are orthonormal inside each eigenspace and the eigenspaces are mutually orthogonal, hence an orthonormal basis.

Part 1. Let $A$ be normal and put

H = \tfrac12 (A + A^\dagger), \qquad K = \tfrac{1}{2i}(A - A^\dagger), \qquad \text{so} \qquad A = H + iK, \quad A^\dagger = H - iK .

Both are Hermitian: $H^\dagger = \tfrac12(A^\dagger + A) = H$ , and $K^\dagger = -\tfrac{1}{2i}(A^\dagger - A) = K$ . Multiplying out,

A A^\dagger = H^2 + K^2 + i(KH - HK), \qquad A^\dagger A = H^2 + K^2 + i(HK - KH),

so $AA^\dagger - A^\dagger A = 2i(KH - HK)$ and the normality hypothesis says precisely that $HK = KH$ . The lemma then gives an orthonormal basis $u_1, \dots, u_n$ with $Hu_k = \alpha_k u_k$ and $K u_k = \beta_k u_k$ , the $\alpha_k, \beta_k$ real, whence

A u_k = (H + iK)u_k = (\alpha_k + i\beta_k)\,u_k =: \lambda_k' u_k .

So $\mathbb{C}^n$ has an orthonormal basis of eigenvectors of $A$ . The matrix $U = (u_1\ \cdots\ u_n)$ has orthonormal columns, so it is unitary (Unitary and orthogonal matrices, part 2 of the characterisation), and

(U^\dagger A U)_{jk} = u_j^\dagger A u_k = \lambda_k'\, u_j^\dagger u_k = \lambda_k'\,\delta_{jk},

that is $U^\dagger A U = \Lambda$ . The eigenspace decomposition $\mathbb{C}^n = E_{\lambda_1} \oplus \cdots \oplus E_{\lambda_p}$ with mutually orthogonal summands is then the "eigenspaces from an eigenbasis" paragraph applied to $A$ .

Part 3. First, for any normal $B$ and any $x$ ,

\|Bx\|^2 = x^\dagger B^\dagger B x = x^\dagger B B^\dagger x = \|B^\dagger x\|^2 .

Next, $B = A - \lambda I$ is normal whenever $A$ is: expanding,

B^\dagger B = A^\dagger A - \overline{\lambda}A - \lambda A^\dagger + |\lambda|^2 I , \qquad B B^\dagger = A A^\dagger - \overline{\lambda}A - \lambda A^\dagger + |\lambda|^2 I ,

and these agree because $A^\dagger A = AA^\dagger$ . Combining the two displays with $B = A - \lambda I$ and $B^\dagger = A^\dagger - \overline{\lambda} I$ ,

A x = \lambda x \iff \|(A - \lambda I)x\| = 0 \iff \|(A^\dagger - \overline{\lambda}I)x\| = 0 \iff A^\dagger x = \overline{\lambda}x ,

using that a vector of norm zero is zero. The same non-zero $x$ witnesses both, so $\sigma(A^\dagger) = \{\overline\lambda : \lambda \in \sigma(A)\}$ . (This is the route of Riley, Hobson and Bence, who note that $A - \lambda I$ inherits normality and then read off $B^\dagger x = 0$ from $x^\dagger B^\dagger B x = 0$ .)

Part 4. Let $Ax = \lambda x$ and $Ay = \mu y$ with $\lambda \neq \mu$ . Using part 3 at $y$ , so that $A^\dagger y = \overline{\mu}y$ , and the conjugate-linearity of $(\cdot,\cdot)$ in its second slot,

\lambda\,(x,y) = (Ax, y) = y^\dagger A x = (x, A^\dagger y) = (x, \overline{\mu}\,y) = \mu\,(x,y) .

Hence $(\lambda - \mu)(x,y) = 0$ , and $\lambda \neq \mu$ forces $(x,y) = 0$ . Within a single eigenspace there is no such constraint, but $E_\lambda$ is a subspace, so any linearly independent spanning set of it can be replaced by an orthonormal basis of it by Gram-Schmidt, and every vector of $E_\lambda$ is still an eigenvector for $\lambda$ .

Part 2. From $U^\dagger A U = \Lambda$ we get $A = U\Lambda U^\dagger$ , and reading the product columnwise, $A = \sum_{k=1}^n \lambda_k'\, u_k u_k^\dagger$ . For each distinct eigenvalue $\lambda_j$ let $I_j = \{k : \lambda_k' = \lambda_j\}$ and $P_j = \sum_{k \in I_j} u_k u_k^\dagger$ . Then $P_j^\dagger = P_j$ , and $P_i P_j = \sum_{k \in I_i}\sum_{l \in I_j} u_k (u_k^\dagger u_l) u_l^\dagger = \delta_{ij}P_j$ by orthonormality, so each $P_j$ is idempotent and $P_iP_j = 0$ for $i \neq j$ . Also $\sum_j P_j = \sum_{k=1}^n u_k u_k^\dagger = UU^\dagger = I$ . The image of $P_j$ is exactly $E_{\lambda_j}$ : it is contained in it because each $u_k$ with $k \in I_j$ lies there, and it contains it because $x \in E_{\lambda_j}$ has $(x, u_k) = 0$ for $k \notin I_j$ by the "eigenspaces from an eigenbasis" paragraph, so $x = \sum_k (x,u_k)u_k = P_j x$ . A Hermitian idempotent with image $E_{\lambda_j}$ is the orthogonal projection onto $E_{\lambda_j}$ . Finally, grouping the terms of $A = \sum_k \lambda_k' u_ku_k^\dagger$ by eigenvalue gives $A = \sum_j \lambda_j P_j$ . $\square$

Theorem(Normality is exactly unitary diagonalizability)Original

Conversely, if $U^\dagger A U = \Lambda$ is diagonal for some unitary $U$ , then $A = U \Lambda U^\dagger$ and

A A^\dagger = U \Lambda \overline{\Lambda} U^\dagger = U \overline{\Lambda} \Lambda U^\dagger = A^\dagger A ,

because diagonal matrices commute. So for an $n \times n$ complex matrix $A$ :

A \text{ is normal} \iff \mathbb{C}^n \text{ has an orthonormal basis of eigenvectors of } A \iff U^\dagger A U \text{ is diagonal for some unitary } U .

Reading off the diagonal gives the sharper classification: a normal matrix is Hermitian exactly when $\sigma(A) \subset \mathbb{R}$ , anti-Hermitian exactly when $\sigma(A) \subset i\mathbb{R}$ , unitary exactly when $\sigma(A)$ lies on the unit circle, and an orthogonal projection exactly when $\sigma(A) \subseteq \{0, 1\}$ .

Idea of the proofOriginal

Once $A = U\Lambda U^\dagger$ , every identity between $A$ and $A^\dagger$ becomes the same identity between $\Lambda$ and $\overline{\Lambda}$ , because conjugating by a fixed unitary matrix is injective and respects products. So each of the four special classes is read off the diagonal entry by entry: being Hermitian is $\lambda = \overline\lambda$ , being unitary is $|\lambda| = 1$ , and so on.

ProofOriginal

The equivalence. If $A$ is normal, part 1 of the previous statement gives an orthonormal eigenbasis and the unitary $U$ with $U^\dagger A U = \Lambda$ . If $\mathbb{C}^n$ has an orthonormal eigenbasis of $A$ , the matrix $U$ of those columns is unitary and the computation $(U^\dagger A U)_{jk} = u_j^\dagger A u_k = \lambda_k'\delta_{jk}$ of that proof makes $U^\dagger A U$ diagonal. If $U^\dagger A U = \Lambda$ is diagonal for some unitary $U$ , then the display above shows $A A^\dagger = A^\dagger A$ , since $A^\dagger = U\overline{\Lambda}U^\dagger$ and diagonal matrices commute. The three conditions therefore imply one another in a cycle.

The classification. Let $A = U\Lambda U^\dagger$ with $U$ unitary, so $A^\dagger = U\overline{\Lambda}U^\dagger$ , and note that $U M U^\dagger = U N U^\dagger$ implies $M = N$ , because $U$ is invertible. Each line below is read entry by entry on the diagonal, and $\sigma(A)$ is the set of diagonal entries of $\Lambda$ .

$A = A^\dagger \iff \Lambda = \overline{\Lambda} \iff$ every $\lambda_k'$ is real $\iff \sigma(A) \subset \mathbb{R}$ .
$A^\dagger = -A \iff \overline{\Lambda} = -\Lambda \iff \operatorname{Re}\lambda_k' = 0$ for every $k$ $\iff \sigma(A) \subset i\mathbb{R}$ .
$A^\dagger A = I \iff \overline{\Lambda}\Lambda = I \iff |\lambda_k'|^2 = 1$ for every $k$ $\iff \sigma(A)$ lies on the unit circle.
$A$ is an orthogonal projection, i.e. $A = A^\dagger = A^2$ , $\iff \Lambda = \overline\Lambda$ and $\Lambda^2 = \Lambda$ $\iff$ every $\lambda_k'$ is real and satisfies $\lambda^2 = \lambda$ $\iff \sigma(A) \subseteq \{0,1\}$ , the only solutions of $\lambda^2 = \lambda$ being $0$ and $1$ .

In each case the hypothesis that $A$ is normal is what supplies the $U$ and $\Lambda$ in the first place; without it none of these spectral conditions says anything, as $\begin{pmatrix} 0 & 1 \\ 0 & 0\end{pmatrix}$ shows — its spectrum is $\{0\} \subseteq \{0,1\}$ and it is not a projection. $\square$

Theorem(Simultaneous diagonalization)AdaptedAdapted from Hassani (2013), Lemma 6.4.17 and Thm 6.4.18, pp. 186-187; Riley, Hobson & Bence (2006), §8.13.5, pp. 278-279; Arfken, Weber & Harris (2013), §6.5, 'Simultaneous Diagonalization', p. 314

Let $A$ and $B$ be normal $n \times n$ complex matrices. Then some single orthonormal basis of $\mathbb{C}^n$ is made of vectors that are eigenvectors of $A$ and eigenvectors of $B$ — equivalently, one unitary $U$ makes both $U^\dagger A U$ and $U^\dagger B U$ diagonal — if and only if

[A, B] = A B - B A = 0 .

In particular this holds for two commuting Hermitian matrices, and then the common basis may be chosen real whenever $A$ and $B$ are real symmetric. The criterion extends to any commuting family of normal matrices.

Idea of the proofOriginal

One direction is immediate: two matrices that are diagonal in the same basis commute, because diagonal matrices do. The other direction turns commutation with $A$ into commutation with each of $A$ 's spectral projections $P_j$ — the reason being that $B$ maps each eigenspace of $A$ into itself, so it cannot mix the projections up. Once $B$ 's projections $Q_\alpha$ and $A$ 's projections $P_j$ all commute with each other, the products $P_jQ_\alpha$ are again orthogonal projections, onto the joint eigenspaces $E_{\lambda_j} \cap N_{\mu_\alpha}$ , and they still sum to the identity. So $\mathbb{C}^n$ splits orthogonally into subspaces on each of which $A$ and $B$ are both scalars, and any orthonormal basis adapted to that splitting is a common eigenbasis.

ProofAdaptedAdapted from Hassani (2013), Lemma 6.4.17 and Thm 6.4.18, pp. 186-187; Riley, Hobson & Bence (2006), §8.13.5, pp. 278-279

Write $A = \sum_j \lambda_j P_j$ and $B = \sum_\alpha \mu_\alpha Q_\alpha$ for the spectral resolutions of part 2 of the spectral theorem, with $P_j$ the orthogonal projection onto $E_{\lambda_j} = \ker(A - \lambda_j I)$ and $Q_\alpha$ that onto $N_{\mu_\alpha} = \ker(B - \mu_\alpha I)$ ; these exist because $A$ and $B$ are normal.

Necessity. Suppose $u_1, \dots, u_n$ is a basis with $A u_k = \alpha_k u_k$ and $B u_k = \beta_k u_k$ . Then

ABu_k = \beta_k A u_k = \alpha_k\beta_k u_k = \alpha_k B u_k = BA u_k

for every $k$ , and two matrices that agree on a basis are equal, so $AB = BA$ . (Only the existence of a common eigenbasis is used here, not its orthonormality.)

A lemma. If a matrix $T$ commutes with the normal matrix $A$ , then $T$ commutes with every spectral projection $P_j$ of $A$ . For if $Ax = \lambda_j x$ then $A(Tx) = T(Ax) = \lambda_j(Tx)$ , so each $E_{\lambda_j}$ is carried into itself by $T$ . Now take any $x$ and write $x = \sum_i P_i x$ with $P_i x \in E_{\lambda_i}$ . For $i \neq j$ we have $P_j T P_i x = 0$ , because $TP_ix \in E_{\lambda_i}$ is fixed by $P_i$ and $P_jP_i = 0$ ; and $P_jTP_jx = TP_jx$ , because $TP_jx$ lies in $E_{\lambda_j}$ , on which $P_j$ is the identity. Hence

P_j T x = \sum_i P_j T P_i x = P_j T P_j x = T P_j x ,

which is the claim. This is the lemma of Hassani, in the form we need.

All four families of projections commute. Applying the lemma to the normal matrix $A$ and $T = B$ gives $BP_j = P_jB$ for every $j$ . Applying it now to the normal matrix $B$ and $T = P_j$ — legitimate because $P_j$ commutes with $B$ — gives $P_jQ_\alpha = Q_\alpha P_j$ for every $j$ and $\alpha$ .

The joint projections. Put $R_{j\alpha} = P_jQ_\alpha$ . Then

R_{j\alpha}^\dagger = Q_\alpha^\dagger P_j^\dagger = Q_\alpha P_j = P_jQ_\alpha = R_{j\alpha}, \qquad R_{j\alpha}^2 = P_jQ_\alpha P_jQ_\alpha = P_j^2 Q_\alpha^2 = P_jQ_\alpha = R_{j\alpha},

the middle step of the second line using the commutation just proved. So each $R_{j\alpha}$ is a Hermitian idempotent, hence the orthogonal projection onto its image, and that image is

W_{j\alpha} = E_{\lambda_j} \cap N_{\mu_\alpha} :

every $R_{j\alpha}x = P_j(Q_\alpha x)$ lies in the image of $P_j$ , which is $E_{\lambda_j}$ , and equals $Q_\alpha(P_jx)$ , which lies in $N_{\mu_\alpha}$ ; conversely a vector in both is fixed by $P_j$ and by $Q_\alpha$ , hence by $R_{j\alpha}$ . Moreover

\sum_{j,\alpha} R_{j\alpha} = \Bigl(\sum_j P_j\Bigr)\Bigl(\sum_\alpha Q_\alpha\Bigr) = I \cdot I = I ,

and $R_{j\alpha}R_{j'\alpha'} = P_jP_{j'}Q_\alpha Q_{\alpha'} = 0$ unless $j = j'$ and $\alpha = \alpha'$ . Therefore every $x$ decomposes as $x = \sum_{j,\alpha}R_{j\alpha}x$ into vectors of the $W_{j\alpha}$ , and these subspaces are mutually orthogonal, since for $(j,\alpha) \neq (j',\alpha')$ and $x \in W_{j\alpha}$ , $y \in W_{j'\alpha'}$ we get $(x,y) = (R_{j\alpha}x, R_{j'\alpha'}y) = y^\dagger R_{j'\alpha'}R_{j\alpha}x = 0$ . Hence $\mathbb{C}^n$ is the orthogonal direct sum of the non-zero $W_{j\alpha}$ .

The common basis. Pick an orthonormal basis of each non-zero $W_{j\alpha}$ (Gram-Schmidt). The union is orthonormal, because the $W_{j\alpha}$ are mutually orthogonal, and spans $\mathbb{C}^n$ , so it is an orthonormal basis; each of its members lies in some $E_{\lambda_j} \cap N_{\mu_\alpha}$ and is therefore an eigenvector of $A$ with eigenvalue $\lambda_j$ and of $B$ with eigenvalue $\mu_\alpha$ . Taking $U$ to be the matrix of these columns, $U$ is unitary and both $U^\dagger A U$ and $U^\dagger B U$ are diagonal.

The real symmetric case. If $A$ and $B$ are real symmetric and commute, then each $W_{j\alpha}$ is the solution space of the real homogeneous linear system $(A - \lambda_j I)x = 0$ , $(B - \mu_\alpha I)x = 0$ — real because the eigenvalues of a real symmetric matrix are real — so it is the complexification of the real solution space $W_{j\alpha} \cap \mathbb{R}^n$ , which has the same dimension. Running Gram-Schmidt over $\mathbb{R}$ inside each real solution space gives a common real orthonormal eigenbasis, and the matrix of those columns is real orthogonal.

A commuting family. Let $\mathcal{F}$ be a set of pairwise commuting normal matrices. Refine as above: start from the eigenspace decomposition of one member and intersect repeatedly with the eigenspaces of further members, choosing at each step a member that splits at least one summand in two. Each such step strictly increases the number of non-zero summands, and that number never exceeds $n$ , so after at most $n-1$ steps no member of $\mathcal{F}$ splits any summand. Call the resulting orthogonal decomposition $\mathbb{C}^n = W_1 \oplus \cdots \oplus W_r$ . Each $W_m$ is an intersection of eigenspaces of finitely many members, so by the lemma — applied once for each of those members — every $A \in \mathcal{F}$ commutes with the projection onto $W_m$ and hence carries $W_m$ into itself; and since $A$ splits no summand, $W_m$ lies inside a single eigenspace of $A$ , i.e. $A$ acts on $W_m$ as a scalar. An orthonormal basis of each $W_m$ is then a common eigenbasis of the whole family. $\square$

NoteOriginal

Normal is the exact condition for "diagonal in some orthonormal frame": the matrix is a list of complex stretch factors along $n$ perpendicular axes. The commutation criterion is the algebraic reason two observables can be measured at once in quantum mechanics — a common eigenbasis exists precisely when the operators commute — and the practical reason that a symmetry of a system lets you block-diagonalize its dynamical matrix.

Remark(Where the hypotheses are used)Original

"Orthonormal" cannot be dropped. $\begin{pmatrix} 1 & 1 \\ 0 & 2 \end{pmatrix}$ has distinct eigenvalues $1, 2$ and so is diagonalizable, but $A A^{\mathsf T} = \begin{pmatrix} 2 & 2 \\ 2 & 4 \end{pmatrix} \neq \begin{pmatrix} 1 & 1 \\ 1 & 5 \end{pmatrix} = A^{\mathsf T} A$ , so it is not normal and no orthonormal eigenbasis exists. Diagonalizable by a general similarity and diagonalizable by a unitary are different conditions, and only the second is normality.

Normality cannot be dropped from the commutation criterion. $A = B = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ commute, yet neither is diagonalizable at all. Commuting buys a common invariant decomposition, not a common eigenbasis, unless each matrix separately has an eigenbasis.

Degeneracy leaves a choice, and the choice matters. If $\lambda$ is a degenerate eigenvalue of $A$ , not every orthonormal eigenbasis of $A$ is an eigenbasis of a commuting $B$ : one must diagonalize $B$ restricted to each eigenspace $E_\lambda$ of $A$ . The theorem asserts that some such choice works, not that any does.

Real normal is weaker than real symmetric. $R(\theta) = \begin{pmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{pmatrix}$ is real and normal, but for $\sin\theta \neq 0$ it has no real eigenvector, so it cannot be diagonalized by a real orthogonal matrix. Among real matrices, orthogonal diagonalizability over $\mathbb{R}$ characterises the symmetric ones, not the normal ones; see unitary and orthogonal matrices for the real canonical form.

Conventions. Physics texts usually write the condition as $[A, A^\dagger] = 0$ and call $A^\dagger$ the adjoint; mathematical texts write $A^*$ . Some books state part 1 of the spectral theorem as "a normal operator is diagonalizable"; read in such a statement that the diagonalizing basis is orthonormal, which is what the proof produces and what the converse above needs.

Example(A normal matrix that is neither Hermitian nor unitary, and a commuting partner)Original

Let $C$ be the cyclic shift of $\mathbb{C}^3$ and $A = I + C$ :

C = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & 0 & 0\end{pmatrix}, \qquad A = \begin{pmatrix} 1 & 1 & 0 \\ 0 & 1 & 1 \\ 1 & 0 & 1\end{pmatrix} .

Step 1: $A$ is normal, and why. $C$ is a permutation matrix, so its columns are the standard basis vectors in some order and are orthonormal: $C$ is unitary, with $C^\dagger = C^{-1} = C^2$ (shifting three times returns to the start, $C^3 = I$ ). Hence

A^\dagger = I + C^2 , \qquad A A^\dagger = (I + C)(I + C^2) = I + C + C^2 + C^3 = 2I + C + C^2 ,

and $A^\dagger A = (I+C^2)(I+C)$ gives the same four terms in the other order, so $AA^\dagger = A^\dagger A$ . Written out, both equal $\begin{pmatrix} 2&1&1\\1&2&1\\1&1&2\end{pmatrix}$ . The structural reason is worth keeping: $A$ and $A^\dagger$ are both polynomials in the one matrix $C$ , and any two polynomials in the same matrix commute.

$A$ is not Hermitian, since $A^{\mathsf T} \neq A$ , and not unitary, as step 2 will show by exhibiting an eigenvalue of modulus $2$ .

Step 2: the eigenvalues and an orthonormal eigenbasis. Let $\omega = e^{2\pi i/3} = -\tfrac12 + \tfrac{\sqrt3}{2} i$ , so $\omega^3 = 1$ and $1 + \omega + \omega^2 = 0$ . For $k = 0, 1, 2$ put

v_k = (1,\ \omega^k,\ \omega^{2k})^{\mathsf T} .

Then $C v_k = (\omega^k, \omega^{2k}, 1)^{\mathsf T} = \omega^k v_k$ , using $\omega^{3k} = 1$ for the last entry, so

A v_k = (1 + \omega^k)\, v_k .

The eigenvalues are therefore

\lambda_0 = 2, \qquad \lambda_1 = 1 + \omega = \tfrac12 + \tfrac{\sqrt3}{2}i = e^{i\pi/3}, \qquad \lambda_2 = 1 + \omega^2 = \tfrac12 - \tfrac{\sqrt3}{2}i = e^{-i\pi/3} .

Two checks: $\operatorname{tr}A = 3 = 2 + 2\cos(\pi/3) = 2 + 1$ , and $\det A = \lambda_0\lambda_1\lambda_2 = 2\,|e^{i\pi/3}|^2 = 2$ , which matches the direct expansion $\det A = 1(1-0) - 1(0-1) = 2$ . Since $|\lambda_0| = 2 \neq 1$ , $A$ is not unitary, as promised.

The $v_k$ are orthogonal: by the geometric-sum formula,

(v_j, v_k) = v_k^\dagger v_j = \sum_{m=0}^{2}\omega^{mj}\,\overline{\omega^{mk}} = \sum_{m=0}^{2}\omega^{m(j-k)} = 3\,\delta_{jk},

because for $j \neq k$ the sum is $1 + \zeta + \zeta^2 = 0$ with $\zeta = \omega^{j-k} \neq 1$ a cube root of unity. So $u_k = v_k/\sqrt3$ is an orthonormal eigenbasis — exactly what part 1 of the spectral theorem promises — and with $U = (u_0\ u_1\ u_2)$ , the discrete Fourier matrix of order 3,

U^\dagger A U = \operatorname{diag}\bigl(2,\ e^{i\pi/3},\ e^{-i\pi/3}\bigr).

Step 3: the adjoint and the spectral resolution. Since $C^2 u_k = \omega^{2k}u_k = \overline{\omega^k}\,u_k$ (because $|\omega| = 1$ , so $\omega^{2k} = \omega^{-k} = \overline{\omega^k}$ ), we get $A^\dagger u_k = (1 + \overline{\omega^k})u_k = \overline{\lambda_k}\,u_k$ : the same eigenvectors, the conjugated eigenvalues, as part 3 requires. The eigenvalues are distinct, so each eigenspace is one-dimensional, $P_k = u_ku_k^\dagger$ , and part 2 reads

A = 2\,u_0u_0^\dagger + e^{i\pi/3}\,u_1u_1^\dagger + e^{-i\pi/3}\,u_2u_2^\dagger ,

which multiplies out to $A$ (one may check the $(1,1)$ entry: $\tfrac13(2 + e^{i\pi/3} + e^{-i\pi/3}) = \tfrac13(2+1) = 1$ , and the $(1,2)$ entry: $\tfrac13(2 + e^{i\pi/3}\overline{\omega} + e^{-i\pi/3}\overline{\omega^2}) = \tfrac13(2 + 1) = 1$ , since $e^{i\pi/3}\overline\omega = e^{i\pi/3}e^{-2\pi i/3} = e^{-i\pi/3}$ and the two conjugate terms sum to $2\cos(\pi/3) = 1$ ).

Step 4: a commuting partner and the role of degeneracy. Let

B = A + A^\dagger = 2I + C + C^2 = \begin{pmatrix} 2&1&1\\1&2&1\\1&1&2\end{pmatrix},

real symmetric, and commuting with $A$ because both are polynomials in $C$ . By the simultaneous-diagonalization theorem the two share an orthonormal eigenbasis, and indeed $Bu_k = (\lambda_k + \overline{\lambda_k})u_k = 2(\operatorname{Re}\lambda_k)u_k$ , giving the eigenvalues

4,\qquad 1,\qquad 1 .

So $B$ has a degenerate eigenvalue $1$ , with the two-dimensional eigenspace $N_1 = \operatorname{span}\{v_1, v_2\}$ , and this is where the third trap above becomes concrete. Solving $(B - I)x = 0$ by hand gives the obvious real basis $(-1,1,0)^{\mathsf T}$ , $(-1,0,1)^{\mathsf T}$ of $N_1$ , and

A\,(-1,1,0)^{\mathsf T} = (0, 1, -1)^{\mathsf T},

which is not a multiple of $(-1,1,0)^{\mathsf T}$ : that perfectly good eigenvector of $B$ is not an eigenvector of $A$ . Only a particular basis of $N_1$ — up to scalars, $\{v_1, v_2\}$ — diagonalizes $A$ as well. The theorem says such a choice exists, not that every choice works.

The same example illustrates the fourth trap. $A$ is real and normal, but its eigenvalues $e^{\pm i\pi/3}$ are not real, so it has no real eigenvector for them and cannot be diagonalized by a real orthogonal matrix, although the real symmetric $B$ can be: for instance by the real orthonormal basis $(1,1,1)^{\mathsf T}/\sqrt3$ , $(1,-1,0)^{\mathsf T}/\sqrt2$ , $(1,1,-2)^{\mathsf T}/\sqrt6$ , which diagonalizes $B$ but not $A$ .

ApplicationOriginal

Commuting observables and good quantum numbers. In quantum mechanics an observable is a Hermitian operator and a measurement leaves the system in an eigenvector of it. Two observables can therefore be assigned definite values at the same time exactly when they have a common eigenvector, and a complete set of such states exists exactly when they commute — which is the simultaneous-diagonalization theorem, and the reason $[\,\cdot\,,\cdot\,]$ is the central object of the theory. Position and momentum do not commute, so no state has both sharp; the classification of the hydrogen atom, by contrast, rests on the fact that $H$ , $L^2$ and $L_z$ do commute pairwise. The proof above explains why the labels come in a tuple: the joint eigenspaces are the subspaces $W_{j\alpha} = E_{\lambda_j} \cap N_{\mu_\alpha}$ , and the quantum numbers $(n, \ell, m)$ are just the names of the eigenvalues whose intersection the state lies in. Degeneracy is the practical point. An energy level $E_n$ of hydrogen is highly degenerate, so "an eigenbasis of $H$ " is far from determined; the extra commuting operators are precisely what selects a basis inside each level, and the fourth trap above warns that an arbitrary basis of a degenerate level will not do.

Symmetry block-diagonalizes a dynamical matrix. The worked example is, read physically, a ring of three identical masses with identical nearest-neighbour springs: the matrix $B$ is (up to a constant) the dynamical matrix of that ring, and $C$ is the operator that rotates the ring by one site. Rotating the ring is a symmetry of it, so $C$ commutes with $B$ , and the theorem says the normal modes can be chosen to be eigenvectors of $C$ as well. Those eigenvectors are the discrete plane waves $v_k$ , with $C v_k = e^{2\pi i k/3}v_k$ — the finite version of Bloch's theorem, the phase $2\pi k/3$ playing the role of the wavenumber. Note what has been bought: the modes were found by diagonalizing the symmetry, which was trivial, rather than the dynamical matrix. For a ring of $N$ masses this replaces an $N \times N$ eigenproblem by inspection, and the same argument in the general form — decompose into joint eigenspaces of the symmetry group — is how the vibrational modes of a molecule are sorted by irreducible representation before any force constant is computed. The degeneracy of the ring ( $\lambda_1 = \lambda_2$ in the example) is itself a consequence of the symmetry, since $v_1$ and $v_2$ are exchanged by reflecting the ring.

Why numerical analysts care whether a matrix is normal. The eigenvalues of a normal matrix are perfectly conditioned: because the diagonalizing matrix $U$ is unitary, a perturbation $A + \Delta A$ moves each eigenvalue by at most $\|\Delta A\|$ in the Euclidean operator norm. For a non-normal matrix the eigenvector matrix can be badly conditioned and the eigenvalues correspondingly violent: the $n \times n$ matrix with ones on the superdiagonal and $\varepsilon$ in the bottom-left corner has eigenvalues of modulus $\varepsilon^{1/n}$ , which is $10^{-1}$ for $\varepsilon = 10^{-10}$ and $n = 10$ , although the unperturbed matrix is nilpotent with every eigenvalue $0$ . So "is this matrix normal?" decides whether computed eigenvalues can be trusted at face value, and it is the reason library routines dispatch Hermitian and unitary input to separate, more accurate algorithms.

Theorem · advanced

Jordan normal form

Adapted

Add to PDF

Every square matrix whose eigenvalues lie in the ground field is similar to a block diagonal matrix of Jordan blocks, unique up to the order of the blocks; the block sizes are read off the nullities of the powers of $A - \lambda I$ , and the form turns powers and power series of a defective matrix into polynomial-times-exponential formulas.

Prerequisites: Eigenvalues and eigenvectors, Vector spaces and bases, Linear maps and matrices, Determinants

Definition(Jordan block)AdaptedAdapted from Dummit & Foote (2004), §12.3, the definition on p. 492

For $\lambda$ in a field $F$ and an integer $m \ge 1$ , the Jordan block $J_m(\lambda)$ is the $m \times m$ matrix carrying $\lambda$ in every diagonal position, $1$ in every position of the first superdiagonal, and $0$ everywhere else:

J_m(\lambda) = \begin{pmatrix} \lambda & 1 & & \\ & \lambda & \ddots & \\ & & \ddots & 1 \\ & & & \lambda \end{pmatrix} = \lambda I_m + N_m , \qquad (N_m)_{pq} = \begin{cases} 1, & q = p+1, \\ 0, & \text{otherwise.}\end{cases}

Dummit and Foote (Dummit & Foote (2004), p. 492) call this the elementary Jordan matrix with eigenvalue $\lambda$ . A matrix is in Jordan canonical form when it is block diagonal with Jordan blocks as its diagonal blocks. The nilpotent part $N_m$ shifts the standard basis down one step, $N_m e_1 = 0$ and $N_m e_{t} = e_{t-1}$ for $t \ge 2$ , so $N_m^m = 0$ while $N_m^{m-1} \neq 0$ .

The block $J_m(\lambda)$ is upper triangular, so its characteristic polynomial is $(x - \lambda)^m$ : the eigenvalue $\lambda$ has algebraic multiplicity $m$ . Its eigenspace, however, is the line spanned by $e_1$ , because $(J_m(\lambda) - \lambda I)x = 0$ reads $x_{t} = 0$ for $t \ge 2$ . So a single block of size $m \ge 2$ is defective in the sense of the eigenvalue problem, with geometric multiplicity $1$ against algebraic multiplicity $m$ .

Definition(Generalized eigenvectors, generalized eigenspaces and Jordan chains)Original

Let $A$ be $n \times n$ over $F$ and $\lambda \in F$ . A vector $x \neq 0$ is a generalized eigenvector of $A$ for $\lambda$ if

(A - \lambda I)^{j} x = 0 \quad \text{for some } j \ge 1 ,

and the smallest such $j$ is its rank. Rank $1$ means an ordinary eigenvector. The generalized eigenspace is

G_\lambda = \ker (A - \lambda I)^{n} ,

which contains every generalized eigenvector for $\lambda$ , because the kernels of the powers stop growing after at most $n$ steps (step 1 of the proof below). Write

d_j(\lambda) = \dim \ker (A - \lambda I)^{j} = n - \operatorname{rank}(A - \lambda I)^{j} , \qquad d_0(\lambda) = 0 ,

so that $d_1(\lambda)$ is the geometric multiplicity of $\lambda$ and $d_n(\lambda) = \dim G_\lambda$ .

A Jordan chain of length $s$ for $\lambda$ is a list $x_1, \dots, x_s$ of non-zero vectors with

(A - \lambda I)x_1 = 0, \qquad (A - \lambda I)x_{t} = x_{t-1} \quad (2 \le t \le s) ,

that is, $x_t = (A - \lambda I)^{s-t}x_s$ with $x_t$ of rank $t$ . Written as the columns of a matrix in this order, a chain is exactly a basis on which $A$ acts as $J_s(\lambda)$ : $Ax_1 = \lambda x_1$ and $Ax_t = \lambda x_t + x_{t-1}$ .

Finally, some non-zero polynomial does annihilate $A$ : the $n^2 + 1$ matrices $I, A, A^2, \dots, A^{n^2}$ cannot be independent in the $n^2$ -dimensional space of $n \times n$ matrices, so a non-trivial linear relation among them exhibits a non-zero polynomial $q$ of degree at most $n^2$ with $q(A) = 0$ , and dividing by its leading coefficient makes it monic. The minimal polynomial $\mu_A$ of $A$ is therefore well defined as the monic polynomial of least degree with $\mu_A(A) = 0$ . It divides every polynomial that annihilates $A$ , since dividing such a polynomial by $\mu_A$ leaves a remainder of lower degree that also annihilates $A$ , which forces the remainder to be $0$ .

Theorem(Jordan normal form)AdaptedAdapted from Dummit & Foote (2004), §12.3, Thm 23 and Cor 24, p. 493

Let $A$ be an $n \times n$ matrix over a field $F$ , and suppose every eigenvalue of $A$ already lies in $F$ : the characteristic polynomial is a product of linear factors there. (Over $\mathbb{C}$ this costs nothing.) Then:

Existence. Some invertible matrix $P$ with entries in $F$ puts $A$ into Jordan canonical form by similarity:

P^{-1} A P = J = \operatorname{diag}\bigl(J_{s_1}(\nu_1),\ J_{s_2}(\nu_2),\ \dots,\ J_{s_b}(\nu_b)\bigr), \qquad s_1 + \cdots + s_b = n ,

each $\nu_i$ an eigenvalue of $A$ , the same eigenvalue permitted in several blocks. The columns of $P$ are $b$ Jordan chains laid end to end. 2. Uniqueness. $J$ is determined by $A$ up to the order of its blocks. 3. Should a diagonal matrix $D$ happen to be similar to $A$ , then $D$ is itself the Jordan form, since diagonal entries are $1 \times 1$ blocks and part 2 leaves no second answer (Dummit & Foote (2004), Cor 24, p. 493). So nothing is lost by always asking for the Jordan form rather than for a diagonalization.

Theorem(The block structure is read off the nullities)Original

With $d_j(\lambda)$ as above, in the Jordan form of $A$ :

the number of blocks carrying the eigenvalue $\lambda$ is $d_1(\lambda)$ , the geometric multiplicity of $\lambda$ ;
the number of those blocks of size at least $j$ is $d_j(\lambda) - d_{j-1}(\lambda)$ , and the number of size exactly $j$ is $2d_j(\lambda) - d_{j-1}(\lambda) - d_{j+1}(\lambda)$ ;
the total of their sizes is $\dim G_\lambda$ , which equals the algebraic multiplicity of $\lambda$ ;
the largest of their sizes is the multiplicity of $\lambda$ as a root of the minimal polynomial $\mu_A$ ; consequently $\mu_A(x) = \prod_\lambda (x - \lambda)^{s(\lambda)}$ with $s(\lambda)$ that largest size.

In particular $A$ is diagonalizable exactly when every block has size $1$ , which by 4 happens exactly when $\mu_A$ has no repeated root (Dummit & Foote (2004), Cor 25, p. 494), and by 1 and 3 exactly when $d_1(\lambda)$ equals the algebraic multiplicity for every $\lambda$ .

Idea of the proofOriginal

Two independent mechanisms are at work, and the proof separates them.

The first is that different eigenvalues do not interfere. If $T = A - \lambda I$ , then the kernels of $T, T^2, T^3, \dots$ grow and must stall, and at the moment they stall the space splits as $\ker T^n \oplus \operatorname{im}T^n$ into a part where $T$ is nilpotent and a part where $T$ is invertible. Peeling off one eigenvalue at a time this way decomposes $F^n$ into the generalized eigenspaces $G_\lambda$ , on each of which $A$ is $\lambda I$ plus a nilpotent matrix.

The second is a statement about one nilpotent matrix $N$ : its domain has a basis made of chains $N^{s-1}u, \dots, Nu, u$ . This is proved by induction on the dimension, applying the result to the smaller space $N(V)$ and then lengthening every chain found there by one step — possible precisely because the chains in $N(V)$ start from vectors that are themselves images.

Uniqueness needs no new idea: the numbers $d_j(\lambda)$ are unchanged by similarity, and a direct count shows a block of size $s$ contributes $\min(j, s)$ to $d_j(\lambda)$ . Inverting that count recovers the block sizes from the $d_j(\lambda)$ , so no two different block patterns can describe the same matrix.

ProofOriginal

Throughout, $V = F^n$ and matrices act on column vectors. Dummit and Foote reach this theorem as a corollary of the elementary divisor decomposition of $V$ as a module over the polynomial ring $F[x]$ (Dummit & Foote (2004), §12.3, pp. 490-493); the argument below is self-contained and uses nothing beyond rank and nullity.

Step 1: the kernels of the powers stall, and the space splits. Let $T : V \to V$ be linear, $d = \dim V$ . Since $T^{j}x = 0$ implies $T^{j+1}x = 0$ , the kernels increase: $\ker T \subseteq \ker T^2 \subseteq \cdots$ . Suppose $\ker T^{j} = \ker T^{j+1}$ for some $j$ . If $T^{j+2}x = 0$ then $T^{j+1}(Tx) = 0$ , so $Tx \in \ker T^{j+1} = \ker T^{j}$ , so $T^{j+1}x = 0$ : the chain is constant from $j$ on. As each strict increase costs at least one dimension and $\dim V = d$ , equality is reached at the latest at $j = d$ . Hence

\ker T^{d} = \ker T^{d+1} = \cdots =: K , \qquad R := \operatorname{im} T^{d} .

Then $V = K \oplus R$ . Indeed $\dim K + \dim R = d$ by rank-nullity applied to $T^d$ , and the sum is direct: if $x = T^{d}y$ lies in $K$ , then $T^{2d}y = 0$ , so $y \in \ker T^{2d} = \ker T^{d}$ and therefore $x = T^{d}y = 0$ . Both summands are carried into themselves by $T$ (and by any matrix commuting with $T$ ), $T$ is nilpotent on $K$ because $T^{d} = 0$ there, and $T$ is injective on $R$ — a vector of $R$ killed by $T$ lies in $K \cap R = \{0\}$ — hence invertible on $R$ .

Step 2: $V$ is the direct sum of the generalized eigenspaces. We show by induction on $\dim V$ that if the characteristic polynomial of $A$ splits over $F$ , with distinct roots $\lambda_1, \dots, \lambda_p$ , then $V = G_{\lambda_1} \oplus \cdots \oplus G_{\lambda_p}$ .

The case $\dim V = 0$ is empty. Otherwise apply step 1 to $T = A - \lambda_1 I$ , giving $V = K \oplus R$ with $K = \ker(A - \lambda_1 I)^{n} = G_{\lambda_1}$ and both summands invariant under $A$ (because $A$ commutes with $T$ ). Here $K \neq \{0\}$ , since $\lambda_1$ is a root of the characteristic polynomial and so has an eigenvector, whence $\dim R < \dim V$ . Choosing a basis adapted to the splitting makes $A$ block diagonal, so its characteristic polynomial is the product of those of $A|_{K}$ and $A|_{R}$ ; in particular the latter splits over $F$ as well. On $K$ we have $A = \lambda_1 I + T$ with $T$ nilpotent, hence strictly upper triangular in a suitable basis of $K$ — choose a basis adapted to the flag $\{0\} \subseteq \ker T \subseteq \ker T^{2} \subseteq \cdots \subseteq K$ , which exhausts $K$ by step 1, and note that $T$ carries $\ker T^{j}$ into $\ker T^{j-1}$ , so each basis vector is sent into the span of the strictly earlier ones (step 3 below gives the sharper chain basis, and does not depend on the present step). Hence the characteristic polynomial of $A|_K$ is $(x - \lambda_1)^{\dim K}$ and the remaining roots $\lambda_2, \dots, \lambda_p$ are exactly the eigenvalues of $A|_R$ . Also, for $\lambda \neq \lambda_1$ the matrix $A - \lambda I = (\lambda_1 - \lambda)I + T$ is invertible on $K$ — a non-zero scalar plus a nilpotent matrix is invertible, its inverse being the finite sum $\sum_{i\ge 0}(-1)^{i}(\lambda_1 - \lambda)^{-i-1}T^{i}$ . Consequently, for $j \ge 2$ ,

G_{\lambda_j} \subseteq R :

write $x \in G_{\lambda_j}$ as $x = y + z$ with $y \in K$ , $z \in R$ . Because $K$ and $R$ are invariant, $(A - \lambda_j I)^{n}y$ lies in $K$ and $(A - \lambda_j I)^{n}z$ lies in $R$ , and their sum is $0$ ; the decomposition of $0$ being unique, each vanishes separately. Then $y = 0$ by the invertibility just noted, so $x = z \in R$ . The inductive hypothesis applied to $A|_R$ decomposes $R$ into the generalized eigenspaces of $A|_R$ , and the one for $\lambda_j$ is $\ker(A - \lambda_j I)^{\dim R} \cap R$ , which equals $G_{\lambda_j}$ : it is contained in $G_{\lambda_j}$ because the kernels only grow with the exponent, and contains it because $G_{\lambda_j} \subseteq R$ and the kernels have stalled by exponent $\dim R$ inside $R$ , by step 1 applied there. Adding $K = G_{\lambda_1}$ gives the claim.

Two consequences. First, the computation just made of the characteristic polynomial of $A|_K$ applies to every $G_\lambda$ , so $\dim G_\lambda$ is the algebraic multiplicity of $\lambda$ . Second, it suffices to produce a Jordan basis of each $G_\lambda$ separately.

Step 3: a nilpotent matrix has a basis of chains. Let $N : V \to V$ be nilpotent, $\dim V = d$ . We claim $V$ has a basis that is a disjoint union of chains $N^{s-1}u, \dots, Nu, u$ with $N^{s}u = 0$ , and that the ends $N^{s-1}u$ of those chains form a basis of $\ker N$ .

Induct on $d$ . For $d = 0$ there is nothing to prove. If $N = 0$ , any basis works, each vector a chain of length one, and the ends span $\ker N = V$ . Otherwise set $W = N(V)$ and $k = \dim \ker N$ , so $\dim W = d - k$ by rank-nullity, and $k \ge 1$ because a nilpotent matrix on a non-zero space has non-trivial kernel (take the largest $i$ with $N^{i} \neq 0$ and any $x$ with $N^{i}x \neq 0$ ; then $N^{i}x \in \ker N$ ). Hence $\dim W < d$ , and $N$ restricts to a nilpotent map of $W$ . By the inductive hypothesis $W$ has a chain basis, say chains

C_i : \quad N^{s_i - 1}u_i, \ \dots,\ N u_i,\ u_i \qquad (i = 1, \dots, r), \qquad N^{s_i}u_i = 0 ,

whose ends $N^{s_i-1}u_i$ form a basis of $\ker N \cap W = \ker(N|_W)$ ; in particular $r = \dim(\ker N \cap W)$ .

Now lengthen. Each $u_i$ lies in $W = N(V)$ , so choose $v_i \in V$ with $Nv_i = u_i$ ; the list

C_i' : \quad N^{s_i}v_i\ (= N^{s_i-1}u_i),\ \dots,\ Nv_i\ (= u_i),\ v_i

is a chain of length $s_i + 1$ , since $N^{s_i+1}v_i = N^{s_i}u_i = 0$ . Extend the $r$ vectors $N^{s_i-1}u_i$ to a basis of $\ker N$ by adjoining $z_1, \dots, z_{k-r}$ , each of which is a chain of length one.

The resulting list has $\sum_{i}(s_i + 1) + (k - r) = (d - k) + r + (k - r) = d$ vectors, so it is enough to show it is independent. Suppose

\sum_{i=1}^{r}\sum_{t=0}^{s_i} a_{i,t}\, N^{t}v_i \ +\ \sum_{j=1}^{k-r} b_j z_j = 0 .

Apply $N$ . The terms $t = s_i$ die because $N^{s_i+1}v_i = 0$ , and the $z_j$ die because they lie in $\ker N$ , leaving $\sum_i \sum_{t=0}^{s_i-1}a_{i,t}N^{t}u_i = 0$ . These vectors are precisely the chain basis of $W$ , so every $a_{i,t}$ with $t \le s_i - 1$ vanishes. What is left of the relation is $\sum_i a_{i,s_i}N^{s_i-1}u_i + \sum_j b_jz_j = 0$ , a relation among a basis of $\ker N$ , so the remaining coefficients vanish too. Finally, the ends of the new chains are the $N^{s_i-1}u_i$ together with the $z_j$ , which is the chosen basis of $\ker N$ , so the second half of the claim is inherited.

Step 4: existence. Apply step 3 to $N = (A - \lambda I)|_{G_\lambda}$ for each eigenvalue $\lambda$ , which is legitimate by step 2. A chain $x_1, \dots, x_s$ of $N$ satisfies $Ax_1 = \lambda x_1$ and $Ax_t = \lambda x_t + x_{t-1}$ , so on the span of that chain, in that order, $A$ is the matrix $J_s(\lambda)$ . Concatenating the chains over all $\lambda$ gives a basis of $V$ by step 2; let $P$ be the matrix of its columns. Then $AP = PJ$ with $J$ block diagonal of Jordan blocks, that is $P^{-1}AP = J$ .

Step 5: uniqueness. Let $J$ be any Jordan matrix similar to $A$ , say $P^{-1}AP = J$ . Since $P^{-1}(A - \lambda I)^{j}P = (J - \lambda I)^{j}$ , similar matrices have the same nullities, so $d_j(\lambda)$ may be computed from $J$ . A kernel of a block diagonal matrix is the direct sum of the kernels of its blocks, so $d_j(\lambda)$ is the sum over blocks $J_s(\nu)$ of $\dim\ker(J_s(\nu) - \lambda I)^{j}$ . For $\nu \neq \lambda$ that block contributes nothing: $J_s(\nu) - \lambda I$ is triangular with the non-zero number $\nu - \lambda$ on its diagonal, hence invertible, and so is its $j$ -th power. For $\nu = \lambda$ the block contributes $\dim \ker N_s^{j}$ , and $N_s^{j}$ maps $e_t \mapsto e_{t-j}$ (zero when $t \le j$ ), so its kernel is spanned by $e_1, \dots, e_{\min(j,s)}$ . Therefore

d_j(\lambda) = \sum_{\text{blocks } J_s(\lambda) \text{ of } J} \min(j, s) , \qquad\text{hence}\qquad d_j(\lambda) - d_{j-1}(\lambda) = \#\{\text{blocks } J_s(\lambda) : s \ge j\} ,

because $\min(j,s) - \min(j-1,s)$ is $1$ when $s \ge j$ and $0$ otherwise. The right-hand side determines, for every $j$ , how many blocks for $\lambda$ have size at least $j$ , hence the whole multiset of block sizes for every eigenvalue, hence $J$ up to the order of the blocks. This proves both the uniqueness statement and parts 1, 2 and 3 of the structure statement — part 1 being the case $j = 1$ , part 3 following from $d_n(\lambda) = \sum_s s = \dim G_\lambda$ and step 2, and the "exactly $j$ " count in part 2 being the difference of the counts for $\ge j$ and $\ge j+1$ .

For part 4, note that for a block, $(J_s(\lambda) - \lambda I)^{i} = N_s^{i}$ vanishes exactly when $i \ge s$ , while $J_s(\lambda) - \nu I$ is invertible for $\nu \neq \lambda$ . So a polynomial $q$ annihilates $J$ if and only if it annihilates every block, and $q(J_s(\lambda)) = 0$ holds if and only if $(x-\lambda)^{s}$ divides $q$ : writing $q(x) = \sum_i c_i (x - \lambda)^{i}$ , we get $q(J_s(\lambda)) = \sum_i c_i N_s^{i}$ , whose superdiagonals are the $c_i$ for $i < s$ , so it vanishes exactly when $c_0 = \cdots = c_{s-1} = 0$ . The monic polynomial of least degree divisible by $(x - \lambda)^{s(\lambda)}$ for every $\lambda$ , with $s(\lambda)$ the largest block size, is $\prod_\lambda(x-\lambda)^{s(\lambda)}$ , and $\mu_A = \mu_J$ because similarity preserves polynomial identities. $\square$

Theorem(Powers and power series of a Jordan block)Original

Let $J_m(\lambda) = \lambda I + N_m$ as above. For every integer $k \ge 0$ ,

J_m(\lambda)^{k} = \sum_{i=0}^{\min(k,\,m-1)} \binom{k}{i}\lambda^{k-i} N_m^{i} , \qquad\text{entrywise}\qquad \bigl(J_m(\lambda)^{k}\bigr)_{pq} = \binom{k}{q-p}\lambda^{\,k-q+p} \quad\text{for } p \le q \le p+k ,

the entry being $0$ at every other position — that is, for $q < p$ and also for $q - p > k$ , so that no negative power of $\lambda$ is ever called for. (Writing the formula with the convention $\binom{k}{i} = 0$ for $i > k$ would make it read $0 \cdot \lambda^{\text{negative}}$ when $\lambda = 0$ and $q - p > k$ , which is why those positions are excluded outright rather than by a convention. At the far edge of that range, $q = p + k$ with $\lambda = 0$ , the entry is $\binom{k}{k}\lambda^{0} = 1$ , using $0^0 = 1$ ; this is the statement that $J_m(0)^{k} = N_m^{k}$ .) More generally, if $f(z) = \sum_{i \ge 0} c_i (z - \lambda)^{i}$ converges in a disc about $\lambda$ , then

f\bigl(J_m(\lambda)\bigr) = \sum_{i=0}^{m-1} \frac{f^{(i)}(\lambda)}{i!}\, N_m^{i} ,

a finite sum: the block sees only the first $m$ Taylor coefficients of $f$ at $\lambda$ . Applying this blockwise through $A = PJP^{-1}$ gives, for an $n \times n$ matrix $A$ with largest block size $s(\lambda)$ at each eigenvalue,

\bigl(A^{k}\bigr)_{pq} = \sum_{\lambda \in \sigma(A),\ \lambda \neq 0} \varphi_{pq}^{\lambda}(k)\, \lambda^{k} ,

with each $\varphi_{pq}^{\lambda}$ a polynomial in $k$ of degree at most $s(\lambda) - 1$ . The sum omits a zero eigenvalue, and the formula then holds for all $k \ge s(0)$ , where $s(0)$ is the largest block size at $0$ (and for all $k \ge 1$ when $A$ is invertible, the omission being vacuous): the blocks at $0$ are nilpotent of index at most $s(0)$ , so they contribute to $A^{k}$ only while $k < s(0)$ , and a term $\varphi(k)\,0^{k}$ could not record that contribution. Hence $A^{k} \to 0$ if and only if every eigenvalue satisfies $|\lambda| < 1$ ; and $A^{k}$ stays bounded as $k \to \infty$ if and only if every eigenvalue satisfies $|\lambda| \le 1$ and every eigenvalue with $|\lambda| = 1$ has $s(\lambda) = 1$ . The same formula with $f(z) = e^{tz}$ gives the polynomial-times-exponential shape of $\exp(tA)$ ; the computational route is in the matrix exponential.

ProofOriginal

$\lambda I$ commutes with $N_m$ , so the binomial theorem holds for $(\lambda I + N_m)^{k}$ exactly as for numbers — the usual proof by induction only ever needs that the two terms commute. Terms with $i \ge m$ drop out because $N_m^{m} = 0$ , and the sum runs only to $i = k$ , so no term carries a negative power of $\lambda$ . The entry form follows from $(N_m^{i})_{pq} = 1$ when $q = p + i$ and $0$ otherwise, which is the statement that $N_m^{i}$ shifts the basis $i$ places: the position $(p,q)$ receives a contribution from the single term $i = q - p$ when $0 \le q - p \le \min(k, m-1)$ , and from no term at all otherwise, which is why the entry vanishes for $q < p$ and for $q - p > k$ whatever $\lambda$ may be.

For the power series, substitute $z = J_m(\lambda)$ into $f(z) = \sum_i c_i(z-\lambda)^{i}$ , so that $(z - \lambda)^i$ becomes $N_m^{i}$ , which vanishes for $i \ge m$ ; only $i \le m - 1$ survives, the sum is finite, and $c_i = f^{(i)}(\lambda)/i!$ is Taylor's formula. (Convergence of the series is therefore not at issue for a single block, though it is for the eigenvalue to lie in the disc of convergence of $f$ at all.)

For $A = PJP^{-1}$ we get $A^{k} = PJ^{k}P^{-1}$ , and $J^k$ is block diagonal with the blocks $J_{s_i}(\nu_i)^{k}$ just computed. Every entry of $A^{k}$ is therefore a fixed linear combination of the entries of the blocks, each of which is $\binom{k}{i}\lambda^{k-i}$ for some $i \le s(\lambda) - 1$ ; and $\binom{k}{i} = k(k-1)\cdots(k-i+1)/i!$ is a polynomial in $k$ of degree $i$ . Grouping by eigenvalue and writing $\binom{k}{i}\lambda^{k-i} = \bigl(\binom{k}{i}\lambda^{-i}\bigr)\lambda^{k}$ gives the stated form with $\deg \varphi^{\lambda}_{pq} \le s(\lambda) - 1$ . That rewriting needs $\lambda \neq 0$ , which is why the eigenvalue $0$ is excluded from the sum; once $k \ge s(0)$ every block at $0$ has already died, since $N_s^{k} = 0$ for $s \le s(0) \le k$ , so the omitted blocks contribute nothing and the formula is exact from that $k$ on.

For the limit: if every $|\lambda| < 1$ then each $k^{i}|\lambda|^{k} \to 0$ , since the geometric factor beats any polynomial, so $A^{k} \to 0$ . Conversely, if $Ax = \lambda x$ with $x \neq 0$ then $A^{k}x = \lambda^{k}x$ , which tends to $0$ only if $|\lambda| < 1$ . For boundedness both conditions are necessary, and here $J^{k} = P^{-1}A^{k}P$ lets us argue on $J$ : the two are bounded or unbounded together. Suppose $A^{k}$ is bounded. If some eigenvalue had $|\lambda| > 1$ , an eigenvector $x \neq 0$ would give $\lVert A^{k}x \rVert = |\lambda|^{k}\lVert x\rVert \to \infty$ ; so every eigenvalue has $|\lambda| \le 1$ . If some eigenvalue with $|\lambda| = 1$ had $s(\lambda) = m \ge 2$ , the entry $\binom{k}{m-1}\lambda^{k-m+1}$ in the corner of a largest block for that $\lambda$ would have modulus $\binom{k}{m-1} \to \infty$ , making $J^{k}$ and hence $A^{k}$ unbounded; so every eigenvalue of modulus $1$ has $s(\lambda) = 1$ . Conversely, if every eigenvalue of modulus $1$ has $s(\lambda) = 1$ and all others have $|\lambda| < 1$ , each entry of $J^{k}$ is either $\lambda^{k}$ with $|\lambda| = 1$ or one of the terms $\binom{k}{i}\lambda^{k-i}$ with $|\lambda| < 1$ , which tend to $0$ ; so $J^{k}$ , and with it $A^{k}$ , is bounded. $\square$

Remark(Where the hypotheses are used)AdaptedAdapted from Hirsch, Smale & Devaney (2013), §5.5, the Proposition on p. 94, and §5.6, p. 100; Grimmett & Stirzaker (2020), §6.6, p. 270

The field must contain the eigenvalues. This hypothesis of Dummit and Foote (Dummit & Foote (2004), Thm 23, p. 493) enters at step 2, where an eigenvalue $\lambda_1 \in F$ is needed to start the peeling, and again in the induction. It is not a formality: over $F = \mathbb{R}$ the rotation $\begin{pmatrix} 0 & -1 \\ 1 & 0\end{pmatrix}$ has characteristic polynomial $x^2 + 1$ , no real eigenvalue and hence no real Jordan form, although over $\mathbb{C}$ it is $\operatorname{diag}(i, -i)$ . What a real matrix does have is the real canonical form of Hirsch, Smale and Devaney (Hirsch, Smale & Devaney (2013), §5.5, Proposition, p. 94): a block diagonal real matrix whose blocks are either real Jordan blocks or, for a conjugate pair $\alpha \pm i\beta$ with $\beta \neq 0$ , block-upper-bidiagonal matrices built from $C_2 = \begin{pmatrix} \alpha & \beta \\ -\beta & \alpha \end{pmatrix}$ on the diagonal and $2 \times 2$ identity matrices one place above. One obtains it from the complex Jordan form by replacing each conjugate pair of chains with the real and imaginary parts of one of them.

Uniqueness is only up to the order of the blocks, and $P$ is far from unique. Even for $A = \lambda I$ , every invertible $P$ works. In general, scaling a whole chain by a non-zero constant, or swapping two chains of the same length for the same eigenvalue, changes $P$ and leaves $J$ alone; the form is an invariant of $A$ , the basis is not.

The eigenvalues and the geometric multiplicities do not determine $J$ . Blocks of sizes $(3,1)$ and $(2,2)$ for the same $\lambda$ both give algebraic multiplicity $4$ and geometric multiplicity $2$ ; they are told apart only by $d_2(\lambda)$ , which is $3$ in the first case and $4$ in the second. This is why the structure statement needs the whole sequence $d_1, d_2, \dots$ and not just $d_1$ .

The form is discontinuous in $A$ , so it cannot be computed in floating point. For $\varepsilon \neq 0$ the matrix $\begin{pmatrix} 0 & 1 \\ \varepsilon & 0\end{pmatrix}$ has distinct eigenvalues $\pm\sqrt{\varepsilon}$ and is diagonalizable; at $\varepsilon = 0$ it is a single $2 \times 2$ Jordan block. Arbitrarily small perturbations therefore change the answer completely, and the ranks $\operatorname{rank}(A - \lambda I)^{j}$ that the method below uses are exactly the quantities that rounding destroys. Jordan form is a tool for exact arithmetic and for theory; numerical work uses the Schur form or the singular value decomposition instead, and the relevant sensitivity is measured by condition numbers. Hirsch, Smale and Devaney (Hirsch, Smale & Devaney (2013), §5.6, p. 100) make the complementary point that a matrix met at random has distinct eigenvalues, so the non-trivial Jordan structure that matters in practice arises from symmetry or from a tuned parameter, not by accident.

Normal matrices never need it. A normal matrix is unitarily diagonalizable (Normal matrices and simultaneous diagonalization), so all its Jordan blocks have size $1$ ; non-trivial Jordan structure is a strictly non-normal phenomenon.

Conventions. Many texts, including Hassani (Hassani (2013), §17.7, p. 539), write the $1$ s on the superdiagonal as here; others put them on the subdiagonal, which is the transpose and equally valid. Some books, among them Grimmett and Stirzaker (Grimmett & Stirzaker (2020), §6.6, p. 270), present the form with one block per distinct eigenvalue, of size equal to that eigenvalue's multiplicity; that is the special case in which each eigenvalue has geometric multiplicity $1$ , and part 1 of the structure statement is what allows several blocks in general. We follow the several-block statement of Dummit and Foote (Dummit & Foote (2004), Thm 23, p. 493) throughout.

Method(Computing the Jordan form and a Jordan basis)AdaptedAdapted from Hirsch, Smale & Devaney (2013), §5.5, the proof on pp. 95-96 and the worked matrices on pp. 96-99

Given an $n \times n$ matrix $A$ over $\mathbb{C}$ with exact entries:

Eigenvalues. Factor the characteristic polynomial $\chi_A$ and record each distinct eigenvalue $\lambda$ with its algebraic multiplicity $a(\lambda)$ .
Nullities. For each $\lambda$ , compute $r_j = \operatorname{rank}(A - \lambda I)^{j}$ for $j = 1, 2, \dots$ until $r_j$ stops decreasing, and set $d_j = n - r_j$ . Stop at the first $j$ with $d_j = a(\lambda)$ ; this $j$ is the largest block size for $\lambda$ .
Block sizes. The number of blocks for $\lambda$ of size at least $j$ is $d_j - d_{j-1}$ . Equivalently, the list $d_1 - d_0,\ d_2 - d_1,\ \dots$ (automatically non-increasing, since a block of size $\ge j+1$ has size $\ge j$ ) is the conjugate partition of the list of block sizes. Check that the sizes sum to $a(\lambda)$ .
Chains, longest first. For the largest size $s$ , pick $v \in \ker(A - \lambda I)^{s}$ not in $\ker(A - \lambda I)^{s-1}$ and generate the chain downwards by repeated multiplication:

v,\quad (A - \lambda I)v,\quad (A-\lambda I)^2 v,\ \dots,\ (A - \lambda I)^{s-1}v .

Every later chain for the same $\lambda$ , whether another of length $s$ or one of a smaller length $s'$ , is started from a vector chosen by one and the same rule. Abbreviate $T = A - \lambda I$ and let $U$ be the span of all chain vectors produced so far; then take

v \in \ker T^{s'} \qquad\text{with}\qquad v \notin \ker T^{s'-1} + U ,

and run the chain $v, Tv, \dots, T^{s'-1}v$ downwards as before.

The second condition is about the sum of the two subspaces, not about each of them. Requiring only $v \notin \ker T^{s'-1}$ and $v \notin U$ is genuinely too weak. Take $\lambda = 0$ and $A$ the direct sum of $J_3(0)$ and $J_2(0)$ , with the chains $a_1, a_2, a_3$ and $b_1, b_2$ as its basis, so $Aa_3 = a_2$ , $Aa_2 = a_1$ , $Aa_1 = 0$ , $Ab_2 = b_1$ , $Ab_1 = 0$ . Steps 1 to 3 give $d_1 = 2$ , $d_2 = 4$ , $d_3 = 5$ and block sizes $3, 2$ . Choosing $v = a_3$ produces $U = \operatorname{span}\{a_1, a_2, a_3\}$ , and then $v' = a_2 + b_1$ satisfies all three of the weaker conditions — $A^{2}v' = 0$ , $Av' = a_1 \neq 0$ , $v' \notin U$ — yet its chain is $a_1,\ a_2 + b_1$ , which repeats $a_1$ : the assembled list has rank $4$ , not $5$ , and $P$ is singular. The rule above rejects $v'$ , since $a_2 + b_1 \in U + \ker A$ , and accepts $b_2$ , which is the correct choice.

Why the rule is the right one. For $v \in \ker T^{s'}$ , the condition $v \notin \ker T^{s'-1} + U$ says precisely that the bottom vector $T^{s'-1}v$ of the new chain is not in the span of the bottom vectors $T^{s_i-1}v_i$ of the chains already built. (All chains built so far have length $s_i \ge s'$ , since the sizes are taken in decreasing order. Suppose $v = y + u$ with $T^{s'-1}y = 0$ and $u \in U$ . From $T^{s'}v = 0$ and $T^{s'}y = 0$ we get $T^{s'}u = 0$ ; expanding $u = \sum_i\sum_{t < s_i} a_{i,t}T^{t}v_i$ in the chain vectors, the only terms of $T^{s'}u$ that survive are those with $t < s_i - s'$ , and they are distinct members of the chain list, so $a_{i,t} = 0$ for every $t < s_i - s'$ . Applying $T^{s'-1}$ to what remains kills every term with $t > s_i - s'$ and gives $T^{s'-1}v = T^{s'-1}u = \sum_i a_{i,\,s_i - s'}\,T^{s_i - 1}v_i$ , a combination of bottom vectors. Conversely, if $T^{s'-1}v = \sum_i c_i T^{s_i - 1}v_i$ , then $v - \sum_i c_i T^{s_i - s'}v_i$ lies in $\ker T^{s'-1}$ , so $v \in \ker T^{s'-1} + U$ .) And chains whose bottom vectors are independent are jointly independent: in a vanishing combination $\sum_i \sum_{t < s_i} a_{i,t}T^{t}v_i = 0$ with some coefficient non-zero, let $c_i$ be the least $t$ with $a_{i,t} \neq 0$ and put $q = \max_i (s_i - c_i)$ over the chains that occur; applying $T^{q-1}$ annihilates every term except those with $s_i - c_i = q$ and $t = c_i$ , and leaves a vanishing combination of distinct bottom vectors with a non-zero coefficient — impossible. Finally, a legal choice is available exactly as often as step 3 predicts: the vectors $T^{s_i - s'}v_i$ taken from the chains already built have independent images in $\ker T^{s'}/\ker T^{s'-1}$ (apply $T^{s'-1}$ and use the independence of the bottoms), that quotient has dimension $d_{s'} - d_{s'-1}$ , the number of blocks of size at least $s'$ , and by the equivalence just proved $\ker T^{s'-1} + U$ meets $\ker T^{s'}$ in exactly the preimage of the span of those images. So the supply of choices runs out only when the chains of length $\ge s'$ already number $d_{s'} - d_{s'-1}$ .

Equivalently, and this is the device Hirsch, Smale and Devaney (Hirsch, Smale & Devaney (2013), §5.5, pp. 95-96) use in their worked $3\times3$ and $4\times4$ examples, start from an eigenvector and back-solve $(A - \lambda I)x_{t} = x_{t-1}$ , going up the chain instead of down. 5. Assemble. Write each chain as consecutive columns ending with the top vector $v$ , that is in the order $(A-\lambda I)^{s-1}v, \dots, (A - \lambda I)v, v$ ; concatenate the chains over all eigenvalues to form $P$ . Then $J = P^{-1}AP$ , with the blocks in the order the chains were laid down. 6. Check without inverting $P$ : verify $AP = PJ$ column by column, and verify $\operatorname{tr}A = \operatorname{tr}J$ and $\det A = \det J$ .

If only the form is wanted — enough for powers, growth rates and the minimal polynomial — stop after step 3; the eigenvectors are needed only for $P$ . Never run this on inexact data: see the remark above.

Example(A defective 4 x 4 matrix and its powers)Original

Let

A = \begin{pmatrix} 2 & -1 & 1 & -1 \\ -1 & 2 & 1 & -1 \\ 2 & 2 & 1 & -2 \\ 4 & 4 & -4 & 3 \end{pmatrix} .

Step 1: eigenvalues. Expanding $\det(xI - A)$ gives

\chi_A(x) = x^4 - 8x^3 + 18x^2 - 27 = (x-3)^3 (x+1) ,

the factorization being confirmed by multiplying out: $(x^3 - 9x^2 + 27x - 27)(x+1) = x^4 - 8x^3 + 18x^2 - 27$ , the linear terms $-27x$ and $+27x$ cancelling. The two standard tests agree: $\operatorname{tr}A = 2 + 2 + 1 + 3 = 8 = 3 + 3 + 3 - 1$ , and $\det A = \chi_A(0) = -27 = 3\cdot3\cdot3\cdot(-1)$ . So $a(3) = 3$ and $a(-1) = 1$ .

Step 2: nullities at $\lambda = 3$ . With $B = A - 3I$ ,

B = \begin{pmatrix} -1 & -1 & 1 & -1 \\ -1 & -1 & 1 & -1 \\ 2 & 2 & -2 & -2 \\ 4 & 4 & -4 & 0 \end{pmatrix} , \qquad B^2 = \begin{pmatrix} 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \\ -16 & -16 & 16 & 0 \\ -16 & -16 & 16 & 0 \end{pmatrix} .

Call the rows of $B$ in order $R_1, \dots, R_4$ . Then $R_2 = R_1$ , and subtracting multiples of $R_1$ from the last two rows leaves the same vector twice, $R_3 + 2R_1 = R_4 + 4R_1 = (0,0,0,-4)$ , so $R_4 = R_3 - 2R_1$ . The row space is therefore spanned by $R_1$ and $R_3$ , which are independent because $R_3$ has a non-zero fourth entry after $R_1$ has been cleared from it: $\operatorname{rank}B = 2$ and $d_1(3) = 2$ . For $B^2$ the two non-zero rows are equal, so $\operatorname{rank}B^2 = 1$ and $d_2(3) = 3 = a(3)$ ; the chain of kernels has stalled.

Step 3: block sizes. The number of blocks for $3$ is $d_1 = 2$ ; the number of size at least $2$ is $d_2 - d_1 = 1$ ; the number of size at least $3$ is $d_3 - d_2 = 0$ . So the blocks for $3$ have sizes $2$ and $1$ , summing to $3 = a(3)$ as required. The eigenvalue $-1$ is simple, giving one block of size $1$ . Hence

J = \begin{pmatrix} 3 & 1 & 0 & 0 \\ 0 & 3 & 0 & 0 \\ 0 & 0 & 3 & 0 \\ 0 & 0 & 0 & -1 \end{pmatrix} ,

and the minimal polynomial is $\mu_A(x) = (x-3)^2(x+1)$ , the largest block size at $3$ being $2$ . Indeed $B^2(A+I) = 0$ while $B(A+I) \neq 0$ : that product is $\begin{pmatrix} -4 & -4 & 4 & -4 \\ -4 & -4 & 4 & -4 \\ -8 & -8 & 8 & -8 \\ 0 & 0 & 0 & 0\end{pmatrix}$ .

Step 4: the chains. Take the top of the long chain to be $v = (0,0,0,1)^{\mathsf T}$ , which lies in $\ker B^2$ (the fourth column of $B^2$ is zero) but not in $\ker B$ (the fourth column of $B$ is $(-1,-1,-2,0)^{\mathsf T} \neq 0$ ). The chain is

x_1 = Bv = (-1,-1,-2,0)^{\mathsf T}, \qquad x_2 = v = (0,0,0,1)^{\mathsf T} ,

and $Bx_1 = 0$ , as the four rows confirm: $1 + 1 - 2 + 0 = 0$ from $R_1$ and again from $R_2$ , then $-2 - 2 + 4 + 0 = 0$ from $R_3$ , then $-4 - 4 + 8 + 0 = 0$ from $R_4$ .

For the second block at $3$ we need an eigenvector independent of $x_1$ . Write a general vector as $\xi = (\xi_1, \xi_2, \xi_3, \xi_4)^{\mathsf T}$ and solve $B\xi = 0$ : rows 1 and 2 give $-\xi_1 - \xi_2 + \xi_3 - \xi_4 = 0$ and row 4 gives $\xi_1 + \xi_2 - \xi_3 = 0$ , whence $\xi_4 = 0$ and $\xi_3 = \xi_1 + \xi_2$ ; row 3 is then automatic. A basis of $\ker B$ is $(-1,1,0,0)^{\mathsf T}$ and $(1,0,1,0)^{\mathsf T}$ , and we may take

w = (-1, 1, 0, 0)^{\mathsf T} ,

which is independent of $x_1 = -(-1,1,0,0)^{\mathsf T} - 2(1,0,1,0)^{\mathsf T}$ .

For $\lambda = -1$ : $(A + I)\xi = 0$ has the one-dimensional solution space spanned by $z = (0,0,1,1)^{\mathsf T}$ , as one checks on the first row, $2\cdot 0 - 1\cdot 0 + 1 \cdot 1 - 1 \cdot 1 = 0$ , and similarly on the others.

Step 5: assemble and check. Laying the chain down first,

P = \begin{pmatrix} -1 & 0 & -1 & 0 \\ -1 & 0 & 1 & 0 \\ -2 & 0 & 0 & 1 \\ 0 & 1 & 0 & 1\end{pmatrix}, \qquad \det P = -2 \neq 0 .

The identity $AP = PJ$ holds column by column: $Ax_1 = 3x_1$ , $Av = 3v + x_1$ (which is $Bv = x_1$ , verified above), $Aw = 3w$ and $Az = -z$ . Hence $P^{-1}AP = J$ .

Powers. By the block-power formula, $J^{k} = \operatorname{diag}\bigl(J_2(3)^k,\ 3^{k},\ (-1)^{k}\bigr)$ with $J_2(3)^{k} = \begin{pmatrix} 3^{k} & k3^{k-1} \\ 0 & 3^{k}\end{pmatrix}$ , so $A^{k} = PJ^{k}P^{-1}$ splits into three fixed matrices:

A^{k} = 3^{k}\,\Pi + k\,3^{k-1} M + (-1)^{k}\,\Pi' ,

\Pi = \begin{pmatrix} 1&0&0&0\\0&1&0&0\\1&1&0&0\\1&1&-1&1\end{pmatrix}, \quad M = \begin{pmatrix} -1&-1&1&-1\\-1&-1&1&-1\\-2&-2&2&-2\\0&0&0&0\end{pmatrix}, \quad \Pi' = \begin{pmatrix} 0&0&0&0\\0&0&0&0\\-1&-1&1&0\\-1&-1&1&0\end{pmatrix} .

Here $\Pi$ and $\Pi'$ are the projections onto $G_3$ and $G_{-1}$ along each other ( $\Pi^2 = \Pi$ , $\Pi'^2 = \Pi'$ , $\Pi + \Pi' = I$ ) and $M = B\Pi$ is the nilpotent part, $M^2 = 0$ . Note that $\Pi$ is not an orthogonal projection: $A$ is not normal, and nothing forces the two generalized eigenspaces to be perpendicular.

Checking the formula. At $k = 1$ it returns $3\Pi + M - \Pi' = A$ ; for instance the $(3,1)$ entry is $3 \cdot 1 + 1\cdot(-2) - (-1) = 2$ , which is $A_{31}$ . At $k = 2$ , squaring $A$ directly gives

A^2 = \begin{pmatrix} 3 & -6 & 6 & -6 \\ -6 & 3 & 6 & -6 \\ -4 & -4 & 13 & -12 \\ 8 & 8 & -8 & 9\end{pmatrix} ,

and the formula gives for the $(3,1)$ entry $9 \cdot 1 + 2 \cdot 3 \cdot (-2) + (-1) = 9 - 12 - 1 = -4$ , which agrees.

A number to check against. Take $k = 10$ . Then

\bigl(A^{10}\bigr)_{31} = 3^{10}\cdot 1 + 10\cdot 3^{9}\cdot(-2) + 1 \cdot(-1) = 59049 - 393660 - 1 = -334612 ,

which is what repeated squaring of $A$ produces. The structure is visible in the arithmetic: the term $-393660$ , carrying the factor $k$ , already dominates $3^{10} = 59049$ at $k = 10$ , and the whole entry grows like $-2k\,3^{k-1}$ rather than like $3^{k}$ . That extra factor of $k$ is the single $2 \times 2$ Jordan block, and nothing in the spectrum $\{3, -1\}$ alone would have predicted it.

ApplicationOriginal

How fast a finite Markov chain forgets its start. For a finite irreducible aperiodic chain with transition matrix $S$ and stationary distribution $\pi$ , the $n$ -step probabilities are the entries of $S^{n}$ . When the eigenvalues are distinct one diagonalizes, and Grimmett and Stirzaker (Grimmett & Stirzaker (2020), §6.6, p. 270) read off the geometric rate: with $\varrho$ the largest modulus among the eigenvalues other than $1$ (they write $\mu$ for it), there is a constant $c$ with $|p_{ij}(n) - \pi_j| \le c\,\varrho^{\,n}$ . They then note that repeated eigenvalues block this route, and that the Jordan form takes its place: $S = B^{-1}JB$ in our notation (they write $M$ for the Jordan matrix), with $S^{n} = B^{-1}J^{n}B$ . Their Perron-Frobenius theorem (Grimmett & Stirzaker (2020), §6.6, Thm (1), p. 269) lists the eigenvalue $1$ once for an irreducible aperiodic chain and puts every other eigenvalue strictly inside the unit circle, so the block at $1$ is $1 \times 1$ and only the other blocks can be large. The block-power formula of this unit then says exactly what changes: an eigenvalue $\lambda \neq 1$ whose largest block has size $s$ contributes terms $\binom{n}{i}\lambda^{n-i}$ with $i \le s-1$ , so with $s$ the largest block size over all eigenvalues other than $1$ the clean bound acquires a polynomial factor and becomes $|p_{ij}(n) - \pi_j| \le c\,n^{s-1}\varrho^{\,n}$ . Since $n^{s-1}\varrho^{\,n} \le c'\,\tau^{\,n}$ for any $\tau$ strictly between $\varrho$ and $1$ , geometric convergence survives, but the pre-asymptotic behaviour does not: for $n$ up to about $(s-1)/\log(1/\varrho)$ the polynomial is still winning, and a chain with a large Jordan block mixes visibly more slowly than its second eigenvalue suggests. This is the discrete-time counterpart of the warning that the spectrum alone does not control $\|A^{k}\|$ .

Defective linear systems and the factor of $t$ . For $\dot x = Ax$ with a repeated eigenvalue and too few eigenvectors, the missing solutions are supplied by the chain. Lee (Lee (2013), Appendix D, pp. 673-674) gives the $2 \times 2$ case explicitly: when $A$ has the single eigenvalue $\lambda$ with eigenvector $Z_0$ and $A - \lambda I \neq 0$ , there is a generalized eigenvector $Z_1$ with $(A - \lambda I)Z_1 = Z_0$ , and the second independent solution is $e^{\lambda t}(tZ_0 + Z_1)$ . In the language above, $Z_0, Z_1$ is a Jordan chain of length two, and the solution is the second column of $P\exp(tJ)$ . The same bookkeeping in general: a chain $x_1, \dots, x_s$ for $\lambda$ carries the $s$ solutions $u_r(t) = e^{\lambda t}\sum_{i=0}^{r-1}(t^{i}/i!)\,x_{r-i}$ for $r = 1, \dots, s$ , polynomial in $t$ of degree up to $s-1$ . Differentiating $u_r$ and using $Ax_{r-i} = \lambda x_{r-i} + x_{r-i-1}$ (with $x_0 = 0$ ) reproduces $\dot u_r = Au_r$ term by term. Physically this is where secular terms come from — critically damped oscillators, resonant forcing, marginal stability in a linearized flow — and it is why a stability analysis must look at the Jordan structure of any eigenvalue sitting on the imaginary axis, not merely at its real part. See linear systems for the solution theory and the matrix exponential for computing $\exp(tA)$ in practice.

Perturbation theory near a degeneracy. In quantum mechanics the Hamiltonian is Hermitian and therefore never defective, but the matrices that govern open and driven systems are not. An effective non-Hermitian Hamiltonian describing a resonance, a gain-loss optical cavity or a damped coupled oscillator typically has isolated parameter values — exceptional points — where two eigenvalues and their eigenvectors coalesce, which is precisely a $2 \times 2$ Jordan block. The splitting of the two eigenvalues near such a point goes like $\sqrt{\varepsilon}$ in the detuning $\varepsilon$ , not like $\varepsilon$ , exactly as in the matrix $\begin{pmatrix} 0 & 1 \\ \varepsilon & 0 \end{pmatrix}$ of the remark above, and that square-root response is what makes exceptional points attractive as sensors. The Jordan form is the right language here because the whole phenomenon is the failure of diagonalizability at one point of parameter space.

A reason to stop short of step 4. Because steps 1 to 3 use only ranks and the characteristic polynomial, the Jordan form is often all one computes: it is what decides whether $A^{k}$ or $\exp(tA)$ carries polynomial factors, what the minimal polynomial is (hence how many powers of $A$ are genuinely independent, which bounds the length of a Krylov recurrence), and whether a projection or involution identity can hold. The eigenvectors, which cost far more, are needed only when the solution itself, and not its shape, is wanted.

Quadratic forms, SVD and matrix functions

Method

Quadratic forms and the Rayleigh quotient

Adapted

Add to PDF

How to reduce a quadratic or Hermitian form to its principal axes, decide its definiteness from the eigenvalue signs or from Sylvester's criterion on the leading principal minors, and bound the eigenvalues by the Rayleigh quotient.

Prerequisites: Spectral theorem for Hermitian matrices, Determinants, Unitary and orthogonal matrices, Orthonormal bases and Gram-Schmidt, Vector spaces and bases

Definition(Quadratic and Hermitian forms, definiteness)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.17, Eqs (8.105)-(8.112), pp. 288-290

A quadratic form on $\mathbb{R}^n$ is a function $Q(x) = x^{\mathsf T} A x$ with $A$ a real $n \times n$ matrix. Only the symmetric part of $A$ contributes: writing $A = \tfrac12(A + A^{\mathsf T}) + \tfrac12(A - A^{\mathsf T})$ , the antisymmetric part $B$ satisfies $x^{\mathsf T} B x = 0$ for every $x$ , because $x^{\mathsf T} B x$ is a scalar and equals its own transpose $-x^{\mathsf T} B x$ . We therefore always take $A = A^{\mathsf T}$ , at the cost of no generality; the symmetric matrix representing a given form is then unique. The complex analogue is the Hermitian form $H(x) = x^\dagger A x$ with $A = A^\dagger$ , which is real-valued for every $x \in \mathbb{C}^n$ since $\overline{H(x)} = x^\dagger A^\dagger x = H(x)$ .

A Hermitian (or real symmetric) $A$ and its form are called

positive definite if $x^\dagger A x > 0$ for all $x \neq 0$ ;
positive semidefinite if $x^\dagger A x \ge 0$ for all $x$ ;
negative definite / negative semidefinite if $-A$ is positive definite / semidefinite;
indefinite if $x^\dagger A x$ takes both signs.

For $1 \le k \le n$ , the $k$ -th leading principal minor of $A$ is $D_k = \det A^{(k)}$ , where $A^{(k)}$ is the top-left $k \times k$ submatrix of $A$ . A principal minor is the determinant of any submatrix obtained by keeping the same set of rows and columns.

Method(Principal axes)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.17, Eqs (8.110)-(8.111), p. 289 and §8.17.2, p. 292

Let $A$ be real symmetric $n \times n$ with eigenvalues $\lambda_1, \dots, \lambda_n$ (with multiplicity) and a real orthonormal eigenbasis $u_1, \dots, u_n$ , as supplied by the spectral theorem, and let $S = (u_1 \ \cdots \ u_n)$ , so $S^{\mathsf T} A S = \Lambda$ . Substituting $x = S x'$ gives

Q(x) = x^{\mathsf T} A x = (x')^{\mathsf T} \Lambda x' = \lambda_1 x_1'^2 + \cdots + \lambda_n x_n'^2 ,

a sum of squares with no cross terms; the new axes, along the $u_k$ , are the principal axes of the form. The same holds for a Hermitian form with $S$ unitary and $x_k'^2$ replaced by $|x_k'|^2$ .

The level set $L = \{x : x^{\mathsf T} A x = 1\}$ is non-empty if and only if at least one $\lambda_k$ is positive. If some $\lambda_k > 0$ , then $L$ meets the line $\mathbb{R}u_k$ , on which the form takes the values $\lambda_k t^2$ , at $t = \pm \lambda_k^{-1/2}$ . If instead every $\lambda_k \le 0$ , then $Q(x) = \sum_k \lambda_k x_k'^2 \le 0 < 1$ for every $x$ , not merely on the axis lines, so $L$ is empty.

Assume $L \neq \emptyset$ and write $p \ge 1$ , $q$ , $z$ for how many of the eigenvalues are respectively positive, negative and zero, so $p + q + z = n$ . Then, with the axes taken along the $u_k$ :

$z = 0$ and $q = 0$ (so $p = n$ ): $L$ is an ellipsoid, compact, with semi-axis $\lambda_k^{-1/2}$ in the direction $u_k$ .
$z = 0$ and $q \ge 1$ : $L$ is a hyperboloid, unbounded — for $n = 2$ a hyperbola.
$z \ge 1$ : $L$ is a cylinder. It is invariant under translation by any multiple of a $u_k$ with $\lambda_k = 0$ , because such a translation does not change $x_k'$ 's contribution to $Q$ ; its cross-section in the span of the $u_k$ with $\lambda_k \neq 0$ is the ellipsoid ( $q = 0$ ) or hyperboloid ( $q \ge 1$ ) of those $p + q$ eigenvalues.

The three cases are disjoint, and the indices $k$ with $\lambda_k \le 0$ are exactly those for which $L$ misses the axis line $\mathbb{R}u_k$ .

Both names are meant in their degenerate senses as well. An "ellipsoid" in $n = 1$ is the two points $\pm\lambda_1^{-1/2}$ and in $n = 2$ an ellipse; a "cylinder" is the product of a cross-section with the $z$ -dimensional space spanned by the null directions, so for $n = 2$ and $\lambda = (1, 0)$ it is the pair of parallel lines $x_1' = \pm 1$ , and for $z = n - 1$ , $p = 1$ a pair of parallel hyperplanes.

Idea of the proofOriginal

The spectral theorem supplies an orthogonal change of coordinates in which $A$ becomes diagonal, and a diagonal matrix has no cross terms: in those coordinates the form is $\lambda_1 x_1'^2 + \cdots + \lambda_n x_n'^2$ , one independent contribution per axis. Because the change of coordinates is orthogonal it changes no length and no angle, so every geometric statement about the level set may be read off in the new coordinates and transported back unchanged. The classification is then a one-variable question asked $n$ times: a positive $\lambda_k$ confines the level set in the $u_k$ direction, a negative one lets it run off to infinity, and a zero one lets it slide freely.

ProofAdaptedAdapted from Riley, Hobson & Bence (2006), §8.17, Eqs (8.109)-(8.111), pp. 288-289 and §8.17.2, p. 292

Step 1: the change of coordinates. Since $A$ is real symmetric, part 6 of the spectral theorem gives a real orthonormal eigenbasis $u_1, \dots, u_n$ and makes $S = (u_1\ \cdots\ u_n)$ real orthogonal with $S^{\mathsf T}AS = \Lambda$ . Then $S^{-1} = S^{\mathsf T}$ , so $x = Sx'$ and $x' = S^{\mathsf T}x$ say the same thing, and $x \mapsto x'$ is a bijection of $\mathbb{R}^n$ that preserves the norm, by part 4 of the characterisation in Unitary and orthogonal matrices. Substituting,

x^{\mathsf T} A x = (Sx')^{\mathsf T}A(Sx') = (x')^{\mathsf T}(S^{\mathsf T}AS)x' = (x')^{\mathsf T}\Lambda x' = \sum_{k=1}^{n}\lambda_k (x_k')^2 ,

with $x_k' = u_k^{\mathsf T}x$ the component of $x$ along $u_k$ . The hypothesis that $A$ is symmetric enters only through the spectral theorem — it is what makes the eigenbasis real and orthonormal, hence $S$ orthogonal rather than merely invertible, and that is what keeps the geometry undistorted. For a Hermitian form the same lines with $S$ unitary, $\dagger$ for $\mathsf T$ , give $x^\dagger A x = \sum_k \lambda_k |x_k'|^2$ , which is real because the $\lambda_k$ are.

Step 2: the axis lines and non-emptiness. On the line $\mathbb{R}u_k$ a point is $x = tu_k$ , whose coordinates are $x' = te_k$ because $u_j^{\mathsf T}u_k = \delta_{jk}$ ; so $Q(tu_k) = \lambda_k t^2$ . The equation $\lambda_kt^2 = 1$ has a real solution exactly when $\lambda_k > 0$ , and then exactly the two solutions $t = \pm\lambda_k^{-1/2}$ . This proves both that $L$ meets $\mathbb{R}u_k$ precisely for $\lambda_k > 0$ , at those two parameter values, and that $L \neq \emptyset$ as soon as some $\lambda_k > 0$ . Conversely if every $\lambda_k \le 0$ then $Q(x) = \sum_k\lambda_k(x_k')^2 \le 0 < 1$ for every $x$ , so $L = \emptyset$ .

Step 3: adapted coordinates. Assume $L \neq \emptyset$ and renumber the eigenvalues, carrying the columns of $S$ with them, so that $\lambda_1, \dots, \lambda_p > 0$ , then $q$ negative ones, then $z$ zeros, with $p \ge 1$ by step 2. In the $x'$ coordinates

L = \Bigl\{x' : \sum_{k \le p}\lambda_k(x_k')^2 - \sum_{p < k \le p+q}|\lambda_k|(x_k')^2 = 1\Bigr\}.

Because $x \mapsto x'$ is a norm-preserving bijection, boundedness, compactness, distances from the origin and the lines contained in $L$ are the same computed in either set of coordinates.

Step 4: $z = 0$ and $q = 0$ , so $p = n$ . Put $\lambda_{\min} = \min_k \lambda_k > 0$ . For $x \in L$ ,

\lambda_{\min}\|x'\|^2 \le \sum_k \lambda_k (x_k')^2 = 1 , \qquad\text{so}\qquad \|x\| = \|x'\| \le \lambda_{\min}^{-1/2} ,

and $L$ is bounded. It is also closed, being the preimage of the closed set $\{1\}$ under the continuous map $Q$ , hence compact. It is the ellipsoid with semi-axes $\lambda_k^{-1/2}$ : the diagonal map $y \mapsto x'$ with $x_k' = \lambda_k^{-1/2}y_k$ is a bijection of $\mathbb{R}^n$ carrying the unit sphere $\sum_k y_k^2 = 1$ exactly onto $L$ , since $\sum_k\lambda_k(x_k')^2 = \sum_k y_k^2$ ; and by step 2 its intersection with the $k$ -th axis is at distance $\lambda_k^{-1/2}$ from the origin.

Step 5: $z = 0$ and $q \ge 1$ . Here $\lambda_1 > 0$ and some $\lambda_m < 0$ . For each real $t$ let $x'(t)$ have $x_m' = t$ , $x_1' = \bigl((1 + |\lambda_m|t^2)/\lambda_1\bigr)^{1/2}$ and all other coordinates zero; the square root is of a positive number, so this is defined for every $t$ , and $\sum_k\lambda_k(x_k')^2 = (1 + |\lambda_m|t^2) - |\lambda_m|t^2 = 1$ . So $x'(t) \in L$ for all $t$ , while $\|x'(t)\| \ge |t| \to \infty$ : $L$ is unbounded. For $n = 2$ the defining equation is $\lambda_1(x_1')^2 - |\lambda_2|(x_2')^2 = 1$ , the standard equation of a hyperbola with respect to its own axes.

Step 6: $z \ge 1$ . Let $Z$ be the span of the $u_k$ with $\lambda_k = 0$ and $Y$ the span of the others, so $\mathbb{R}^n = Y \oplus Z$ orthogonally. The expression $\sum_k\lambda_k(x_k')^2$ contains no term in a coordinate with $\lambda_k = 0$ , so $Q(y + c u_k) = Q(y)$ whenever $\lambda_k = 0$ : translating by any vector of $Z$ moves no point of $L$ off $L$ , which is the stated invariance. Writing a general $x$ as $y + z$ with $y \in Y$ , $z \in Z$ gives $Q(x) = Q(y)$ , so

x \in L \iff y \in L \cap Y, \qquad \text{that is} \qquad L = (L \cap Y) + Z .

Now $L \cap Y$ is the level set at $1$ of the restriction of the form to the $(p+q)$ -dimensional space $Y$ , on which the eigenvalues are exactly the non-zero $\lambda_k$ with the same eigenvectors; steps 4 and 5 applied inside $Y$ make it an ellipsoid when $q = 0$ and a hyperboloid when $q \ge 1$ . So $L$ is that cross-section times $Z$ .

Step 7: disjointness and the degenerate readings. The triple $(p,q,z)$ counts the positive, negative and zero eigenvalues of $A$ , so it is determined by $A$ and exactly one of the three cases applies. For the names: in step 4 with $n = 1$ , $L = \{\pm\lambda_1^{-1/2}\}$ , a pair of points, and with $n = 2$ it is an ellipse, both being the image of the unit sphere of the relevant dimension under a diagonal stretch. In step 6 with $p + q = 1$ and that one eigenvalue positive, $L \cap Y$ is a pair of points at distance $\lambda^{-1/2}$ from the origin, so $L$ is a pair of parallel hyperplanes; for $n = 2$ with $\lambda = (1,0)$ the equation is $(x_1')^2 = 1$ , the two lines $x_1' = \pm 1$ . $\square$

Method(Eigenvalue test for definiteness)AdaptedAdapted from Hassani (2013), §6.4, Cor 6.4.10, p. 181; Riley, Hobson & Bence (2006), §8.17, the paragraph following Eq. (8.112), p. 290

Let $A$ be Hermitian $n \times n$ (in particular, real symmetric). Then $A$ is positive definite if and only if every eigenvalue of $A$ is positive, and positive semidefinite if and only if every eigenvalue is $\ge 0$ .

The strict half is (Riley, Hobson & Bence (2006), §8.17, p. 290), which states that $Q > 0$ for every $x$ forces every eigenvalue to be positive. The corresponding result in (Hassani (2013), Cor 6.4.10, p. 181) is the semidefinite one, because "positive" there is Hassani's non-strict notion (Def 4.3.9, p. 117); his strict notion is written $A > 0$ . The semidefinite half is the same reading of the sum of squares $Q = \sum_k \lambda_k x_k'^2$ with $\ge$ in place of $>$ , which is how (Riley, Hobson & Bence (2006), §8.17, p. 290) arrives at both terms. The consequence that $\operatorname{tr} A > 0$ and $\det A > 0$ for a positive definite $A$ is ours — it is immediate, since the trace is the sum and the determinant the product of the $n$ eigenvalues — and is on neither cited page, so do not attribute it there. It is only a one-way test: see the traps below.

Idea of the proofOriginal

In the principal-axis coordinates the form is the weighted sum $\sum_k \lambda_k |x_k'|^2$ with non-negative weights $|x_k'|^2$ , and such a sum is positive for every non-zero $x$ exactly when every coefficient is positive: if one $\lambda_j$ fails, the single eigenvector $u_j$ exhibits the failure, and if none fails, the smallest eigenvalue bounds the whole sum from below.

ProofAdaptedAdapted from Riley, Hobson & Bence (2006), §8.17, the paragraph following Eq. (8.112), p. 290

By the spectral theorem choose an orthonormal eigenbasis $u_1,\dots,u_n$ of $A$ with $Au_k = \lambda_k u_k$ and the $\lambda_k$ real, let $U$ be the unitary matrix of those columns and put $x' = U^\dagger x$ , so that by step 1 of the previous proof

x^\dagger A x = \sum_{k=1}^{n}\lambda_k |x_k'|^2, \qquad \|x'\| = \|x\| .

The map $x \mapsto x'$ is a bijection, so $x \neq 0$ if and only if $x' \neq 0$ .

All eigenvalues positive implies positive definite. Let $\lambda_{\min} = \min_k\lambda_k > 0$ and let $x \neq 0$ . Then

x^\dagger A x = \sum_k \lambda_k|x_k'|^2 \ \ge\ \lambda_{\min}\sum_k |x_k'|^2 = \lambda_{\min}\|x\|^2 \ >\ 0 ,

the first inequality because every $|x_k'|^2 \ge 0$ and the last because $\|x\| > 0$ .

Positive definite implies all eigenvalues positive. Suppose some $\lambda_j \le 0$ . The eigenvector $u_j$ is non-zero and $u_j^\dagger A u_j = \lambda_j u_j^\dagger u_j = \lambda_j \le 0$ , so the defining inequality fails at $x = u_j$ and $A$ is not positive definite.

The semidefinite half. If every $\lambda_k \ge 0$ then $x^\dagger A x = \sum_k\lambda_k|x_k'|^2 \ge 0$ for every $x$ , each term being a product of two non-negative numbers. If some $\lambda_j < 0$ then $u_j^\dagger A u_j = \lambda_j < 0$ . This is the same reading of the sum of squares with $\ge$ in place of $>$ , exactly as (Riley, Hobson & Bence (2006), §8.17, p. 290) obtains both terms.

The trace and determinant. If $A$ is positive definite then all $\lambda_k > 0$ , and part 5 of the spectral theorem gives $\operatorname{tr}A = \sum_k\lambda_k > 0$ as a sum of positive numbers and $\det A = \prod_k\lambda_k > 0$ as a product of them. Neither converse holds: $\operatorname{diag}(-1,-1)$ has trace $-2$ but determinant $+1$ , and $\operatorname{diag}(3,-1)$ has trace $+2$ with determinant $-3$ , so each of the two conditions can hold while $A$ fails to be positive definite. $\square$

Method(Sylvester's criterion)AdaptedAdapted from Gelfand & Fomin (1963), §30, p. 127

Let $A$ be real symmetric $n \times n$ , with leading principal minors $D_k = \det A^{(k)}$ for $k = 1, \dots, n$ . Then positive definiteness of the form $x^{\mathsf T} A x$ is equivalent to positivity of the whole descending chain

D_1 = a_{11} > 0, \qquad D_2 = \begin{vmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{vmatrix} > 0, \qquad \dots, \qquad D_n = \det A > 0 .

Each of the $n$ inequalities is needed, and no condition beyond them is.

Applying this to $-A$ and using $D_k(-A) = (-1)^k D_k(A)$ : $A$ is negative definite if and only if $(-1)^k D_k > 0$ for $k = 1, \dots, n$ , i.e. the leading principal minors alternate in sign beginning with $D_1 < 0$ .

Idea of the proofOriginal

One direction is easy: restricting a positive definite form to the first $k$ coordinates leaves it positive definite, so each $A^{(k)}$ has only positive eigenvalues and its determinant $D_k$ is positive. The other direction is an induction in which $A^{(n-1)}$ is already known to be positive definite. That means the form is positive on an $(n-1)$ -dimensional subspace, and two subspaces of $\mathbb{R}^n$ whose dimensions add to more than $n$ must share a non-zero vector; so $A$ cannot have two non-positive eigenvalues, because the plane they span would meet that subspace. At most one eigenvalue is therefore in doubt, and $D_n = \det A > 0$ settles its sign.

ProofOriginal

Throughout, $A^{(k)}$ is the top-left $k \times k$ block and $V_k = \operatorname{span}\{e_1,\dots,e_k\}$ . For $x \in V_k$ , only the first $k$ entries of $x$ are non-zero, so

x^{\mathsf T} A x = \sum_{i,j \le k} a_{ij}x_ix_j = \tilde x^{\mathsf T}A^{(k)}\tilde x , \qquad \tilde x = (x_1,\dots,x_k)^{\mathsf T} :

the form of $A^{(k)}$ is the form of $A$ restricted to $V_k$ . Each $A^{(k)}$ is symmetric, being a block of a symmetric matrix cut out by the same rows and columns.

Lemma (overlapping subspaces). If $V, W \subseteq \mathbb{R}^n$ are subspaces with $\dim V + \dim W > n$ , then $V \cap W$ contains a non-zero vector. Indeed, let $v_1,\dots,v_a$ be a basis of $V$ and $w_1,\dots,w_b$ one of $W$ , with $a + b > n$ . More than $n$ vectors in an $n$ -dimensional space are linearly dependent (Vector spaces and bases), so $\sum_i c_iv_i + \sum_j d_jw_j = 0$ with the coefficients not all zero. Put $x = \sum_i c_iv_i = -\sum_j d_jw_j$ , which lies in $V$ and in $W$ . If $x$ were $0$ , independence of the $v_i$ would force every $c_i = 0$ and independence of the $w_j$ every $d_j = 0$ , contradicting the choice; so $x \neq 0$ .

Necessity. Suppose $A$ is positive definite. For $1 \le k \le n$ and $\tilde x \in \mathbb{R}^k$ non-zero, the vector $x \in V_k$ with those first $k$ entries is non-zero, so $\tilde x^{\mathsf T}A^{(k)}\tilde x = x^{\mathsf T}Ax > 0$ : each $A^{(k)}$ is positive definite. By the eigenvalue test, its eigenvalues are all positive, so $D_k = \det A^{(k)}$ , their product, is positive.

Sufficiency, by induction on $n$ . Suppose $D_1, \dots, D_n > 0$ .

Base case $n = 1$ . Here $A = (a_{11})$ with $a_{11} = D_1 > 0$ , and $x^{\mathsf T}Ax = a_{11}x_1^2 > 0$ for $x_1 \neq 0$ .

Induction step. Let $n \ge 2$ and assume the criterion for $n-1$ . The leading principal minors of $A^{(n-1)}$ are $D_1, \dots, D_{n-1}$ — a leading block of a leading block is a leading block — so the induction hypothesis makes $A^{(n-1)}$ positive definite, and therefore

x^{\mathsf T}Ax > 0 \qquad \text{for every non-zero } x \in V_{n-1} . \tag{$*$}

Order the eigenvalues of $A$ as $\lambda_1 \le \cdots \le \lambda_n$ with a real orthonormal eigenbasis $u_1,\dots,u_n$ (Spectral theorem for Hermitian matrices, part 6).

Suppose, for contradiction, that $\lambda_2 \le 0$ , hence also $\lambda_1 \le 0$ . Let $W = \operatorname{span}\{u_1,u_2\}$ , of dimension $2$ . For $x = c_1u_1 + c_2u_2 \in W$ , orthonormality gives

x^{\mathsf T}Ax = \lambda_1c_1^2 + \lambda_2c_2^2 \le 0 .

Since $\dim W + \dim V_{n-1} = 2 + (n-1) = n+1 > n$ , the lemma supplies a non-zero $x \in W \cap V_{n-1}$ . That $x$ has $x^{\mathsf T}Ax \le 0$ by the display and $x^{\mathsf T}Ax > 0$ by $(*)$ — a contradiction. Hence $\lambda_2 > 0$ , and so $\lambda_2, \dots, \lambda_n$ are all positive.

Finally $D_n = \det A = \lambda_1\lambda_2\cdots\lambda_n$ by part 5 of the spectral theorem. The hypothesis $D_n > 0$ and the positivity of $\lambda_2\cdots\lambda_n$ force $\lambda_1 > 0$ . All eigenvalues are then positive and the eigenvalue test makes $A$ positive definite, completing the induction.

Every one of the $n$ inequalities is needed. Dropping the $k$ -th leaves a false criterion, as these diagonal matrices show. For $k \le n-1$ take $A$ diagonal with $a_{kk} = a_{k+1,k+1} = -1$ and every other diagonal entry $+1$ : then $D_j = 1$ for $j < k$ , $D_k = -1$ , and $D_j = (-1)(-1) = 1$ for $j \ge k+1$ , so only the $k$ -th inequality fails, while $A$ is not positive definite because $e_k^{\mathsf T}Ae_k = -1$ . For $k = n$ take $A = \operatorname{diag}(1,\dots,1,-1)$ , which has $D_j = 1$ for $j < n$ and $D_n = -1$ . And no further condition is needed, because the induction above derives positive definiteness from these $n$ alone.

The negative definite form. $A$ is negative definite exactly when $-A$ is positive definite, which by the above is exactly $D_k(-A) > 0$ for every $k$ . Now $(-A)^{(k)} = -A^{(k)}$ and scaling a $k \times k$ matrix by $-1$ scales its determinant by $(-1)^k$ (the determinant is homogeneous of degree $k$ in the rows, Determinants), so $D_k(-A) = (-1)^kD_k(A)$ and the condition reads $(-1)^kD_k > 0$ : $D_1 < 0$ , $D_2 > 0$ , $D_3 < 0$ , and so on. $\square$

Definition(Rayleigh quotient)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.17.1, p. 291

Let $A$ be Hermitian $n \times n$ . For $x \in \mathbb{C}^n$ , $x \neq 0$ , the Rayleigh quotient of $A$ at $x$ is

\mathcal{R}(x) = \frac{x^\dagger A x}{x^\dagger x} ,

which is real, because the numerator and denominator are, and unchanged by rescaling $x$ by any non-zero scalar.

Method(Rayleigh bounds)Original

Let $A$ be Hermitian $n \times n$ with eigenvalues ordered $\lambda_1 \le \lambda_2 \le \cdots \le \lambda_n$ and $E_\lambda$ the eigenspace of $\lambda$ . Then $\lambda_1 \le \mathcal{R}(x) \le \lambda_n$ for every $x \neq 0$ ; the lower bound is an equality exactly for $x \in E_{\lambda_1} \setminus \{0\}$ and the upper bound exactly for $x \in E_{\lambda_n} \setminus \{0\}$ . Hence

\lambda_1 = \min_{x \neq 0} \mathcal{R}(x), \qquad \lambda_n = \max_{x \neq 0} \mathcal{R}(x) ,

both attained. (Expand $x$ in an orthonormal eigenbasis: $\mathcal{R}(x)$ is the weighted average of the $\lambda_k$ with weights $|x_k'|^2 / \|x\|^2$ summing to $1$ .)

Idea of the proofOriginal

In an orthonormal eigenbasis the Rayleigh quotient is literally a weighted average of the eigenvalues, the weight on $\lambda_k$ being the fraction of $\|x\|^2$ that $x$ puts along $u_k$ . An average of numbers lies between the smallest and the largest of them, and equals the smallest only if all the weight sits on the smallest — which is what " $x$ lies in the eigenspace of $\lambda_1$ " says.

ProofOriginal

Fix an orthonormal eigenbasis $u_1,\dots,u_n$ of $A$ with $Au_k = \lambda_k u_k$ and $\lambda_1 \le \cdots \le \lambda_n$ (Spectral theorem for Hermitian matrices). For $x \neq 0$ write $c_k = (x, u_k)$ , so that $x = \sum_k c_ku_k$ and, by the expansion in an orthonormal basis (Orthonormal bases and Gram-Schmidt),

x^\dagger x = \sum_k |c_k|^2 =: \sigma > 0 , \qquad x^\dagger A x = \sum_k \lambda_k|c_k|^2 ,

the second because $Ax = \sum_k \lambda_kc_ku_k$ and the basis is orthonormal. Hence

\mathcal{R}(x) = \sum_{k=1}^{n} w_k\lambda_k , \qquad w_k = \frac{|c_k|^2}{\sigma} \ge 0, \qquad \sum_k w_k = 1 .

The bounds. Since the $w_k$ are non-negative and sum to one,

\mathcal{R}(x) - \lambda_1 = \sum_k w_k(\lambda_k - \lambda_1) \ge 0, \qquad \lambda_n - \mathcal{R}(x) = \sum_k w_k(\lambda_n - \lambda_k) \ge 0 ,

each summand being a product of two non-negative numbers, by the ordering of the eigenvalues. So $\lambda_1 \le \mathcal{R}(x) \le \lambda_n$ .

The equality cases. In the first display, equality forces every summand to vanish, so $w_k = 0$ — that is $c_k = 0$ — for every $k$ with $\lambda_k > \lambda_1$ . Then $x = \sum_{k : \lambda_k = \lambda_1}c_ku_k$ , a combination of eigenvectors for $\lambda_1$ , so $x \in E_{\lambda_1}$ . Conversely if $x \in E_{\lambda_1}$ and $x \neq 0$ then $Ax = \lambda_1 x$ and $\mathcal{R}(x) = \lambda_1 x^\dagger x/x^\dagger x = \lambda_1$ . The same argument with $\lambda_n$ gives the upper equality case. Note that the whole eigenspace appears, not one vector: if $\lambda_1$ is degenerate, every non-zero vector of $E_{\lambda_1}$ attains the minimum.

Attainment. $u_1 \in E_{\lambda_1}$ and $u_n \in E_{\lambda_n}$ are non-zero, so the bounds are attained and the infimum and supremum are a minimum and a maximum. $\square$

Method(Stationary values of the quotient)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.17.1, Eqs (8.114)-(8.116) and the worked result for lambda(x), p. 291

Let $A$ be Hermitian $n \times n$ . Then $\mathcal{R}$ is stationary at $x \neq 0$ — its first variation vanishes for every variation $\Delta x$ — exactly when $A x = \mathcal{R}(x)\, x$ , that is, exactly when $x$ is an eigenvector of $A$ ; the stationary value is then the eigenvalue belonging to $x$ . Equivalently, the critical points of $x^\dagger A x$ on the sphere $x^\dagger x = 1$ are the normalised eigenvectors of $A$ , and the critical value at each is its eigenvalue.

Idea of the proofOriginal

Differentiate the quotient once. Because $A$ is Hermitian, the two cross terms produced by varying $x$ in $x^\dagger A x$ are complex conjugates of each other, so the first variation collapses to $2\operatorname{Re}$ of a single inner product against the vector $Ax - \mathcal{R}(x)x$ . Demanding that this vanish for every variation is demanding that the vector itself vanish, which is the eigenvalue equation with $\mathcal{R}(x)$ as the eigenvalue. Nothing in the argument needs the eigenvalues to be known in advance; the eigenvalue equation is the output.

ProofAdaptedAdapted from Riley, Hobson & Bence (2006), §8.17.1, Eqs (8.113)-(8.116) and the variational computation for lambda(x), pp. 290-291

Write $N(x) = x^\dagger A x$ and $D(x) = x^\dagger x$ , so $\mathcal{R} = N/D$ on $\mathbb{C}^n\setminus\{0\}$ , and fix $x \neq 0$ and a variation $h \in \mathbb{C}^n$ .

Step 1: the first variations of $N$ and $D$ . Expanding,

N(x+h) - N(x) = h^\dagger A x + x^\dagger A h + h^\dagger A h .

The first two terms are complex conjugates of each other, because $\overline{x^\dagger A h} = h^\dagger A^\dagger x = h^\dagger A x$ using $A = A^\dagger$ — this is the one place the Hermitian hypothesis is used, and without it the two terms do not combine. So

N(x+h) - N(x) = 2\operatorname{Re}\bigl(x^\dagger A h\bigr) + O(\|h\|^2), \qquad D(x+h) - D(x) = 2\operatorname{Re}\bigl(x^\dagger h\bigr) + O(\|h\|^2),

the second by the same computation with $A = I$ .

Step 2: the first variation of the quotient. Since $D(x) > 0$ , the quotient is differentiable at $x$ , and keeping only the terms linear in $h$ ,

\delta\mathcal{R} = \frac{D(x)\cdot 2\operatorname{Re}(x^\dagger A h) - N(x)\cdot 2\operatorname{Re}(x^\dagger h)}{D(x)^2} = \frac{2}{x^\dagger x}\,\operatorname{Re}\Bigl( x^\dagger A h - \mathcal{R}(x)\, x^\dagger h\Bigr) .

Now $x^\dagger A h = (Ax)^\dagger h$ , again because $A$ is Hermitian, so with

r = A x - \mathcal{R}(x)\,x

the variation is $\delta\mathcal{R} = \dfrac{2}{x^\dagger x}\operatorname{Re}\bigl(r^\dagger h\bigr)$ .

Step 3: stationarity is the eigenvalue equation. If $r = 0$ then $\delta\mathcal{R} = 0$ for every $h$ . Conversely, if $\delta\mathcal{R} = 0$ for every $h$ , take $h = r$ : then $\operatorname{Re}(r^\dagger r) = \|r\|^2 = 0$ , so $r = 0$ . Hence $\mathcal{R}$ is stationary at $x$ exactly when $Ax = \mathcal{R}(x)x$ , that is exactly when $x$ is an eigenvector, and then the eigenvalue is $\mathcal{R}(x)$ , which is the stationary value. Conversely any eigenvector $Ax = \lambda x$ has $\mathcal{R}(x) = \lambda$ and $r = 0$ , so it is stationary; the quantifier "for every variation" is essential, since $\operatorname{Re}(r^\dagger h)$ vanishes for the $h$ orthogonal to $r$ however large $r$ is.

Step 4: the version on the sphere. $\mathcal{R}$ is unchanged by rescaling, so $\mathcal{R}(x) = N(x)$ whenever $\|x\| = 1$ , and $\delta\mathcal{R}$ vanishes on the radial variation $h = x$ for every $x$ : indeed $r^\dagger x = x^\dagger A x - \mathcal{R}(x)x^\dagger x = 0$ by the definition of $\mathcal{R}$ . A variation tangent to the sphere at $x$ is one with $\operatorname{Re}(x^\dagger h) = 0$ , obtained by differentiating $\|x\|^2 = 1$ ; for such $h$ , step 2 with $D(x) = 1$ gives

\delta\bigl(N|_{\text{sphere}}\bigr) = 2\operatorname{Re}(x^\dagger A h) = 2\operatorname{Re}(r^\dagger h) + 2\mathcal{R}(x)\operatorname{Re}(x^\dagger h) = 2\operatorname{Re}(r^\dagger h) = \delta\mathcal{R} .

So the critical points of $N$ on the sphere are exactly the unit vectors at which $\operatorname{Re}(r^\dagger h)$ vanishes for all tangent $h$ . That is the same as vanishing for all $h \in \mathbb{C}^n$ , because any $h$ splits as $h = h_{\mathrm{t}} + cx$ with $c = \operatorname{Re}(x^\dagger h)$ real and $h_{\mathrm{t}} = h - cx$ tangent — check $\operatorname{Re}(x^\dagger h_{\mathrm{t}}) = \operatorname{Re}(x^\dagger h) - c = 0$ — while $\operatorname{Re}(r^\dagger x) = 0$ always. Hence it is the same as $r = 0$ . The critical points on the sphere are therefore the normalised eigenvectors, with critical values $N(x) = \mathcal{R}(x) = \lambda$ . Riley, Hobson and Bence reach the same conclusion with a Lagrange multiplier imposing $x^\dagger x = 1$ , the multiplier turning out to be the eigenvalue. $\square$

Method(Courant-Fischer min-max)Original

With $A$ and the ordering $\lambda_1 \le \cdots \le \lambda_n$ as above, for every $k = 1, \dots, n$

\lambda_k = \min_{\substack{V \subseteq \mathbb{C}^n \\ \dim V = k}} \ \max_{\substack{x \in V \\ x \neq 0}} \mathcal{R}(x) = \max_{\substack{W \subseteq \mathbb{C}^n \\ \dim W = n-k+1}} \ \min_{\substack{x \in W \\ x \neq 0}} \mathcal{R}(x) ,

the inner extrema being attained and the outer ones attained at $V = \operatorname{span}\{u_1, \dots, u_k\}$ and $W = \operatorname{span}\{u_k, \dots, u_n\}$ respectively. The cases $k = 1$ and $k = n$ are the Rayleigh bounds above. Restricting the minimum to a smaller family of subspaces can only raise it, which is what makes this a practical source of upper bounds for $\lambda_k$ .

Idea of the proofOriginal

Two inequalities, in opposite directions, pin $\lambda_k$ down. One is a calculation: on the particular subspace spanned by the first $k$ eigenvectors the quotient never exceeds $\lambda_k$ and reaches it, so the minimum over all $k$ -dimensional subspaces is at most $\lambda_k$ . The other is a counting argument: any $k$ -dimensional subspace must meet the $(n-k+1)$ -dimensional span of the last $n-k+1$ eigenvectors, because their dimensions add to $n+1$ ; on that span the quotient is at least $\lambda_k$ , so the maximum over any $k$ -dimensional subspace is at least $\lambda_k$ . The max-min formula is the min-max formula read for $-A$ .

ProofOriginal

Keep the orthonormal eigenbasis $u_1,\dots,u_n$ with $\lambda_1 \le \cdots \le \lambda_n$ and the weighted-average form $\mathcal{R}(x) = \sum_k w_k\lambda_k$ of the Rayleigh proof. Two consequences of it will be used repeatedly: if $x \neq 0$ lies in $V_k = \operatorname{span}\{u_1,\dots,u_k\}$ then the weights $w_j$ vanish for $j > k$ , so

\mathcal{R}(x) = \sum_{j \le k}w_j\lambda_j \le \lambda_k ; \tag{A}

and if $x \neq 0$ lies in $W_k = \operatorname{span}\{u_k,\dots,u_n\}$ then the weights vanish for $j < k$ , so

\mathcal{R}(x) = \sum_{j \ge k}w_j\lambda_j \ge \lambda_k . \tag{B}

The inner extrema are attained. Let $V$ be a subspace with $\dim V = d \ge 1$ . $\mathcal{R}$ is continuous on $\mathbb{C}^n\setminus\{0\}$ , the unit sphere $\Sigma_V = \{x \in V : \|x\| = 1\}$ is closed and bounded in the finite-dimensional space $V$ , hence compact, and $\mathcal{R}$ is unchanged by rescaling, so the extrema of $\mathcal{R}$ over the non-zero vectors of $V$ are the extrema over $\Sigma_V$ and are attained there.

Upper bound for the min-max. Take $V = V_k$ , which has dimension $k$ because the $u_j$ are independent. By (A), $\max_{x \in V_k\setminus\{0\}}\mathcal{R}(x) \le \lambda_k$ , and $u_k \in V_k$ gives $\mathcal{R}(u_k) = \lambda_k$ , so the maximum over $V_k$ equals $\lambda_k$ . Hence the minimum over all $k$ -dimensional subspaces is at most $\lambda_k$ .

Lower bound for the min-max. Let $V$ be any subspace of dimension $k$ . Then $\dim V + \dim W_k = k + (n-k+1) = n+1 > n$ , so by the overlapping-subspaces lemma of the Sylvester proof — whose argument is unchanged over $\mathbb{C}$ — there is a non-zero $y \in V \cap W_k$ . By (B), $\mathcal{R}(y) \ge \lambda_k$ , and therefore

\max_{x \in V\setminus\{0\}}\mathcal{R}(x) \ \ge\ \mathcal{R}(y)\ \ge\ \lambda_k .

As $V$ was arbitrary, the minimum over all $k$ -dimensional subspaces is at least $\lambda_k$ . With the previous paragraph it equals $\lambda_k$ and is attained at $V = V_k$ .

The max-min form. Apply what is proved to $-A$ , which is Hermitian with eigenvalues $-\lambda_n \le \cdots \le -\lambda_1$ , so that the $j$ -th smallest eigenvalue of $-A$ is $-\lambda_{n-j+1}$ , and with $\mathcal{R}_{-A} = -\mathcal{R}_A$ . For $j = n-k+1$ the min-max formula for $-A$ reads

-\lambda_{k} = \min_{\dim V = n-k+1}\ \max_{x \in V\setminus\{0\}}\bigl(-\mathcal{R}_A(x)\bigr) = \min_{\dim V = n-k+1}\ \Bigl(-\min_{x \in V\setminus\{0\}}\mathcal{R}_A(x)\Bigr) = -\max_{\dim V = n-k+1}\ \min_{x \in V\setminus\{0\}}\mathcal{R}_A(x) ,

using $\max(-f) = -\min f$ twice. Multiplying by $-1$ gives the stated max-min formula over subspaces $W$ of dimension $n-k+1$ . The optimal subspace transports too: for $-A$ the optimum was the span of its first $n-k+1$ eigenvectors, which are $u_n, u_{n-1}, \dots, u_k$ , that is $W = \operatorname{span}\{u_k,\dots,u_n\}$ .

The cases $k = 1$ and $k = n$ . For $k = 1$ the only $1$ -dimensional subspaces are the lines $\mathbb{C}x$ , on each of which $\mathcal{R}$ is constant, so the min-max reduces to $\min_{x \neq 0}\mathcal{R}(x) = \lambda_1$ ; for $k = n$ the only $n$ -dimensional subspace is $\mathbb{C}^n$ itself and the formula reads $\max_{x\neq 0}\mathcal{R}(x) = \lambda_n$ . These are the Rayleigh bounds.

Why restricting the family gives upper bounds. A minimum over a subfamily of the $k$ -dimensional subspaces is at least the minimum over all of them, which is $\lambda_k$ . So evaluating $\max_{x \in V}\mathcal{R}(x)$ for any single conveniently chosen $V$ of dimension $k$ — a span of $k$ trial vectors, say — delivers a number $\ge \lambda_k$ , and equality holds exactly when that $V$ is optimal. $\square$

Method(Analysing a quadratic form)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.17 and §8.17.1, pp. 288-291

Given a form written as $x^{\mathsf T} M x$ with $M$ any real square matrix, or as an explicit polynomial such as $x_1^2 + 4 x_1 x_2 - x_2^2$ :

Symmetrise. Replace $M$ by $A = \tfrac12 (M + M^{\mathsf T})$ . From a polynomial, set $a_{ii}$ to the coefficient of $x_i^2$ and $a_{ij} = a_{ji}$ to half the coefficient of $x_i x_j$ . Skip this and you will diagonalize the wrong matrix: the form determines only the symmetric part.
If you only need definiteness, apply Sylvester's criterion: compute $D_1, \dots, D_n$ . All positive means positive definite; alternating from $D_1 < 0$ means negative definite. Any other pattern with all $D_k \neq 0$ means indefinite. If some $D_k = 0$ the criterion is silent — go to step 3.
Otherwise diagonalize. Compute the eigenvalues of $A$ from $\det(A - \lambda I) = 0$ for small $n$ , or with a symmetric eigensolver (numpy.linalg.eigh) for larger $n$ . The signs of the eigenvalues give the definiteness class, with no exceptional cases; the triple (number positive, number negative, number zero) is the inertia of the form.
For the principal axes, compute an orthonormal eigenbasis, assemble $S$ with those columns, and read off $Q = \sum_k \lambda_k x_k'^2$ where $x' = S^{\mathsf T} x$ . Order the eigenvalues as you like, but keep the columns of $S$ in the matching order.
For the geometry of a level set $x^{\mathsf T} A x = c$ with $c > 0$ : first test non-emptiness — the set is non-empty if and only if some $\lambda_k > 0$ (for $A = \operatorname{diag}(-1,-1)$ and $c = 1$ it is empty, and there is no geometry to describe). Given that, the axis directions are the $u_k$ ; for each $\lambda_k > 0$ the set meets the line $\mathbb{R} u_k$ at distance $(c/\lambda_k)^{1/2}$ from the origin, and for each $\lambda_k \le 0$ it does not meet that line at all. The set is unbounded precisely when some $\lambda_k \le 0$ ; it contains, through each of its points, the whole line in the direction $u_k$ — equivalently, it is invariant under translation along $u_k$ — precisely when $\lambda_k = 0$ . These are two different conditions: a hyperbola ( $A = \operatorname{diag}(1,-1)$ , $c = 1$ ) is unbounded along both of its axis directions while containing no line at all, since no eigenvalue vanishes. When every $\lambda_k > 0$ it is an ellipsoid with those semi-axes; otherwise classify it by the counts $(p, q, z)$ as in "Principal axes" — the set for level $c$ is the set for level $1$ scaled by $c^{1/2}$ , so every semi-axis is multiplied by $c^{1/2}$ and the classification is unchanged.
For quick eigenvalue bounds, evaluate $\mathcal{R}$ at any convenient trial vectors: each value is an upper bound for $\lambda_1$ and a lower bound for $\lambda_n$ , and the bound is good when the trial vector is close to the eigenvector, because $\mathcal{R}$ is stationary there.

The recipe needs $A$ Hermitian throughout; steps 2-6 are false without it. For a Hermitian complex $A$ use $x^\dagger A x$ , $A^\dagger$ and a unitary $S$ in place of the real transposes.

NoteOriginal

A quadratic form is the second-order part of any smooth function near a critical point, so its definiteness decides whether a potential has a minimum, an equilibrium is stable or an energy is bounded below. Diagonalizing it chooses coordinates in which those questions become $n$ questions about single numbers.

Remark(Traps)Original

Sylvester's criterion does not test semidefiniteness. $A = \operatorname{diag}(0, -1)$ has $D_1 = 0$ and $D_2 = 0$ , both $\ge 0$ , yet $A$ is not positive semidefinite. Positive semidefiniteness is equivalent to all principal minors (not just the leading ones) being $\ge 0$ , or simply to all eigenvalues being $\ge 0$ . For definiteness the leading minors suffice; for semidefiniteness they do not.

A positive determinant is not positive definiteness. $\operatorname{diag}(-1, -1)$ has determinant $1$ . Only the full chain $D_1, \dots, D_n$ settles the question.

The equality case needs the whole eigenspace. If $\lambda_1$ is degenerate, $\mathcal{R}(x) = \lambda_1$ for every non-zero $x \in E_{\lambda_1}$ , not only for one eigenvector.

Min-max is over subspaces, not vectors. In the Courant-Fischer formula the outer minimum runs over $k$ -dimensional subspaces; replacing it by a minimum over vectors would give $\lambda_1$ for every $k$ .

Sylvester's criterion for Hermitian $A$ . Stated above for real symmetric $A$ , as (Gelfand & Fomin (1963), §30, p. 127) gives it. It holds word for word for Hermitian $A$ : each leading submatrix $A^{(k)}$ is itself Hermitian, so $D_k = \det A^{(k)}$ is real, and the criterion reads the same. That extension is not on the cited page, so do not attribute it there.

Conventions: the factor $\tfrac12$ doubles the eigenvalues, it does not halve them. Fix a form $Q$ . Writing it our way, $Q(x) = x^{\mathsf T} A x$ , makes $A$ half the Hessian of $Q$ . Texts that write $Q(x) = \tfrac12 x^{\mathsf T} \tilde A x$ — common in mechanics and optimisation, so that $\tilde A$ is the Hessian — have $\tilde A = 2A$ , whose eigenvalues are twice ours. For the polynomial $Q = x_1^2 + 4x_1x_2 - x_2^2$ used to illustrate step 1 of the method, our $A = \begin{pmatrix} 1 & 2 \\ 2 & -1\end{pmatrix}$ has eigenvalues $\pm\sqrt 5$ , while the Hessian $\tilde A = \begin{pmatrix} 2 & 4 \\ 4 & -2\end{pmatrix}$ has $\pm 2\sqrt 5$ . Going the other way — holding the matrix fixed and reading it in the half convention — halves the form and hence the eigenvalues; what the factor does depends on which of the two you hold fixed, so say which. Either way the definiteness class, the inertia and the principal axes are unchanged, since the two matrices differ by a positive scalar; only the numbers $\lambda_k$ and therefore the semi-axis lengths $\lambda_k^{-1/2}$ change.

Ordering. Order the eigenvalues increasingly, as here, or decreasingly, as is common in numerical linear algebra — but then the min-max formula swaps its two halves.

Example(A form in three variables, start to finish)Original

Suppose a calculation has produced the form $Q(x) = x^{\mathsf T}Mx$ with

M = \begin{pmatrix} 3 & 4 & 0 \\ -2 & 3 & -3 \\ -2 & 1 & 5\end{pmatrix},

which is not symmetric. Follow the method.

Step 1: symmetrise. $A = \tfrac12(M + M^{\mathsf T})$ has entries $a_{ii} = M_{ii}$ and $a_{ij} = \tfrac12(M_{ij}+M_{ji})$ :

A = \begin{pmatrix} 3 & 1 & -1 \\ 1 & 3 & -1 \\ -1 & -1 & 5\end{pmatrix}, \qquad Q(x) = 3x_1^2 + 3x_2^2 + 5x_3^2 + 2x_1x_2 - 2x_1x_3 - 2x_2x_3 .

Reading the polynomial back off $A$ is the check: the diagonal gives the squares and each off-diagonal entry is half the coefficient of the corresponding cross term, so $a_{12} = 1$ goes with $2x_1x_2$ .

Step 2: definiteness by Sylvester. The leading principal minors are

D_1 = 3, \qquad D_2 = \begin{vmatrix} 3 & 1 \\ 1 & 3\end{vmatrix} = 8, \qquad D_3 = \det A = 3(15-1) - 1(5-1) + (-1)(-1+3) = 42 - 4 - 2 = 36 ,

all positive, so $A$ is positive definite. In particular $Q(x) > 0$ for every $x \neq 0$ , and the level set $Q = 1$ will be an ellipsoid.

Step 3: the eigenvalues and eigenvectors. The characteristic polynomial is

\det(A - \lambda I) = -\lambda^3 + 11\lambda^2 - 36\lambda + 36 = -(\lambda-2)(\lambda-3)(\lambda-6),

whose coefficients check against $\operatorname{tr}A = 11 = 2+3+6$ and $\det A = 36 = 2\cdot 3\cdot 6$ . Solving $(A-\lambda I)x = 0$ for each root,

\lambda_1 = 2:\ u_1 = \tfrac{1}{\sqrt2}(1,-1,0)^{\mathsf T}, \qquad \lambda_2 = 3:\ u_2 = \tfrac{1}{\sqrt3}(1,1,1)^{\mathsf T}, \qquad \lambda_3 = 6:\ u_3 = \tfrac{1}{\sqrt6}(1,1,-2)^{\mathsf T},

each verified directly: $A(1,-1,0)^{\mathsf T} = (2,-2,0)^{\mathsf T}$ , $A(1,1,1)^{\mathsf T} = (3,3,3)^{\mathsf T}$ , $A(1,1,-2)^{\mathsf T} = (6,6,-12)^{\mathsf T}$ . The three are mutually orthogonal, as the spectral theorem requires for distinct eigenvalues, and all positive, confirming step 2 independently.

Step 4: the principal axes. With $S = (u_1\ u_2\ u_3)$ and $x' = S^{\mathsf T}x$ ,

Q = 2x_1'^2 + 3x_2'^2 + 6x_3'^2 , \qquad x_1' = \tfrac{x_1-x_2}{\sqrt2}, \quad x_2' = \tfrac{x_1+x_2+x_3}{\sqrt3}, \quad x_3' = \tfrac{x_1+x_2-2x_3}{\sqrt6} .

Step 5: the level set $Q = 1$ . The inertia is $(p,q,z) = (3,0,0)$ , so this is an ellipsoid, with semi-axes

\lambda_1^{-1/2} = \tfrac{1}{\sqrt2} \approx 0.7071 \text{ along } u_1, \quad \tfrac{1}{\sqrt3} \approx 0.5774 \text{ along } u_2, \quad \tfrac{1}{\sqrt6} \approx 0.4082 \text{ along } u_3 .

A spot check on the first: the point $u_1/\sqrt2 = (1,-1,0)^{\mathsf T}/2$ has $Q = \tfrac14(3 + 3 - 2) = 1$ , and its distance from the origin is $\sqrt{2}/2 = 1/\sqrt2$ . For the level $Q = 4$ every semi-axis is multiplied by $\sqrt4 = 2$ .

Step 6: Rayleigh bounds from trial vectors. Every trial vector gives $2 \le \mathcal{R}(x) \le 6$ . With $x = (2,-1,0)^{\mathsf T}$ , a rough guess at $u_1$ ,

\mathcal{R}(x) = \frac{3\cdot 4 + 3\cdot 1 + 2\cdot 2\cdot(-1)}{4 + 1} = \frac{12 + 3 - 4}{5} = \frac{11}{5} = 2.2 ,

an upper bound for $\lambda_1 = 2$ , high by $10\%$ although the trial vector points $18.4^\circ$ away from $u_1$ . Stationarity sharpens this: perturbing the true eigenvector to $x = (1,-1,\delta)^{\mathsf T}$ gives

\mathcal{R}(x) = \frac{4 + 5\delta^2}{2+\delta^2} = 2 + \tfrac32\delta^2 + O(\delta^4),

so at $\delta = 0.1$ — a $7\%$ tilt of the vector — the quotient is $135/67 = 2.01493$ , high by only $0.75\%$ . The error in the eigenvalue is quadratic in the error of the eigenvector, which is exactly the content of the stationarity statement and the reason trial-vector bounds are useful.

Step 7: Courant–Fischer for the middle eigenvalue. Take the $2$ -dimensional trial subspace $V = \operatorname{span}\{e_1, e_2\}$ . For $x \in V$ the quotient is the Rayleigh quotient of the block $A^{(2)} = \begin{pmatrix} 3&1\\1&3\end{pmatrix}$ , whose eigenvalues are $2$ and $4$ , so

\max_{x \in V\setminus\{0\}}\mathcal{R}(x) = 4 \ \ge\ \lambda_2 = 3 ,

an upper bound for $\lambda_2$ from a subspace chosen without any eigenvector information. The optimal choice $V = \operatorname{span}\{u_1,u_2\}$ does give $3$ : in the orthonormal basis $\{u_1,u_2\}$ of that plane, $A$ is represented by $\operatorname{diag}(2,3)$ , whose largest eigenvalue is $3$ .

Step 8: a one-entry change makes it indefinite. Replace the entry $a_{33} = 5$ by $0$ , i.e. drop the $5x_3^2$ term. Now $D_1 = 3$ , $D_2 = 8$ and

D_3 = 3(0-1) - 1(0-1) + (-1)(-1+3) = -3 + 1 - 2 = -4 ,

so the pattern is $(+,+,-)$ with no zero: by step 2 of the method the form is indefinite, and Sylvester tells us so without any eigenvalue computation. For confirmation, the characteristic polynomial is $-(\lambda-2)(\lambda^2-4\lambda-2)$ , with roots $2$ and $2 \pm \sqrt6 \approx 4.4495,\ -0.4495$ — inertia $(2,1,0)$ . The level set is then a hyperboloid, unbounded along $u$ for the negative eigenvalue, and $Q$ takes both signs: $Q(e_3) = 0$ for the modified form while $Q(0,0,1)$ was $5$ before, and $Q(1,1,1) = 3+3+0+2-2-2 = 4 > 0$ while $Q(1,1,3) = 3+3+0+2-6-6 = -4 < 0$ .

ApplicationOriginal

The second-derivative test, and why stability is an eigenvalue question. Let $f$ be twice continuously differentiable near a critical point $a$ , so $\nabla f(a) = 0$ . Taylor's theorem gives

f(a + h) = f(a) + \tfrac12\, h^{\mathsf T} H h + o(\|h\|^2), \qquad H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}(a),

and $H$ is symmetric, so the leading behaviour near $a$ is governed by a quadratic form and this unit settles it: $H$ positive definite makes $a$ a strict local minimum, $H$ negative definite a strict local maximum, and $H$ indefinite a saddle, with directions of increase and decrease given by eigenvectors for eigenvalues of either sign. Sylvester's criterion is what one actually computes, since it needs only $n$ determinants rather than the full spectrum; for $n = 2$ it is the familiar test $f_{xx} > 0$ and $f_{xx}f_{yy} - f_{xy}^2 > 0$ . In mechanics the same statement is the stability of an equilibrium: with $f$ the potential energy, positive definiteness of $H$ is the condition for a restoring force in every direction, and the eigenvalues of $H$ (divided by the masses) are the squared frequencies of the small oscillations about it — see normal modes. A semidefinite $H$ leaves the question open at second order, which is where the trap about Sylvester's criterion and semidefiniteness bites: $f(x,y) = x^2 + y^3$ and $f(x,y) = x^2 + y^4$ have the same Hessian at the origin and only the second has a minimum there.

Rayleigh–Ritz: a variational upper bound on a ground-state energy. In quantum mechanics the energy levels of a system are the eigenvalues $\lambda_1 \le \lambda_2 \le \cdots$ of a Hermitian Hamiltonian, and the Rayleigh bound says the ground-state energy is

\lambda_1 = \min_{\psi \neq 0} \frac{\psi^\dagger H \psi}{\psi^\dagger \psi} .

So any trial state gives an upper bound on the ground-state energy — never an underestimate, which is what makes the method trustworthy — and because the quotient is stationary at the true eigenvector, a trial state wrong by a relative error $\varepsilon$ gives an energy wrong only by order $\varepsilon^2$ . That is the whole of the variational method: pick a family of trial states depending on parameters, minimise the quotient over the parameters, and report the result as an upper bound. Courant–Fischer extends it to excited states: taking a $k$ -dimensional trial space and computing the largest eigenvalue of the $k \times k$ matrix of $H$ restricted to it gives an upper bound for $\lambda_k$ , which is exactly what step 7 of the example does with $k = 2$ . Enlarging the trial space can only lower each bound, and this is why the method converges as the basis is refined. The same mathematics appears as Rayleigh's method for the fundamental frequency of a vibrating structure and as the finite element method for eigenvalue problems.

Principal axes of a physical tensor. Many physical quantities are symmetric matrices whose quadratic form has a direct meaning, and diagonalizing them is the first step of every calculation. The inertia tensor $I$ of a rigid body has $\omega^{\mathsf T}I\omega = 2T$ twice the rotational kinetic energy; it is positive definite for a genuine three-dimensional body, so its eigenvectors are the three mutually perpendicular principal axes about which rotation produces no wobble, its eigenvalues the principal moments, and the level set $\omega^{\mathsf T}I\omega = 1$ is the ellipsoid of inertia with semi-axes $I_k^{-1/2}$ . The Cauchy stress tensor is symmetric, and its eigenvalues are the principal stresses with eigenvectors the directions of pure tension or compression — a negative eigenvalue is compression, so the inertia of the form is the statement of what kind of loading the material is under. The strain tensor, the electric susceptibility of an anisotropic crystal and the diffusion tensor of an anisotropic medium all work the same way: the eigenvectors give the material's own axes, and the numbers one quotes are the eigenvalues.

Application

Generalized eigenproblems and normal modes

Adapted

Add to PDF

States the spectral theory of the pencil Ax = lambda Bx for Hermitian A and positive definite B, and applies it to the small oscillations of a conservative system, whose normal frequencies and normal modes are the eigenvalues and B-orthonormal eigenvectors of the pencil formed by the potential and kinetic energy matrices.

Prerequisites: Quadratic forms and the Rayleigh quotient, Spectral theorem for Hermitian matrices, Determinants, Unitary and orthogonal matrices, Orthonormal bases and Gram-Schmidt, Inner products and norms

Definition(The generalized eigenvalue problem)AdaptedAdapted from Riley, Hobson & Bence (2006), §9.1, Eqs (9.9)-(9.10), p. 319

Let $A$ and $B$ be Hermitian $n \times n$ matrices with $B$ positive definite. A number $\lambda \in \mathbb{C}$ is an eigenvalue of the pencil $(A, B)$ if

A x = \lambda B x \qquad \text{for some } x \neq 0 ,

and such an $x$ is a corresponding eigenvector. The equation $A x = \lambda B x$ has a non-zero solution exactly when $A - \lambda B$ is singular, that is exactly when

\det(A - \lambda B) = 0 ,

the characteristic equation of the pencil. That equivalence is just the singularity criterion and needs nothing of $B$ ; what invertibility of $B$ contributes is that the left-hand side is a polynomial in $\lambda$ of degree exactly $n$ , so that the pencil has $n$ eigenvalues and not fewer (part 1 of the next statement). Two vectors $x, y$ are $B$ -orthogonal if $y^\dagger B x = 0$ ; because $B$ is Hermitian positive definite, $(x, y)_B = y^\dagger B x$ is an inner product on $\mathbb{C}^n$ , and $B$ -orthogonality is orthogonality in it. Taking $B = I$ recovers the ordinary eigenproblem.

Application(Spectral theorem for a definite pencil)Original

Let $A$ be Hermitian and $B$ Hermitian positive definite, both $n \times n$ . Then:

$\det(A - \lambda B)$ is a polynomial in $\lambda$ of degree exactly $n$ , with leading coefficient $(-1)^n \det B \neq 0$ ; so the pencil has exactly $n$ eigenvalues counted with multiplicity, and all of them are real.
There is a basis $x^1, \dots, x^n$ of $\mathbb{C}^n$ with

A x^k = \lambda_k B x^k , \qquad (x^j)^\dagger B\, x^k = \delta_{jk} , \qquad (x^j)^\dagger A\, x^k = \lambda_k \delta_{jk} .

The eigenvectors are thus simultaneously $B$ -orthonormal and $A$ -orthogonal. If $A$ and $B$ are real symmetric, the $x^k$ may be taken real. 3. The problem is equivalent to an ordinary Hermitian eigenproblem: with $B^{1/2}$ the Hermitian positive definite square root of $B$ , the matrix $C = B^{-1/2} A B^{-1/2}$ is Hermitian, and $A x = \lambda B x$ holds if and only if $C y = \lambda y$ with $y = B^{1/2} x$ . In particular the pencil and $C$ have the same eigenvalues with the same multiplicities, and $x \mapsto B^{1/2}x$ carries a $B$ -orthonormal eigenbasis of the pencil to an orthonormal eigenbasis of $C$ . (A Cholesky factor $B = L L^\dagger$ serves equally well, with $C = L^{-1} A L^{-\dagger}$ .) 4. Quotient bounds. With $\lambda_1 \le \cdots \le \lambda_n$ ,

\lambda_1 \le \frac{x^\dagger A x}{x^\dagger B x} \le \lambda_n \qquad \text{for every } x \neq 0 ,

both bounds attained, and the quotient is stationary at $x$ exactly when $x$ is an eigenvector of the pencil, with stationary value the corresponding eigenvalue. $A$ is positive definite if and only if every $\lambda_k > 0$ , and positive semidefinite if and only if every $\lambda_k \ge 0$ .

Idea of the proofOriginal

A positive definite $B$ has a positive definite square root, so the substitution $y = B^{1/2}x$ is available and it is invertible. Under it the pencil equation $Ax = \lambda Bx$ becomes the ordinary eigenvalue equation $Cy = \lambda y$ for the Hermitian matrix $C = B^{-1/2}AB^{-1/2}$ , and the $B$ -inner product $(u,v)_B = v^\dagger B u$ becomes the ordinary one, since $(u,v)_B = (B^{1/2}v)^\dagger(B^{1/2}u)$ . So the whole theorem — reality of the spectrum, the count, the double orthogonality, the quotient bounds — is the Hermitian spectral theorem and the Rayleigh bounds transported through one change of variable. Definiteness of $B$ is what makes the square root, and hence the whole argument, exist; mere invertibility is not enough.

ProofOriginal

Step 1: the square root. $B$ is Hermitian, so by the spectral theorem $B = V\Delta V^\dagger$ with $V$ unitary and $\Delta = \operatorname{diag}(\beta_1,\dots,\beta_n)$ real; $B$ is positive definite, so by the eigenvalue test of Quadratic forms and the Rayleigh quotient every $\beta_k > 0$ . Define

B^{1/2} = V\,\operatorname{diag}\bigl(\sqrt{\beta_1},\dots,\sqrt{\beta_n}\bigr)\,V^\dagger .

It is Hermitian, its eigenvalues $\sqrt{\beta_k}$ are positive so it is positive definite, $(B^{1/2})^2 = V\Delta V^\dagger = B$ , and it is invertible with $B^{-1/2} = V\operatorname{diag}(\beta_k^{-1/2})V^\dagger$ , also Hermitian. It is the only Hermitian positive definite square root: if $X$ is one, then $X$ has an orthonormal eigenbasis with positive eigenvalues $\gamma$ , and $X^2 = B$ makes each such eigenvector an eigenvector of $B$ with eigenvalue $\gamma^2$ ; since $t \mapsto t^2$ is injective on the positive reals, $X$ must act as $\sqrt\beta$ on the eigenspace $E_\beta(B)$ for every $\beta$ , which determines $X$ completely.

Step 2: part 3, the reduction. Put $C = B^{-1/2}AB^{-1/2}$ . It is Hermitian, since $B^{-1/2}$ and $A$ are: $C^\dagger = B^{-1/2}A^\dagger B^{-1/2} = C$ . Because $B^{1/2}CB^{1/2} = A$ and $B^{1/2}(\lambda I)B^{1/2} = \lambda B$ ,

A - \lambda B = B^{1/2}\,(C - \lambda I)\,B^{1/2} . \tag{$\ast$}

Hence, with $y = B^{1/2}x$ — a bijection of $\mathbb{C}^n$ carrying $0$ to $0$ and nothing else there —

A x = \lambda B x \iff B^{1/2}(C-\lambda I)B^{1/2}x = 0 \iff (C - \lambda I)y = 0 \iff Cy = \lambda y ,

the middle step because $B^{1/2}$ is invertible. For the Cholesky variant, if $B = LL^\dagger$ with $L$ invertible then $A - \lambda B = L(L^{-1}AL^{-\dagger} - \lambda I)L^\dagger$ by the same algebra, and $C' = L^{-1}AL^{-\dagger}$ is Hermitian; everything below goes through with $L^\dagger$ in place of $B^{1/2}$ .

Step 3: part 1. Taking determinants in $(\ast)$ and using multiplicativity (Determinants),

\det(A - \lambda B) = \bigl(\det B^{1/2}\bigr)^2\det(C-\lambda I) = \det B \cdot \det(C - \lambda I),

the last equality because $\det B^{1/2}\det B^{1/2} = \det\bigl((B^{1/2})^2\bigr) = \det B$ . Now $\det(C-\lambda I)$ is a polynomial in $\lambda$ of degree $n$ with leading coefficient $(-1)^n$ , so $\det(A-\lambda B)$ is a polynomial of degree exactly $n$ with leading coefficient $(-1)^n\det B$ , which is non-zero because $B$ is positive definite and hence $\det B > 0$ . A polynomial of degree $n$ over $\mathbb{C}$ has exactly $n$ roots with multiplicity, and by step 2 those roots are the eigenvalues of the pencil; they are the eigenvalues of the Hermitian matrix $C$ , with the same multiplicities, hence all real.

Step 4: part 2. By the spectral theorem $C$ has an orthonormal eigenbasis $y^1,\dots,y^n$ , $Cy^k = \lambda_ky^k$ . Put $x^k = B^{-1/2}y^k$ . These are a basis of $\mathbb{C}^n$ , being the images of a basis under an invertible matrix, and by step 2 they satisfy $Ax^k = \lambda_kBx^k$ . Moreover, using that $B^{-1/2}$ is Hermitian,

(x^j)^\dagger B\, x^k = (y^j)^\dagger B^{-1/2}B B^{-1/2} y^k = (y^j)^\dagger y^k = \delta_{jk},

(x^j)^\dagger A\, x^k = (y^j)^\dagger B^{-1/2}A B^{-1/2} y^k = (y^j)^\dagger C\, y^k = \lambda_k (y^j)^\dagger y^k = \lambda_k\delta_{jk} .

So the $x^k$ are $B$ -orthonormal and $A$ -orthogonal at once. If $A$ and $B$ are real symmetric, then $V$ may be taken real orthogonal (part 6 of the spectral theorem), hence $B^{\pm 1/2}$ and $C$ are real symmetric, the $y^k$ may be taken real, and so may the $x^k$ .

Step 5: part 4, the bounds. For $x \neq 0$ put $y = B^{1/2}x \neq 0$ . Then

x^\dagger A x = y^\dagger B^{-1/2}AB^{-1/2}y = y^\dagger C y , \qquad x^\dagger B x = y^\dagger B^{-1/2}BB^{-1/2}y = y^\dagger y ,

so the generalized quotient at $x$ equals the ordinary Rayleigh quotient $\mathcal{R}_C(y)$ . The denominator $x^\dagger Bx = \|y\|^2$ is positive for $x \neq 0$ , which is where positive definiteness of $B$ is needed to make the quotient defined at all. The Rayleigh bounds of Quadratic forms and the Rayleigh quotient now give $\lambda_1 \le \mathcal{R}_C(y) \le \lambda_n$ with both bounds attained, at $y = y^1$ and $y = y^n$ , that is at $x = x^1$ and $x = x^n$ .

Step 6: part 4, stationarity and definiteness. The map $x \mapsto y = B^{1/2}x$ is a linear bijection, so for any variation $h$ at $x$ the corresponding variation at $y$ is $B^{1/2}h$ , and $h \mapsto B^{1/2}h$ runs over all of $\mathbb{C}^n$ as $h$ does. Hence the first variation of the generalized quotient vanishes for every $h$ exactly when the first variation of $\mathcal{R}_C$ vanishes for every variation at $y$ , which by the stationary-value statement of Quadratic forms and the Rayleigh quotient happens exactly when $Cy = \mathcal{R}_C(y)\,y$ , that is exactly when $x$ is an eigenvector of the pencil, with the stationary value its eigenvalue. Finally, the first display of step 5 shows that $x^\dagger Ax$ is positive at every non-zero $x$ exactly when $y^\dagger Cy$ is positive at every non-zero $y$ , since $x \mapsto y$ is a bijection fixing only $0$ ; and by the eigenvalue test that holds exactly when every eigenvalue of $C$ — that is, every $\lambda_k$ — is positive. The same sentence with $\ge$ gives the semidefinite case. $\square$

Definition(Small oscillations)AdaptedAdapted from Riley, Hobson & Bence (2006), Ch. 9 introduction, Eqs (9.1)-(9.2), pp. 316-317, and §9.1, Eq. (9.7), p. 318

Let a conservative system be described by generalized coordinates $q = (q_1, \dots, q_n)^{\mathsf T}$ , chosen to vanish at an equilibrium configuration, with a potential $V(q)$ that does not depend on the velocities and with $V(0) = 0$ . For small $q$ and $\dot q$ the kinetic and potential energies are quadratic forms

T = \tfrac12\, \dot q^{\mathsf T} M\, \dot q , \qquad V = \tfrac12\, q^{\mathsf T} K q + O(\|q\|^3) ,

where $M$ and $K$ are constant real symmetric $n \times n$ matrices, $K_{ij} = \partial^2 V(0)/\partial q_i \partial q_j$ the Hessian of $V$ at the equilibrium, and the linear term is absent because $\partial V(0)/\partial q_i = 0$ at an equilibrium. $M$ is the mass or kinetic-energy matrix and is positive definite, since $T > 0$ whenever $\dot q \neq 0$ ; $K$ is the stiffness or potential-energy matrix. Keeping only these quadratic terms, Lagrange's equations $\frac{d}{dt}\frac{\partial T}{\partial \dot q_i} - \frac{\partial T}{\partial q_i} + \frac{\partial V}{\partial q_i} = 0$ give the linear system

M \ddot q + K q = 0 .

This is Eq. (9.7) of Riley, Hobson and Bence, who reach it instead from conservation of energy, $\frac{d}{dt}(T + V) = 0$ , and defer the Lagrangian derivation to their chapter 22; the route through Lagrange's equations used here is ours.

The middle term disappears because $M$ has been frozen at its equilibrium value: in general $T = \tfrac12 \dot q^{\mathsf T} M(q) \dot q$ , so that

\frac{\partial T}{\partial q_i} = \tfrac12\, \dot q^{\mathsf T} \bigl(\partial_i M(q)\bigr) \dot q ,

which carries two factors of $\dot q$ and, since $\partial_i M(0)$ need not vanish, no factor of $q$ : it is of second order in the small quantities $q, \dot q$ , one order beyond the first-order terms $M\ddot q$ and $Kq$ that are kept, and is therefore dropped in the linearization, so that only $\frac{d}{dt}(M\dot q) = M\ddot q$ survives from the kinetic part. Counted in the energies rather than in the equations of motion, the same statement is that the neglected part of $T$ itself, $\tfrac12 \dot q^{\mathsf T}\bigl(M(q) - M(0)\bigr)\dot q = O(\|q\|\,\|\dot q\|^2)$ , is third order against the second-order forms kept. (The example below has exactly such a $q$ -dependent $T$ , through a factor $\cos(q_1 - q_2)$ , which step 1 there evaluates at $q = 0$ ; because that factor is even in $q_1 - q_2$ it is the special case $\partial_i M(0) = 0$ , in which $\partial T/\partial q_i$ is third order and not merely second.) The displayed system is the equation of motion of the system linearized about the equilibrium; everything below is a statement about it, not about the exact nonlinear motion.

Application(Normal frequencies and normal modes)AdaptedAdapted from Riley, Hobson & Bence (2006), §9.1, Eqs (9.8)-(9.13), pp. 318-321 and §9.3, Eqs (9.16), (9.19), pp. 327-328

With $M, K$ as above, substituting $q(t) = x\, e^{-i\omega t}$ (or $x \cos \omega t$ ) into $M \ddot q + K q = 0$ gives

(K - \omega^2 M)\, x = 0 ,

the pencil $(K, M)$ with $\lambda = \omega^2$ . By the previous statement there are $n$ real values $\lambda_1 \le \cdots \le \lambda_n$ and real $M$ -orthonormal vectors $x^1, \dots, x^n$ with $K x^k = \lambda_k M x^k$ . Then:

The normal coordinates $\eta_k(t) = (x^k)^{\mathsf T} M q(t)$ satisfy the uncoupled normal equations

\ddot\eta_k + \lambda_k \eta_k = 0, \qquad k = 1, \dots, n ,

and $q(t) = \sum_k \eta_k(t)\, x^k$ . In these coordinates $T = \tfrac12 \sum_k \dot\eta_k^2$ and $V = \tfrac12 \sum_k \lambda_k \eta_k^2$ . 2. Hence every solution is

q(t) = \sum_{k} \bigl(a_k \cos \omega_k t + b_k \sin \omega_k t\bigr) x^k \quad \text{over the } k \text{ with } \lambda_k = \omega_k^2 > 0 ,

plus a term $(a_k + b_k t)\, x^k$ for each $k$ with $\lambda_k = 0$ and a term $(a_k e^{\gamma_k t} + b_k e^{-\gamma_k t}) x^k$ with $\gamma_k = \sqrt{-\lambda_k}$ for each $k$ with $\lambda_k < 0$ . The positive $\omega_k = \sqrt{\lambda_k}$ are the normal frequencies (eigenfrequencies) and the motions $q(t) = \eta_k(t) x^k$ are the normal modes. 3. Every solution is bounded for all $t$ , for every choice of initial data, if and only if every $\lambda_k > 0$ , i.e. if and only if $K$ is positive definite; this is the case when the equilibrium is a strict local minimum of $V$ to second order. A negative $\lambda_k$ gives exponential growth along $x^k$ (unstable equilibrium), and a zero $\lambda_k$ gives a drift linear in $t$ along $x^k$ . 4. Orthogonality and bounds. $(x^j)^{\mathsf T} M x^k = \delta_{jk}$ and $(x^j)^{\mathsf T} K x^k = \lambda_k \delta_{jk}$ : modes of distinct frequencies are orthogonal with respect to both energies, and within a degenerate eigenspace the basis can be chosen so that the two displayed relations hold for every $j \neq k$ as well, degenerate or not — that is what part 2 of the pencil theorem delivers and what step 5 of the method below constructs. In neither case are the modes in general orthogonal with respect to the plain dot product $(x^j)^{\mathsf T} x^k$ . For any trial vector $x \neq 0$ ,

\lambda_1 \le \frac{x^{\mathsf T} K x}{x^{\mathsf T} M x} \le \lambda_n ,

with $\lambda_1 = \omega_1^2$ and $\lambda_n = \omega_n^2$ whenever those eigenvalues are positive — parts 2 and 3 allow $\lambda_k \le 0$ , and no frequency $\omega_k$ is defined for such a $k$ . So at a stable equilibrium a guessed mode shape gives an upper bound for the lowest frequency and a lower bound for the highest.

Idea of the proofOriginal

Everything follows from the one change of variables $\eta_k = (x^k)^{\mathsf T}Mq$ , which reads off the amount of the $k$ -th mode present in the configuration $q$ . Because the modes are $M$ -orthonormal, this change of variables inverts itself as $q = \sum_k\eta_kx^k$ ; and because they are also $K$ -orthogonal, applying it to $M\ddot q + Kq = 0$ kills every cross term and leaves $n$ separate equations $\ddot\eta_k + \lambda_k\eta_k = 0$ . A single scalar equation of that form is the harmonic oscillator, whose solution everyone knows and whose character depends on the sign of $\lambda_k$ alone: oscillation, drift, or exponential growth. So the behaviour of the whole coupled system is read off from $n$ signs.

ProofAdaptedAdapted from Riley, Hobson & Bence (2006), §9.1, Eqs (9.8)-(9.13), pp. 318-321

The eigenvalue problem. Substituting $q(t) = x\,e^{-i\omega t}$ into $M\ddot q + Kq = 0$ and using $\ddot q = -\omega^2 x\,e^{-i\omega t}$ gives $(-\omega^2M + K)x\,e^{-i\omega t} = 0$ ; the scalar factor never vanishes, so $(K - \omega^2 M)x = 0$ . With $\lambda = \omega^2$ this is the pencil $(K,M)$ , and $M$ is positive definite by hypothesis while both are real symmetric, so the previous statement applies: there are $n$ real $\lambda_1 \le \cdots \le \lambda_n$ and real $M$ -orthonormal $x^1,\dots,x^n$ with $Kx^k = \lambda_kMx^k$ , $(x^j)^{\mathsf T}Mx^k = \delta_{jk}$ and $(x^j)^{\mathsf T}Kx^k = \lambda_k\delta_{jk}$ . The trial form $x\cos\omega t$ leads to the same equation, since it too has $\ddot q = -\omega^2 q$ .

Part 1, the change of variables inverts itself. The $x^k$ are a basis, so any $q$ can be written $q = \sum_k c_kx^k$ ; then

(x^j)^{\mathsf T}Mq = \sum_k c_k\,(x^j)^{\mathsf T}Mx^k = \sum_k c_k\delta_{jk} = c_j ,

using $M$ -orthonormality. So $c_j = \eta_j$ and $q(t) = \sum_k\eta_k(t)\,x^k$ , as claimed. This is the only place the normalisation $(x^k)^{\mathsf T}Mx^k = 1$ is used; with a different normalisation the formula would carry a factor.

Part 1, the normal equations. Differentiating $\eta_k = (x^k)^{\mathsf T}Mq$ twice in $t$ — the $x^k$ and $M$ are constant — and then using the equation of motion in the form $M\ddot q = -Kq$ ,

\ddot\eta_k = (x^k)^{\mathsf T}M\ddot q = -(x^k)^{\mathsf T}Kq = -\sum_j \eta_j\,(x^k)^{\mathsf T}Kx^j = -\lambda_k\eta_k ,

the last step by $K$ -orthogonality, which is what removes every term but $j = k$ . So $\ddot\eta_k + \lambda_k\eta_k = 0$ , and the $n$ equations are uncoupled.

Part 1, the energies. Substituting $q = \sum_k\eta_kx^k$ and $\dot q = \sum_k\dot\eta_kx^k$ ,

T = \tfrac12\dot q^{\mathsf T}M\dot q = \tfrac12\sum_{j,k}\dot\eta_j\dot\eta_k\,(x^j)^{\mathsf T}Mx^k = \tfrac12\sum_k\dot\eta_k^2 , \qquad V = \tfrac12 q^{\mathsf T}Kq = \tfrac12\sum_{j,k}\eta_j\eta_k\,(x^j)^{\mathsf T}Kx^k = \tfrac12\sum_k\lambda_k\eta_k^2 ,

where in the second computation the double sum collapses by $K$ -orthogonality to the terms $j = k$ , each contributing $\lambda_k\eta_k^2$ .

Part 2. The scalar equation $\ddot\eta + \lambda\eta = 0$ is a second-order linear equation with constant coefficients, so its solution space is two-dimensional — a solution is determined by the two numbers $\eta(0), \dot\eta(0)$ , and the general statement for linear equations of order $n$ belongs to the theory of linear differential equations, which comes later in the handbook and is not needed as a prerequisite here. A two-parameter family of solutions is therefore all of them. For $\lambda = \omega^2 > 0$ , $\cos\omega t$ and $\sin\omega t$ are solutions and are independent, giving $\eta = a\cos\omega t + b\sin\omega t$ . For $\lambda = 0$ the equation is $\ddot\eta = 0$ , so $\eta = a + bt$ . For $\lambda < 0$ , put $\gamma = \sqrt{-\lambda} > 0$ ; then $e^{\pm\gamma t}$ are independent solutions and $\eta = ae^{\gamma t} + be^{-\gamma t}$ . Assembling $q = \sum_k\eta_kx^k$ gives the stated general solution, and the motion with $\eta_j \equiv 0$ for $j \neq k$ is $q(t) = \eta_k(t)x^k$ : the whole system moves in the fixed pattern $x^k$ , with every coordinate oscillating at the single frequency $\omega_k = \sqrt{\lambda_k}$ and passing through zero together. That is the normal mode.

Part 3. Suppose every $\lambda_k > 0$ . Then $|\eta_k(t)| = |a_k\cos\omega_kt + b_k\sin\omega_kt| \le (a_k^2+b_k^2)^{1/2}$ for all $t$ , by the Cauchy-Schwarz inequality in $\mathbb{R}^2$ applied to $(a_k,b_k)$ and $(\cos\omega_kt, \sin\omega_kt)$ , so

\|q(t)\| \le \sum_k |\eta_k(t)|\,\|x^k\| \le \sum_k (a_k^2+b_k^2)^{1/2}\|x^k\| ,

a bound independent of $t$ : every solution is bounded. Conversely suppose $\lambda_m \le 0$ for some $m$ , and choose the initial data $\eta_m(0) = 0$ , $\dot\eta_m(0) = 1$ and $\eta_k(0) = \dot\eta_k(0) = 0$ for $k \neq m$ , which is legitimate because the map from $(q(0), \dot q(0))$ to $(\eta_k(0), \dot\eta_k(0))$ is a bijection. Then $q(t) = \eta_m(t)x^m$ with $\eta_m(t) = t$ if $\lambda_m = 0$ and $\eta_m(t) = \gamma_m^{-1}\sinh(\gamma_mt)$ if $\lambda_m < 0$ , both unbounded, and $x^m \neq 0$ . So boundedness of every solution is equivalent to all $\lambda_k > 0$ , which by part 4 of the pencil theorem applied to $(K,M)$ is equivalent to $K$ being positive definite. Since $V = \tfrac12q^{\mathsf T}Kq$ to second order, $K$ positive definite says exactly that the quadratic part of $V$ is positive in every direction, i.e. the equilibrium is a strict local minimum of $V$ to second order. The two remaining cases are the computations just made: $\lambda_m < 0$ gives $e^{\gamma_m t}$ growth along $x^m$ , and $\lambda_m = 0$ gives drift linear in $t$ along $x^m$ .

Part 4. The orthogonality relations are part 2 of the pencil theorem, restated for $(K,M)$ . They do not imply $(x^j)^{\mathsf T}x^k = 0$ : $M$ -orthogonality is orthogonality in the inner product $(x,y)_M = y^{\mathsf T}Mx$ , and unless $M$ is a multiple of the identity that is a different relation — the example below exhibits two modes whose unnormalised shapes $u = (1,\sqrt3)^{\mathsf T}$ and $v = (1,-\sqrt3)^{\mathsf T}$ satisfy $u^{\mathsf T}Mv = 0$ while $u^{\mathsf T}v = -2 \neq 0$ ; rescaling to the $M$ -orthonormal $x^1, x^2$ changes only the size of that dot product, to $-2/(ml^2\sqrt{24}) = -0.408248/(ml^2)$ , not its vanishing. The bound is part 4 of the pencil theorem with $A = K$ , $B = M$ , and $\lambda_k = \omega_k^2$ for those $\lambda_k$ that are positive. $\square$

Method(Finding the normal modes of a system)AdaptedAdapted from Riley, Hobson & Bence (2006), Ch. 9 introduction and §9.1, pp. 316-321

Choose coordinates $q_i$ that vanish at the equilibrium, as many as the system has degrees of freedom and no more. A redundant coordinate makes $M$ singular and the pencil ill-posed.
Expand the energies to second order. Write $T$ in terms of the $\dot q_i$ , keeping terms quadratic in $\dot q$ , to get $M$ ; expand $V$ about $q = 0$ to second order to get $K$ . Both come out symmetric, or can be symmetrised as in quadratic forms.
Check that $M$ is positive definite (Sylvester's criterion, or its eigenvalues). It will be, if the coordinates are independent.
Solve $\det(K - \omega^2 M) = 0$ for the $n$ roots $\lambda_k = \omega_k^2$ . For $n \le 3$ expand the determinant by hand; otherwise use a symmetric generalized eigensolver (scipy.linalg.eigh(K, M)), not eig(inv(M) @ K).
For each root solve $(K - \lambda_k M) x = 0$ for the mode shape and normalise it by $(x^k)^{\mathsf T} M x^k = 1$ . For a degenerate $\lambda_k$ the solution space has dimension $> 1$ ; pick any basis of it and make it $M$ -orthonormal by Gram-Schmidt in the $M$ -inner product, which is the ordinary process with $(x,y)_M = y^{\mathsf T} M x$ in place of the dot product.
Check $\sum_k \lambda_k = \operatorname{tr}(M^{-1}K)$ and $(x^j)^{\mathsf T} M x^k = \delta_{jk}$ before interpreting the answer.
Read off the physics. $\lambda_k = 0$ means a free motion with no restoring force (a rigid translation or rotation, or a soft direction of $V$ ); $\lambda_k < 0$ means an unstable equilibrium; the signs within $x^k$ say which parts move in phase.

The recipe requires $M$ and $K$ symmetric with $M$ positive definite. If $M$ is only semidefinite, or if the forces are not derived from a potential (damping, magnetic forces, gyroscopic terms), the pencil is no longer definite, its eigenvalues need not be real, and this method does not apply.

NoteOriginal

A coupled oscillating system looks complicated only in the wrong coordinates. The pencil $(K, M)$ finds the coordinates in which it falls apart into $n$ independent harmonic oscillators: the eigenvalues are the squared frequencies you can measure, and each eigenvector is the pattern of motion that one frequency belongs to.

Remark(Conventions and where the hypotheses are used)AdaptedAdapted from Riley, Hobson & Bence (2006), Ch. 9 introduction and §9.1, pp. 316-317, 321; Arfken, Weber & Harris (2013), §6.5, pp. 322-324

The factor of one half. Riley, Hobson and Bence write $T = \dot q^{\mathsf T} A \dot q$ and $V = q^{\mathsf T} B q$ with no $\tfrac12$ , so their $A$ and $B$ are half of our $M$ and $K$ . Since $\det(B - \omega^2 A) = 2^{-n} \det(K - \omega^2 M)$ , the normal frequencies and mode shapes are unaffected; only the normalisation constant of the modes changes. Use either convention, but not both in one calculation.

Positive definite $K$ is about strict minima. A minimum of $V$ gives $K$ positive semidefinite; definiteness is stronger and fails exactly when $V$ is flat to second order in some direction. Such directions are common: the linear triatomic molecule of Riley, Hobson and Bence §9.1 (p. 321) has $\omega^2 = 0$ for the mode in which all three masses translate together, since $V$ depends only on the bond extensions. Part 3 above therefore separates "bounded motion" ( $K > 0$ ) from "no exponential growth" ( $K \ge 0$ ). Note that Riley, Hobson and Bence restrict Chapter 9 from the start to a potential with a local minimum at the equilibrium and state there (p. 317) that this makes the potential matrix positive definite, like the kinetic one; the cases $\lambda_k \le 0$ that parts 2 and 3 above admit therefore lie outside their setting, and are covered here by the pencil theorem, which assumes definiteness of $M$ only.

Do not throw away the symmetry. Multiplying by $M^{-1}$ gives the ordinary eigenproblem $M^{-1}K x = \omega^2 x$ , with the same eigenvalues and eigenvectors; but $M^{-1}K$ is in general neither symmetric nor normal, as Arfken, Weber and Harris point out for their three-mass chain (§6.5, pp. 322-324). Its eigenvectors are then not orthogonal in the plain dot product, and the left and right eigenvectors differ. $M$ -orthogonality is the correct replacement, and solvers that take $M$ as a second argument preserve it.

$B$ must be definite, not merely invertible. For $A = \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix}$ and $B = \begin{pmatrix} 1 & 0 \\ 0 & -1 \end{pmatrix}$ the equation $\det(A - \lambda B) = -\lambda^2 - 1 = 0$ gives $\lambda = \pm i$ : an indefinite $B$ destroys the reality of the spectrum, which is why part 1 assumes $B > 0$ and not just $\det B \neq 0$ .

Example(A double pendulum with unequal masses)Original

Two point masses hang in a vertical plane on light rigid rods of equal length $l$ : the upper mass is $2m$ , at the end of the first rod, and the lower mass is $m$ , at the end of the second rod hinged to the first. Let $q_1$ and $q_2$ be the angles the two rods make with the downward vertical, so that $q = 0$ is the hanging equilibrium. Write $\mu = g/l$ throughout.

Step 1: the energies to second order. The upper mass has speed $l\dot q_1$ ; the lower mass has velocity $l\dot q_1$ along the first rod's perpendicular plus $l\dot q_2$ along the second's, with angle $q_1 - q_2$ between them, so its squared speed is $l^2(\dot q_1^2 + \dot q_2^2 + 2\dot q_1\dot q_2\cos(q_1-q_2))$ . Hence

T = \tfrac12 (2m) l^2\dot q_1^2 + \tfrac12 m l^2\bigl(\dot q_1^2 + \dot q_2^2 + 2\dot q_1\dot q_2\cos(q_1-q_2)\bigr) .

Keeping only terms quadratic in the velocities means evaluating the cosine at $q = 0$ , which gives $1$ , so

T = \tfrac12\,\dot q^{\mathsf T}M\dot q , \qquad M = ml^2\begin{pmatrix} 3 & 1 \\ 1 & 1\end{pmatrix}.

The heights below the pivot are $l\cos q_1$ and $l\cos q_1 + l\cos q_2$ , so, taking the pivot as the zero of potential,

V = -(2m)gl\cos q_1 - mg\bigl(l\cos q_1 + l\cos q_2\bigr) = \text{const} + \tfrac12 mgl\bigl(3q_1^2 + q_2^2\bigr) + O(\|q\|^4),

using $\cos q = 1 - q^2/2 + O(q^4)$ . The constant is $V(0) = -4mgl$ , the energy of the hanging configuration measured from the pivot; dropping it changes no force and no mode, and puts $V$ in the normalisation $V(0) = 0$ that the definition of small oscillations above asks for. Hence

V = \tfrac12 q^{\mathsf T}Kq , \qquad K = mgl\begin{pmatrix} 3 & 0 \\ 0 & 1\end{pmatrix}.

Step 2: $M$ is positive definite. Its leading principal minors are $3ml^2 > 0$ and $(3-1)(ml^2)^2 = 2m^2l^4 > 0$ , so Sylvester's criterion (Quadratic forms and the Rayleigh quotient) applies and the pencil is definite. $K$ is diagonal with positive entries, so it is positive definite too, and the equilibrium will turn out stable.

Step 3: the characteristic equation. Factor out $ml^2$ from both matrices, which does not change the roots:

K - \lambda M = ml^2\begin{pmatrix} 3\mu - 3\lambda & -\lambda \\ -\lambda & \mu - \lambda\end{pmatrix}, \qquad \det(K-\lambda M) = (ml^2)^2\bigl[3(\mu-\lambda)^2 - \lambda^2\bigr] .

Expanding, $3(\mu-\lambda)^2 - \lambda^2 = 2\lambda^2 - 6\mu\lambda + 3\mu^2$ , a quadratic of degree exactly $n = 2$ with leading coefficient $2 = (-1)^2\det\begin{pmatrix}3&1\\1&1\end{pmatrix}$ , as part 1 of the pencil theorem predicts. Its roots are

\lambda = \frac{6\mu \pm \sqrt{36\mu^2 - 24\mu^2}}{4} = \frac{6\mu \pm 2\sqrt3\,\mu}{4}, \qquad\text{so}\qquad \lambda_1 = \frac{3-\sqrt3}{2}\,\mu , \quad \lambda_2 = \frac{3+\sqrt3}{2}\,\mu .

Numerically $\lambda_1 = 0.633975\,\mu$ and $\lambda_2 = 2.366025\,\mu$ , both positive and real as they must be. The frequencies are

\omega_1 = 0.796225\sqrt{g/l}, \qquad \omega_2 = 1.538189\sqrt{g/l}, \qquad \frac{\omega_2}{\omega_1} = 1.931852 .

Step 4: the checks of step 6 of the method. $\lambda_1 + \lambda_2 = 3\mu$ , and

M^{-1}K = \frac{1}{2}\begin{pmatrix} 1 & -1 \\ -1 & 3\end{pmatrix}\cdot\mu\begin{pmatrix} 3&0\\0&1\end{pmatrix} = \frac\mu2\begin{pmatrix} 3 & -1 \\ -3 & 3\end{pmatrix}, \qquad \operatorname{tr}(M^{-1}K) = 3\mu \ \checkmark

Also $\lambda_1\lambda_2 = (9-3)\mu^2/4 = \tfrac32\mu^2$ , which matches $\det K/\det M = 3\mu^2/2$ . (Note that $M^{-1}K$ is not symmetric, exactly as the third remark above warns.)

Step 5: the mode shapes. The first row of $(K-\lambda M)x = 0$ reads $3(\mu-\lambda)x_1 = \lambda x_2$ , so $x_2/x_1 = 3(\mu-\lambda)/\lambda$ . For $\lambda_1$ , $\mu - \lambda_1 = \tfrac{\sqrt3-1}{2}\mu$ and

\frac{x_2}{x_1} = \frac{3(\sqrt3-1)/2}{(3-\sqrt3)/2} = \frac{3(\sqrt3-1)}{\sqrt3(\sqrt3-1)} = \sqrt3 ,

while for $\lambda_2$ , $\mu - \lambda_2 = -\tfrac{1+\sqrt3}{2}\mu$ and the same computation gives $x_2/x_1 = -\sqrt3$ . So, before normalisation,

x^1 \propto (1,\ \sqrt3)^{\mathsf T}, \qquad x^2 \propto (1,\ -\sqrt3)^{\mathsf T} .

In the slow mode the two rods swing the same way, the lower one through $\sqrt3 \approx 1.732$ times the angle; in the fast mode they swing in opposition. Both are visible in the signs, which is the reading asked for in step 7 of the method.

Step 6: $M$ -orthonormalisation and the orthogonality checks. With $\tilde M = \begin{pmatrix}3&1\\1&1\end{pmatrix}$ so that $M = ml^2\tilde M$ ,

(1, \sqrt3)\,\tilde M\,(1,-\sqrt3)^{\mathsf T} = (1,\sqrt3)\cdot(3-\sqrt3,\ 1-\sqrt3)^{\mathsf T} = (3-\sqrt3) + (\sqrt3 - 3) = 0 ,

so the two modes are $M$ -orthogonal, as part 4 requires. Their $M$ -norms are

(1,\sqrt3)\,\tilde M\,(1,\sqrt3)^{\mathsf T} = 6 + 2\sqrt3 = 9.464102 , \qquad (1,-\sqrt3)\,\tilde M\,(1,-\sqrt3)^{\mathsf T} = 6 - 2\sqrt3 = 2.535898 ,

so the normalised modes are

x^1 = \frac{(1,\sqrt3)^{\mathsf T}}{\sqrt{ml^2(6+2\sqrt3)}} = \frac{(0.325058,\ 0.563016)^{\mathsf T}}{\sqrt{ml^2}}, \qquad x^2 = \frac{(1,-\sqrt3)^{\mathsf T}}{\sqrt{ml^2(6-2\sqrt3)}} = \frac{(0.627963,\ -1.087664)^{\mathsf T}}{\sqrt{ml^2}} .

They are not orthogonal in the plain dot product: $(1,\sqrt3)\cdot(1,-\sqrt3) = 1 - 3 = -2 \neq 0$ . And $K$ -orthogonality holds: $(1,\sqrt3)\operatorname{diag}(3,1)(1,-\sqrt3)^{\mathsf T} = 3 - 3 = 0$ , while $(1,\pm\sqrt3)\operatorname{diag}(3,1)(1,\pm\sqrt3)^{\mathsf T} = 6$ for both, and indeed $6\mu/(6+2\sqrt3) = \lambda_1$ and $6\mu/(6-2\sqrt3) = \lambda_2$ , as $(x^k)^{\mathsf T}Kx^k = \lambda_k$ demands.

Step 7: a trial-vector bound. Guess the slow mode as $x = (1,1)^{\mathsf T}$ — both rods swinging together through equal angles, which is in the right ballpark but has the wrong amplitude ratio. Then

\frac{x^{\mathsf T}Kx}{x^{\mathsf T}Mx} = \frac{\mu(3+1)}{3+1+1+1} = \frac{4\mu}{6} = 0.666667\,\mu \ \ge\ \lambda_1 = 0.633975\,\mu ,

an upper bound, high by $5.2\%$ in $\lambda_1$ and only $2.5\%$ in $\omega_1$ because of the square root. That one line of arithmetic gets the fundamental frequency to better than three percent with no eigenvalue calculation at all.

Step 8: the motion. Since both $\lambda_k > 0$ , $K$ is positive definite and part 3 says every motion is bounded; explicitly

q(t) = (a_1\cos\omega_1t + b_1\sin\omega_1t)\,x^1 + (a_2\cos\omega_2t + b_2\sin\omega_2t)\,x^2 .

The frequency ratio is

\frac{\omega_2}{\omega_1} = \sqrt{\frac{3+\sqrt3}{3-\sqrt3}} = \sqrt{2+\sqrt3} = \frac{\sqrt2+\sqrt6}{2} = 1.931852 ,

using $(3+\sqrt3)/(3-\sqrt3) = (3+\sqrt3)^2/6 = 2+\sqrt3$ and $\bigl((\sqrt2+\sqrt6)/2\bigr)^2 = (8+4\sqrt3)/4 = 2+\sqrt3$ . This is irrational, so a general motion never exactly repeats: the linearized double pendulum is quasi-periodic, not periodic, unless one of the two amplitudes vanishes.

ApplicationOriginal

Vibrational spectroscopy. A molecule of $N$ atoms is a conservative system with $3N$ coordinates, and the apparatus of this unit is how its infrared and Raman spectrum is computed. The kinetic-energy matrix $M$ in Cartesian displacements is diagonal, each atom contributing its mass three times; the stiffness matrix $K$ is the Hessian of the electronic energy at the equilibrium geometry, which a quantum chemistry program computes by differentiating the energy twice. Diagonalizing the pencil $(K, M)$ gives $3N$ eigenvalues, of which several vanish: the rigid motions change no bond and so leave $V$ flat, precisely the $\lambda_k = 0$ case of part 3. For a nonlinear molecule there are six such directions — three rigid translations and three rigid rotations — leaving $3N-6$ vibrational frequencies $\omega_k = \sqrt{\lambda_k}$ to compare with the measured absorption lines. For a linear molecule only five vanish, because every atom sits on the axis and a rotation about that axis displaces no atom at all, so it is not a direction in configuration space; such a molecule has $3N-5$ vibrations. Thus a linear triatomic, CO $_2$ for instance, has $3 \cdot 3 - 5 = 4$ vibrational frequencies rather than the $3$ that $3N-6$ would give, and a diatomic has the one frequency of its stretching mode, where $3N-6$ would predict none. The three-mass chain of the second remark above is the axial part of this calculation for such a molecule: only the three displacements along the axis are retained, so it yields three modes — the rigid translation with $\omega^2 = 0$ and two axial vibrations — while the remaining two of the four vibrations are bending motions transverse to the axis, which that reduced model does not carry. The eigenvectors are the pictures that accompany each line in a spectroscopy paper: which atoms move, how far, and in or out of phase. A negative eigenvalue in such a calculation is the standard diagnostic that the geometry found is not a minimum but a saddle point of the energy surface — a transition state — and the corresponding eigenvector is the reaction coordinate along which the molecule falls apart. Part 3 is exactly that statement: a negative $\lambda_k$ means exponential departure along $x^k$ .

Modal analysis of structures, and the practical use of the quotient bound. A building, a bridge or a turbine blade is modelled by a finite element mesh whose nodal displacements $q$ carry a mass matrix $M$ and a stiffness matrix $K$ , both large, sparse and symmetric with $M$ positive definite. Engineers compute the lowest few eigenvalues of $(K,M)$ because resonance with those frequencies is what destroys structures: a design is checked by confirming that the expected excitation — traffic, wind, rotor imbalance, the dominant frequencies of regional earthquakes — is far from every $\omega_k$ . The method's warning in step 4 matters here: a solver given $M$ as a second argument preserves the symmetry of the problem and the $M$ -orthogonality of the computed modes, whereas forming $M^{-1}K$ destroys both, and on a large mesh it also destroys the sparsity that made the computation feasible. Rayleigh's method is the classical hand version of step 7 of the example: assume a plausible deflection shape and evaluate $x^{\mathsf T}Kx/x^{\mathsf T}Mx$ . What the quotient bounds is $\lambda_1 = \omega_1^2$ , from above; the square root of the quotient is then an upper bound on the fundamental frequency itself. The stationarity statement is what makes the estimate sharp: since the quotient is stationary at the true mode, its error is quadratic in the error of the shape, so a shape wrong by ten percent gives $\lambda_1$ wrong by about one percent and $\omega_1$ wrong by only about half of one percent, the square root halving the relative error. Step 7 of the example is that arithmetic in miniature — a crude shape overestimates $\lambda_1$ by $5.2\%$ and $\omega_1$ by $2.5\%$ .

Instability as a negative eigenvalue. The same pencil analysed about an unstable equilibrium delivers the growth rate rather than a frequency. An inverted pendulum, a column under a compressive load past its buckling threshold, or a plasma column in an adverse field all have a stiffness matrix $K$ with at least one negative eigenvalue, and part 3 then gives the fastest-growing disturbance as the eigenvector $x^m$ for the most negative $\lambda_m$ , with growth rate $\gamma_m = \sqrt{-\lambda_m}$ . Linear stability analysis in this form — assemble $K$ , test its definiteness, and if it fails read off the shape of the unstable mode — is the standard first calculation for any equilibrium whose stability is in question, and the buckling load of a column is found by increasing the load until the smallest eigenvalue of $K$ first crosses zero.

Theorem

Singular value decomposition

Adapted

Add to PDF

Every complex m-by-n matrix factors as A = USV* with U, V unitary and S non-negative diagonal; the singular values are the square roots of the min(m,n) largest eigenvalues of A*A, truncating the factorisation gives the best approximation of each rank, and inverting the non-zero part gives the pseudoinverse that solves least squares with minimum norm.

Prerequisites: Spectral theorem for Hermitian matrices, Orthogonal projection and least squares, Normal matrices and simultaneous diagonalization, Orthonormal bases and Gram-Schmidt

Definition(Singular values and the singular value decomposition)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.18.3, Eqs (8.131)-(8.133), pp. 302-303

Let $A$ be a complex $m \times n$ matrix and $p = \min(m, n)$ . A singular value decomposition (SVD) of $A$ is a factorisation

A = U S V^\dagger

in which $U$ is $m \times m$ unitary, $V$ is $n \times n$ unitary, and $S$ is $m \times n$ with $S_{ij} = 0$ for $i \neq j$ and diagonal entries

s_1 \ge s_2 \ge \cdots \ge s_p \ge 0 ,

the singular values of $A$ . The columns $u_1, \dots, u_m$ of $U$ are the left singular vectors and the columns $v_1, \dots, v_n$ of $V$ the right singular vectors. For real $A$ one may take $U$ and $V$ real orthogonal.

Theorem(Existence and basic properties)AdaptedAdapted from Reed & Simon (1980), §VI.5, Thm VI.17, pp. 203-204; Riley, Hobson & Bence (2006), §8.18.3, Eqs (8.132)-(8.136) and pp. 303-304

Every complex $m \times n$ matrix $A$ has a singular value decomposition, and:

Where the singular values come from. $A^\dagger A$ ( $n \times n$ ) and $A A^\dagger$ ( $m \times m$ ) are Hermitian and positive semidefinite, so their eigenvalues are real and $\ge 0$ . The numbers $s_1^2, \dots, s_p^2$ are the $p$ largest eigenvalues of each, in decreasing order; the remaining eigenvalues of the larger of the two matrices are zero. Equivalently $s_i = \sqrt{\lambda_i}$ with $\lambda_i$ the $i$ -th largest eigenvalue of $A^\dagger A$ . One may take for $V$ any unitary whose columns are an orthonormal eigenbasis of $A^\dagger A$ ordered by decreasing eigenvalue; every column of $U$ belonging to a non-zero singular value is then forced, $u_i = A v_i / s_i$ for each $i$ with $s_i > 0$ , and the remaining ones are any orthonormal completion. The columns of $U$ so obtained are an orthonormal eigenbasis of $A A^\dagger$ . The two families cannot be chosen independently of each other.
Coupling of the two families. For $i = 1, \dots, p$ ,

A v_i = s_i u_i , \qquad A^\dagger u_i = s_i v_i ,

while $A v_i = 0$ for $i > p$ and $A^\dagger u_i = 0$ for $i > p$ . 3. Rank, range, null space, dyadic form. Let $r$ be the number of non-zero singular values. Then $r = \operatorname{rank} A$ ; $u_1, \dots, u_r$ are an orthonormal basis of the range of $A$ ; $v_{r+1}, \dots, v_n$ are an orthonormal basis of its null space; $v_1, \dots, v_r$ are an orthonormal basis of the range of $A^\dagger$ ; and

A = \sum_{i=1}^{r} s_i\, u_i v_i^\dagger .

Uniqueness. The singular values are uniquely determined by $A$ . The singular vectors are not, and the non-zero and the zero parts are free in different ways. Let $r = \operatorname{rank} A$ .
- Non-zero singular values. Let $W \subseteq \{1, \dots, r\}$ be a maximal block of indices on which $s_i$ takes one common value $s > 0$ . Writing $U_W$ and $V_W$ for the matrices of the corresponding columns, the pair may be replaced by $U_W Q$ and $V_W Q$ for any $|W| \times |W|$ unitary $Q$ — the same $Q$ on both sides, as the coupling $A v_i = s\, u_i$ of part 2 forces. The common phase $u_i \mapsto e^{i\alpha} u_i$ , $v_i \mapsto e^{i\alpha} v_i$ is the case $|W| = 1$ .
- Zero singular values. The trailing columns $u_{r+1}, \dots, u_m$ and $v_{r+1}, \dots, v_n$ are constrained only by being orthonormal completions, of a basis of $\ker A^\dagger$ and of $\ker A$ respectively. They contribute nothing to $A = \sum_{i \le r} s_i u_i v_i^\dagger$ , so here no matching is required: the two completions may be changed by unrelated unitaries, of sizes $m - r$ and $n - r$ , which need not even be equal.
Every SVD of $A$ arises from any one of them by such changes.
Relation to the eigenvalues. For a general square $A$ the singular values are the eigenvalues of $|A| = (A^\dagger A)^{1/2}$ , and bear no simple relation to the eigenvalues of $A$ itself. If $A$ is normal — that is, $A A^\dagger = A^\dagger A$ , which holds in particular for Hermitian and for unitary $A$ — with eigenvalues $\lambda_1, \dots, \lambda_n$ listed with multiplicity, then the singular values are the numbers $|\lambda_k|$ sorted decreasingly.

Idea of the proofOriginal

There is only one Hermitian matrix in sight, $A^\dagger A$ , and the spectral theorem hands us an orthonormal eigenbasis for it. Its eigenvalues measure $\|Ax\|_2^2$ along that basis, so their square roots are the stretching factors we are after, and dividing each image $A v_i$ by its own length produces an orthonormal family on the target side. Between those two bases $A$ is diagonal, and that is the factorisation. The rest of the statement — the second eigenvalue list, the coupling, rank, range, null space, the dyadic form, and exactly how much freedom is left — is read off from that one diagonalisation.

ProofAdaptedAdapted from Reed & Simon (1980), §VI.5, Thm VI.17, pp. 203-204; Riley, Hobson & Bence (2006), §8.18.3, Eqs (8.132)-(8.136), pp. 302-304

$A^\dagger A$ is Hermitian and positive semidefinite. $(A^\dagger A)^\dagger = A^\dagger A$ , since $(XY)^\dagger = Y^\dagger X^\dagger$ and $(A^\dagger)^\dagger = A$ . For every $x \in \mathbb{C}^n$ ,

x^\dagger A^\dagger A x = (A x)^\dagger (A x) = \|A x\|_2^2 \ \ge\ 0 .

Each eigenvalue $\lambda$ of $A^\dagger A$ is real by the spectral theorem, and taking $x$ to be an eigenvector for it the display reads $\lambda \|x\|_2^2 \ge 0$ , so $\lambda \ge 0$ . The same two lines with $A^\dagger$ in place of $A$ settle $A A^\dagger$ .

An orthonormal eigenbasis on the right. By the spectral theorem pick an orthonormal basis $v_1, \dots, v_n$ of $\mathbb{C}^n$ with $A^\dagger A\, v_i = \lambda_i v_i$ , numbered so that $\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_n \ge 0$ . Put $s_i = \sqrt{\lambda_i} \ge 0$ , and let $r$ be the number of indices with $\lambda_i > 0$ ; by the ordering these are $i = 1, \dots, r$ .
At most $p$ of the $s_i$ are non-zero. If $A^\dagger A x = 0$ then $\|Ax\|_2^2 = x^\dagger A^\dagger A x = 0$ by step 1, so $\ker(A^\dagger A) = \ker A$ and hence $\operatorname{rank}(A^\dagger A) = \operatorname{rank} A$ (least squares, part 4). Since $A^\dagger A$ is diagonalisable with exactly $r$ non-zero eigenvalues, $r = \operatorname{rank} A \le \min(m, n) = p$ . In particular $\lambda_i = 0$ for every $i > p$ , so the numbers $s_1 \ge \cdots \ge s_p$ carry all of the information and fit on the diagonal of an $m \times n$ matrix.
Normalising the images. For $i \le r$ put $u_i = A v_i / s_i$ , which is legitimate because $s_i > 0$ . These vectors are orthonormal:

u_i^\dagger u_j = \frac{(A v_i)^\dagger (A v_j)}{s_i s_j} = \frac{v_i^\dagger A^\dagger A v_j}{s_i s_j} = \frac{\lambda_j\, v_i^\dagger v_j}{s_i s_j} = \frac{s_j}{s_i}\, \delta_{ij} = \delta_{ij} ,

by the definition of $u_i$ , then the definition of the adjoint, then step 2, then the orthonormality of the $v_j$ . For $i > r$ we have $\lambda_i = 0$ , so $\|A v_i\|_2^2 = v_i^\dagger A^\dagger A v_i = 0$ and therefore $A v_i = 0$ .

Completing the basis gives the factorisation. Since $r \le m$ by step 3, the orthonormal set $u_1, \dots, u_r$ extends to an orthonormal basis $u_1, \dots, u_m$ of $\mathbb{C}^m$ : take any basis of $(\operatorname{span}\{u_1, \dots, u_r\})^\perp$ and orthonormalise it by Gram-Schmidt. Let $U$ and $V$ be the matrices with those columns; both are unitary, their columns being orthonormal bases. Column $j$ of $U^\dagger A V$ is $U^\dagger A v_j$ , which by step 4 is $s_j U^\dagger u_j = s_j e_j$ when $j \le r$ and $0$ when $j > r$ . As $s_j = 0$ for $r < j \le p$ , that is precisely the matrix $S$ of the definition, so $U^\dagger A V = S$ , that is, $A = U S V^\dagger$ . The construction also establishes the recipe asserted in part 1: $V$ may be any unitary whose columns are an orthonormal eigenbasis of $A^\dagger A$ in decreasing order of eigenvalue, and then $u_i = A v_i / s_i$ is forced whenever $s_i > 0$ , the remaining columns being any orthonormal completion.
Part 1, the two eigenvalue lists. Conversely, from any SVD $A = U S V^\dagger$ , unitarity of $U$ and $V$ gives

A^\dagger A = V (S^\dagger S) V^\dagger , \qquad A A^\dagger = U (S S^\dagger) U^\dagger .

Since $S_{ij} = 0$ off the diagonal, $(S^\dagger S)_{jj} = \sum_i |S_{ij}|^2$ is $s_j^2$ for $j \le p$ and $0$ for $j > p$ , and likewise $(S S^\dagger)_{ii}$ is $s_i^2$ for $i \le p$ and $0$ for $i > p$ ; both matrices are diagonal. A unitary similarity does not change eigenvalues, so the eigenvalues of $A^\dagger A$ are $s_1^2, \dots, s_p^2$ together with $n - p$ zeros, those of $A A^\dagger$ the same numbers with $m - p$ zeros, and the columns of $V$ and of $U$ are orthonormal eigenbases of the two matrices.

Part 2, the coupling. Multiplying $A = U S V^\dagger$ on the right by $V$ gives $A V = U S$ . Column $i$ of the left-hand side is $A v_i$ ; column $i$ of the right-hand side is $s_i u_i$ for $i \le p$ and $0$ for $i > p$ , because column $i$ of $S$ then vanishes. That is the first identity. Taking adjoints, $A^\dagger = V S^\dagger U^\dagger$ , so $A^\dagger U = V S^\dagger$ , and reading columns the same way — $S$ is real, so $S^\dagger = S^{\mathsf T}$ — gives $A^\dagger u_i = s_i v_i$ for $i \le p$ and $A^\dagger u_i = 0$ for $i > p$ .
Part 3, rank, range, null space and the dyadic form. By part 2 together with $s_i = 0$ for $r < i \le p$ , $A v_i = 0$ for every $i > r$ . Expanding an arbitrary $x$ as $x = \sum_{i=1}^n (v_i^\dagger x)\, v_i$ ,

A x = \sum_{i=1}^{n} (v_i^\dagger x)\, A v_i = \sum_{i=1}^{r} s_i (v_i^\dagger x)\, u_i = \Bigl( \sum_{i=1}^{r} s_i u_i v_i^\dagger \Bigr) x ,

valid for every $x$ , which is the dyadic form. It gives $\operatorname{im} A \subseteq \operatorname{span}\{u_1, \dots, u_r\}$ , and the reverse inclusion holds because $A v_i = s_i u_i$ with $s_i \neq 0$ ; the $u_i$ are orthonormal, hence independent, so $\operatorname{rank} A = r$ and $u_1, \dots, u_r$ is an orthonormal basis of the range. For the null space, $A x = 0$ says $\sum_{i \le r} s_i (v_i^\dagger x) u_i = 0$ , which by independence of the $u_i$ and $s_i \neq 0$ forces $v_i^\dagger x = 0$ for every $i \le r$ , that is $x \in \operatorname{span}\{v_{r+1}, \dots, v_n\}$ ; and conversely those $v_i$ are in $\ker A$ . Applying what has just been proved to $A^\dagger = \sum_{i \le r} s_i v_i u_i^\dagger$ shows that $v_1, \dots, v_r$ is an orthonormal basis of $\operatorname{im} A^\dagger$ .

Part 4, uniqueness of the singular values. By step 6 the numbers $s_1^2 \ge \cdots \ge s_p^2$ are the $p$ largest eigenvalues of $A^\dagger A$ , hence determined by $A$ alone, and the constraint $s_i \ge 0$ fixes the square roots.
Part 4, the freedom in the singular vectors. Let $A = U S V^\dagger = U' S V'^\dagger$ be two SVDs, with the same $S$ by step 9. By step 6 the columns of $V$ and of $V'$ are both orthonormal eigenbases of $A^\dagger A$ in decreasing order of eigenvalue. Two orthonormal bases adapted to the same eigenspaces in the same order differ exactly by a unitary change of basis inside each eigenspace. Those eigenspaces are $\operatorname{span}\{v_i : i \in W\}$ for each maximal block $W$ of indices carrying one common value $s > 0$ , and $\operatorname{span}\{v_{r+1}, \dots, v_n\} = \ker A$ for the eigenvalue $0$ . So, columnwise, $V'_W = V_W Q_W$ with $Q_W$ unitary of size $|W|$ , while $v'_{r+1}, \dots, v'_n$ is an arbitrary orthonormal basis of $\ker A$ . For $i \le r$ the left vectors are then forced, $u'_i = A v'_i / s_i$ by step 5, so

U'_W = A V'_W / s = A V_W Q_W / s = U_W Q_W

with the same unitary $Q_W$ . The trailing columns $u'_{r+1}, \dots, u'_m$ complete $u'_1, \dots, u'_r$ , a basis of $\operatorname{im} A$ , to an orthonormal basis of $\mathbb{C}^m$ , so they form an arbitrary orthonormal basis of $(\operatorname{im} A)^\perp = \ker A^\dagger$ , of dimension $m - r$ and unrelated to the $n - r$ dimensions available on the other side. Conversely each such change does produce an SVD, because inside a block $U_W Q_W (V_W Q_W)^\dagger = U_W Q_W Q_W^\dagger V_W^\dagger = U_W V_W^\dagger$ leaves the dyadic sum of part 3 unchanged, and the trailing columns never enter it.

Part 5, the normal case. For square $A$ (so $m = n = p$ ), step 6 gives $A^\dagger A = V \operatorname{diag}(s_1^2, \dots, s_n^2) V^\dagger$ , and $B = V \operatorname{diag}(s_1, \dots, s_n) V^\dagger$ is positive semidefinite with $B^2 = A^\dagger A$ . It is the only such matrix, so the notation $|A| = (A^\dagger A)^{1/2}$ is unambiguous: a positive semidefinite $C$ with $C^2 = A^\dagger A$ commutes with $A^\dagger A$ , hence maps each eigenspace of $A^\dagger A$ into itself, and on the eigenspace for $\mu$ it is a positive semidefinite matrix whose square is $\mu I$ , so it is Hermitian with every eigenvalue equal to $\sqrt{\mu}$ and therefore equals $\sqrt{\mu}\, I$ there. The eigenvalues of $|A| = B$ are thus $s_1, \dots, s_n$ . If in addition $A$ is normal, normal matrices supplies a unitary $W$ with $A = W \Lambda W^\dagger$ and $\Lambda = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$ ; then $A^\dagger A = W \overline{\Lambda} \Lambda W^\dagger = W \operatorname{diag}(|\lambda_1|^2, \dots, |\lambda_n|^2) W^\dagger$ , so the eigenvalues of $A^\dagger A$ are the $|\lambda_k|^2$ and the singular values, their square roots in decreasing order, are the $|\lambda_k|$ sorted decreasingly. $\square$

Theorem(Spectral norm and best low-rank approximation)Original

Let $A$ be $m \times n$ with singular values $s_1 \ge \cdots \ge s_p \ge 0$ and rank $r$ , let $\|A\|_2 = \max_{x \neq 0} \|Ax\|_2/\|x\|_2$ and $\|A\|_F = \bigl(\sum_{i,j} |A_{ij}|^2\bigr)^{1/2}$ . Then $\|A\|_2 = s_1$ and $\|A\|_F = \bigl(\sum_{i=1}^{p} s_i^2\bigr)^{1/2}$ , and for each integer $k$ with $1 \le k < r$ the truncation

A_k = \sum_{i=1}^{k} s_i\, u_i v_i^\dagger

(of rank exactly $k$ ) is a best rank- $k$ approximation to $A$ in both norms: over all $m \times n$ matrices $B$ with $\operatorname{rank} B \le k$ ,

\min_B \|A - B\|_2 = \|A - A_k\|_2 = s_{k+1} , \qquad \min_B \|A - B\|_F = \|A - A_k\|_F = \Bigl(\sum_{i>k} s_i^2\Bigr)^{1/2} .

The Frobenius case is the Eckart-Young theorem (1936), whose integral-operator analogue goes back to Schmidt (1907); Mirsky (1960) extended it to every unitarily invariant norm, which is where the spectral-norm case comes from. The Frobenius minimiser is unique exactly when $s_k > s_{k+1}$ ; the spectral-norm minimiser is in general not unique. For $k \ge r$ the minimum is $0$ , attained by $A$ itself.

Idea of the proofOriginal

Both norms are blind to the unitary factors, so each is a function of the singular values alone, and the error made by chopping the dyadic sum is immediate. The two lower bounds are the real content. In the spectral norm, a matrix of rank at most $k$ must annihilate some direction inside the $(k+1)$ -dimensional span of $v_1, \dots, v_{k+1}$ , and $A$ stretches every direction there by at least $s_{k+1}$ . In the Frobenius norm, split $A - B$ along the range of $B$ : the part of $A$ pointing out of that range survives untouched, and a subspace of dimension $k$ can capture at most the $k$ largest eigenvalues' worth of $A A^\dagger$ .

ProofOriginal

Throughout, fix an SVD $A = U S V^\dagger$ as in the existence statement, so that $A = \sum_{i \le r} s_i u_i v_i^\dagger$ , and write $(X, Y)_F = \operatorname{tr}(Y^\dagger X)$ , which is an inner product on $m \times n$ matrices with $\|X\|_F^2 = (X, X)_F$ and is linear in its first slot.

The dyads are orthonormal in the Frobenius inner product. For $i, j \le p$ , using $\operatorname{tr}(x y^\dagger) = y^\dagger x$ for column vectors,

(u_i v_i^\dagger,\, u_j v_j^\dagger)_F = \operatorname{tr}\bigl( v_j u_j^\dagger u_i v_i^\dagger \bigr) = (u_j^\dagger u_i)\, \operatorname{tr}(v_j v_i^\dagger) = \delta_{ij}\, (v_i^\dagger v_j) = \delta_{ij} ,

by the orthonormality of the $u$ 's and then of the $v$ 's. Hence $\bigl\| \sum_i c_i u_i v_i^\dagger \bigr\|_F^2 = \sum_i |c_i|^2$ for any scalars $c_i$ .

The Frobenius norm and the Frobenius truncation error. Applying step 1 to $A = \sum_{i \le r} s_i u_i v_i^\dagger$ gives $\|A\|_F^2 = \sum_{i \le r} s_i^2 = \sum_{i \le p} s_i^2$ , the omitted terms being zero. Since $A - A_k = \sum_{k < i \le r} s_i u_i v_i^\dagger$ , the same step gives $\|A - A_k\|_F^2 = \sum_{i > k} s_i^2$ .
The spectral norm and the spectral truncation error. For any $x$ put $y = V^\dagger x$ , so $\|y\|_2 = \|x\|_2$ as $V$ is unitary. Then $\|Ax\|_2 = \|U S y\|_2 = \|S y\|_2$ , again by unitarity, and

\|S y\|_2^2 = \sum_{i \le p} s_i^2 |y_i|^2 \ \le\ s_1^2 \|y\|_2^2 ,

with equality at $x = v_1$ (then $y = e_1$ ); so $\|A\|_2 = s_1$ . Running the same computation for $A - A_k$ , whose dyadic form has only the indices $k < i \le r$ , gives $\|(A - A_k)x\|_2^2 = \sum_{k < i \le r} s_i^2 |v_i^\dagger x|^2 \le s_{k+1}^2 \|x\|_2^2$ with equality at $x = v_{k+1}$ , so $\|A - A_k\|_2 = s_{k+1}$ . By the range argument of part 3 of the existence statement, applied to $A_k = \sum_{i \le k} s_i u_i v_i^\dagger$ with $s_1, \dots, s_k > 0$ (here $k < r$ ), $\operatorname{im} A_k = \operatorname{span}\{u_1, \dots, u_k\}$ , so $\operatorname{rank} A_k = k$ exactly.

Spectral lower bound. Let $\operatorname{rank} B \le k$ . Then $\dim \ker B \ge n - k$ , while $\dim \operatorname{span}\{v_1, \dots, v_{k+1}\} = k+1$ , and $(n-k) + (k+1) > n$ , so the two subspaces of $\mathbb{C}^n$ intersect in a non-zero vector. Normalise it: $w = \sum_{i \le k+1} c_i v_i$ with $\sum_{i \le k+1} |c_i|^2 = 1$ and $B w = 0$ . Then, by part 2 of the existence statement and the orthonormality of the $u_i$ ,

\|(A - B)w\|_2 = \|A w\|_2 = \Bigl\| \sum_{i \le k+1} c_i s_i u_i \Bigr\|_2 = \Bigl( \sum_{i \le k+1} |c_i|^2 s_i^2 \Bigr)^{1/2} \ \ge\ s_{k+1} ,

the last step because $s_i \ge s_{k+1}$ for every $i \le k+1$ and the $|c_i|^2$ sum to $1$ . Since $\|w\|_2 = 1$ , this gives $\|A - B\|_2 \ge s_{k+1}$ , which with step 3 proves the spectral-norm minimum.

A capture lemma. Let $M$ be an $m \times m$ Hermitian positive semidefinite matrix with eigenvalues $\mu_1 \ge \cdots \ge \mu_m \ge 0$ and an orthonormal eigenbasis $w_1, \dots, w_m$ , and let $q_1, \dots, q_j$ be orthonormal in $\mathbb{C}^m$ with $j \le k$ . Then $\sum_{l \le j} q_l^\dagger M q_l \le \sum_{i \le k} \mu_i$ . To see it, expand $M = \sum_i \mu_i w_i w_i^\dagger$ and set $t_i = \sum_{l \le j} |w_i^\dagger q_l|^2$ , so that $\sum_{l \le j} q_l^\dagger M q_l = \sum_i \mu_i t_i$ . Now $t_i$ is the squared norm of the orthogonal projection of $w_i$ onto $\operatorname{span}\{q_1, \dots, q_j\}$ , hence $0 \le t_i \le \|w_i\|_2^2 = 1$ (orthogonal projection, parts 3 and 4), while summing over $i$ instead gives $\sum_i t_i = \sum_{l \le j} \|q_l\|_2^2 = j \le k$ , because $w_1, \dots, w_m$ is an orthonormal basis. Therefore

\sum_i \mu_i t_i - \sum_{i \le k} \mu_i = \sum_{i \le k} \mu_i (t_i - 1) + \sum_{i > k} \mu_i t_i \ \le\ \mu_k \sum_{i \le k} (t_i - 1) + \mu_k \sum_{i > k} t_i = \mu_k \Bigl( \sum_i t_i - k \Bigr) \ \le\ 0 ,

where the first inequality uses $t_i - 1 \le 0$ with $\mu_i \ge \mu_k$ for $i \le k$ and $t_i \ge 0$ with $\mu_i \le \mu_k$ for $i > k$ , and the second uses $\mu_k \ge 0$ with $\sum_i t_i \le k$ .

Frobenius lower bound. Let $\operatorname{rank} B \le k$ , let $L = \operatorname{im} B$ , of dimension $j \le k$ , and let $Q$ be the orthogonal projection of $\mathbb{C}^m$ onto $L$ , so $Q B = B$ . Split

A - B = (I - Q)A + (QA - B) .

Every column of the first term lies in $L^\perp$ and every column of the second in $L$ , so $(\,(I-Q)A,\, QA - B\,)_F = 0$ , the trace being the sum of the columnwise inner products. Hence

\|A - B\|_F^2 = \|(I-Q)A\|_F^2 + \|QA - B\|_F^2 \ \ge\ \|(I-Q)A\|_F^2 = \|A\|_F^2 - \|QA\|_F^2 ,

the last equality being Pythagoras applied to each column of $A$ . Writing $Q = \sum_{l \le j} q_l q_l^\dagger$ for an orthonormal basis $q_1, \dots, q_j$ of $L$ and using $Q^\dagger Q = Q$ ,

\|QA\|_F^2 = \operatorname{tr}(A^\dagger Q A) = \operatorname{tr}(Q A A^\dagger) = \sum_{l \le j} q_l^\dagger A A^\dagger q_l \ \le\ \sum_{i \le k} s_i^2 ,

by step 5 with $M = A A^\dagger$ , whose eigenvalues are $s_1^2, \dots, s_p^2$ and zeros by part 1 of the existence statement. Combining, $\|A - B\|_F^2 \ge \|A\|_F^2 - \sum_{i \le k} s_i^2 = \sum_{i > k} s_i^2$ , which with step 2 proves the Frobenius minimum.

The Frobenius minimiser is unique when $s_k > s_{k+1}$ . Suppose $\operatorname{rank} B \le k$ and $\|A - B\|_F^2 = \sum_{i>k} s_i^2$ . Equality in step 6 forces $B = QA$ and $\sum_{l \le j} q_l^\dagger A A^\dagger q_l = \sum_{i \le k} s_i^2$ , so both inequalities of step 5 are tight for $M = A A^\dagger$ , $\mu_i = s_i^2$ . Here $\mu_k = s_k^2 > 0$ because $k < r$ , so tightness of the second gives $\sum_i t_i = j = k$ . Tightness of the first says

\sum_{i \le k} (\mu_i - \mu_k)(1 - t_i) + \sum_{i > k} (\mu_k - \mu_i) t_i = 0 ,

a sum of non-negative terms. Choosing $w_i = u_i$ as the eigenbasis of $A A^\dagger$ , the hypothesis $s_k > s_{k+1}$ makes $\mu_k - \mu_i > 0$ for every $i > k$ , so $t_i = 0$ there; then $\sum_{i \le k} t_i = k$ with each $t_i \le 1$ forces $t_i = 1$ for $i \le k$ , that is $u_i \in L$ for $i \le k$ . As $\dim L = k$ , this gives $L = \operatorname{span}\{u_1, \dots, u_k\}$ and $Q = \sum_{i \le k} u_i u_i^\dagger$ . Finally $u_i^\dagger A = (A^\dagger u_i)^\dagger = s_i v_i^\dagger$ by part 2 of the existence statement, so

B = QA = \sum_{i \le k} u_i (u_i^\dagger A) = \sum_{i \le k} s_i u_i v_i^\dagger = A_k .

This also shows $A_k$ does not depend on which SVD was fixed at the start, as it must not: it is the unique minimiser.

It is not unique when $s_k = s_{k+1}$ . Write $s$ for the common value, which is positive because $k < r$ . For real $\theta$ let

u'_k = \cos\theta\, u_k + \sin\theta\, u_{k+1}, \quad u'_{k+1} = -\sin\theta\, u_k + \cos\theta\, u_{k+1} ,

and define $v'_k, v'_{k+1}$ by the same formulas. By part 4 of the existence statement — the case of a block $W \supseteq \{k, k+1\}$ of equal singular values, rotated by the same real orthogonal $Q$ on both sides — this is again an SVD of $A$ . Its rank- $k$ truncation $A'_k = \sum_{i<k} s_i u_i v_i^\dagger + s\, u'_k v'^{\dagger}_k$ is a best rank- $k$ approximation, by steps 2 and 6 applied to that SVD. But $v'^{\dagger}_k v_{k+1} = \sin\theta$ , so $A'_k v_{k+1} = s \sin\theta\, u'_k$ , while $A_k v_{k+1} = 0$ . For $\sin\theta \neq 0$ the two differ, so there are infinitely many minimisers.

The case $k \ge r$ . Then $\operatorname{rank} A = r \le k$ , so $B = A$ is admissible and both minima are $0$ . $\square$

Definition(Pseudoinverse)AdaptedAdapted from Stakgold & Holst (2011), Exercises 5.5.3-5.5.4, pp. 325-326

With $A = U S V^\dagger$ as above and $r$ non-zero singular values, the Moore-Penrose pseudoinverse of $A$ is the $n \times m$ matrix

A^{+} = V S^{+} U^\dagger , \qquad (S^{+})_{ii} = \begin{cases} 1/s_i, & s_i > 0, \\ 0, & s_i = 0, \end{cases} \quad (S^{+})_{ij} = 0 \ (i \neq j),

so that $A^{+} = \sum_{i=1}^{r} s_i^{-1} v_i u_i^\dagger$ . It does not depend on which SVD of $A$ is used. For any $b \in \mathbb{C}^m$ , the vector $x^{+} = A^{+} b$ is the minimum-norm least-squares solution: it minimises the residual $\|A x - b\|_2$ over all $x \in \mathbb{C}^n$ , and among the minimisers it is the unique one of smallest $\|x\|_2$ .

That last property is what (Stakgold & Holst (2011), Exercises 5.5.3-5.5.4, pp. 325-326) takes as the definition of the pseudoinverse, for an operator with closed range and with no factorisation in sight. The formula $A^{+} = V S^{+} U^\dagger$ , and the identification of the two, are ours; the cited pages do not contain them. Consequently

$A A^{+}$ is the orthogonal projection of $\mathbb{C}^m$ onto the range of $A$ , and $A^{+} A$ the orthogonal projection of $\mathbb{C}^n$ onto the range of $A^\dagger$ , the orthogonal complement of $\ker A$ ;
if $A$ is square and invertible then $A^{+} = A^{-1}$ ;
if $A$ has full column rank $n$ then $A^{+} = (A^\dagger A)^{-1} A^\dagger$ , recovering the normal equations of least squares.

Idea of the proofOriginal

In the singular bases $A$ is a diagonal matrix, and for a diagonal matrix the two demands are transparent: the residual splits into coordinates the unknown can set exactly and coordinates it cannot touch at all, and the norm of the unknown splits into coordinates the data forces and coordinates that are free. Setting the free ones to zero is the whole content of the pseudoinverse. Since this description of $A^{+}b$ never mentions a basis, it cannot depend on which SVD was used.

Proof(Properties of the pseudoinverse)Original

Fix an SVD $A = U S V^\dagger$ with $r$ non-zero singular values, let $b \in \mathbb{C}^m$ , and for $x \in \mathbb{C}^n$ put $y = V^\dagger x$ and $c = U^\dagger b$ . Unitarity gives $\|x\|_2 = \|y\|_2$ , and the correspondence $x \leftrightarrow y$ is a bijection of $\mathbb{C}^n$ .

The residual in the singular bases. Since $U$ is unitary, $\|Ax - b\|_2 = \|U(Sy - c)\|_2 = \|Sy - c\|_2$ . The matrix $S$ has $(Sy)_i = s_i y_i$ for $i \le p$ and $(Sy)_i = 0$ for $p < i \le m$ , and $s_i = 0$ for $r < i \le p$ , so

\|Ax - b\|_2^2 = \sum_{i=1}^{r} |s_i y_i - c_i|^2 + \sum_{i=r+1}^{m} |c_i|^2 .

The second sum does not involve $x$ at all. Hence $x$ minimises the residual if and only if $y_i = c_i / s_i$ for every $i \le r$ , the coordinates $y_{r+1}, \dots, y_n$ being unconstrained, and the minimal residual is $\bigl( \sum_{i > r} |c_i|^2 \bigr)^{1/2}$ .

Minimum norm. Among those minimisers, $\|x\|_2^2 = \|y\|_2^2 = \sum_{i \le r} |c_i/s_i|^2 + \sum_{i > r} |y_i|^2$ , which is smallest exactly when $y_i = 0$ for every $i > r$ , and then uniquely so. That choice of $y$ is precisely $S^{+} c$ , so the minimum-norm least-squares solution is $x = V S^{+} U^\dagger b = A^{+} b$ , as claimed. Taking $b$ to run over $\mathbb{C}^m$ identifies $A^{+}$ with the operator of the cited exercises.
Independence of the SVD. Steps 1 and 2 characterise $A^{+}b$ as the unique minimum-norm minimiser of $\|Ax - b\|_2$ , a description that mentions only $A$ and $b$ . So $A^{+}$ is the same matrix for every SVD of $A$ .
The two projections. $S S^{+}$ is the $m \times m$ diagonal matrix with $r$ ones followed by zeros, since $(S S^{+})_{ij} = S_{ii} (S^{+})_{ij} = s_i s_i^{-1} \delta_{ij}$ for $i \le r$ and vanishes otherwise. Hence

A A^{+} = U S V^\dagger V S^{+} U^\dagger = U (S S^{+}) U^\dagger = \sum_{i=1}^{r} u_i u_i^\dagger ,

which is Hermitian and idempotent with range $\operatorname{span}\{u_1, \dots, u_r\} = \operatorname{im} A$ , so it is the orthogonal projection onto $\operatorname{im} A$ (orthogonal projection, part 5). Identically, $S^{+} S$ is $n \times n$ diagonal with $r$ ones, so $A^{+} A = \sum_{i \le r} v_i v_i^\dagger$ is the orthogonal projection onto $\operatorname{span}\{v_1, \dots, v_r\} = \operatorname{im} A^\dagger = (\ker A)^\perp$ .

Square and invertible. Then $r = m = n$ , so $S$ is square with all $s_i > 0$ and $S^{+} = S^{-1}$ ; hence $A^{+} = V S^{-1} U^\dagger = (U S V^\dagger)^{-1} = A^{-1}$ .
Full column rank. If $\operatorname{rank} A = n$ then $r = n \le m$ , and $S^\dagger S = \operatorname{diag}(s_1^2, \dots, s_n^2)$ is invertible, so $A^\dagger A = V (S^\dagger S) V^\dagger$ is invertible with inverse $V (S^\dagger S)^{-1} V^\dagger$ . Therefore

(A^\dagger A)^{-1} A^\dagger = V (S^\dagger S)^{-1} V^\dagger \cdot V S^\dagger U^\dagger = V \bigl( (S^\dagger S)^{-1} S^\dagger \bigr) U^\dagger ,

and $(S^\dagger S)^{-1} S^\dagger$ is the $n \times m$ matrix with $(i,i)$ entry $s_i / s_i^2 = 1/s_i$ and all other entries zero, which is $S^{+}$ . So $(A^\dagger A)^{-1} A^\dagger = V S^{+} U^\dagger = A^{+}$ , and by step 1 the minimiser is unique because no coordinates $y_i$ with $i > r = n$ exist. $\square$

NoteOriginal

The SVD says that every linear map, of any shape and any rank, is the same thing in disguise: choose the right orthonormal basis in the source and in the target, and the map simply scales $r$ perpendicular directions by $s_1, \dots, s_r$ and annihilates the rest. Because the scaling factors come out sorted, truncating the list is the optimal way to simplify the map — which is why the SVD underlies principal component analysis, image and model compression, rank determination, and the regularisation of ill-posed least-squares problems.

Remark(Traps and conventions)Original

Singular values are not eigenvalues. For $A = \begin{pmatrix} 0 & 2 \\ 0 & 0 \end{pmatrix}$ both eigenvalues are $0$ while the singular values are $2$ and $0$ . The two lists agree up to absolute values and ordering exactly for normal $A$ , meaning $A A^\dagger = A^\dagger A$ (see normal matrices) — not only for Hermitian $A$ : a $2 \times 2$ rotation matrix and $\operatorname{diag}(i, 1)$ are normal without being Hermitian, and both have singular values $1, 1$ and eigenvalues of modulus $1$ . (Normality is necessary as well as sufficient, since $\sum_i s_i^2 = \|A\|_F^2 \ge \sum_k |\lambda_k|^2$ for every square $A$ , with equality precisely when $A$ is normal.) The two lists coincide, with no absolute values, exactly for positive semidefinite $A$ .

$s_i \ge 0$ and the ordering are conventions, and they are what makes the factorisation nearly unique. A sign can always be moved into $u_i$ or $v_i$ ; reversing the order permutes the columns of $U$ and $V$ together. Do not compare two SVDs column by column without fixing both conventions.

Full versus reduced. Many texts and all numerical libraries offer a "thin" or "economy" form in which $U$ is $m \times p$ and $V$ is $n \times p$ with orthonormal columns and $S$ is $p \times p$ square. The thin factors are isometries, so $U^\dagger U = I_p$ and $V^\dagger V = I_p$ hold in both forms, while the identities with the factors the other way round can fail. Which one fails depends on the shape: since $p = \min(m, n)$ , the thin $U$ is square — hence already unitary, with $U U^\dagger = I_m$ — exactly when $m \le n$ , and the thin $V$ is square exactly when $n \le m$ . For $m > n$ the thin $U$ is $m \times n$ and $U U^\dagger$ is the rank- $n$ orthogonal projection onto $\operatorname{span}\{u_1, \dots, u_n\}$ , not $I_m$ ; for $n > m$ the same happens to $V V^\dagger$ . At most one of the two can fail, and when $m \neq n$ exactly one does; when $m = n$ the thin form is the full form, both factors are unitary and neither fails. So "the thin factors are not unitary" is a statement about one named factor, decided by the shape, never about both at once.

The pseudoinverse is not continuous in $A$ . A singular value that passes through zero makes $A^{+}$ blow up: for $A_\epsilon = \operatorname{diag}(1, \epsilon)$ , $A_\epsilon^{+} = \operatorname{diag}(1, 1/\epsilon)$ for $\epsilon \neq 0$ but $A_0^{+} = \operatorname{diag}(1, 0)$ . This is why practical computations zero out all singular values below a tolerance rather than inverting them, and it is the quantitative content of conditioning.

$A^{+} = (A^\dagger A)^{-1} A^\dagger$ needs full column rank. With dependent columns $A^\dagger A$ is singular and the normal equations have many solutions; the pseudoinverse selects the one of least norm, which the normal equations by themselves do not.

Notation. $A^{+}$ , $A^{\dagger}$ and $A^{\#}$ all appear in the literature for the pseudoinverse; we reserve $A^\dagger$ for the Hermitian conjugate throughout, as in the spectral theorem. Singular values are written $s_i$ here and $\sigma_i$ in much of the numerical literature.

Example(A rank-deficient 3 x 3 matrix, worked through completely)Original

Take the real matrix

A = \begin{pmatrix} 6 & 9 & 6 \\ 0 & -6 & -12 \\ 6 & 6 & 0 \end{pmatrix} ,

so $m = n = p = 3$ and $A^\dagger = A^{\mathsf T}$ .

1. Form $A^{\mathsf T} A$ . Its $(i,j)$ entry is the dot product of columns $i$ and $j$ of $A$ :

A^{\mathsf T} A = \begin{pmatrix} 72 & 90 & 36 \\ 90 & 153 & 126 \\ 36 & 126 & 180 \end{pmatrix} .

2. Its eigenvalues. The characteristic polynomial of a $3 \times 3$ matrix is $\lambda^3 - e_1 \lambda^2 + e_2 \lambda - e_3$ with $e_1$ the trace, $e_2$ the sum of the three $2 \times 2$ principal minors and $e_3$ the determinant. Here $e_1 = 72 + 153 + 180 = 405$ ,

e_2 = (72 \cdot 153 - 90^2) + (72 \cdot 180 - 36^2) + (153 \cdot 180 - 126^2) = 2916 + 11664 + 11664 = 26244 ,

and $e_3 = 0$ , because the columns of $A$ satisfy $2 c_1 - 2 c_2 + c_3 = 0$ — check the first row: $2 \cdot 6 - 2 \cdot 9 + 6 = 0$ — so $A$ , and hence $A^{\mathsf T} A$ , is singular. Thus

\lambda^3 - 405 \lambda^2 + 26244 \lambda = \lambda (\lambda^2 - 405\lambda + 26244) = \lambda(\lambda - 324)(\lambda - 81) ,

since $324 + 81 = 405$ and $324 \cdot 81 = 26244$ . So the singular values are

s_1 = \sqrt{324} = 18, \qquad s_2 = \sqrt{81} = 9, \qquad s_3 = 0 ,

and $r = \operatorname{rank} A = 2$ .

3. Right singular vectors. Solving $(A^{\mathsf T} A - \lambda I)v = 0$ for each eigenvalue and normalising gives

v_1 = \tfrac13 (1, 2, 2)^{\mathsf T}, \qquad v_2 = \tfrac13 (2, 1, -2)^{\mathsf T}, \qquad v_3 = \tfrac13 (2, -2, 1)^{\mathsf T} .

Each has norm $\tfrac13\sqrt{1+4+4} = 1$ , and the three are mutually orthogonal ( $1\cdot2 + 2\cdot1 - 2\cdot2 = 0$ , and similarly for the other two pairs). Verify one eigenvector equation: $A^{\mathsf T} A (1,2,2)^{\mathsf T} = (72 + 180 + 72,\ 90 + 306 + 252,\ 36 + 252 + 360)^{\mathsf T} = (324, 648, 648)^{\mathsf T} = 324\,(1,2,2)^{\mathsf T}$ , as required.

4. Left singular vectors. By the construction, $u_i = A v_i / s_i$ for $i \le r = 2$ :

A v_1 = \tfrac13 (36, -36, 18)^{\mathsf T} = (12, -12, 6)^{\mathsf T} = 18 \cdot \tfrac13 (2, -2, 1)^{\mathsf T}, \quad\text{so}\quad u_1 = \tfrac13 (2, -2, 1)^{\mathsf T} ,

A v_2 = \tfrac13 (9, 18, 18)^{\mathsf T} = (3, 6, 6)^{\mathsf T} = 9 \cdot \tfrac13 (1, 2, 2)^{\mathsf T}, \quad\text{so}\quad u_2 = \tfrac13 (1, 2, 2)^{\mathsf T} .

The third column is any unit vector orthogonal to these two; $u_3 = \tfrac13 (2, 1, -2)^{\mathsf T}$ serves. Checks demanded by the statement: $A v_3 = 0$ , so $v_3$ spans $\ker A$ ; $A^{\mathsf T} u_3 = 0$ , so $u_3$ spans $\ker A^{\mathsf T} = (\operatorname{im} A)^\perp$ ; $A^{\mathsf T} u_1 = (6, 12, 12)^{\mathsf T} = 18 v_1$ and $A^{\mathsf T} u_2 = (6, 3, -6)^{\mathsf T} = 9 v_2$ , which is part 2.

5. The decomposition. With $U = (u_1\ u_2\ u_3)$ , $V = (v_1\ v_2\ v_3)$ and $S = \operatorname{diag}(18, 9, 0)$ ,

A = 18\, u_1 v_1^{\mathsf T} + 9\, u_2 v_2^{\mathsf T} = \begin{pmatrix} 4 & 8 & 8 \\ -4 & -8 & -8 \\ 2 & 4 & 4 \end{pmatrix} + \begin{pmatrix} 2 & 1 & -2 \\ 4 & 2 & -4 \\ 4 & 2 & -4 \end{pmatrix} ,

and the two matrices do add up to $A$ . As a check on the singular values, $\|A\|_F^2 = 36 + 81 + 36 + 36 + 144 + 36 + 36 = 405$ (the three zero entries of $A$ contribute nothing) and $s_1^2 + s_2^2 = 324 + 81 = 405$ .

6. Best rank-one approximation. $A_1 = 18\, u_1 v_1^{\mathsf T}$ is the first matrix displayed above, and $A - A_1$ is the second. Its spectral and Frobenius norms are both $s_2 = 9$ : indeed $A - A_1 = 9\, u_2 v_2^{\mathsf T}$ is a single dyad, and $\|A - A_1\|_F^2 = 4+1+4+16+4+16+16+4+16 = 81$ . No matrix of rank $1$ comes closer in either norm, and because $s_1 > s_2$ this $A_1$ is the only Frobenius minimiser.

7. Pseudoinverse. $A^{+} = \tfrac{1}{18} v_1 u_1^{\mathsf T} + \tfrac19 v_2 u_2^{\mathsf T}$ , which works out to

A^{+} = \frac{1}{54} \begin{pmatrix} 2 & 2 & 3 \\ 2 & 0 & 2 \\ 0 & -4 & -2 \end{pmatrix} .

Check the projection property: $A A^{+} = I - u_3 u_3^{\mathsf T} = \tfrac19 \begin{pmatrix} 5 & -2 & 4 \\ -2 & 8 & 2 \\ 4 & 2 & 5\end{pmatrix}$ , which is symmetric, squares to itself, and kills $u_3$ — the orthogonal projection onto $\operatorname{im} A$ .

8. Minimum-norm least squares. For $b = (54, 0, 0)^{\mathsf T}$ ,

x^{+} = A^{+} b = (2, 2, 0)^{\mathsf T}, \qquad A x^{+} = (30, -12, 24)^{\mathsf T}, \qquad b - A x^{+} = (24, 12, -24)^{\mathsf T} .

The residual is $36\, u_3$ , so it is orthogonal to $\operatorname{im} A$ as it must be, and its norm is $36$ . Every least-squares solution is $x^{+} + t v_3$ with $t \in \mathbb{R}$ , and since $x^{+} \cdot (2,-2,1) = 4 - 4 + 0 = 0$ the vector $x^{+}$ is orthogonal to $\ker A$ , giving

\|x^{+} + t v_3\|_2^2 = \|x^{+}\|_2^2 + t^2 = 8 + t^2 ,

minimal only at $t = 0$ . So $\|x^{+}\|_2 = 2\sqrt2 \approx 2.8284$ is the smallest norm among all least-squares solutions, and $\|b - A x^{+}\|_2 = 36$ is the common residual of all of them.

ApplicationOriginal

Empirical orthogonal functions in climate science. Arrange a gridded field — monthly sea-surface temperature anomalies, say — as a matrix with one row per time and one column per grid point, and compute its SVD. The right singular vectors are the spatial patterns (EOFs), the left singular vectors scaled by the singular values are their time series (principal components), and $s_i^2 / \sum_j s_j^2$ is the fraction of variance that pattern carries. The best low-rank approximation statement is what justifies keeping the first few: no other combination of that many patterns reproduces the data field better, in the least-squares sense. The same computation under the name principal component analysis reduces the dimension of any data matrix.
Quantum entanglement. A pure state of two subsystems, $|\psi\rangle = \sum_{jk} C_{jk} |j\rangle \otimes |k\rangle$ , has a coefficient matrix $C$ ; its SVD is the Schmidt decomposition $|\psi\rangle = \sum_i s_i |u_i\rangle \otimes |\overline{v_i}\rangle$ . The singular values are the Schmidt coefficients, $\operatorname{rank} C$ is the Schmidt rank, the state is a product state exactly when that rank is $1$ , and the reduced density matrices $C C^\dagger$ and $C^{\mathsf T} \overline{C}$ have the common eigenvalues $s_i^2$ , from which the entanglement entropy $-\sum_i s_i^2 \log s_i^2$ (for $\sum_i s_i^2 = 1$ ) is computed. Truncating to the largest few $s_i$ is the compression step at the heart of matrix-product-state and DMRG methods.
Regularising an ill-posed inversion. In tomography, deconvolution and geophysical inversion one must solve $Ax = b$ for a discretised operator whose singular values decay towards zero. The pseudoinverse multiplies the data component along $u_i$ by $1/s_i$ , so noise in the directions with tiny $s_i$ is amplified without limit. Truncated SVD regularisation keeps only the terms with $s_i$ above a threshold, which replaces $A^{+}$ by the pseudoinverse of the best low-rank approximation $A_k$ and trades resolution for stability; the quantitative version of the trade-off is the subject of condition numbers.
Rank, and how many digits of it you can believe. The theoretical rank of a measured matrix is almost never attained numerically, because rounding and measurement error perturb every zero singular value away from zero. The spectral-norm statement says that $A$ is within $s_{k+1}$ of a matrix of rank $k$ , so counting the singular values above the noise level is the defensible way to determine rank — the standard test for the order of a linear model, the number of independent reaction pathways in a kinetics matrix, or the dimension of an attractor reconstructed by delay embedding.

Method

Functions of matrices and the matrix exponential

Adapted

Add to PDF

Defines a function of a matrix by its power series, states when the series converges, collects the properties of exp(A) including the exact condition — that A and B commute — for exp(sA+tB) = exp(sA)exp(tB) to hold for all s and t near 0, which is more than the single identity exp(A+B) = exp(A)exp(B) requires, and gives the recipes for computing exp(tA) by diagonalization and for defective matrices.

Prerequisites: Eigenvalues and eigenvectors, Power series, Normal matrices and simultaneous diagonalization, Jordan normal form, Absolute and conditional convergence, Convergence tests for series, Determinants

Definition(Power series in a matrix)AdaptedAdapted from Riley, Hobson & Bence (2006), §8.5, Eq. (8.38), p. 255; Hirsch, Smale & Devaney (2013), §6.4, p. 123

Powers of a square matrix are defined by $A^0 = I$ and $A^k = A A^{k-1}$ , so for scalars $c_0, c_1, \dots$ one can form

f(A) = \sum_{k=0}^{\infty} c_k A^k ,

a matrix of the same size as $A$ , provided the series converges. Convergence means entrywise convergence, which is the same as convergence in any norm on the space of matrices. The matrix exponential is the case $c_k = 1/k!$ :

\exp A = e^{A} = \sum_{k=0}^{\infty} \frac{A^k}{k!} .

The same construction defines $\sin A$ and $\cos A$ , which converge for every $A$ , and the geometric (Neumann) series $\sum_k A^k$ , which converges to $(I - A)^{-1}$ when $\rho(A) < 1$ and diverges when $\rho(A) > 1$ . Here the spectral radius of $A$ is $\rho(A) = \max\{|\lambda| : \lambda \in \sigma(A)\}$ , and the convergence statement below is what supplies these conditions.

Method(Convergence)Original

Let $\sum_k c_k z^k$ have radius of convergence $R \in (0, \infty]$ and let $A$ be an $n \times n$ complex matrix.

If $\rho(A) < R$ , then $\sum_k c_k A^k$ converges absolutely, entry by entry. If $\rho(A) > R$ , it diverges.
The boundary case $\rho(A) = R$ may go either way, and is not decided by $\rho(A)$ alone. At $A = I$ , where $\rho(A) = 1$ , both the coefficient sequence $c_k = 1$ and the coefficient sequence $c_0 = 0$ , $c_k = 1/k^2$ for $k \ge 1$ , have $R = 1$ ; the first matrix series diverges and the second converges.

The exponential has $R = \infty$ , so part 1 covers it for every $A$ . The next statement covers it again with an explicit bound and no reference to the spectrum.

Idea of the proofOriginal

A matrix power series can only do to $A$ what the scalar series does to each of its eigenvalues, so what matters is how fast the entries of $A^k$ can grow. The Jordan form answers that: they grow like a polynomial in $k$ times $\rho(A)^k$ , and a polynomial factor cannot defeat the geometric margin available strictly inside the disc of convergence. Outside the disc a single eigenvector is enough to wreck everything, because on that vector the matrix series is the scalar series evaluated at $z = \lambda$ . On the circle the geometric margin is gone, and the polynomial factor that it used to absorb becomes decisive instead: at $\rho(A) = R$ the answer depends on the Jordan structure of $A$ and not on the spectrum or the coefficients alone, so no condition on $\rho(A)$ can settle it. The degenerate case $A = I$ , where that structure drops out entirely and the matrix series is just $\bigl(\sum_k c_k\bigr) I$ , is therefore the place to look when one wants two coefficient sequences of the same radius to behave differently at the same $A$ .

ProofOriginal

Entrywise growth of the powers. Claim: for every $r > \rho(A)$ there is a constant $C$ , depending on $A$ and $r$ only, with $|(A^k)_{il}| \le C r^k$ for all $k \ge 0$ and all $i, l$ . Write $A = T J T^{-1}$ with $J$ the Jordan normal form of $A$ , so $A^k = T J^k T^{-1}$ ; since $T$ and $T^{-1}$ are fixed, each entry of $A^k$ is a fixed finite combination of entries of $J^k$ , and it is enough to bound those. $J$ is a direct sum of blocks $\lambda I_q + N_q$ with $q \le n$ , $N_q$ the nilpotent shift ( $N_q^q = 0$ , entries $0$ or $1$ ) and $\lambda$ an eigenvalue, so $|\lambda| \le \rho(A) < r$ . As $\lambda I_q$ commutes with $N_q$ , the binomial theorem gives

(\lambda I_q + N_q)^k = \sum_{j=0}^{\min(k,\, q-1)} \binom{k}{j} \lambda^{k-j} N_q^{\,j} ,

so every entry of the block of $J^k$ is $\binom{k}{j}\lambda^{k-j}$ for some $j \le q-1$ , or $0$ . If $\lambda = 0$ only $j = k$ survives, which needs $k \le q - 1$ : finitely many $k$ , absorbed into $C$ . If $\lambda \neq 0$ then $|\lambda^{k-j}| \le \max(1, |\lambda|^{-(n-1)})\, |\lambda|^k$ and $\binom{k}{j} \le k^{\,n-1}$ for $k \ge 1$ , so the entry is at most a constant times $k^{\,n-1} |\lambda|^k \le k^{\,n-1} \rho(A)^k$ . Finally $k^{\,n-1}\rho(A)^k / r^k \to 0$ as $k \to \infty$ , because $\rho(A) < r$ makes the geometric factor win, so that sequence is bounded and the claim follows.

Part 1, convergence. Suppose $\rho(A) < R$ and choose $r$ with $\rho(A) < r < R$ . A power series converges absolutely at every point strictly inside its disc of convergence (power series), so $\sum_k |c_k| r^k < \infty$ . By step 1,

\sum_{k=0}^{\infty} |c_k|\, |(A^k)_{il}| \ \le\ C \sum_{k=0}^{\infty} |c_k| r^k \ <\ \infty

for every $i, l$ : the matrix series converges absolutely, entry by entry.

Part 1, divergence. Suppose $\rho(A) > R$ and pick an eigenvalue $\lambda$ with $|\lambda| = \rho(A)$ and an eigenvector $v \neq 0$ . Then $A^k v = \lambda^k v$ by induction, so the partial sums $P_K = \sum_{k \le K} c_k A^k$ satisfy $P_K v = \bigl(\sum_{k \le K} c_k \lambda^k\bigr) v$ . If the matrix series converged entrywise then each $(P_K v)_i$ , a fixed linear combination of entries of $P_K$ , would converge; choosing $i$ with $v_i \neq 0$ makes $\sum_{k \le K} c_k \lambda^k = (P_K v)_i / v_i$ a convergent scalar sequence. But $|\lambda| > R$ , and a power series diverges at every point outside its disc of convergence. So the matrix series diverges.
Part 2, the boundary. For $A = I$ every power is $I$ , so $\sum_k c_k A^k = \bigl(\sum_k c_k\bigr) I$ converges exactly when the scalar series $\sum_k c_k$ does, while $\rho(I) = 1$ . Taking $c_k = 1$ gives $R = 1$ (the coefficients have $|c_k|^{1/k} = 1$ ) and a divergent $\sum_k 1$ . Taking $c_0 = 0$ and $c_k = 1/k^2$ gives $R = 1$ as well, since $(1/k^2)^{1/k} \to 1$ , and $\sum_{k \ge 1} 1/k^2$ converges. So $\rho(A) = R$ decides nothing.
The Neumann series. For $\sum_k z^k$ we have $R = 1$ , so by part 1 the series $\sum_k A^k$ converges when $\rho(A) < 1$ and diverges when $\rho(A) > 1$ . In the convergent case, telescoping gives $(I - A)\sum_{k \le K} A^k = I - A^{K+1}$ , and by step 1 with any $r \in (\rho(A), 1)$ the entries of $A^{K+1}$ are at most $C r^{K+1} \to 0$ . Letting $K \to \infty$ gives $(I-A)\sum_k A^k = I$ , and the same computation on the other side gives $\sum_k A^k (I - A) = I$ , so the sum is $(I-A)^{-1}$ . $\square$

Method(The exponential converges for every matrix)AdaptedAdapted from Hirsch, Smale & Devaney (2013), §6.4, p. 125

Let $A$ be an $n \times n$ complex matrix and put $a = \max_{i,j}|A_{ij}|$ . Then

\bigl| (A^k)_{ij} \bigr| \ \le\ n^{\,k-1} a^k \qquad (k \ge 1) ,

and consequently every entry of $\sum_k A^k/k!$ is an absolutely convergent scalar series, with

\sum_{k=0}^{\infty} \frac{|(A^k)_{ij}|}{k!} \ \le\ 1 + \frac{e^{na} - 1}{n} .

So $\exp A$ is defined for every $A$ , under no hypothesis at all. The same bound makes $\sum_k c_k A^k$ converge absolutely whenever $n a < R$ : a crude but spectrum-free test, which needs no eigenvalue of $A$ to be known. A sharp test of that kind uses a submultiplicative norm — one with $\|XY\| \le \|X\|\,\|Y\|$ , so that $\|A^k\| \le \|A\|^k$ — together with the fact that $\rho(A) \le \|A\|$ ; such norms are the subject of a later unit, matrix norms and conditioning, and nothing below depends on them.

Idea of the proofOriginal

Each entry of $A^{k+1}$ is a sum of $n$ products of an entry of $A^k$ with an entry of $A$ , so passing from one power to the next costs at most one factor $n$ and one factor $a$ ; iterating that is the bound. Dividing by $k!$ then leaves, up to the factor $1/n$ , a term of the scalar series for $e^{na}$ , and the comparison test finishes the job. Nothing about the spectrum of $A$ is used anywhere, which is what makes the resulting test crude but usable with the entries of $A$ alone.

ProofAdaptedAdapted from Hirsch, Smale & Devaney (2013), §6.4, p. 125

The bound on the powers. Induction on $k$ . For $k = 1$ the claim reads $|A_{il}| \le a$ , which is the definition of $a$ . Assume it for $k$ . Each entry of $A^{k+1}$ is $\sum_{q=1}^{n} (A^k)_{iq} A_{ql}$ , a sum of $n$ terms, so by the triangle inequality, the inductive hypothesis and $|A_{ql}| \le a$ ,

|(A^{k+1})_{il}| \ \le\ \sum_{q=1}^{n} n^{\,k-1} a^k \cdot a \ =\ n^{\,k} a^{k+1} .

Comparison with the scalar exponential series. Split off the term $k = 0$ , where $(A^0)_{il} = \delta_{il}$ has modulus at most $1$ . For $k \ge 1$ step 1 gives $|(A^k)_{il}|/k! \le (na)^k/(n\,k!)$ , so

\sum_{k=0}^{\infty} \frac{|(A^k)_{il}|}{k!} \ \le\ 1 + \frac{1}{n} \sum_{k=1}^{\infty} \frac{(na)^k}{k!} \ =\ 1 + \frac{e^{na} - 1}{n} \ <\ \infty ,

the middle equality because $\sum_{k \ge 1}(na)^k/k! = e^{na} - 1$ . Each entry series is therefore dominated by a convergent series of non-negative numbers, hence absolutely convergent by the comparison test (convergence tests). This is the route of (Hirsch, Smale & Devaney (2013), §6.4, p. 125), which bounds the entry by $e^{na}$ without separating the $k = 0$ term.

The spectrum-free test. Run the same chain with the coefficients in place: $\sum_{k \ge 1} |c_k|\,|(A^k)_{il}| \le \frac{1}{n}\sum_{k \ge 1} |c_k| (na)^k$ by step 1. If $na < R$ then $na$ lies strictly inside the disc of convergence of $\sum_k c_k z^k$ , where a power series converges absolutely (power series), so the right-hand side is finite and $\sum_k c_k A^k$ converges absolutely entry by entry.
Relation to the previous statement. The conclusion for $\exp A$ is also the case $R = \infty$ of part 1 above. The bound just proved is nevertheless the one used in the rest of this unit, because it is explicit in the entries of $A$ and survives being applied to $sA + tB$ with unknown spectrum. $\square$

Method(Properties of the exponential)AdaptedAdapted from Hirsch, Smale & Devaney (2013), §6.4, the Propositions on pp. 126 and 128 and the Theorem on p. 129; Riley, Hobson & Bence (2006), §8.16, Eq. (8.104), pp. 287-288

Let $A, B$ be $n \times n$ complex matrices and $T$ invertible. Then:

$\exp(T^{-1} A T) = T^{-1} (\exp A) T$ .
If $A B = B A$ , then $\exp(A + B) = (\exp A)(\exp B)$ .
$\exp(-A) = (\exp A)^{-1}$ ; in particular $\exp A$ is invertible for every $A$ .
If $A v = \lambda v$ with $v \neq 0$ , then $(\exp A) v = e^{\lambda} v$ .
$\dfrac{d}{dt} \exp(tA) = A \exp(tA) = \exp(tA)\, A$ , and $x(t) = \exp(tA)\, x_0$ is a solution of $\dot x = A x$ with $x(0) = x_0$ and is the only one ((Hirsch, Smale & Devaney (2013), §6.4, Theorem, p. 129)).
$\det(\exp A) = e^{\operatorname{tr} A}$ .

The commuting hypothesis is what makes the proof of property 2 work: that proof goes through the binomial theorem for $(A+B)^k$ , which is valid only when $AB = BA$ . It is not necessary for the identity itself — see part 3 of the statement "When the product rule holds" below, where a non-commuting pair satisfying $\exp(A+B) = \exp(A)\exp(B)$ is exhibited.

Idea of the proofOriginal

Each property is a scalar identity about $e^x$ transplanted to matrices, and the transplant is always the same two moves: do the algebra on the partial sums, where everything is a finite sum, then pass to the limit, which is legitimate because multiplying by a fixed matrix, or applying a fixed linear functional, is continuous entrywise. Two of them need more than that. The product rule has two infinite series to multiply, which is the Cauchy product, available because the exponential series converges absolutely; it is the binomial theorem afterwards that insists on $AB = BA$ . And the derivative in property 5 needs an estimate rather than a limit: a difference quotient has to be controlled as $h \to 0$ , and what controls it is the quantitative entrywise tail bound of the previous statement applied to $\bigl(\exp(hA) - I\bigr)/h - A$ . The determinant identity is of a different kind: put $A$ into triangular form and both sides can be read off the diagonal.

ProofAdaptedAdapted from Hirsch, Smale & Devaney (2013), §6.4, the Propositions on pp. 126 and 128, the Lemma on pp. 126-127 and the Theorem on p. 129; Riley, Hobson & Bence (2006), §8.16, Eq. (8.104), pp. 287-288

Property 1. $(T^{-1} A T)^k = T^{-1} A^k T$ , by induction: the inductive step cancels the adjacent $T T^{-1}$ in $(T^{-1}A^kT)(T^{-1}AT)$ . Hence the partial sums satisfy $\sum_{k \le K} (T^{-1}AT)^k/k! = T^{-1}\bigl(\sum_{k \le K} A^k/k!\bigr) T$ . Every entry of the right-hand side is one fixed linear combination of the entries of $\sum_{k\le K}A^k/k!$ , with coefficients built from $T$ and $T^{-1}$ , so it converges to the same combination of the entries of $\exp A$ . Letting $K \to \infty$ gives $\exp(T^{-1}AT) = T^{-1}(\exp A)T$ .
Property 2. Fix entries $i, l$ . By the statement "The exponential converges for every matrix" each scalar series $\sum_j (A^j)_{iq}/j!$ and $\sum_k (B^k)_{ql}/k!$ converges absolutely, so the Cauchy product of two absolutely convergent series (absolute convergence) applies to each of the $n$ pairs indexed by $q$ ; summing the $n$ resulting identities,

(\exp A \, \exp B)_{il} = \sum_{q=1}^{n} \Bigl( \sum_{j} \frac{(A^j)_{iq}}{j!} \Bigr) \Bigl( \sum_{k} \frac{(B^k)_{ql}}{k!} \Bigr) = \sum_{N=0}^{\infty} \ \sum_{j+k=N} \frac{(A^j B^k)_{il}}{j!\, k!} ,

with no hypothesis on $A$ and $B$ so far. Now the hypothesis enters: since $AB = BA$ , also $B A^j = A^j B$ for every $j$ , and the binomial theorem holds in the form $(A+B)^N = \sum_{j+k=N} \frac{N!}{j!\,k!} A^j B^k$ (induction on $N$ , the step using $B A^j = A^j B$ to move every $B$ past every $A$ ). So the inner sum is $(A+B)^N/N!$ and the right-hand side is $\exp(A+B)$ .

Property 3. $A$ and $-A$ commute, so property 2 gives $(\exp A)(\exp(-A)) = \exp(A - A) = \exp 0 = I$ , and with the roles exchanged $(\exp(-A))(\exp A) = I$ . Hence $\exp A$ is invertible with inverse $\exp(-A)$ .
Property 4. From $Av = \lambda v$ we get $A^k v = \lambda^k v$ by induction, so $\bigl(\sum_{k \le K} A^k/k!\bigr) v = \bigl(\sum_{k \le K}\lambda^k/k!\bigr) v$ . The left side converges to $(\exp A)v$ because $\exp A$ converges entrywise, the right to $e^\lambda v$ because the scalar exponential series converges. Hence $(\exp A)v = e^{\lambda} v$ .
Property 5. For fixed $t$ and $h$ the matrices $tA$ and $hA$ commute, so property 2 gives $\exp((t+h)A) = \exp(tA)\exp(hA)$ and therefore

\frac{\exp((t+h)A) - \exp(tA)}{h} = \exp(tA) \cdot \frac{\exp(hA) - I}{h} .

Now $\frac{\exp(hA)-I}{h} - A = \sum_{k \ge 2} h^{k-1}A^k/k!$ , whose entries are bounded, for $0 < |h| \le 1$ , using the entrywise bound $|(A^k)_{il}| \le n^{k-1}a^k$ with $a = \max_{i,l}|A_{il}|$ , by

\sum_{k \ge 2} \frac{|h|^{k-1} n^{k-1} a^k}{k!} = a \sum_{k \ge 2} \frac{(na|h|)^{k-1}}{k!} \ \le\ a \cdot n a |h| \sum_{j \ge 0} \frac{(na|h|)^{j}}{j!} \ \le\ n a^2 e^{na}\, |h| ,

which tends to $0$ with $h$ . So $\frac{\exp(hA)-I}{h} \to A$ entrywise, and the difference quotient converges to $\exp(tA)\,A$ : that is the derivative. Moreover $A$ commutes with every $A^k$ , hence with every partial sum of the series, hence with $\exp(tA)$ in the limit, so the derivative equals $A\exp(tA)$ too. Consequently $x(t) = \exp(tA)x_0$ satisfies $\dot x = A\exp(tA)x_0 = Ax(t)$ and $x(0) = x_0$ . For uniqueness, let $y$ be any solution with $y(0) = x_0$ and set $z(t) = \exp(-tA)y(t)$ . By the product rule and the derivative just computed, applied with $-A$ ,

\dot z = -A \exp(-tA) y + \exp(-tA) \dot y = \exp(-tA)\bigl(-A y + A y\bigr) = 0 ,

where the commuting of $A$ with $\exp(-tA)$ was used to move $-A$ across. So $z$ is constant, $z(t) = z(0) = x_0$ , and multiplying by $\exp(tA)$ — invertible by property 3, with inverse $\exp(-tA)$ — gives $y(t) = \exp(tA)x_0$ . This is the existence-and-uniqueness theorem cited in the statement.

Property 6. Let $A = TJT^{-1}$ with $J$ the Jordan form, which is upper triangular with the eigenvalues $\lambda_1, \dots, \lambda_n$ of $A$ on its diagonal (listed with algebraic multiplicity). Multiplying two upper triangular matrices $X, Y$ leaves the result upper triangular and multiplies the diagonals entrywise, as $(XY)_{ii} = \sum_q X_{iq}Y_{qi} = X_{ii}Y_{ii}$ shows — only $q = i$ survives, since $X_{iq} = 0$ for $q < i$ and $Y_{qi} = 0$ for $q > i$ . So $J^k$ is upper triangular with $(J^k)_{ii} = \lambda_i^k$ . Each partial sum of $\exp J$ is therefore upper triangular with $(i,i)$ entry $\sum_{k \le K}\lambda_i^k/k!$ , and these properties survive the entrywise limit: $\exp J$ is upper triangular with diagonal entries $e^{\lambda_i}$ . The determinant of a triangular matrix is the product of its diagonal entries, so

\det \exp J = \prod_{i=1}^{n} e^{\lambda_i} = e^{\sum_i \lambda_i} = e^{\operatorname{tr} A} ,

the last step because $\operatorname{tr} A = \operatorname{tr} J$ is the sum of the eigenvalues. By property 1, $\exp A = T (\exp J) T^{-1}$ , whence $\det \exp A = \det T \cdot \det \exp J \cdot (\det T)^{-1} = \det \exp J$ by the product rule for determinants. This is the route of (Riley, Hobson & Bence (2006), §8.16, Eq. (8.104), pp. 287-288), which runs it with a diagonalising similarity; the triangular form is what covers defective $A$ as well. $\square$

Method(Adjoints, unitarity and positivity)Original

For every $n \times n$ complex $A$ , $\exp(A^\dagger) = (\exp A)^\dagger$ . Consequently:

if $A$ is anti-Hermitian ( $A^\dagger = -A$ ) then $\exp A$ is unitary;
if $A$ is Hermitian then $\exp A$ is Hermitian, and it is positive definite: $x^\dagger (\exp A) x > 0$ for every $x \neq 0$ .

Neither cited source states these; they are ours.

Idea of the proofOriginal

Conjugate transposition does two convenient things at once: it is continuous entrywise, being conjugation composed with a relabelling of indices, and it turns $A^k$ into $(A^\dagger)^k$ . So it slides through the partial sums and hence through the limit, which is the whole of the first claim. The two consequences are then bookkeeping with the properties already proved: an anti-Hermitian exponent makes the adjoint equal to the inverse, and a Hermitian exponent lets the exponential be split as $B^\dagger B$ with $B$ invertible, which is what positive definiteness asks for.

ProofOriginal

Powers and the adjoint. $(A^k)^\dagger = (A^\dagger)^k$ for every $k \ge 0$ , by induction: the case $k = 0$ is $I^\dagger = I$ , and $(A^{k+1})^\dagger = (A\,A^k)^\dagger = (A^k)^\dagger A^\dagger = (A^\dagger)^k A^\dagger = (A^\dagger)^{k+1}$ , using $(XY)^\dagger = Y^\dagger X^\dagger$ and then the inductive hypothesis.
The adjoint passes to the limit. Fix $i, l$ . By definition $\bigl( X^\dagger \bigr)_{il} = \overline{X_{li}}$ , so

\Bigl( \Bigl( \sum_{k \le K} \frac{A^k}{k!} \Bigr)^{\!\dagger} \Bigr)_{il} = \overline{ \sum_{k \le K} \frac{(A^k)_{li}}{k!} } = \sum_{k \le K} \frac{\overline{(A^k)_{li}}}{k!} = \sum_{k \le K} \frac{\bigl((A^\dagger)^k\bigr)_{il}}{k!}

by step 1. The outer series on the left converges to $\overline{(\exp A)_{li}} = \bigl((\exp A)^\dagger\bigr)_{il}$ , because complex conjugation is continuous, and the right-hand side converges to $(\exp(A^\dagger))_{il}$ . Hence $\exp(A^\dagger) = (\exp A)^\dagger$ .

Anti-Hermitian exponent. If $A^\dagger = -A$ , then by step 2 and property 3,

(\exp A)^\dagger (\exp A) = \exp(A^\dagger)\exp(A) = \exp(-A)\exp(A) = I ,

and exchanging the two factors gives $(\exp A)(\exp A)^\dagger = I$ in the same way. So $\exp A$ is unitary.

Hermitian exponent. If $A^\dagger = A$ then $(\exp A)^\dagger = \exp(A^\dagger) = \exp A$ by step 2, so $\exp A$ is Hermitian. Also $\tfrac12 A$ is Hermitian, so $B = \exp\bigl(\tfrac12 A\bigr)$ satisfies $B^\dagger = B$ by the same step, and $B$ is invertible by property 3. Since $\tfrac12 A$ commutes with itself, property 2 gives

\exp A = \exp\bigl(\tfrac12 A + \tfrac12 A\bigr) = B^2 = B^\dagger B .

Therefore $x^\dagger (\exp A) x = x^\dagger B^\dagger B x = \|Bx\|_2^2$ for every $x$ , and for $x \neq 0$ invertibility of $B$ gives $Bx \neq 0$ , so the value is strictly positive: $\exp A$ is positive definite. $\square$

Method(When the product rule holds)Original

For $n \times n$ complex $A, B$ :

$A B = B A$ implies $\exp(sA + tB) = \exp(sA)\exp(tB)$ for all $s, t \in \mathbb{C}$ .
Conversely, if $\exp(sA + tB) = \exp(sA)\exp(tB)$ holds for all $s, t$ in a neighbourhood of $0$ in $\mathbb{R}^2$ , then $AB = BA$ .
Commuting is not necessary for the single identity $\exp(A + B) = \exp(A)\exp(B)$ . For

A = \begin{pmatrix} 0 & 0 \\ 0 & 2\pi i \end{pmatrix} , \qquad B = \begin{pmatrix} 0 & 1 \\ 0 & 2\pi i \end{pmatrix}

one has $\exp(A+B) = I = \exp(A)\exp(B)$ while $[A, B] = \begin{pmatrix} 0 & -2\pi i \\ 0 & 0 \end{pmatrix} \neq 0$ . Consistently with part 2, the identity fails for this pair at other values of $s, t$ , for instance at $s = 1$ , $t = \tfrac12$ . 4. For $A$ and $B$ small enough, $\exp(A)\exp(B) = \exp\bigl(A + B + \tfrac12 [A,B] + \cdots\bigr)$ , the Baker-Campbell-Hausdorff series, whose further terms are iterated commutators. The series converges only under such a smallness condition — and the matrix logarithm is not single-valued, as the remark below records — so for general $A, B$ it is to be read as a formal expansion. What does hold for all $A, B$ is its leading content: the discrepancy between the two sides is of second order in the size of $A$ and $B$ , with $\tfrac12[A,B]$ as the second-order term, which is the error committed by operator-splitting schemes.

Idea of the proofOriginal

One direction is property 2 applied to the pair $sA$ , $tB$ , which commute as soon as $A$ and $B$ do. The converse is a matter of reading off coefficients: expand both sides to second order in $(s, t)$ and the terms of order $st$ differ by exactly $\tfrac12[A,B]$ , so an identity holding on a whole neighbourhood of the origin forces the commutator to vanish. The counterexample shows why "for all $s, t$ " is not decoration: at one isolated pair of values the identity can hold by accident, because the scalar exponential is periodic along the imaginary axis and can hide a non-zero commutator.

ProofOriginal

Part 1. If $AB = BA$ then $(sA)(tB) = st\,AB = st\,BA = (tB)(sA)$ for all scalars $s, t$ , so property 2 of the previous statement applies to the commuting pair $sA$ , $tB$ and gives $\exp(sA + tB) = \exp(sA)\exp(tB)$ .
A second-order expansion. Let $c = \max_{i,l} \max(|A_{il}|, |B_{il}|)$ and $\sigma = |s| + |t|$ , and put $X = sA + tB$ , so $\max_{i,l}|X_{il}| \le \sigma c$ . By the entrywise bound, applied to $X$ , the entries of the tail $\exp X - I - X - \tfrac12 X^2 = \sum_{k \ge 3} X^k/k!$ are bounded in modulus by

\sum_{k \ge 3} \frac{n^{k-1}(\sigma c)^k}{k!} \ \le\ \frac{(n \sigma c)^3}{6n} \sum_{j \ge 0} \frac{(n\sigma c)^j}{j!} \ \le\ K_1 \sigma^3 \qquad (\sigma \le 1)

for a constant $K_1$ depending only on $n$ and $c$ . Expanding $X^2 = s^2A^2 + st(AB + BA) + t^2B^2$ ,

\exp(sA + tB) = I + sA + tB + \tfrac12 s^2 A^2 + \tfrac12 st (AB+BA) + \tfrac12 t^2 B^2 + O(\sigma^3) ,

where $O(\sigma^3)$ means entries bounded by a constant times $\sigma^3$ for $\sigma \le 1$ . Applying the same bound to $\exp(sA)$ and to $\exp(tB)$ separately and multiplying the two expansions — all discarded terms have total degree at least $3$ in $(s,t)$ , and each factor has entries bounded by a constant for $\sigma \le 1$ , so they too are $O(\sigma^3)$ —

\exp(sA)\exp(tB) = I + sA + tB + \tfrac12 s^2 A^2 + st\, AB + \tfrac12 t^2 B^2 + O(\sigma^3) .

Part 2. Assume the identity holds for all $(s,t)$ in a neighbourhood of the origin in $\mathbb{R}^2$ . Subtracting the two displays of step 2, for all small $(s,t)$

0 = st \bigl( \tfrac12(AB + BA) - AB \bigr) + O(\sigma^3) = -\tfrac12 st\, [A, B] + O(\sigma^3) .

Take $s = t = h$ with $h$ real and small, divide by $h^2$ (so $\sigma^3/h^2 = 8|h| \to 0$ ), and let $h \to 0$ : every entry of $[A,B]$ vanishes, that is $AB = BA$ .

Part 3, the counterexample. For $M = \begin{pmatrix} 0 & \beta \\ 0 & d \end{pmatrix}$ with $d \neq 0$ , induction gives $M^k = \begin{pmatrix} 0 & \beta d^{\,k-1} \\ 0 & d^{\,k} \end{pmatrix}$ for $k \ge 1$ , so summing the series termwise (legitimate because the exponential series converges absolutely entry by entry),

\exp M = \begin{pmatrix} 1 & \beta\,(e^{d} - 1)/d \\ 0 & e^{d} \end{pmatrix} .

With $d = 2\pi i$ we have $e^d = 1$ , so this is $I$ : that covers $B$ (take $\beta = 1$ , $d = 2\pi i$ ) and $A + B = \begin{pmatrix} 0 & 1 \\ 0 & 4\pi i\end{pmatrix}$ (take $\beta = 1$ , $d = 4\pi i$ , again $e^d = 1$ ), while $A$ is diagonal with $\exp A = \operatorname{diag}(1, e^{2\pi i}) = I$ . Hence $\exp(A+B) = I = (\exp A)(\exp B)$ . The commutator does not vanish:

AB = \begin{pmatrix} 0 & 0 \\ 0 & (2\pi i)^2 \end{pmatrix}, \qquad BA = \begin{pmatrix} 0 & 2\pi i \\ 0 & (2\pi i)^2 \end{pmatrix}, \qquad [A,B] = \begin{pmatrix} 0 & -2\pi i \\ 0 & 0 \end{pmatrix} .

At $s = 1$ , $t = \tfrac12$ the formula with $\beta = \tfrac12$ and $d = 3\pi i$ (so $e^d = -1$ ) gives $\exp(A + \tfrac12 B) = \begin{pmatrix} 1 & i/(3\pi) \\ 0 & -1\end{pmatrix}$ , using $-1/(3\pi i) = i/(3\pi)$ , while $\exp(A)\exp(\tfrac12 B) = \exp(\tfrac12 B) = \begin{pmatrix} 1 & i/\pi \\ 0 & -1 \end{pmatrix}$ by the same formula with $d = \pi i$ . The two differ, so for this pair $\exp(sA+tB) = \exp(sA)\exp(tB)$ is not an identity in $s$ and $t$ : the coincidence at $(s,t) = (1,1)$ is isolated. Part 2 says more than this one computation does — it forces failure at points arbitrarily close to the origin, which $(1, \tfrac12)$ is not — but a single explicit pair of values is the cheapest way to see that the identity does not persist.

Part 4 is not proved here. The expansion of step 2 does give its leading content: replacing $A$ by $\epsilon A$ and $B$ by $\epsilon B$ and setting $s = t = 1$ ,

\exp(\epsilon A)\exp(\epsilon B) - \exp(\epsilon A + \epsilon B) = \tfrac12 \epsilon^2 [A, B] + O(\epsilon^3) ,

which is the quantitative statement that the error of operator splitting is second order in the step size. The full Baker-Campbell-Hausdorff series, and the proof that all of its terms are iterated commutators, is beyond the scope of this unit; the standard reference is the theory of Lie algebras and the Lie group-Lie algebra correspondence. $\square$

Method(Computing exp(tA))Original

If $A$ is diagonalizable, $A = S \Lambda S^{-1}$ with $\Lambda = \operatorname{diag}(\lambda_1, \dots, \lambda_n)$ and the columns of $S$ an eigenbasis. Then by property 1,

\exp(tA) = S \exp(t\Lambda) S^{-1} = S \operatorname{diag}\bigl(e^{\lambda_1 t}, \dots, e^{\lambda_n t}\bigr) S^{-1} .

The same formula computes any $f(A)$ with $f(A) = S \operatorname{diag}(f(\lambda_k)) S^{-1}$ , provided every $\lambda_k$ lies inside the disc of convergence. 2. If $A$ is normal, $S$ may be taken unitary (see normal matrices), so $S^{-1} = S^\dagger$ and no inverse has to be computed; this is also the numerically stable case. 3. If $A$ is defective, split it as $A = D + N$ with $DN = ND$ , $D$ diagonalizable and $N$ nilpotent ( $N^q = 0$ for some $q \le n$ ). Those three properties of $D$ and $N$ are all the step uses, however the splitting was obtained. One source of it is the Jordan normal form: if $S^{-1} A S = \Lambda + \widehat N$ with $\Lambda$ its diagonal part and $\widehat N$ its superdiagonal nilpotent part, then in the original basis $D = S \Lambda S^{-1}$ and $N = S \widehat N S^{-1}$ , so $D$ is diagonalizable but not in general diagonal. Then by property 2

\exp(tA) = \exp(tD) \exp(tN) = \exp(tD) \sum_{k=0}^{q-1} \frac{t^k N^k}{k!} ,

a finite sum, with $\exp(tD)$ computed by step 1. The entries of $\exp(tA)$ are then sums of terms (polynomial in $t$ ) $\times\, e^{\lambda t}$ over the eigenvalues $\lambda$ , the degree of the polynomial multiplying $e^{\lambda t}$ being at most one less than the size of the largest Jordan block for $\lambda$ — at most, because cancellation in the particular entry can lower it, and the degree is exactly that only for the entries where the top power of $\widehat N$ survives. 4. For $n$ small, or to avoid eigenvectors altogether, use that $A^n$ is a combination of $I, A, \dots, A^{n-1}$ by the Cayley-Hamilton theorem — a matrix satisfies its own characteristic equation, that is, the characteristic polynomial $p(\lambda) = \det(\lambda I - A)$ of the eigenvalue problem satisfies $p(A) = 0$ ((Arfken, Weber & Harris (2013), §6.4, Exercise 6.4.1, Note, p. 317)) — so $\exp(tA) = \sum_{j=0}^{n-1} \alpha_j(t) A^j$ for scalar functions $\alpha_j$ . Determine the $\alpha_j$ by requiring $e^{\lambda t} = \sum_{j} \alpha_j(t) \lambda^j$ at each distinct eigenvalue, together with its first $m - 1$ derivatives with respect to $\lambda$ there when that eigenvalue has algebraic multiplicity $m$ . That is $n$ linear conditions on the $n$ unknowns $\alpha_0(t), \dots, \alpha_{n-1}(t)$ , and what makes them determine the $\alpha_j$ is that the coefficient matrix — a confluent Vandermonde matrix in the distinct eigenvalues — is nonsingular: if $\sum_j \alpha_j \lambda^j$ and its first $m-1$ derivatives vanish at an eigenvalue of multiplicity $m$ , then the characteristic polynomial divides this polynomial of degree $< n$ , which forces it to be $0$ , so the homogeneous system has only the zero solution. 5. Numerically, call a library routine (scipy.linalg.expm), which uses scaling and squaring with a rational approximation. Verify with property 6: $\det \exp(tA)$ should equal $e^{t \operatorname{tr} A}$ .

How each route fails. Summing the defining series directly loses accuracy badly when $\|tA\|$ is large, because the terms grow before they decay (and catastrophically for $A$ with eigenvalues of both signs). Route 1 is useless for a defective $A$ and unreliable for a nearly defective one, since $S$ is then ill-conditioned and $S^{-1}$ amplifies the error; see the later unit on condition numbers. Route 3 requires the Jordan structure, which is itself not computable stably. Never use $\exp(A)\exp(B)$ in place of $\exp(A+B)$ unless $[A, B] = 0$ .

NoteOriginal

$\exp(tA)$ is the propagator of the linear system $\dot x = Ax$ : it carries the state at time $0$ to the state at time $t$ . When $A$ is diagonalizable it does so by multiplying each eigendirection by $e^{\lambda t}$ , so the eigenvalues alone read off the dynamics — decay for $\operatorname{Re}\lambda < 0$ , growth for $\operatorname{Re}\lambda > 0$ , oscillation at frequency $\operatorname{Im}\lambda$ . When $A$ is defective that picture is incomplete: each Jordan block of size $m$ contributes $e^{\lambda t}$ times a polynomial in $t$ of degree $m-1$ , which is why $A = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$ has $\exp(tA) = \begin{pmatrix} 1 & t \\ 0 & 1 \end{pmatrix}$ and grows without bound although its only eigenvalue is $0$ . The Traps below say what survives of the eigenvalue reading in that case. The same object appears as the time evolution $e^{-iHt/\hbar}$ of quantum mechanics and as the transition matrix of a continuous-time Markov chain.

Remark(Traps)Original

$\exp A$ is not the matrix of entrywise exponentials. For $A = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$ , $\exp A = \begin{pmatrix} 1 & 1 \\ 0 & 1 \end{pmatrix}$ , not $\begin{pmatrix} 1 & e \\ 1 & 1 \end{pmatrix}$ .

$\exp$ is not surjective onto invertible matrices over $\mathbb{R}$ . $\det \exp A = e^{\operatorname{tr} A} > 0$ for real $A$ , so no real matrix of negative determinant, such as $\operatorname{diag}(1,-1)$ , is a real exponential. Over $\mathbb{C}$ every invertible matrix is an exponential.

Eigenvalues, not the spectral radius, govern growth. $\|\exp(tA)\|$ is bounded as $t \to +\infty$ only if $\operatorname{Re}\lambda \le 0$ for every eigenvalue and every eigenvalue with $\operatorname{Re}\lambda = 0$ has equal algebraic and geometric multiplicity: $A = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$ has $\rho(A) = 0$ yet $\exp(tA) = \begin{pmatrix} 1 & t \\ 0 & 1 \end{pmatrix}$ grows.

The eigenvalues of $\exp A$ are the $e^{\lambda}$ , but the map on eigenvalues is not injective. $\lambda$ and $\lambda + 2\pi i$ give the same $e^\lambda$ , which is why there is no single-valued matrix logarithm and why the counterexample in part 3 of the product-rule statement exists at all.

Conventions. Physics writes $e^{-iHt/\hbar}$ and mathematics $\exp(tA)$ ; both are this object. A few texts define $f(A)$ for non-diagonalizable $A$ by the Jordan form or by a Cauchy integral rather than by a power series. The three definitions agree wherever more than one applies, but only the series version is used above.

Example(exp(tA) for a defective 3 x 3 matrix)Original

Take

A = \begin{pmatrix} 0 & 1 & 0 \\ 0 & 0 & 1 \\ 1 & -3 & 3 \end{pmatrix} .

1. Eigenvalues. $A$ is the companion matrix of the polynomial whose coefficients stand in its last row, so

\det(\lambda I - A) = \lambda^3 - 3\lambda^2 + 3\lambda - 1 = (\lambda - 1)^3 ,

one eigenvalue $\lambda = 1$ of algebraic multiplicity $3$ .

2. $A$ is defective, so route 1 is unavailable. Put

N = A - I = \begin{pmatrix} -1 & 1 & 0 \\ 0 & -1 & 1 \\ 1 & -3 & 2 \end{pmatrix} .

Its third row is $-1$ times the first plus $2$ times the second, and its first two rows are independent, so $\operatorname{rank} N = 2$ and $\dim \ker N = 1$ . The eigenvalue $1$ therefore has geometric multiplicity $1$ against algebraic multiplicity $3$ : there is no eigenbasis, and $A$ has a single Jordan block of size $3$ .

3. The splitting of step 3 of the method. Take $D = I$ and $N = A - I$ . Then $D$ is diagonalizable, $DN = ND$ because $D = I$ , and $N$ is nilpotent:

N^2 = \begin{pmatrix} 1 & -2 & 1 \\ 1 & -2 & 1 \\ 1 & -2 & 1 \end{pmatrix} , \qquad N^3 = 0 ,

all three rows of $N^2$ being equal, every column of $N^2$ is a multiple of $(1,1,1)^{\mathsf T}$ , and $N(1,1,1)^{\mathsf T} = (-1+1+0,\ 0-1+1,\ 1-3+2)^{\mathsf T} = 0$ , so $N^3 = N \cdot N^2 = 0$ .

4. Assemble. Since $tD = tI$ commutes with $tN$ , property 2 gives $\exp(tA) = e^{t}\bigl(I + tN + \tfrac12 t^2 N^2\bigr)$ , a finite sum:

\exp(tA) = e^{t} \begin{pmatrix} 1 - t + \tfrac12 t^2 & t - t^2 & \tfrac12 t^2 \\[2pt] \tfrac12 t^2 & 1 - t - t^2 & t + \tfrac12 t^2 \\[2pt] t + \tfrac12 t^2 & -3t - t^2 & 1 + 2t + \tfrac12 t^2 \end{pmatrix} .

Every entry is a quadratic polynomial times $e^{t}$ , the degree being one less than the size of the Jordan block, exactly as step 3 of the method predicts.

5. Two checks. At $t = 0$ the matrix is $I$ . Differentiating, the derivative of $e^t\bigl(I + tN + \tfrac12 t^2 N^2\bigr)$ is $e^t\bigl(I + tN + \tfrac12 t^2N^2\bigr) + e^t\bigl(N + tN^2\bigr)$ , which at $t = 0$ is $I + N = A$ , as property 5 requires.

6. A number. At $t = 1$ ,

\exp A = \frac{e}{2}\begin{pmatrix} 1 & 0 & 1 \\ 1 & -2 & 3 \\ 3 & -8 & 7 \end{pmatrix} \approx \begin{pmatrix} 1.359141 & 0 & 1.359141 \\ 1.359141 & -2.718282 & 4.077423 \\ 4.077423 & -10.873127 & 9.513986 \end{pmatrix} .

Pulling the factor $e/2$ out of all three rows multiplies the determinant by $(e/2)^3$ , and expanding the integer matrix along its first row,

\det \begin{pmatrix} 1 & 0 & 1 \\ 1 & -2 & 3 \\ 3 & -8 & 7\end{pmatrix} = 1\cdot(-14 + 24) - 0 + 1\cdot(-8 + 6) = 10 - 2 = 8 ,

so $\det \exp A = (e/2)^3 \cdot 8 = e^3 = 20.085537$ , which is $e^{\operatorname{tr} A}$ with $\operatorname{tr} A = 0 + 0 + 3 = 3$ : property 6 confirmed. Note in passing that $(\exp A)_{12} = 0$ although $A_{12} = 1$ , and that the entrywise exponential of $A$ would have no zero entry at all: the exponential redistributes the entries completely.

7. What it solves. $\dot x = Ax$ for this companion matrix is the scalar equation $y''' - 3y'' + 3y' - y = 0$ written for $x = (y, y', y'')^{\mathsf T}$ , whose general solution is $(c_0 + c_1 t + c_2 t^2)e^{t}$ . The first column of $\exp(tA)$ must therefore be the solution with $y(0) = 1$ , $y'(0) = y''(0) = 0$ , namely $y(t) = e^t\bigl(1 - t + \tfrac12 t^2\bigr)$ ; differentiating gives $y'(t) = \tfrac12 t^2 e^t$ and $y''(t) = (t + \tfrac12 t^2)e^t$ , which are precisely the second and third entries of that column. The matrix exponential has solved the third-order equation without any eigenvector being computed.

ApplicationOriginal

Unitarity of quantum time evolution. For a time-independent Hamiltonian, the state obeys $i\hbar\, \dot\psi = H\psi$ , so $\psi(t) = \exp(-iHt/\hbar)\,\psi(0)$ by property 5. If $H$ is Hermitian then $(-iHt/\hbar)^\dagger = +iHt/\hbar = -(-iHt/\hbar)$ for real $t$ , so the exponent is anti-Hermitian and the adjoints statement makes the propagator unitary. That is exactly the conservation of total probability, $\|\psi(t)\| = \|\psi(0)\|$ , and it is a consequence of the algebra of the exponential rather than of anything about the particular $H$ . The eigenvalue property 4 then says that an energy eigenstate only picks up the phase $e^{-iE t/\hbar}$ , which is why stationary states are stationary.
Stability of a linear system, read off the spectrum. For $\dot x = Ax$ the solution is $x(t) = \exp(tA)x_0$ , and property 1 together with step 3 of the method says that every entry of $\exp(tA)$ is a sum of terms, each a polynomial in $t$ times $e^{\lambda t}$ , one group of terms for each eigenvalue $\lambda$ . So all solutions decay when every eigenvalue has $\operatorname{Re}\lambda < 0$ , and the trap noted above — a purely imaginary eigenvalue whose Jordan block is larger than $1 \times 1$ produces secular growth like $t\,e^{i\omega t}$ — is the mathematics behind resonance in a driven oscillator and behind the marginal cases in the stability analysis of a steady state.
Operator splitting in numerical simulation. Split the generator of an evolution equation into parts that can each be exponentiated exactly: a Laplacian handled in Fourier space and a nonlinear or reaction term handled pointwise, or the kinetic and potential parts of a Hamiltonian. The scheme advances by $\exp(hA)\exp(hB)$ in place of $\exp(h(A+B))$ , and part 5 of the product-rule proof shows the error per step is $\tfrac12 h^2[A, B] + O(h^3)$ . This is the error estimate behind the split-step Fourier method for the nonlinear Schrodinger equation, Strang splitting in atmosphere and ocean models, and the leapfrog and higher-order symplectic integrators of molecular dynamics, where symmetrising the composition as $\exp(\tfrac12 hA)\exp(hB)\exp(\tfrac12 hA)$ cancels the commutator term and leaves an $O(h^3)$ error per step.
Rotations from angular velocity. A real antisymmetric $3 \times 3$ matrix $\Omega$ with $\Omega x = \omega \times x$ satisfies $\Omega^{\mathsf T} = -\Omega$ , so $\exp(t\Omega)$ is real orthogonal by the adjoints statement, and $\det \exp(t\Omega) = e^{t \operatorname{tr}\Omega} = 1$ by property 6: it is a rotation, through angle $|\omega| t$ about $\omega$ . Rigid-body kinematics is the statement $\dot R = \Omega R$ , solved by this exponential, and the failure of $\exp(A)\exp(B) = \exp(A+B)$ for non-commuting $A, B$ is the familiar fact that rotations about different axes do not commute.
Continuous-time Markov chains. A generator $Q$ has non-negative off-diagonal entries and rows summing to zero, and the transition matrix over time $t$ is $P(t) = \exp(tQ)$ . Both defining properties of a stochastic matrix come straight out of this unit: $Q\mathbf{1} = 0$ gives $Q^k \mathbf{1} = 0$ for $k \ge 1$ and hence $P(t)\mathbf{1} = \mathbf{1}$ , so the rows sum to one; and writing $\exp(tQ) = e^{-ct}\exp\bigl(t(Q + cI)\bigr)$ with $c = \max_i |Q_{ii}|$ , permitted by property 2 because $cI$ commutes with everything, exhibits $P(t)$ as a positive multiple of the exponential of a matrix with non-negative entries. For $t \ge 0$ — the only range in which a transition matrix is asked for — every term $t^k(Q+cI)^k/k!$ of that exponential series is then entrywise non-negative, and so is $P(t)$ . The Chapman-Kolmogorov relation $P(s+t) = P(s)P(t)$ is property 2 again.

Definition

Matrix norms and condition numbers

Adapted

Add to PDF

Defines induced matrix norms and the spectral norm, defines the condition number and expresses it through the singular values, and states how far the solution of a linear system can move when the data are perturbed.

Prerequisites: Singular value decomposition, Inner products and norms, Quadratic forms and the Rayleigh quotient, Compactness

Definition(Induced matrix norm)AdaptedAdapted from Stakgold & Holst (2011), §10.6, p. 772

Fix one norm $\|\cdot\|$ , given on $\mathbb{C}^k$ for every $k$ and so available on the domain and on the codomain alike; as in (Stakgold & Holst (2011), §10.6, p. 772), where the operator acts on a single space, it is the same norm on both sides. The induced (or operator) norm of an $m \times n$ complex matrix $A$ is

\|A\| = \max_{x \neq 0} \frac{\|A x\|}{\|x\|} = \max_{\|x\| = 1} \|A x\| ,

the maximum being attained because the unit sphere is compact and $x \mapsto \|Ax\|$ continuous (compactness). It is the smallest number $c$ with $\|Ax\| \le c \|x\|$ for all $x$ . Because one and the same norm is used on both spaces, induced norms satisfy $\|I\| = 1$ and are submultiplicative: $\|A B\| \le \|A\| \|B\|$ whenever the product is defined. Both properties need that agreement: if different norms are allowed on the two spaces, $\|\cdot\|_\infty$ on the domain and $\|\cdot\|_1$ on the codomain give $\|I_n\| = n$ , and $\|\cdot\|_1$ on the domain with $\|\cdot\|_\infty$ on the codomain give $\|M\| = \max_{i,j}|M_{ij}|$ , for which the $2 \times 2$ all-ones $A$ has $\|A^2\| = 2 > 1 = \|A\|^2$ .

Taking the vector norms $\|x\|_q = \bigl(\sum_i |x_i|^q\bigr)^{1/q}$ for $1 \le q < \infty$ , and $\|x\|_\infty = \max_i |x_i|$ , gives

\|A\|_1 = \max_{j} \sum_{i} |A_{ij}| \ \ (\text{largest column sum}), \qquad \|A\|_\infty = \max_{i} \sum_{j} |A_{ij}| \ \ (\text{largest row sum}),

and the spectral norm $\|A\|_2$ , induced by the Euclidean norm. The Frobenius norm $\|A\|_F = \bigl(\sum_{i,j}|A_{ij}|^2\bigr)^{1/2}$ is submultiplicative but is not an induced norm, since $\|I_n\|_F = \sqrt{n} \neq 1$ for $n \ge 2$ . It is not induced even by a pair of norms, one on each space — but that value $\sqrt n$ does not show it, being itself $\|I_n\|$ for the pair ( $\|\cdot\|_\infty$ on the domain, $\|\cdot\|_2$ on the codomain). What shows it is the behaviour on rank-one matrices; see step 0 of the proof below.

Definition(The spectral norm and the spectrum)AdaptedAdapted from Stakgold & Holst (2011), §10.6, pp. 771-772 (including Lemma 10.6.1)

Let $A$ be a complex matrix and $\rho(A) = \max\{|\lambda| : \lambda \in \sigma(A)\}$ its spectral radius (defined for square $A$ ).

$\|A\|_2 = s_1(A)$ , the largest singular value of $A$ , equivalently $\|A\|_2 = \sqrt{\lambda_{\max}(A^\dagger A)}$ ; see the singular value decomposition.
If $A$ is square then $\rho(A) \le \|A\|$ for every induced norm.
If $A$ is Hermitian then $\|A\|_2 = \rho(A) = \max_k |\lambda_k|$ , and $\lambda_{\min}(A) \le x^\dagger A x / x^\dagger x \le \lambda_{\max}(A)$ for $x \neq 0$ .
Inequality 2 can be strict: $A = \begin{pmatrix} 0 & 1 \\ 0 & 0 \end{pmatrix}$ has $\rho(A) = 0$ and $\|A\|_2 = 1$ .

Idea of the proofOriginal

Everything here comes from one change of coordinates. In the singular bases $A$ is a diagonal matrix, so its effect on the unit sphere is a list of independent scalings, and the norm is the largest of them. The spectral radius, by contrast, only tests the single direction of an eigenvector, so it can never exceed a quantity defined as a maximum over all directions; it misses the stretching that happens off the eigenvectors, which is why a nilpotent matrix can have radius zero and norm one. When $A$ is Hermitian there is no off-eigenvector stretching left to miss, and the two agree.

ProofAdaptedAdapted from Stakgold & Holst (2011), §10.6, pp. 771-772, including Lemma 10.6.1

The claims of the definition. $\|I\| = 1$ because $\|Ix\| = \|x\|$ for every $x$ — which is a statement about one norm, and is where the convention that the same norm is used on both spaces is needed. Submultiplicativity: $\|ABx\| \le \|A\|\,\|Bx\| \le \|A\|\,\|B\|\,\|x\|$ , by applying the defining inequality $\|My\| \le \|M\|\,\|y\|$ first to $M = A$ with $y = Bx$ and then to $M = B$ ; here too the intermediate vector $Bx$ is measured in the one norm shared by the two spaces. For the $1$ -norm, $\|Ax\|_1 = \sum_i \bigl| \sum_j A_{ij}x_j \bigr| \le \sum_j |x_j| \sum_i |A_{ij}| \le \bigl( \max_j \sum_i |A_{ij}| \bigr) \|x\|_1$ by the triangle inequality, with equality at $x = e_{j^\ast}$ for a column $j^\ast$ of largest sum. For the $\infty$ -norm, $\|Ax\|_\infty = \max_i \bigl| \sum_j A_{ij}x_j \bigr| \le \bigl( \max_i \sum_j |A_{ij}| \bigr)\|x\|_\infty$ , with equality at the $x$ having $\|x\|_\infty = 1$ and $x_j = \overline{A_{i^\ast j}}/|A_{i^\ast j}|$ (or $1$ where that entry vanishes) for a row $i^\ast$ of largest sum, since then $(Ax)_{i^\ast} = \sum_j |A_{i^\ast j}|$ . Finally $\|I_n\|_F = \sqrt{n} \neq 1$ for $n \ge 2$ , and every induced norm has $\|I\| = 1$ , so the Frobenius norm is not an induced norm.

Not even a pair of norms induces $\|\cdot\|_F$ , for $n \ge 2$ . This last argument is ours; the cited page does not discuss the Frobenius norm. Suppose some norm $a$ on the domain $\mathbb{C}^n$ and some norm $b$ on the codomain $\mathbb{C}^n$ had $\max_{\|x\|_a = 1}\|Ax\|_b = \|A\|_F$ for every $A$ . Write $\|v\|_{a^\ast} = \max_{\|x\|_a = 1} |v^\dagger x|$ , a norm on $\mathbb{C}^n$ . Test the hypothesis on the rank-one matrices $A = u v^\dagger$ : since $(uv^\dagger)x = (v^\dagger x)\,u$ ,

\max_{\|x\|_a = 1} \|u v^\dagger x\|_b = \|u\|_b \max_{\|x\|_a = 1}|v^\dagger x| = \|u\|_b\,\|v\|_{a^\ast} , \qquad \|u v^\dagger\|_F = \|u\|_2 \|v\|_2 ,

the second because $\sum_{i,j}|u_i \overline{v_j}|^2 = \|u\|_2^2\|v\|_2^2$ . So $\|u\|_b\,\|v\|_{a^\ast} = \|u\|_2\|v\|_2$ for all $u, v$ . Fixing a $v_0$ with $\|v_0\|_2 = 1$ and putting $c = \|v_0\|_{a^\ast} > 0$ gives $\|u\|_b = \|u\|_2/c$ for every $u$ ; feeding that back with any fixed $u \neq 0$ gives $\|v\|_{a^\ast} = c\,\|v\|_2$ for every $v$ . Now take any $x$ with $\|x\|_a = 1$ and use $v = x$ in the definition of $\|\cdot\|_{a^\ast}$ : $c\,\|x\|_2 = \|x\|_{a^\ast} \ge |x^\dagger x| = \|x\|_2^2$ , so $\|x\|_2 \le c$ on the whole unit sphere of $a$ . Hence

\max_{\|x\|_a = 1}\|I_n x\|_b = \max_{\|x\|_a = 1} \frac{\|x\|_2}{c} \ \le\ 1 ,

whereas the hypothesis demands the value $\|I_n\|_F = \sqrt{n} > 1$ . No such pair exists. (This is the argument the definition block promises; the value $\|I_n\|_F = \sqrt n$ alone does not settle the question, because $\sqrt n$ is the pair-induced norm of $I_n$ for $\|\cdot\|_\infty$ on the domain and $\|\cdot\|_2$ on the codomain.)

Part 1. Take an SVD $A = U S V^\dagger$ (the singular value decomposition). For $x \neq 0$ put $y = V^\dagger x$ , so $\|y\|_2 = \|x\|_2$ since $V$ is unitary, and

\|Ax\|_2^2 = \|U S y\|_2^2 = \|S y\|_2^2 = \sum_{i \le p} s_i^2 |y_i|^2 \ \le\ s_1^2 \|y\|_2^2 = s_1^2\|x\|_2^2 ,

the first equality again by unitarity. Equality holds at $x = v_1$ , where $y = e_1$ . Hence $\|A\|_2 = s_1$ , and $s_1^2$ is the largest eigenvalue of $A^\dagger A$ by part 1 of the existence statement for the SVD, so $\|A\|_2 = \sqrt{\lambda_{\max}(A^\dagger A)}$ .

Part 2. Let $\lambda$ be an eigenvalue of $A$ with $|\lambda| = \rho(A)$ and $x \neq 0$ an eigenvector. For any induced norm, $\|A\| \ge \|Ax\|/\|x\| = \|\lambda x\|/\|x\| = |\lambda|$ by absolute homogeneity of the vector norm, so $\|A\| \ge \rho(A)$ .
Part 3. A Hermitian matrix is normal, so by part 5 of the SVD existence statement its singular values are the numbers $|\lambda_k|$ sorted decreasingly. With part 1 this gives $\|A\|_2 = \max_k |\lambda_k| = \rho(A)$ , which is the matrix case of Lemma 10.6.1 of the cited section. The two-sided Rayleigh estimate $\lambda_{\min}(A) \le x^\dagger A x / x^\dagger x \le \lambda_{\max}(A)$ for $x \neq 0$ is the Rayleigh bounds.
Part 4. For $A = \begin{pmatrix} 0 & 1 \\ 0 & 0\end{pmatrix}$ the characteristic polynomial is $\lambda^2$ , so $\sigma(A) = \{0\}$ and $\rho(A) = 0$ . But $A^\dagger A = \operatorname{diag}(0, 1)$ , whose largest eigenvalue is $1$ , so $\|A\|_2 = 1$ by part 1. The inequality of part 2 is therefore strict here, and it cannot be reversed in general.
A lemma for the blocks that follow: the inverse. Nothing in parts 1-4 needs this step; it is recorded here, where the singular bases are at hand, because the condition number defined in the next block and the sensitivity bounds after it both use it. Let $A$ be square and invertible, so $p = n$ and $s_n > 0$ . Inverting $A = USV^\dagger$ gives $A^{-1} = V S^{-1} U^\dagger$ , and reversing the order of the columns of $V$ and of $U$ turns this into an SVD of $A^{-1}$ with singular values $1/s_n \ge \cdots \ge 1/s_1$ . So $\|A^{-1}\|_2 = 1/s_n$ by part 1, and

\kappa_2(A) = \|A\|_2 \|A^{-1}\|_2 = \frac{s_1}{s_n} ,

which is the formula asserted in the definition below. If $A$ is Hermitian positive definite its singular values are its eigenvalues, all positive — by part 5 of the SVD existence statement (the singular value decomposition), as in step 3 above, with the absolute values dropped because the eigenvalues are positive — so $\kappa_2(A) = \lambda_{\max}/\lambda_{\min}$ . A unitary $Q$ has $Q^\dagger Q = I$ , so every eigenvalue of $Q^\dagger Q$ is $1$ and every singular value of $Q$ is $1$ , giving $\kappa_2(Q) = 1$ . Two further consequences used below: applying $A^{-1}$ to $Av_i = s_i u_i$ gives $A^{-1}u_i = v_i/s_i$ , and the computation of part 1 read from the other end gives $\|Ax\|_2^2 = \sum_i s_i^2|y_i|^2 \ge s_n^2\|x\|_2^2$ , that is $\|Ax\|_2 \ge s_n \|x\|_2$ for every $x$ . $\square$

Definition(Condition number)AdaptedAdapted from Stakgold & Holst (2011), §10.6, p. 772

For an invertible square matrix $A$ and an induced norm $\|\cdot\|$ , the condition number is

\kappa(A) = \|A\| \, \|A^{-1}\| \ \ge \ \|A A^{-1}\| = \|I\| = 1 .

For a singular $A$ one sets $\kappa(A) = \infty$ . In the spectral norm, with singular values $s_1 \ge \cdots \ge s_n > 0$ ,

\kappa_2(A) = \frac{s_1}{s_n} ,

and if $A$ is Hermitian positive definite this is $\lambda_{\max}/\lambda_{\min}$ . A matrix with $\kappa$ close to $1$ is called well-conditioned, and one with $\kappa$ large ill-conditioned. Unitary and orthogonal matrices have $\kappa_2 = 1$ .

What the cited page displays is the more general $A$ -condition number $\kappa_A(M) = \|M\|_A\,\|M^{-1}\|_A$ , built from the norm of an $A$ -inner product and introduced there as an extension of "the standard notion"; the $\kappa$ above is that standard notion, the case in which the inner product is the Euclidean one.

Definition(Sensitivity of a linear system)Original

Let $A$ be an invertible $n \times n$ matrix, $b \neq 0$ , and $x$ the solution of $A x = b$ . Fix any induced norm and write $\kappa = \kappa(A)$ .

Perturbed right-hand side. If $A(x + \delta x) = b + \delta b$ , then

\frac{1}{\kappa}\, \frac{\|\delta b\|}{\|b\|} \ \le\ \frac{\|\delta x\|}{\|x\|} \ \le\ \kappa\, \frac{\|\delta b\|}{\|b\|} .

In the spectral norm the upper bound is attained. So $\kappa$ is exactly the worst-case amplification of relative error, and no sharper constant exists. 2. Perturbed matrix. If $A + \delta A$ is invertible and $(A + \delta A)\hat{x} = b$ , then

\frac{\|\hat x - x\|}{\|\hat x\|} \ \le\ \kappa\, \frac{\|\delta A\|}{\|A\|} .

Distance to singularity. In the spectral norm,

\min\bigl\{\|\delta A\|_2 : A + \delta A \text{ is singular}\bigr\} = s_n = \frac{\|A\|_2}{\kappa_2(A)} ,

the minimum being attained at $\delta A = -s_n u_n v_n^\dagger$ . An ill-conditioned matrix is therefore literally a matrix that is relatively close to a singular one. 4. Rule of thumb. Solving $A x = b$ in arithmetic with unit roundoff $\varepsilon$ gives a relative error of order $\kappa(A)\, \varepsilon$ : about $\log_{10}\kappa(A)$ of the available decimal digits are lost, whatever algorithm is used.

Idea of the proofOriginal

Both bounds are the defining inequality of an induced norm used twice, once for $A$ and once for $A^{-1}$ — and the condition number is precisely the product of those two uses. Which way round you use them decides whether you get the upper or the lower bound, which is why the true amplification is trapped between $\kappa$ and $1/\kappa$ and not merely below $\kappa$ . Sharpness is then a matter of aiming: put the data along the direction $A$ stretches most and the error along the one it stretches least. The distance to singularity is the same arithmetic read geometrically: $A$ moves every vector by at least $s_n$ , so no smaller perturbation can manufacture a null vector, and one of size exactly $s_n$ can.

ProofOriginal

Write $\|\cdot\|$ for the fixed induced norm and $\kappa = \|A\|\,\|A^{-1}\|$ . Note $x \neq 0$ , since $b = Ax \neq 0$ .

Part 1, upper bound. Subtracting $Ax = b$ from $A(x + \delta x) = b + \delta b$ gives $A\,\delta x = \delta b$ , so $\delta x = A^{-1}\delta b$ and $\|\delta x\| \le \|A^{-1}\|\,\|\delta b\|$ . Separately $\|b\| = \|Ax\| \le \|A\|\,\|x\|$ , which rearranges to $1/\|x\| \le \|A\|/\|b\|$ . Multiplying the two inequalities,

\frac{\|\delta x\|}{\|x\|} \ \le\ \|A^{-1}\|\,\|\delta b\| \cdot \frac{\|A\|}{\|b\|} = \kappa\, \frac{\|\delta b\|}{\|b\|} .

Part 1, lower bound. Run the same two steps the other way round: $\delta b = A\,\delta x$ gives $\|\delta b\| \le \|A\|\,\|\delta x\|$ , and $x = A^{-1}b$ gives $\|x\| \le \|A^{-1}\|\,\|b\|$ , that is $1/\|b\| \le \|A^{-1}\|/\|x\|$ . Multiplying,

\frac{\|\delta b\|}{\|b\|} \ \le\ \|A\|\,\|\delta x\| \cdot \frac{\|A^{-1}\|}{\|x\|} = \kappa\, \frac{\|\delta x\|}{\|x\|} ,

which is the lower bound after dividing by $\kappa > 0$ .

Part 1, the upper bound is attained in the spectral norm. Let $A = USV^\dagger$ be an SVD. Take $b = s_1 u_1$ ; then $x = A^{-1}b = s_1 \cdot v_1/s_1 = v_1$ by step 5 of the previous proof, so $\|x\|_2 = 1$ and $\|b\|_2 = s_1$ . Take $\delta b = \epsilon\, u_n$ with $\epsilon \neq 0$ ; then $\delta x = A^{-1}\delta b = \epsilon\, v_n/s_n$ , so $\|\delta x\|_2 = |\epsilon|/s_n$ . Hence

\frac{\|\delta x\|_2/\|x\|_2}{\|\delta b\|_2/\|b\|_2} = \frac{|\epsilon|/s_n}{|\epsilon|/s_1} = \frac{s_1}{s_n} = \kappa_2(A) ,

so no constant smaller than $\kappa_2$ can replace it in part 1. (For $n = 1$ the two directions coincide, but then $\kappa_2 = 1$ and the bound is an equality for every $\delta b$ anyway.)

Part 2. Subtracting $Ax = b$ from $(A + \delta A)\hat x = b$ gives $A\hat x + \delta A\,\hat x = Ax$ , hence $A(\hat x - x) = -\delta A\, \hat x$ and $\hat x - x = -A^{-1}\,\delta A\,\hat x$ . Taking norms, $\|\hat x - x\| \le \|A^{-1}\|\,\|\delta A\|\,\|\hat x\|$ , and $\hat x \neq 0$ because $(A + \delta A)\hat x = b \neq 0$ . Dividing by $\|\hat x\|$ and inserting $\|A\|/\|A\|$ ,

\frac{\|\hat x - x\|}{\|\hat x\|} \ \le\ \|A^{-1}\|\,\|A\| \cdot \frac{\|\delta A\|}{\|A\|} = \kappa\, \frac{\|\delta A\|}{\|A\|} .

The bound is in terms of $\|\hat x\|$ , the computed vector, not $\|x\|$ ; that is what makes it an identity-free consequence of one line of algebra.

Part 3, no singular matrix is nearer than $s_n$ . Suppose $\|\delta A\|_2 < s_n$ . For every $x \neq 0$ , the triangle inequality and $\|Ax\|_2 \ge s_n\|x\|_2$ (step 5 of the previous proof) give

\|(A + \delta A)x\|_2 \ \ge\ \|Ax\|_2 - \|\delta A\, x\|_2 \ \ge\ \bigl( s_n - \|\delta A\|_2 \bigr)\|x\|_2 \ >\ 0 ,

so $A + \delta A$ has trivial kernel and is invertible. Hence every $\delta A$ making $A + \delta A$ singular has $\|\delta A\|_2 \ge s_n$ .

Part 3, and one at exactly that distance. Put $\delta A = -s_n u_n v_n^\dagger$ . For any $x$ , $\|u_n v_n^\dagger x\|_2 = |v_n^\dagger x| \,\|u_n\|_2 = |v_n^\dagger x| \le \|x\|_2$ by Cauchy-Schwarz, with equality at $x = v_n$ , so $\|\delta A\|_2 = s_n$ . And $(A + \delta A)v_n = s_n u_n - s_n u_n = 0$ with $v_n \neq 0$ , so $A + \delta A$ is singular. The minimum is therefore $s_n$ , attained there, and $s_n = s_1/\kappa_2(A) = \|A\|_2/\kappa_2(A)$ by step 5 of the previous proof.
Part 4 is a rule of thumb, and the only part of this statement not proved. Its basis is part 2 together with backward stability: a backward-stable solver returns an $\hat x$ that is the exact solution of a nearby system, $(A + \delta A)\hat x = b$ with $\|\delta A\|/\|A\|$ of order $\varepsilon$ (times a modest factor depending on $n$ and on the algorithm). Part 2 then bounds the relative error of $\hat x$ by $\kappa(A)$ times that, and $\log_{10}$ of the bound is the count of lost decimal digits. The clause "whatever algorithm is used" rests on step 3: merely rounding $b$ into the machine perturbs it by a relative $\varepsilon$ , and for the worst $b$ and $\delta b$ that perturbation alone is amplified by exactly $\kappa$ , so no algorithm can be uniformly better. What the rule of thumb adds to this is the empirical claim that the orders of magnitude are typically attained, which is not a theorem. $\square$

NoteOriginal

A matrix norm measures how much a matrix can stretch a vector; the condition number measures how much worse the stretching can be in one direction than in another. That ratio, not the size of the determinant, is what decides whether a linear system can be solved from imperfect data: $\kappa(A)$ is the factor by which relative errors in $b$ or in $A$ may be magnified in the answer, and the quantity to inspect before trusting a numerical solution or a fitted parameter.

Remark(Traps and conventions)AdaptedAdapted from Strauss (2008), §1.5, pp. 26-27; Arfken, Weber & Harris (2013), Exercise 5.1.3 note, p. 267

A small determinant does not mean ill-conditioned, and a large one does not mean well-conditioned. $\varepsilon I_n$ has determinant $\varepsilon^n$ , as small as you like, yet $\kappa_2 = 1$ : it is perfectly conditioned, merely scaled. Conversely $\operatorname{diag}(10^{6}, 10^{-6})$ has determinant $1$ and $\kappa_2 = 10^{12}$ . Conditioning is about the ratio of the extreme singular values, which is invariant under scaling $A \mapsto cA$ , while the determinant is not.

Conditioning is a property of the problem, not of the algorithm. Strauss (§1.5, pp. 26-27) places it as the stability requirement in well-posedness: a nonsingular matrix with one very small eigenvalue gives a unique solution that nevertheless moves a great deal when $b$ is perturbed slightly, and no choice of solver can repair that. The sharp condition is a small smallest singular value, which is the weaker requirement: $s_n \le |\lambda|$ for every eigenvalue $\lambda$ , since $\|Ax\|_2 \ge s_n\|x\|_2$ applied to an eigenvector gives $|\lambda| \ge s_n$ , so a small eigenvalue forces a small $s_n$ but not conversely. A standard example is the Hilbert matrix $A_{ij} = 1/(i + j + 1)$ with $i, j = 0, \dots, n-1$ , which arises when a function is fitted by a power series in the mean-square sense and whose determinant and smallest singular value shrink extremely fast with $n$ (Arfken, Weber and Harris note $\det A = 3.7 \times 10^{-12}$ already at $n = 5$ , Exercise 5.1.3, p. 267).

$\kappa$ depends on the norm. Different induced norms give different values, though they are equivalent up to factors depending only on $n$ (for example $n^{-1}\kappa_\infty \le \kappa_2 \le n\,\kappa_\infty$ ), so "ill-conditioned" is norm-independent as a qualitative statement. Always say which norm you mean; $\kappa$ with no subscript usually means $\kappa_2$ .

$\rho(A)$ is not a norm. It fails the triangle inequality and vanishes on non-zero nilpotent matrices, so it cannot be used in the bounds above; inequality 2 of the spectral-norm statement runs one way only.

Non-square matrices. $\kappa_2(A) = s_1/s_r$ with $r = \operatorname{rank} A$ is also used for rectangular $A$ and governs least-squares sensitivity, but then $\|A\|_2 \|A^{+}\|_2$ replaces $\|A\|\|A^{-1}\|$ and the bounds of parts 1-2 need modification.

Example(A 2 x 2 matrix with condition number exactly 100)Original

Take

A = \begin{pmatrix} 101 & 99 \\ 99 & 101 \end{pmatrix} .

1. Spectrum and singular values. $A$ is real symmetric, and $A(1,1)^{\mathsf T} = (200, 200)^{\mathsf T}$ , $A(1,-1)^{\mathsf T} = (2, -2)^{\mathsf T}$ , so the eigenvalues are $200$ and $2$ with orthonormal eigenvectors

w_1 = \tfrac{1}{\sqrt2}(1,1)^{\mathsf T}, \qquad w_2 = \tfrac{1}{\sqrt2}(1,-1)^{\mathsf T} .

Both eigenvalues are positive, so $A$ is symmetric positive definite and its singular values are its eigenvalues: $s_1 = 200$ , $s_2 = 2$ , with $u_1 = v_1 = w_1$ and $u_2 = v_2 = w_2$ .

2. The norms. By part 1 of the spectral-norm statement, $\|A\|_2 = 200$ . The largest column sum and the largest row sum are both $101 + 99 = 200$ , so $\|A\|_1 = \|A\|_\infty = 200$ as well, and $\|A\|_F = \sqrt{2(101^2 + 99^2)} = \sqrt{40004} = 2\sqrt{10001} \approx 200.0100$ .

3. The inverse and the condition numbers. $\det A = 101^2 - 99^2 = (101-99)(101+99) = 400$ , so

A^{-1} = \frac{1}{400}\begin{pmatrix} 101 & -99 \\ -99 & 101\end{pmatrix} .

Then $\|A^{-1}\|_2 = 1/s_2 = \tfrac12$ and $\|A^{-1}\|_1 = (101+99)/400 = \tfrac12$ , giving

\kappa_2(A) = \frac{s_1}{s_2} = \frac{200}{2} = 100 , \qquad \kappa_1(A) = 200 \cdot \tfrac12 = 100 .

Note that $\det A = 400$ is not small: conditioning and the determinant are unrelated, as the remark above says.

4. The worst case is attained. Take $b = (200, 200)^{\mathsf T}$ , which is a multiple of $u_1$ ; then $x = (1, 1)^{\mathsf T}$ , since $Ax = (200, 200)^{\mathsf T}$ . Perturb along $u_2$ : $\delta b = (0.1, -0.1)^{\mathsf T}$ . Then

\delta x = A^{-1}\delta b = \frac{1}{400}\begin{pmatrix} 101 \cdot 0.1 + 99 \cdot 0.1 \\ -99 \cdot 0.1 - 101 \cdot 0.1 \end{pmatrix} = \frac{1}{400}\begin{pmatrix} 20 \\ -20 \end{pmatrix} = \begin{pmatrix} 0.05 \\ -0.05\end{pmatrix} .

The relative sizes are

\frac{\|\delta b\|_2}{\|b\|_2} = \frac{0.1\sqrt2}{200\sqrt2} = 5 \times 10^{-4} , \qquad \frac{\|\delta x\|_2}{\|x\|_2} = \frac{0.05\sqrt2}{\sqrt2} = 5 \times 10^{-2} ,

a ratio of exactly $100 = \kappa_2(A)$ : a perturbation of the data in the fourth decimal place has moved the answer in the second.

5. The best case, for contrast. Keep the same $b$ and the same $\|\delta b\|_2$ , but point the perturbation along $u_1$ instead: $\delta b = (0.1, 0.1)^{\mathsf T}$ . Then $\delta x = \frac{1}{400}(0.2, 0.2)^{\mathsf T} = (5\times10^{-4}, 5\times10^{-4})^{\mathsf T}$ and $\|\delta x\|_2/\|x\|_2 = 5 \times 10^{-4}$ , an amplification of $1$ . The two experiments differ only in the direction of $\delta b$ , and part 1 confines every intermediate case between the factors $1/100$ and $100$ .

6. Distance to singularity. Part 3 says the nearest singular matrix is at spectral distance $s_2 = 2$ and is reached at $\delta A = -s_2 u_2 u_2^{\mathsf T}$ . Here

\delta A = -2 \cdot \tfrac12 \begin{pmatrix} 1 & -1 \\ -1 & 1\end{pmatrix} = \begin{pmatrix} -1 & 1 \\ 1 & -1 \end{pmatrix} , \qquad A + \delta A = \begin{pmatrix} 100 & 100 \\ 100 & 100\end{pmatrix} ,

which is indeed singular, and $\|\delta A\|_2 = 2$ . Relative to $\|A\|_2 = 200$ , this matrix is only $1\% = 1/\kappa_2(A)$ away from being singular — the geometric meaning of $\kappa_2 = 100$ .

7. Digits. $\log_{10}\kappa_2(A) = 2$ , so part 4 predicts the loss of about two decimal digits. In IEEE double precision, where the unit roundoff is $\varepsilon \approx 1.1 \times 10^{-16}$ , a solution of $Ax = b$ should be trusted to a relative accuracy of about $10^{-14}$ and no better.

ApplicationOriginal

Never form the normal equations if you can avoid it. The singular values of $A^\dagger A$ are the squares of those of $A$ , so $\kappa_2(A^\dagger A) = \kappa_2(A)^2$ . Solving a least-squares problem through $A^\dagger A x = A^\dagger b$ therefore squares the condition number and doubles the number of decimal digits lost — from $\log_{10}\kappa_2(A)$ to $2\log_{10}\kappa_2(A)$ , so that double precision and $\kappa_2(A) = 10^{4}$ leave about eight good digits instead of twelve — while a QR or SVD factorisation of $A$ itself works with $\kappa_2(A)$ . This single inequality is why numerical libraries fit models by orthogonal factorisation rather than by the textbook normal equations.
Fitting a polynomial, and why high degree fails. Least-squares fitting of a function on $[0,1]$ by a power series leads to the Hilbert matrix $A_{ij} = 1/(i+j+1)$ with $i, j = 0, \dots, n-1$ , whose singular values decay geometrically: $\kappa_2 \approx 4.8 \times 10^{5}$ already at $n = 5$ and $\approx 1.6 \times 10^{13}$ at $n = 10$ . At $n = 10$ double precision leaves about three correct digits in the fitted coefficients, whatever the solver. The cure is not a better solver but a better basis — orthogonal polynomials, for which the corresponding Gram matrix is diagonal, and the identity once they are normalised, so that $\kappa_2 = 1$ . Without normalisation $\kappa_2$ is the ratio of the extreme squared norms, which stays modest: for the Legendre polynomials, with $\int_{-1}^{1} P_k^2 = 2/(2k+1)$ , degrees up to $10$ give $\kappa_2 = 21$ .
Discretised differential operators get worse as the mesh is refined. The standard second-difference approximation of $-d^2/dx^2$ on $[0,1]$ with Dirichlet conditions and $N$ interior points, spacing $h = 1/(N+1)$ , is symmetric positive definite with eigenvalues $(4/h^2)\sin^2\bigl(k\pi h/2\bigr)$ , so

\kappa_2 = \cot^2\!\Bigl( \frac{\pi h}{2} \Bigr) \approx \frac{4}{\pi^2 h^2} ,

which is $4.13 \times 10^{3}$ at $N = 100$ and grows like $N^2$ . Accuracy and conditioning therefore pull in opposite directions, the iteration count of conjugate gradients grows like $\sqrt{\kappa_2} \sim N$ , and this is the reason multigrid and other preconditioners exist: they replace the system by an equivalent one whose condition number does not blow up with $h$ .

Inverse problems in the geosciences and in imaging. Recovering a source distribution from remote measurements — seismic velocity from travel times, atmospheric composition from radiances, an image from projections — discretises to an $Ax = b$ whose singular values decay towards zero, so $\kappa_2$ is enormous and the relation of part 1 says the inversion will amplify measurement noise by that factor. The practical response is to accept a different problem: truncate the singular value expansion or add a Tikhonov penalty, replacing $A$ by a nearby operator of moderate condition number and trading resolution for stability. The condition number is the number that quantifies the trade.
Trusting an eigenvector computation. Route 1 of the matrix exponential computes $f(A) = S f(\Lambda) S^{-1}$ from an eigenbasis. The relevant conditioning is not $\kappa(A)$ but $\kappa(S)$ , which blows up as $A$ approaches a defective matrix, because two eigenvectors then become nearly parallel. The same quantity controls the sensitivity of computed eigenvalues, which is why nearly degenerate normal modes of a structure, or nearly crossing energy levels, demand care that well-separated ones do not.