Hilbert Spaces & Orthogonal Projection

A Banach space provides limits; a Hilbert space adds angles. The inner product turns approximation into geometry: the best approximation from a closed subspace is found by making the residual orthogonal to every allowed direction. This single idea underlies least squares, Fourier coefficients, Galerkin methods, and weak formulations of differential equations.

This chapter uses inner products linear in the first argument. Thus over $\mathbb C$, $\langle x,y\rangle=\overline{\langle y,x\rangle}$ and the Riesz functional represented by $y$ is $x\mapsto\langle x,y\rangle$.

Inner product and Hilbert structure #

An inner product satisfies conjugate symmetry, linearity in the first argument, and positive definiteness. Cauchy–Schwarz gives

$$|\langle x,y\rangle|\le\|x\|\|y\|, \qquad \|x\|=\sqrt{\langle x,x\rangle}.$$

The induced norm obeys the parallelogram law

$$\|x+y\|^2+\|x-y\|^2=2\|x\|^2+2\|y\|^2.$$

Conversely, a norm satisfying this law comes from an inner product via a polarization identity. A Hilbert space is complete in the induced norm. $\ell^2$, $L^2(\Omega)$, and finite-dimensional Euclidean/unitary spaces are the main examples; $L^p$ is not Hilbert when $p\ne2$.

Projection theorem #

Let $C$ be a nonempty closed convex subset of a Hilbert space $H$. For every $x\in H$, there is a unique $p\in C$ minimizing $\|x-p\|$. Closedness keeps the limit in $C$; convexity and the parallelogram law force a minimizing sequence to be Cauchy and guarantee uniqueness.

If $C=M$ is a closed linear subspace, the minimizer is characterized by

$$x-P_Mx\perp M.$$

Hence every vector decomposes uniquely as

$$x=P_Mx+(I-P_M)x,qquad H=M\oplus M^\perp.$$

The projection is linear, bounded, idempotent, and self-adjoint: $P_M^2=P_M=P_M^*$. If $M\ne\{0\}$, then $\|P_M\|=1$.

Why the normal equations are correct #

Suppose $M=\mathop{\mathrm{span}}\{v_1,\ldots,v_n\}$ and $p=\sum_j c_jv_j$. Orthogonality of the residual gives

$$\left\langle x-\sum_jc_jv_j,v_i\right\rangle=0 \quad(i=1,\ldots,n).$$

Thus the coefficients solve the Gram system

$$Gc=b,qquad G_{ij}=\langle v_j,v_i\rangle, \qquad b_i=\langle x,v_i\rangle.$$

If the spanning vectors are linearly independent, $G$ is positive definite. For an orthonormal family, $G=I$ and $P_Mx=\sum_j\langle x,v_j\rangle v_j$.

Worked example: best affine fit in $L^2[0,1]$ #

Project $f(x)=x^2$ onto $M=\mathop{\mathrm{span}}\{1,x\}$. Write $p(x)=a+bx$. Orthogonality to $1$ and $x$ gives

$$\int_0^1(x^2-a-bx)\,dx=0, \qquad \int_0^1x(x^2-a-bx)\,dx=0.$$

Therefore

$$\begin{bmatrix}1&1/2\\1/2&1/3\end{bmatrix} \begin{bmatrix}a\\b\end{bmatrix} = \begin{bmatrix}1/3\\1/4\end{bmatrix},$$

so $a=-1/6$ and $b=1$. The best affine approximation is $p(x)=x-1/6$; the residual is orthogonal to every affine perturbation.

Riesz representation in a Hilbert space #

For every bounded linear functional $F\in H^*$, there is a unique $y\in H$ such that

$$F(x)=\langle x,y\rangle\quad\text{for all }x\in H, \qquad \|F\|=\|y\|.$$

If $F\ne0$, project any point with $F(x)\ne0$ onto the orthogonal complement of $\ker F$; that complement is one-dimensional and supplies the representing vector. This identifies $H$ anti-linearly with its continuous dual under the chosen convention. It is a special Hilbert-space theorem, not a representation of arbitrary Banach duals.

Projection checklist #

• Verify that the feasible subspace is closed; otherwise a nearest point may not exist.
• State the inner-product convention in complex spaces.
• Write the residual orthogonality conditions before coordinates.
• Distinguish orthogonal projection from an arbitrary idempotent operator.
• Use the Pythagorean identity $\|x-m\|^2=\|x-P_Mx\|^2+\|P_Mx-m\|^2$ to prove optimality.