Linear-Quadratic Regulation & Riccati Equations

The linear-quadratic regulator (LQR) is the rare optimal-control problem whose nonlinear HJB equation closes into a matrix differential equation. Linear dynamics preserve quadratic value functions, and a positive-definite quadratic input penalty makes the pointwise minimizer unique. The result is linear state feedback, but its validity depends on the model, weights, horizon, and absence of hard constraints.

Finite-horizon problem #

Let

$$\dot x(t)=A(t)x(t)+B(t)u(t),$$

and minimize

$$J[u]=\frac12x(t_f)^TS_fx(t_f) +\frac12\int_{t_0}^{t_f} \left[x(t)^TQ(t)x(t)+u(t)^TR(t)u(t)\right]dt,$$

with

$$S_f\succeq0,\qquad Q(t)\succeq0,\qquad R(t)\succ0.$$

The factor $1/2$ simplifies derivatives and does not change the minimizer. Positive definiteness of $R$ makes control effort strictly convex and ensures $R^{-1}$ exists.

Quadratic HJB ansatz #

The terminal value is quadratic, and the dynamics are linear, so try

$$V(t,x)=\frac12x^TP(t)x, \qquad P(t)=P(t)^T.$$

Then

$$V_t=\frac12x^T\dot P x, \qquad \nabla_xV=P x.$$

The HJB equation becomes

$$0=\frac12x^T\dot P x +\min_u\left[ \frac12x^TQx +\frac12u^TRu +x^TP(Ax+Bu) \right].$$

Only two terms depend on $u$:

$$\frac12u^TRu+x^TPBu.$$

Differentiate with respect to $u$:

$$Ru+B^TPx=0.$$

Therefore

$$\boxed{u^*(t)=-K(t)x(t), \qquad K(t)=R(t)^{-1}B(t)^TP(t).}$$

Completing the square makes global pointwise optimality explicit:

$$\frac12u^TRu+x^TPBu =\frac12(u+R^{-1}B^TPx)^TR(u+R^{-1}B^TPx) -\frac12x^TPBR^{-1}B^TPx.$$

Differential Riccati equation #

Substitute the minimizing control into HJB and use

$$x^TPAx=\frac12x^T(PA+A^TP)x.$$

Because the resulting quadratic form must vanish for every $x$,

$$\boxed{ -\dot P =A^TP+PA-PBR^{-1}B^TP+Q, \qquad P(t_f)=S_f.}$$

This is the continuous-time Riccati differential equation (RDE). Its boundary condition is at $t_f$, so $P$ is integrated backward. The state has its boundary condition at $t_0$ and is integrated forward under

$$\dot x=(A-BK)x.$$

The optimal cost from $(t_0,x_0)$ is

$$\boxed{J^*=V(t_0,x_0)=\frac12x_0^TP(t_0)x_0.}$$

Direct PMP cross-check #

Using the maximum-convention Hamiltonian

$$H=p^T(Ax+Bu)-\frac12(x^TQx+u^TRu),$$

maximization gives

$$B^Tp-Ru=0, \qquad u=R^{-1}B^Tp.$$

Because this convention has $p=-V_x$, set

$$p=-P x.$$

Then

$$u=-R^{-1}B^TPx,$$

and substituting into the costate equation recovers the same RDE. This is a useful sign audit: if PMP produces positive feedback for the regulator, the Hamiltonian/costate convention has been mixed.

Infinite horizon and the algebraic Riccati equation #

For constant $A,B,Q,R$, consider

$$J=\frac12\int_0^\infty(x^TQx+u^TRu)dt.$$

If the finite-horizon stabilizing solution converges to a constant $P$, then $\dot P=0$ and

$$\boxed{A^TP+PA-PBR^{-1}B^TP+Q=0.}$$

This is the continuous algebraic Riccati equation (CARE). The feedback and closed-loop matrix are

$$K=R^{-1}B^TP, \qquad A_{cl}=A-BK.$$

An algebraic equation can have multiple symmetric solutions. LQR uses the stabilizing solution: the one for which $A-BK$ is Hurwitz.

A standard sufficient set of conditions is:

• $(A,B)$ is stabilizable: every uncontrollable mode is already stable;
• $(Q^{1/2},A)$ is detectable: every mode invisible to the state penalty is stable;
• $R\succ0$ and $Q\succeq0$.

Full controllability and $Q\succ0$ are stronger than necessary. Conversely, a numerical CARE routine returning a matrix does not by itself prove that the chosen branch is stabilizing; inspect $P=P^T$, its definiteness, the residual, and the eigenvalues of $A-BK$.

Scalar CARE worked completely #

Take

$$\dot x=x+u, \qquad J=\frac12\int_0^\infty(x^2+u^2)dt.$$

Here $A=B=Q=R=1$. The CARE is

$$2P-P^2+1=0,$$

or

$$P^2-2P-1=0.$$

The two algebraic roots are

$$P=1\pm\sqrt2.$$

The feedback is $u=-Px$, so the closed-loop pole is

$$A-BK=1-P.$$

For $P=1-\sqrt2$, the pole is $+\sqrt2$ and the system is unstable. For

$$\boxed{P=1+\sqrt2,\qquad K=1+\sqrt2,}$$

the pole is $-\sqrt2$. The stabilizing requirement selects the correct Riccati root.

Discrete-time LQR #

For

$$x_{k+1}=A_kx_k+B_ku_k$$

and

$$J=\frac12x_N^TS_Nx_N +\frac12\sum_{k=0}^{N-1} (x_k^TQ_kx_k+u_k^TR_ku_k),$$

assume $V_{k+1}(x)=\tfrac12x^TP_{k+1}x$. Minimizing the one-step Bellman expression gives

$$\boxed{ K_k=(R_k+B_k^TP_{k+1}B_k)^{-1}B_k^TP_{k+1}A_k,}$$

$$\boxed{u_k^*=-K_kx_k,}$$

and the backward discrete Riccati recursion

$$\boxed{ P_k=Q_k+A_k^TP_{k+1}A_k -A_k^TP_{k+1}B_k (R_k+B_k^TP_{k+1}B_k)^{-1} B_k^TP_{k+1}A_k,}$$

with $P_N=S_N$. The continuous formula $K=R^{-1}B^TP$ must not be pasted into a sampled system; the $B^TPB$ term is a real consequence of one-step state propagation.

Choosing state and input weights #

Bryson's rule is an initial normalization, not an optimizer of design intent. For acceptable magnitudes $x_{i,\max}$ and $u_{j,\max}$,

$$\boxed{Q_{ii}=\frac{1}{x_{i,\max}^2}, \qquad R_{jj}=\frac{1}{u_{j,\max}^2}.}$$

Each normalized variable contributes roughly one when it reaches its declared scale. The inverse square is essential: a larger acceptable excursion receives a smaller weight. The KPI lecture's opposite proportionality is corrected here using the NASA-sponsored Stanford report's explicit statement of the rule.

Then iterate using physical outcomes:

• transient state peaks and settling;
• actuator magnitude, rate, and thermal load;
• closed-loop poles and dominant eigenvectors;
• sensitivity to model uncertainty and sensor noise;
• units and coordinate scaling.

Multiplying both $Q$ and $R$ by the same positive scalar leaves $K$ unchanged, because only their relative tradeoff matters.

What unconstrained LQR does not solve #

The derivation minimizes over all $u\in\mathbb R^m$. If the actuator saturates, clipping

$$u=\mathop{\mathrm{sat}}(-Kx)$$

creates a nonlinear closed loop that is not the solution of the constrained LQR problem. Stability and performance must be re-evaluated, and an observer or integrator can wind up.

Likewise, LQR does not enforce:

• state constraints;
• input rate or slew limits;
• a prescribed sparse or low-order feedback architecture;
• robustness to an uncertainty set;
• estimation from partial measurements.

Use observers for state reconstruction, constrained optimal control or MPC for hard limits, and fixed-structure optimization when the controller architecture itself is restricted.