Dynamic Programming & the Hamilton-Jacobi-Bellman Equation

Dynamic programming asks a stronger question than “what is the optimal trajectory from this initial state?” It asks for the optimal remaining cost from every state and time. The resulting value function directly generates feedback, but it must satisfy a nonlinear partial differential equation: the Hamilton–Jacobi–Bellman (HJB) equation.

Value function and principle of optimality #

For

$$\dot x=f(t,x,u),\qquad u(t)\in U,$$

with cost

$$J_{t,x}[u]=\Phi(x(t_f)) +\int_t^{t_f}L(\tau,x(\tau),u(\tau))d\tau,$$

define

$$\boxed{V(t,x)=\inf_{u(\cdot)}J_{t,x}[u].}$$

For a short interval $[t,t+h]$, Bellman's principle of optimality says

$$V(t,x)=\inf_{u(\cdot)} \left[ \int_t^{t+h}L(\tau,x(\tau),u(\tau))d\tau +V(t+h,x(t+h)) \right].$$

Whatever control is optimal now must leave a state from which the remaining tail is itself optimal. Otherwise replacing that tail would improve the whole trajectory.

Infinitesimal derivation #

For continuous $u$ over a short step and differentiable $V$,

$$x(t+h)=x+h f(t,x,u)+o(h),$$

$$\int_t^{t+h}L\,d\tau=hL(t,x,u)+o(h),$$

and

$$V(t+h,x(t+h)) =V(t,x)+hV_t+h\nabla_xV^Tf+o(h).$$

Substitute into Bellman's identity, cancel $V(t,x)$, divide by $h$, and let $h\downarrow0$:

$$0=V_t(t,x)+min_{u\in U} \left[L(t,x,u)+\nabla_xV(t,x)^Tf(t,x,u)\right].$$

Thus

$$\boxed{-V_t=\min_{u\in U} \left[L+\nabla_xV^Tf\right], \qquad V(t_f,x)=\Phi(x).}$$

The terminal cost is the boundary condition propagated backward through the PDE.

Feedback from the minimizer #

Once $V$ is known, the optimal feedback is

$$\boxed{ u^*(t,x)\in\arg\min_{u\in U} \left[L(t,x,u)+\nabla_xV(t,x)^Tf(t,x,u)\right].}$$

This is a state-feedback law defined over the region where $V$ is valid. In contrast, a PMP shooting solution supplies one state-costate trajectory for one boundary condition unless additional synthesis turns those extremals into a switching surface.

Verification: when HJB is sufficient #

Suppose a continuously differentiable candidate $W$ satisfies

$$-W_t\le L(t,x,u)+\nabla W^Tf(t,x,u)$$

for every admissible $u$, with $W(t_f,x)=\Phi(x)$. Along any admissible trajectory,

$$\frac{d}{dt}W(t,x(t))=W_t+\nabla W^Tf\ge-L.$$

Integrating gives

$$W(t_0,x_0)\le \Phi(x(t_f))+\int_{t_0}^{t_f}Ldt =J[u].$$

If a feedback $u^*$ attains equality in the HJB minimization along its trajectory, then

$$W(t_0,x_0)=J[u^*]\le J[u]$$

for every admissible $u$. Therefore $W=V$ and $u^*$ is globally optimal on the verified domain. This is stronger than merely satisfying PMP.

Free-final-time target problems #

For an autonomous problem that terminates upon reaching a target set $X_f$, the stationary value $V(x)$ satisfies

$$\boxed{0=\min_{u\in U} \left[L(x,u)+\nabla V(x)^Tf(x,u)\right], \qquad V(x)=0\ \text{on }X_f.}$$

For minimum time, $L\equiv1$. The boundary data belong on the target, not at an arbitrary final clock time.

Worked minimum-time integrator #

Let

$$\dot x=u,qquad |u|\le1,$$

and stop at $x=0$ with minimum time. Geometrically, $V(x)=|x|$. Away from the origin,

$$V_x=\mathop{\mathrm{sgn}}x.$$

The stationary HJB equation is

$$0=1+\min_{|u|\le1}V_xu =1-|V_x|.$$

Its minimizer is

$$u^*(x)=-\mathop{\mathrm{sgn}}x,$$

which reaches the origin in $|x|$ seconds. At $x=0$, $V=|x|$ is not differentiable. The controller switches there, so the nonsmoothness is structural rather than a defect in the answer.

Viscosity solutions and switching surfaces #

Classical differentiability is often lost where different optimal trajectories have equal cost, at target boundaries, and across switching surfaces. Demanding a classical solution can therefore reject the true value function.

A viscosity solution replaces pointwise derivatives with comparisons against smooth test functions touching $V$ from above or below. It preserves three properties needed by control:

• nonsmooth value functions such as $|x|$ are admissible;
• appropriate boundary and growth conditions can restore uniqueness;
• monotone, stable, consistent numerical schemes can converge to the correct generalized solution.

The word “viscosity” is historical; no physical fluid term is added to the original control problem. A small artificial diffusion can be a numerical regularization, but its limit and boundary effects must be checked.

Connection to PMP #

With the maximum-convention Hamiltonian

$$H=p^Tf-L,$$

set

$$p=-\nabla_xV.$$

Then HJB's minimization of $L+V_x^Tf$ is PMP's maximization of $p^Tf-L$. Differentiating $p=-V_x$ along a smooth optimal trajectory yields

$$\dot p=-H_x.$$

Thus PMP trajectories are characteristics of HJB where the value is smooth. When $V$ has a corner, multiple costates or generalized gradients may correspond to the same state.

Why HJB is difficult #

If each of $n$ state dimensions is discretized with $N$ nodes, a full grid has $N^n$ points. This exponential growth is the curse of dimensionality. HJB remains practical for low-dimensional synthesis, structured approximations, decomposition, or special analytic forms such as LQR. For large systems, direct trajectory optimization, model predictive control, approximate dynamic programming, or learned value approximations may be needed—but each replaces exact global PDE solution with a new approximation whose residuals and constraint behavior must be verified.

Diagnostic checklist #

• Is $V$ a cost-to-go from the stated $(t,x)$, with the correct terminal or target boundary?
• Does the PDE use minimization or maximization consistently with the value definition?
• Does the control actually attain the pointwise extremum over the declared set $U$?
• Are state constraints represented as boundary/state-constraint conditions rather than merely ignored?
• Is a claimed classical solution differentiable where the derivation uses its gradient?
• Does the selected feedback keep the state inside the domain where $V$ was solved?