From the Perceptron to Backpropagation

A neural network is a parameterized composition of functions. The perceptron supplies the linear decision unit; nonlinear hidden layers let compositions represent boundaries a single hyperplane cannot; backpropagation computes gradients through that composition without separately redoing shared calculations for every parameter.

The perceptron is a mistake-driven linear classifier #

For $y\in\{-1,+1\}$, predict

$$\hat y=\mathop{\mathrm{sign}}(w^Tx+b).$$

When an example is misclassified, update

$$w\leftarrow w+\eta yx,\qquad b\leftarrow b+\eta y.$$

If a positive point is on the negative side, $yx=x$ moves the score upward for similar points. If a negative point is predicted positive, $yx=-x$ moves it downward. With linearly separable data and suitable repeated presentation, the perceptron converges to some separating hyperplane—not necessarily the maximum-margin one.

If classes overlap, updates need not settle. The raw threshold also supplies no probability. Averaging iterates, using a margin loss, or moving to logistic/SVM objectives produces different algorithms rather than repairing the same guarantee.

Why a hidden layer changes expressiveness #

XOR labels $(0,0)$ and $(1,1)$ negative, but $(0,1)$ and $(1,0)$ positive. No line separates these sets. Hidden units can create intermediate half-spaces and combine them into a non-convex decision region.

A feedforward layer computes

$$h=\phi(Wx+b),$$

where $\phi$ acts elementwise. Without nonlinear $\phi$, stacking layers collapses:

$$W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2),$$

still one affine map. ReLU, sigmoid, tanh, or other nonlinearities prevent this collapse. An MLP is therefore not powerful because it has many matrix multiplications; its alternating linear maps and nonlinearities create a learned piecewise or smoothly nonlinear representation.

Backpropagation is reverse-mode chain rule #

Suppose a network produces loss $L$ through intermediate values:

flowchart TD
  X["input x"] --> Z1["z₁ = W₁x + b₁"]
  Z1 --> H["h = φ(z₁)"]
  H --> Z2["z₂ = W₂h + b₂"]
  Z2 --> L["loss L(z₂, y)"]

The forward pass computes and stores intermediates. The reverse pass starts with $\partial L/\partial L=1$ and propagates adjoints backward. For the first-layer weights,

$$\frac{\partial L}{\partial W_1} =\frac{\partial L}{\partial z_2} \frac{\partial z_2}{\partial h} \frac{\partial h}{\partial z_1} \frac{\partial z_1}{\partial W_1}.$$

The algorithm reuses each downstream derivative for every upstream parameter. Reverse-mode automatic differentiation makes the cost of all parameter gradients a modest multiple of the forward computation, rather than one full perturbation per parameter.

Backprop computes gradients. An optimizer such as SGD or Adam decides how to update parameters from them. Cross-entropy or squared error defines what is optimized. These are separate layers of the training mechanism.

A scalar worked pass #

Take $x=2$, first weight $w_1=0.5$, ReLU activation, output weight $w_2=3$, target $y=1$, and

$$L=\frac12(\hat y-y)^2.$$

Forward:

$$z_1=w_1x=1,\quad h=\mathop{\mathrm{ReLU}}(1)=1,\quad \hat y=w_2h=3,\quad L=2.$$

Backward:

$$\frac{\partial L}{\partial w_2}=(\hat y-y)h=2,$$

$$\frac{\partial L}{\partial w_1}=(\hat y-y)w_2\mathop{\mathrm{ReLU}}'(z_1)x=2\cdot3\cdot1\cdot2=12.$$

With a small learning rate, both weights move downward. If $z_1<0$, the ReLU derivative is zero and this example sends no gradient through $w_1$—the mechanism behind a “dead” ReLU.

Output layer and loss must agree #

Binary classification commonly uses one logit with binary cross-entropy; multiclass mutually exclusive outcomes use softmax cross-entropy; regression may use a linear output with squared or robust loss. Applying a sigmoid before a numerically stable “with logits” loss duplicates the transform and can damage gradients.

The loss encodes statistical and decision assumptions. Squared error emphasizes large residuals; cross-entropy rewards probability quality; class weights change the effective training distribution. A neural network does not infer the correct objective from labels.

Gradient checks and misconceptions #

Finite differences can check a small network:

$$\frac{\partial L}{\partial\theta}\approx \frac{L(\theta+\epsilon)-L(\theta-\epsilon)}{2\epsilon}.$$

Use double precision and a range of $\epsilon$ for diagnostics; finite differences are too expensive and noisy for ordinary training.

Backprop is not “the error flowing backward” as a physical substance, and unit activations are not automatically interpretable concepts. Universal approximation does not promise learnability, finite-data generalization, or efficient optimization. The architecture, initialization, objective, data, and optimizer determine which representable function is actually found.