Backprop, By Hand

MST0052 · Lecture 14 supplement · Optional

## Backprop, By Hand

### Vegard H. Larsen — BI Norwegian Business School

---

## What this deck is

- **Optional** — nothing here is required for the project
- **Backprop derived** on the smallest network worth differentiating
- Why **ReLU replaced sigmoid**, and why `alpha` **is ridge**
- Every gradient **checked numerically**
- New required material

</div>

---

01

## The smallest network worth differentiating

---

## One hidden layer, every symbol named

$$h = g(W\_1 x + b\_1) \qquad \hat{y} = w\_2^\top h + b\_2 \qquad L = \tfrac{1}{2}(\hat{y} - y)^2$$

- $g$ = sigmoid on the hidden layer · **identity** output (regression)
- Remove the hidden layer and this is **L4's one neuron**, verbatim

---

## The picture

![The tiny network with forward connections and the backward deltas above](../figures/nn-backprop.svg)

Forward: left to right. **Backward: the two δ's**, right to left.

---

02

## The chain rule, layer by layer

---

## The output layer — an old friend

$$\delta\_2 = \frac{\partial L}{\partial \hat{y}} = \hat{y} - y$$

$$\frac{\partial L}{\partial w\_2} = \delta\_2 h \qquad \frac{\partial L}{\partial b\_2} = \delta\_2$$

The error signal is **the residual** — and these are L4's one-neuron gradients, with $h$ as the inputs.

---

## The hidden layer — one more link in the chain

Chain through $\hat{y}$, then through the activation:

$$\delta\_1 = \frac{\partial L}{\partial z\_1} = \left( w\_2 \odot g'(z\_1) \right) \delta\_2$$

$$\frac{\partial L}{\partial W\_1} = \delta\_1 x^\top \qquad \frac{\partial L}{\partial b\_1} = \delta\_1$$

Same shape as the output layer: **error signal times input**.

---

## Backprop is bookkeeping

- Compute $\delta$ at the output — **one subtraction**
- Each layer back: multiply by its **local derivative** ($w\_2$, $g'$)
- Every parameter's gradient: **its layer's δ times its layer's input**
- Cost: one forward pass + one backward pass — **not** one pass per parameter

"The chain rule, automated" — the main deck's slide, unpacked.

---

## Why ReLU replaced sigmoid

The backward pass **multiplies by $g'$ once per layer**:

- **Sigmoid:** $g' = g(1-g) \leq \tfrac{1}{4}$ — ten layers shrink $\delta$ by $\geq 4^{10}$ → **vanishing gradients**
- **ReLU:** $g'$ is $0$ or $1$ — the signal passes **undamped** (or not at all)

The main deck's "sigmoid kills gradients for large |z|" — now mechanical.

---

03

## Weight decay is ridge

---

## Add ridge's penalty, watch the update

Penalised loss: $L + \frac{\alpha}{2} \lVert w \rVert^2$ — the gradient gains $\alpha w$, so each step becomes

$$w \leftarrow (1 - \eta\alpha) w - \eta \nabla L$$

- Every step first **shrinks the weights** by $(1-\eta\alpha)$ — hence "weight **decay**"
- This is **ridge's penalty** riding along with **L4's gradient descent**
- `MLPClassifier(alpha=...)` — the same dial as `Ridge(alpha=...)`

---

## See it verified

```python
delta2 = y_hat - y                    # dL/dy_hat -- the residual
g_w2, g_b2 = delta2 * h, delta2
delta1 = (w2 * h * (1 - h)) * delta2  # chain rule through sigmoid
g_W1, g_b1 = np.outer(delta1, x), delta1
```

checked against central finite differences

```text
parameters checked:        21
max |backprop - numeric|:  1.72e-10
```

Pre-tested: <code>scripts/examples/lecture_14_backprop_supplement.py</code> — the whole network in ~30 lines of NumPy.

---

## What this buys you at the oral exam

- **What backprop is** — the chain rule per layer, reusing the downstream error
- **Why deep sigmoid nets stall** — a $\leq \tfrac{1}{4}$ factor per layer
- **What `alpha` does** — ridge's penalty, shrinking weights every step
- **Why the output layer is familiar** — it's L4's neuron on learned features

---

## Questions

Backup slides below ↓

## Backup — the cross-entropy surprise

Binary task: sigmoid output $\hat{p} = \sigma(z)$, cross-entropy loss. Chain the two:

$$\frac{\partial L}{\partial \hat{p}} \cdot \frac{\partial \hat{p}}{\partial z} = \hat{p} - y$$

- The sigmoid's derivative **cancels** against the loss — δ is again **the residual**
- Set the hidden layer aside and this is **logistic regression's gradient** (L5)

## Backup — the batch form

One example used vectors; a minibatch of $n$ rows stacks them:

$$\Delta\_2 = \hat{Y} - Y \qquad \nabla\_{W\_2} = \frac{1}{n} \Delta\_2^\top H \qquad \Delta\_1 = (\Delta\_2 W\_2) \odot g'(Z\_1)$$

- Same recursion, **matrices instead of vectors** — this is what GPUs are for
- The $\frac{1}{n}$ is why minibatch gradients are **averages** — L14's noise-as-a-feature slide

---

## What's next

The supplement arc, complete:

- **L4:** one neuron, two roads to β̂ · **L6:** the error, decomposed
- **L9:** the eigenvalues · **L10:** the average · **L11:** the loss families
- **L14:** the chain rule that trains them all