The Bias-Variance Decomposition

MST0052 · Lecture 6 supplement · Optional

## The Bias-Variance Decomposition, Derived

### Vegard H. Larsen — BI Norwegian Business School

---

## What this deck is

- **Optional** — nothing here is required for the project
- The **proof** behind L6's three-term formula
- Each term **measured** in a simulation you can rerun
- New required material

</div>

---

01

## The setup

---

## Where data comes from

Assume every observation is **truth plus noise**:

$$y = f(x) + \varepsilon, \qquad E[\varepsilon] = 0, \qquad \mathrm{Var}(\varepsilon) = \sigma^2$$

We never see $f$ — only noisy samples from it.

---

## A fitted model is a random object

- Draw a **training set** $D$ → fit → get $\hat{f}\_D$
- Draw a **different** $D$ → fit → get a **different** $\hat{f}\_D$
- The fresh-draw figure in L6 showed exactly this

Two sources of randomness: the noise on a new observation, and the sample behind the fit.

---

## The quantity we decompose

Expected squared error at a **fixed new point** $x\_0$:

$$E\left[\left(y\_0 - \hat{f}\_D(x\_0)\right)^2\right]$$

The expectation runs over **both** the noise $\varepsilon\_0$ in $y\_0$ and the sample $D$.

---

02

## The derivation

---

## Step 1 — split off the noise

Add and subtract the truth $f(x\_0)$, then expand the square:

$$E\left[(y\_0 - \hat{f})^2\right] = \underbrace{E\left[(y\_0 - f)^2\right]}\_{\sigma^2} + E\left[(f - \hat{f})^2\right] + 2 E\left[\varepsilon\_0 (f - \hat{f})\right]$$

The cross term is **zero**: $\varepsilon\_0$ is fresh noise — independent of a model fitted on $D$.

---

## Step 2 — split the estimation error

Add and subtract the **average fit** $E[\hat{f}]$:

$$E\left[(f - \hat{f})^2\right] = \underbrace{\left(f - E[\hat{f}]\right)^2}\_{\text{Bias}^2} + \underbrace{E\left[(\hat{f} - E[\hat{f}])^2\right]}\_{\text{Var}}$$

This cross term is zero **by definition**: $E\left[\hat{f} - E[\hat{f}]\right] = 0$.

---

## The decomposition

$$E\left[(y\_0 - \hat{f}(x\_0))^2\right] = \sigma^2 + \text{Bias}^2\left[\hat{f}(x\_0)\right] + \mathrm{Var}\left[\hat{f}(x\_0)\right]$$

- **Exact** — no approximation, no assumptions beyond $E[\varepsilon]=0$, independence
- Specific to **squared loss** (backup)
- Every term is **non-negative** — σ² really is a floor

---

03

## Measure it

---

## Why simulation

- On **real data** you can't: $f$ and $\sigma^2$ are unknown, and you got **one** $D$
- On **synthetic data** you can: known truth, unlimited fresh training sets
- Same setup as L6's worked example: $y = \sin(2\pi x) + \varepsilon$, $\sigma = 0.3$, $n = 30$

---

## The recipe

```python
preds = np.empty((N_REPS, x_grid.size))
for r in range(N_REPS):                      # 2000 fresh training sets
    x_tr = rng.uniform(0, 1, size=30)
    y_tr = np.sin(2 * np.pi * x_tr) + rng.normal(0, 0.3, size=30)
    model = make_pipeline(PolynomialFeatures(degree=degree),
                          LinearRegression()).fit(x_tr.reshape(-1, 1), y_tr)
    preds[r] = model.predict(x_grid.reshape(-1, 1))

bias2 = ((preds.mean(axis=0) - f_true) ** 2).mean()
var = preds.var(axis=0).mean()
```

Pre-tested: <code>scripts/examples/lecture_06_bias_variance_supplement.py</code>

---

## The numbers

2000 training sets · same DGP as the L6 worked example

| degree | bias² | variance | σ² | sum | measured error |
|--------|-------|----------|-----|------|----------------|
| 1 | 0.155 | 0.021 | 0.090 | 0.265 | 0.266 |
| 4 | 0.003 | 0.020 | 0.090 | **0.112** | 0.113 |
| 15 | 0.033 | **27.346** | 0.090 | 27.469 | 27.467 |

The sum **equals** the error measured against fresh observations — the theorem, verified.

---

## The anatomy of the U-curve

![Measured bias-squared, variance, and total error vs polynomial degree](../figures/bias-variance-decomposition.svg)

L6's U-curve, **taken apart**: bias² collapses, variance climbs, σ² never moves.

---

## Where the variance comes from

For a linear model, the L4 supplement priced it exactly:

$$\mathrm{Var}\left[\hat{f}(x\_0)\right] = \sigma^2 x\_0^\top (X^\top X)^{-1} x\_0$$

- Near-collinear features → the inverse blows up → **variance blows up**
- Ridge's $+\lambda I$ tames the inverse — **buying variance with bias**

The tradeoff is not a metaphor — it is visible in the formulas.

---

## How the rest of the course attacks each term

- **L7 cross-validation** — estimates the *sum*, finds the minimum
- **Regularisation** (L4) — adds bias to cut variance
- **L10 bagging** — averages high-variance fits: variance **÷ B** (idealised)
- **L12 boosting** — stacks weak learners to grind down **bias**
- **σ²** — untouchable, by every method, forever

---

## Questions

Backup slides below ↓

## Backup — k-NN, decomposed exactly

The k-NN prediction is an average of $k$ noisy neighbours, so:

$$\mathrm{Var}\left[\hat{f}(x\_0)\right] = \frac{\sigma^2}{k} \qquad \text{Bias}\left[\hat{f}(x\_0)\right] = \frac{1}{k}\sum\_{i=1}^{k} f(x\_{(i)}) - f(x\_0)$$

- Small $k$: tiny bias, variance $\to \sigma^2$ — the **jagged** corner
- Large $k$: variance $\to 0$, but far-away neighbours **drag the average off target**

## Backup — why squared loss is special

- Squared loss makes the cross terms **vanish** — the decomposition is exact
- **0-1 classification loss** has no clean additive version — competing definitions exist
- The *intuition* still transfers; the *arithmetic* does not

The classification analogue of σ² is the Bayes error — see the main deck's backup.

## Backup — the bookkeeping

- $x\_0$ is **fixed** — bias and variance are functions of *where you predict*
- $D$ is random — its draw produces $\hat{f}\_D$
- $\varepsilon\_0$ is random — fresh noise, independent of $D$
- The figure **averages** bias²($x\_0$) and Var($x\_0$) over a grid of $x\_0$

---

## What's next

Back to the **main track** — this deck echoes forward:

- **Lecture 7:** CV estimates the sum you just saw split apart
- **Lecture 10:** bagging attacks the variance term head-on