Bias-Variance Tradeoff

<p class="kicker">MST0052 · Lecture 6 · Fall 2026</p>

## Bias-Variance<br>Tradeoff

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L4 and L5 each ended with a **knob** tuned by CV. Today: *why* that works.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

Every model can fail in exactly **two ways** — too rigid, or too flexible.

The most important idea in this course. Everything after today leans on it.

---

## Today's plan

<p class="kicker">90 minutes · one idea · seven figures</p>

- The **two failure modes**
- Decomposing error: **bias, variance, noise**
- The **U-curve** — and how to read yours
- **Worked example:** polynomials on noisy data

---

## Two ways to fail

---

## The question every model must answer

> How wrong, on average, on a **new** point?

$\hat{f}$ was fitted on **one sample**. A different sample → a different fit.

---

## Two failure modes

<div class="cols">
<div class="callout">
<h3>Too rigid</h3>
<p>Misses real structure — <strong>underfitting</strong>.</p>
</div>
<div class="callout">
<h3>Too flexible</h3>
<p>Chases noise in this sample — <strong>overfitting</strong>.</p>
</div>
</div>

Both fail on new data — for **opposite** reasons.

---

## The picture

![Three polynomial fits to noisy sine data](../figures/polynomial-fits.svg)

Same data. Same target. Which one is *right*?

---

## A fresh draw

![Three polynomial fits across fresh samples](../figures/polynomial-resample.svg)

Refit on new samples: the line **barely moves**, the wiggles **go wild**.

---

## The decomposition

---

## Expected squared error

$$E\left[(y - \hat{f}(x))^2\right]$$

The expectation runs over the **noise in $y$** and the **sample** behind $\hat{f}$.

---

## Three sources of error

$$E[(y - \hat{f}(x))^2] = \text{Bias}^2[\hat{f}(x)] + \text{Var}[\hat{f}(x)] + \sigma^2$$

<div class="cols-3">
<div class="callout">
<h3>Bias²</h3>
<p>Systematic — the model is too simple.</p>
</div>
<div class="callout">
<h3>Variance</h3>
<p>Instability — sensitive to the sample.</p>
</div>
<div class="callout">
<h3>σ²</h3>
<p>Irreducible — the floor.</p>
</div>
</div>

---

## Bias — the model's blind spot

$$\text{Bias}[\hat{f}(x)] = E[\hat{f}(x)] - f(x)$$

Averaged over **all possible training samples** — the model still misses.

The straight line in the left panel.

---

## Variance — the model's sensitivity

$$\text{Var}[\hat{f}(x)] = E\left[(\hat{f}(x) - E[\hat{f}(x)])^2\right]$$

Retrain on a fresh sample — how much does the prediction **move**?

The degree-15 wiggles.

---

## Irreducible noise

**No model, no amount of data, removes σ².** Randomness in the world, not in your model.

</div>

Two people with identical features can have different outcomes.

---

## The U-curve — and reading it

---

## They move in opposite directions

<div class="cols">
<div class="stat stat--word stat--blue">
<span class="stat-value">Bias ↓</span>
<span class="stat-label">more flexibility<br>fits more shapes</span>
</div>
<div class="stat stat--word stat--red">
<span class="stat-value">Variance ↑</span>
<span class="stat-label">more flexibility<br>chases the sample</span>
</div>
</div>

Somewhere in between: the **sweet spot.**

---

## The U-curve

![Bias-variance tradeoff: training and test error vs complexity](../figures/bias-variance.svg)

Training error always falls. **Test error turns.**

The minimum: where marginal bias-reduction equals marginal variance cost.

---

**Training error** tells you nothing about generalisation.

A model that memorises scores perfectly — and fails every new point.

---

## Would more data help?

<p class="kicker">Learning curves · error vs training-set size</p>

<div class="cols">
<div class="callout">
<h3>High bias — No.</h3>
<p>Both curves converge to a high error. The model is the bottleneck.</p>
</div>
<div class="callout">
<h3>High variance — Yes.</h3>
<p>A large train–validation gap that closes as data grows.</p>
</div>
</div>

---

## Diagnosing your project

<p class="kicker">Reference — screenshot this</p>

| Symptom | Diagnosis | What to try |
|---------|-----------|-------------|
| Train error high, validation close behind | High bias | More flexible model, better features |
| Train error low, validation much higher | High variance | Regularise, more data, simpler model |
| Both low and close | Near the sweet spot | Stop tinkering |

---

## Managing the dial

---

## One dial, three models

Every hyperparameter you have tuned **is** this dial.

<div class="cols-3">
<div class="callout">
<h3>k-NN — k</h3>
<p>k = 1 bends to every neighbour. k = 50 blurs everything.</p>
</div>
<div class="callout">
<h3>Trees — depth</h3>
<p>Deep memorises the sample. Shallow misses the structure.</p>
</div>
<div class="callout">
<h3>Ridge — λ</h3>
<p>λ = 0 is raw OLS. λ → ∞ shrinks everything flat.</p>
</div>
</div>

---

## You can't compute it — you can manage it

You never see bias and variance separately — CV sees their **sum**.

- **Define** candidate models across complexity levels
- **Estimate** out-of-sample error for each — honestly
- **Pick** the best, and check it's **stable across folds**

</div>

---

When two models tie out-of-sample, the **simpler one wins**.

Lower variance today beats speculative bias reduction tomorrow.

---

## What the examiner expects

<p class="kicker">Your project · the minimum bar</p>

- Name your model family's **complexity knob**
- Show **CV scores** across a range of it
- **Justify** the chosen value in bias-variance language
- Say whether **more data would help** — and how you know

</div>

---

## Worked example — polynomials

---

## Step 1 — A truth we know

<p class="kicker">Step 1 · Synthetic data</p>

$$y = \sin(2\pi x) + \varepsilon, \quad \varepsilon \sim \mathcal{N}(0, 0.3^2)$$

```python
import numpy as np
rng = np.random.default_rng(0)
x_train = rng.uniform(0, 1, size=30)
y_train = np.sin(2 * np.pi * x_train) + rng.normal(0, 0.3, size=30)
```

$n = 30$ — and we know $f$, so we can *see* bias and variance.

---

## Step 2 — One pipeline, one dial

<p class="kicker">Step 2 · The model</p>

```python
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

model = make_pipeline(PolynomialFeatures(degree=d),
                      LinearRegression())
model.fit(x_train.reshape(-1, 1), y_train)
```

`degree` **is** the dial.

---

## Step 3 — Degree 1: the high-bias corner

<p class="kicker">Step 3 · Underfit</p>

![Degree-1 polynomial fit to sine data](../figures/polynomial-underfit.svg)

Train and test error: both high — and **close**.

<p class="muted">Would more data help? <strong>No</strong> — the model is the bottleneck.</p>

---

## Step 4 — Degree 15: the high-variance corner

<p class="kicker">Step 4 · Overfit</p>

![Degree-15 polynomial fit to sine data](../figures/polynomial-overfit.svg)

Training error ≈ zero. Test error: **terrible.**

<p class="muted">Would more data help? <strong>Yes</strong> — it constrains the wiggles.</p>

---

## Step 5 — Sweep the dial

<p class="kicker">Step 5 · Degrees 1–15</p>

```python
from sklearn.metrics import mean_squared_error

train_errs, test_errs = [], []
for d in range(1, 16):
    m = make_pipeline(PolynomialFeatures(degree=d), LinearRegression())
    m.fit(x_train.reshape(-1, 1), y_train)
    train_errs.append(mean_squared_error(
        y_train, m.predict(x_train.reshape(-1, 1))))
    test_errs.append(mean_squared_error(
        y_test, m.predict(x_test.reshape(-1, 1))))
```

The same loop you'll run on your project — knob on the x-axis.

---

## Step 5 — The U-curve, on real numbers

<p class="kicker">Step 5 · The result</p>

![Train and test MSE vs polynomial degree](../figures/polynomial-sweep.svg)

Training falls forever. Test bottoms out at **degree 3–5**, then climbs.

---

## Step 6 — Fix the model, grow the data

<p class="kicker">Step 6 · Learning curve at degree 15</p>

![Training and test MSE vs sample size at degree 15](../figures/polynomial-learning-curve.svg)

Degree 15 *can* be right — **with enough data.**

---

## What the experiment shows

- **Training error is not a model selection rule**
- The right complexity **depends on how much data you have**
- The U-shape is **real** — you will see it in your project

</div>

---

## Wrap-up

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Decompose</span>
<span class="stat-label">bias² + variance + noise</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">Diagnose</span>
<span class="stat-label">read the train–val gap</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Simplify</span>
<span class="stat-label">the simplest model that<br>generalises wins</span>
</div>
</div>

---

## Before Lecture 7

- **Re-run** today's polynomial experiment on your machine
- **Plot train and CV scores** against your project's knob — $k$, depth, $\lambda$
- Read ahead: L7 is **model selection and cross-validation** in depth
- **L8 showcase signup** in Canvas — closes **today**

</div>

All the code is on the course site.

---

## Questions

Backup slides below — press down ↓

## The derivation, in three lines

<p class="kicker">For the curious · not on the exam</p>

$$E[(y - \hat{f})^2] = E[(y - f + f - \hat{f})^2]$$

$$= E[(y - f)^2] + E[(f - \hat{f})^2] + 2 E[(y - f)(f - \hat{f})]$$

$$= \sigma^2 + \text{Bias}^2[\hat{f}] + \text{Var}[\hat{f}]$$

The cross term vanishes: the noise $y - f$ is independent of the fit $\hat{f}$.

<p class="muted">The full derivation, with each step justified: the optional L6 supplement deck.</p>

## Same U-shape, real dataset

<p class="kicker">k-NN sweep · breast cancer</p>

```python
from sklearn.datasets import load_breast_cancer
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)
for k in [1, 3, 5, 11, 21, 51, 101]:
    pipe = make_pipeline(StandardScaler(),
                         KNeighborsClassifier(n_neighbors=k))
    s = cross_val_score(pipe, X, y, cv=5, scoring='f1')
    print(f"k={k:3d}: F1 = {s.mean():.3f} ± {s.std():.3f}")
```

The tradeoff is **universal** — different family, same shape.

## Double descent

<p class="kicker">Beyond this course</p>

For very over-parameterised models, test error can **fall again** past the interpolation threshold.

The U becomes a "U, then a second descent." Active research — mentioned so you recognise the term.

## Validation curves vs learning curves

<p class="kicker">Two diagnostics · often confused</p>

| Plot | x-axis | Tells you |
|------|--------|-----------|
| **Validation curve** | Hyperparameter ($\lambda$, $k$, depth) | Where on the U-curve you sit |
| **Learning curve** | Training set size $n$ | Whether more data would help |

Use both — they answer different questions.

## Bayes error

<p class="kicker">Terminology</p>

In classification, $\sigma^2$ is often called the **Bayes error rate** — the error of the optimal predictor that knows the true class probabilities.

Same idea, different name. The textbook uses both.

---

## What's next

**Lecture 7:** Model selection and cross-validation

- k-fold CV in depth
- Hyperparameter tuning with GridSearchCV
- Reporting model selection results