Why Averaging Works

<p class="kicker">MST0052 · Lecture 10 supplement · Optional</p>

## Why Averaging Works

### Vegard H. Larsen — BI Norwegian Business School

---

## What this deck is

- **Optional** — nothing here is required for the project
- **One formula** — the variance of an average of correlated fits
- Why `max_features` exists, in **numbers**
- New required material

</div>

---

## The formula

---

## The setup

$B$ trees, each a random object (bootstrap + data), identically distributed:

$$\mathrm{Var}(\hat{f}\_b) = \sigma^2 \qquad \mathrm{Corr}(\hat{f}\_b, \hat{f}\_{b'}) = \rho$$

- Same recipe → same $\sigma^2$ for every tree
- Same data → their errors are **correlated**: $\rho > 0$

---

## Expand the variance of the average

$$\mathrm{Var}\left( \frac{1}{B} \sum\_b \hat{f}\_b \right) = \frac{1}{B^2} \left[ B \sigma^2 + B(B-1) \rho \sigma^2 \right]$$

$B$ variance terms, $B(B-1)$ covariance terms — collect them:

$$\mathrm{Var}(\text{average}) = \rho \sigma^2 + \frac{1-\rho}{B} \sigma^2$$

---

## Read the two terms

<div class="cols">
<div class="callout">
<h3>(1−ρ)σ²/B</h3>
<p>The independent part. Averaging <strong>divides it by B</strong> — buy as many trees as you can afford.</p>
</div>
<div class="callout">
<h3>ρσ²</h3>
<p>The shared part. <strong>No amount of trees touches it.</strong> The floor.</p>
</div>
</div>

The main deck's "÷B is an upper bound" — this is the exact statement.

---

## The picture

![Variance of the average vs ensemble size for several correlations](../figures/ensemble-variance.svg)

More trees ride the curve **down to the floor** — only **lower ρ moves the floor**.

---

## What the forest does about it

---

## Two knobs, two terms

- **Bootstrap** alone (bagging): trees still see the same strong features → $\rho$ stays high
- **`max_features`** (the forest's coin flip): trees split on different features → **$\rho$ drops**
- The price: each tree gets worse — **$\sigma^2$ rises**

The bet: $\rho$ falls **faster** than $\sigma^2$ rises.

---

## See it in numbers

<p class="kicker">25 trees · 200 fresh-data ensembles · measured at 50 test points</p>

| `max_features` | σ² (one tree) | ρ | formula | measured |
|----------------|--------------|------|---------|----------|
| `None` (bagging) | 9.7 | 0.103 | 1.348 | 1.354 |
| `'sqrt'` (forest) | **14.5** | **0.038** | **1.116** | 1.107 |

Worse trees, **better ensemble** — and the formula predicts the measurement within 1%.

<p class="muted">Pre-tested: <code>scripts/examples/lecture_10_ensembles_supplement.py</code></p>

---

## Why bias doesn't average away

$$E\left[ \frac{1}{B} \sum\_b \hat{f}\_b \right] = E[\hat{f}\_1]$$

- The average of $B$ identically distributed fits has **the same expectation** as one fit
- Averaging attacks **variance only** — the L6 decomposition's other term is untouched
- That's boosting's job (L12) — and why forests use **deep, low-bias** trees

---

## Questions

Backup slides below ↓

## Backup — where the 63% comes from

A bootstrap draw of $n$ rows misses row $i$ with probability

$$\left( 1 - \frac{1}{n} \right)^n \xrightarrow{\ n \to \infty\ } e^{-1} \approx 0.37$$

- So each tree sees $\approx 1 - e^{-1} \approx 63\%$ of distinct rows
- The missing 37% are the **out-of-bag** rows — free validation (main deck)

## Backup — the formula's fine print

- Assumes **identically distributed** trees with **equal pairwise** $\rho$ — true by symmetry of the recipe
- $\rho$ varies across input space — the table's $\rho$ is an **average** over test points
- Classification by majority vote doesn't decompose this cleanly — the **intuition** transfers, the **algebra** is for averages

---

## What's next

Back to the **main track** — the supplement story continues:

- **L12:** boosting attacks the **bias** term this deck proved averaging cannot touch
- **L14 supplement:** the chain rule behind the gradients boosting borrowed