Gradient Boosting

<p class="kicker">MST0052 · Lecture 12 · Fall 2026</p>

## Gradient Boosting

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L10 **averaged** trees; L11 optimised one boundary. Today: trees again, fitted **one mistake at a time** — the **last new supervised family** of the course.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

A forest fits three hundred trees **independently** and averages. Boosting fits them **one at a time** — each trained on the previous one's mistakes.

Parallel averaging vs sequential correction. The same protocol judges both.

---

## Today's plan

<p class="kicker">90 minutes · the last new family · three families, one protocol</p>

- **Sequential correction** vs bagging
- The algorithm **in one line**
- Three knobs: **learning rate**, trees, depth
- **Worked example:** boosting vs L10's forest vs L11's SVM

---

## Sequential correction

---

## Two ways to combine weak trees

<div class="cols">
<div class="callout">
<h3>Bagging — L10</h3>
<p>Independent bootstrap fits, averaged. Variance down, bias unchanged.</p>
</div>
<div class="callout">
<h3>Boosting — today</h3>
<p>Sequential fits, each targets what's still wrong. Bias down too.</p>
</div>
</div>

Same weak trees — opposite **assembly**.

---

## More powerful, more dangerous

<div class="cols">
<div class="callout">
<h3>Bagging plateaus</h3>
<p>Once the variance is averaged away, there's nothing left to remove.</p>
</div>
<div class="callout warn">
<h3>Boosting keeps attacking</h3>
<p>It goes after the residuals — and chases noise if the residuals are noise.</p>
</div>
</div>

---

## The algorithm in one line

---

## A first small tree

Fit a depth-3 tree. It captures the broad signal — and gets a lot wrong.

The **residuals** — what it missed — are themselves a **target**.

</div>

---

## Fit the next tree to the miss

$$r\_i = y\_i - \hat{y}\_i \qquad\qquad F\_2(x) = F\_1(x) + \eta \cdot h\_2(x)$$

The second tree predicts the **residuals**, not $y$. The learning rate $\eta$ keeps each correction **small**.

Repeat.

---

## Residuals are gradients

$$y\_i - \hat{y}\_i = -\frac{\partial}{\partial \hat{y}} \tfrac{1}{2}(y\_i - \hat{y}\_i)^2$$

Swap the loss, swap the target — same algorithm. That's the **gradient** in gradient boosting.

---

The entire algorithm is one line: the model you have, plus a **small step** toward fixing **its mistakes**.

<p class="muted">F ₘ = F ₘ₋₁ + η · hₘ — everything else is engineering.</p>

---

## Boosting, iteration by iteration

![Decision boundary after 1, 10 and 100 boosting iterations](../figures/boosting-iterations.svg)

Same data, three snapshots — the boundary **adds detail** with every tree.

---

## The three knobs

---

## The three knobs

<p class="kicker">Reference — screenshot this</p>

| Knob | What it controls | Start here |
|------|------------------|------------|
| `learning_rate` (η) | step size per tree | 0.05 |
| `max_iter` | total trees | 2000 + early stopping |
| `max_depth` | capacity per tree | 3–6 |

<p class="muted">They interact — lower η needs more trees; deeper trees want a smaller η.</p>

---

## The learning rate is the dial

<div class="cols">
<div class="callout">
<h3>Small η — 0.01 to 0.1</h3>
<p>Tiny corrections, many trees. Robust, slow.</p>
</div>
<div class="callout warn">
<h3>Large η — 0.5 and up</h3>
<p>Big corrections, few trees. Fast, overfits.</p>
</div>
</div>

L6's dial again — but here it's **η × n_trees together**, a 2D dial.

---

## Early stopping

```python
from sklearn.ensemble import HistGradientBoostingClassifier

gb = HistGradientBoostingClassifier(
    learning_rate=0.05, max_depth=4,
    max_iter=2000, early_stopping=True,
    random_state=42,
)
```

Don't grid-search the tree count — give a big budget and let validation **stop the training**.

---

## The recipe

<p class="kicker">The pattern · L7 machinery, new model</p>

1. **Split** off the test set
2. **Defaults + early stopping** first — no grid
3. **Tune** `learning_rate` × `max_depth` on a small grid
4. `subsample` / `l2_regularization` — **only if needed**
5. **Test** — once

---

## Boosting in practice

---

## Four engines, one algorithm

**Built in**   `HistGradientBoostingClassifier`

**Industry default**   `XGBoost`

**Fastest at scale**   `LightGBM`

**Categoricals**   `CatBoost`

</div>

Any is **defensible** — pick one, document it.

---

## The histogram trick

<div class="cols">
<div class="callout">
<h3>Raw values</h3>
<p>Evaluate splits on every threshold — slow.</p>
</div>
<div class="callout">
<h3>256 bins</h3>
<p>Split on bins — hundreds of times faster, same accuracy.</p>
</div>
</div>

<p class="muted">Why <code>HistGradientBoosting</code>, not the older <code>GradientBoostingClassifier</code>. Extra libraries: <code>pip install</code> — and commit your <code>requirements.txt</code>.</p>

---

## Pitfall 1 — Tuning everything at once

<p class="kicker">Pitfall 1 · Six knobs, exploding grid</p>

- Tune `learning_rate` × `max_depth` **first**
- Early stopping owns the tree count
- The rest — only on **evidence** of overfitting

---

## Pitfall 2 — Tilting the comparison

<p class="kicker">Pitfall 2 · Boosting wins by accident</p>

Different folds, different seeds, more compute budget — and boosting "wins."

Same split, same splitter, same metric — **L7's rule**.

---

## Pitfall 3 — feature_importances_ isn't there

<p class="kicker">Pitfall 3 · HistGBM has no feature_importances_</p>

```python
gb.feature_importances_        # AttributeError!

from sklearn.inspection import permutation_importance
r = permutation_importance(gb, X_test, y_test,
                           scoring='f1', n_repeats=20,
                           random_state=42)
```

Histogram trees don't ship impurity importances — use **permutation importance**.

---

## When boosting is the right call

<ul class="checklist">
<li><strong>Medium-to-large tabular</strong> data</li>
<li>Messy inputs — <strong>NaN and categoricals</strong> handled natively</li>
<li>You can afford <strong>early stopping + a small grid</strong></li>
<li class="no">Tiny clean datasets — simpler families tie (today's example proves it)</li>
<li class="no">Untuned, as a reflex</li>
</ul>

---

## Worked example — boosting vs forest vs SVM

---

## The protocol

<p class="kicker">load_breast_cancer() · 80/20 stratified · random_state=42 · 5-fold CV · f1</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">569</span>
<span class="stat-label">rows</span>
</div>
<div class="stat stat--green">
<span class="stat-value">30</span>
<span class="stat-label">numeric features</span>
</div>
<div class="stat">
<span class="stat-value">3</span>
<span class="stat-label">families · third lecture,<br>same protocol</span>
</div>
</div>

L10's forest and L11's SVM carry over as **incumbents**.

---

## Workflow 1 — defaults + early stopping

<p class="kicker">Workflow 1 · No grid at all</p>

```python
gb = HistGradientBoostingClassifier(
    learning_rate=0.05, max_depth=4,
    max_iter=2000, early_stopping=True, random_state=42,
)
scores = cross_val_score(gb, X_train, y_train, cv=cv, scoring='f1')
```

CV f1 **0.972 ± 0.009** — **ties the forest** out of the box. Early stopping spent **95** of 2000 trees.

---

## Where it stopped

![Training and validation loss with the early stop marked](../figures/boosting-early-stopping.svg)

Validation flattens at **85**, patience runs out at **95** — the other **1905** trees were never needed.

---

## Workflow 2 — the grid

<p class="kicker">Workflow 2 · learning_rate × max_depth, 12 cells</p>

```python
grid = {
    'learning_rate': [0.02, 0.05, 0.1, 0.2],
    'max_depth':     [3, 4, 6],
}

search = GridSearchCV(gb, grid, cv=cv, scoring='f1', n_jobs=-1)
search.fit(X_train, y_train)
```

Best lr = 0.02, depth = 3 — CV f1 **0.976 ± 0.012**, up **0.004**.

---

## Read the heatmap

![CV f1 over the learning_rate × max_depth grid](../figures/boosting-cv-heatmap.svg)

A **plateau** — every cell within **0.005**. The corner best is L7's flag — moot when the surface is this flat.

---

## XGBoost, same algorithm

<p class="kicker">Point at the code · not run today</p>

```python
from xgboost import XGBClassifier

xgb = XGBClassifier(
    learning_rate=0.05, max_depth=4, n_estimators=2000,
    eval_metric='logloss', early_stopping_rounds=50,
    random_state=42, n_jobs=-1,
)
xgb.fit(X_train, y_train,
        eval_set=[(X_val, y_val)], verbose=False)
```

Different API, same one-line update — expect similar numbers.

<p class="muted"><code>pip install xgboost</code> — and commit your <code>requirements.txt</code>.</p>

---

## Read the numbers

<p class="kicker">Test f1 · CV mean ± std underneath</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">0.973</span>
<span class="stat-label">HistGBM tuned<br>CV 0.976 ± 0.012</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.958</span>
<span class="stat-label">random forest (L10)<br>CV 0.972 ± 0.014</span>
</div>
<div class="stat stat--yellow">
<span class="stat-value">0.986</span>
<span class="stat-label">linear SVM (L11)<br>CV 0.981 ± 0.008</span>
</div>
</div>

<p class="muted">Every CV gap is <strong>inside one fold-std</strong> — a three-way tie. The 114-row test set leans SVM.</p>

---

## What actually moved the score

<p class="kicker">CV f1 gaps · the knobs were forgiving</p>

<div class="cols-3">
<div class="stat stat--green">
<span class="stat-value">0.004</span>
<span class="stat-label">tuning gain<br>over defaults</span>
</div>
<div class="stat stat--yellow">
<span class="stat-value">0.005</span>
<span class="stat-label">best-to-worst across<br>all 12 grid cells</span>
</div>
<div class="stat stat--blue">
<span class="stat-value">95</span>
<span class="stat-label">trees used,<br>of a 2000 budget</span>
</div>
</div>

No cliffs here — L11's gamma cliff was **0.21**. Boosting's safety net is **early stopping**, not the grid.

---

## Which one would you ship?

<ul class="checklist">
<li>The <strong>linear SVM</strong> stays champion — leads CV, one knob, simplest to defend</li>
<li>Report boosting <strong>honestly</strong> — tied the forest at defaults, +0.004 from tuning</li>
<li class="no">A "boosting wins tabular" headline — not on these rows, and the gap is inside one std</li>
<li class="no">Touching the test set again — it was spent <strong>once</strong></li>
</ul>

---

## Wrap-up

---

Fit the **miss**, shrink the **step**, stop when validation says stop — and the supervised toolbox is **complete**.

<p class="muted">Linear models, trees, forests, SVMs, boosting — from here on, the craft is using them well.</p>

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Residual</span>
<span class="stat-label">each tree learns what the<br>ensemble still misses</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">Shrink</span>
<span class="stat-label">the learning rate keeps every<br>correction small — tiny steps</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Stop</span>
<span class="stat-label">early stopping picks the tree<br>count — never the grid</span>
</div>
</div>

---

## Before Lecture 13

- **Run** today's three-family comparison on your machine
- Project: add boosting as a model family under your **L7 protocol**
- Read ahead: **L13 is clustering**

</div>

<p class="muted">L13 drops the labels: finding structure with <strong>no y at all</strong>.</p>

---

## Questions

Backup slides below — press down ↓

## AdaBoost, the original

<p class="kicker">History · reweighting, not gradients</p>

```python
from sklearn.ensemble import AdaBoostClassifier

ada = AdaBoostClassifier(n_estimators=200, random_state=42)
```

- Up-weight the **misclassified** points each round; combine learners by accuracy
- Predates gradient boosting — rarely used on tabular data today

## Native categorical features

<p class="kicker">Messy data · categorical_features=</p>

```python
HistGradientBoostingClassifier(categorical_features=[2, 5, 7])
```

- HistGBM, LightGBM and CatBoost handle categoricals **natively**
- Beats one-hot encoding in both **speed and accuracy** on many-category columns

## Missing values, no imputation

<p class="kicker">Messy data · NaN just works</p>

```python
gb.fit(X_with_nans, y)   # just works
```

- Each split learns an **optimal direction** to send missing values
- Contrast L3 — no imputation step needed before the model

## SHAP values

<p class="kicker">Interpretation · per-prediction</p>

```python
import shap

explainer = shap.TreeExplainer(gb)
shap_values = explainer(X_test)
shap.plots.beeswarm(shap_values)
```

Per-prediction feature contributions — the **next level** after permutation importance.

## When the grid grows

<p class="kicker">Search budgets · RandomizedSearchCV</p>

```python
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform, randint

param_dist = {
    'learning_rate': loguniform(0.01, 0.3),
    'max_depth':     randint(3, 8),
    'l2_regularization': loguniform(1e-3, 10),
}
search = RandomizedSearchCV(gb, param_dist, n_iter=40,
                            cv=cv, scoring='f1', n_jobs=-1)
```

Boosting + a large grid is where randomised (and halving — L7 backup) search shines.

---

## What's next

**Lecture 13:** Clustering

- K-means in depth
- Hierarchical clustering
- Choosing the number of clusters