Model Selection and Cross-Validation

<p class="kicker">MST0052 · Lecture 7 · Fall 2026</p>

## Model Selection<br>and Cross-Validation

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L6 gave the **vocabulary**. Today, the **toolkit**.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

Finding the sweet spot is easy. Finding it **without cheating** is the whole game.

Every slide today is a cheat-prevention device.

---

## Today's plan

<p class="kicker">90 minutes · one protocol · three families</p>

- **Cross-validation** in depth — k-fold, stratified, time-series
- **Hyperparameter search** with `GridSearchCV`
- The **metric** is a modelling choice
- **Worked example:** three families, one rule

---

## From L6 to a rule

---

## L6 in one sentence

> Out-of-sample error has a sweet spot in model complexity. The job is to find it without cheating.

Today: *how* — in code, on real data.

---

## Model selection is workflow selection

> **preprocessing + features + algorithm + hyperparameters**

You compare *workflows*, not algorithms.

---

## The three sets

<div class="cols-3">
<div class="callout">
<h3>Training</h3>
<p>Fit the model. Touched every time.</p>
</div>
<div class="callout">
<h3>Validation</h3>
<p>Tune and compare. Many times — never the final score.</p>
</div>
<div class="callout warn">
<h3>Test</h3>
<p>One final, honest number.</p>
</div>
</div>

---

The test set is touched **exactly once**.

Tune against it and it stops being a test set.

---

## Why one split isn't enough

<div class="cols">
<div class="callout">
<h3>One split</h3>
<p>One noisy number. The seed decides the winner.</p>
</div>
<div class="callout">
<h3>Cross-validation</h3>
<p>Averages over the luck of the split.</p>
</div>
</div>

---

## Cross-validation

---

## k-fold cross-validation

![K-fold cross-validation](../figures/kfold-cv.svg)

Fit on $k-1$ folds, score the held-out one — report **mean ± std**.

---

## What CV estimates

**CV scores the workflow, not a fitted model.**

</div>

After CV picks, **refit on the full training set** — that's the model you ship.

---

## Stratified k-fold

<div class="cols">
<div class="callout">
<h3>Classifier + cv=5</h3>
<p><strong>Stratified</strong> by default — class ratios preserved per fold.</p>
</div>
<div class="callout">
<h3>Regressor + cv=5</h3>
<p>Plain KFold — no stratification.</p>
</div>
</div>

`cv=5` means **different things** for classifiers and regressors.

---

## cross_val_score

```python
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = make_pipeline(StandardScaler(), LogisticRegression(max_iter=5000))
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring='f1')
print(f"CV F1: {scores.mean():.3f} ± {scores.std():.3f}")
```

<p class="muted">Pass a splitter object for full control — <code>shuffle=True</code> matters when rows are ordered.</p>

---

## When k-fold is wrong: time series

Shuffled k-fold trains on the **future** to predict the **past**. That's a leak.

</div>

```text
Fold 1:  [train: 0..n1]   [val: n1..n2]
Fold 2:  [train: 0..n2]   [val: n2..n3]
Fold 3:  [train: 0..n3]   [val: n3..n4]
```

`TimeSeriesSplit` — train up to a cut point, validate on the next block.

---

## Match the splitter to your data

<p class="kicker">Reference — screenshot this</p>

| Splitter | When |
|----------|------|
| `KFold` | Regression on i.i.d. data |
| `StratifiedKFold` | Classification — preserves class ratios |
| `ShuffleSplit` | Many random train/val splits |
| `GroupKFold` | Grouped rows (patients, users) that must not cross folds |
| `TimeSeriesSplit` | Temporal data |

Defaults are safe only for unstructured i.i.d. data.

---

## The search

---

## The search problem

Every knob value is a **different workflow** — with its own CV score.

Hand-tuning ("I tried 0.1 and it looked OK") is **not reproducible.** We need a search.

---

## GridSearchCV

```python
from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

pipe = make_pipeline(StandardScaler(),
                     RandomForestClassifier(random_state=42))

param_grid = {
    'randomforestclassifier__n_estimators':     [100, 300],
    'randomforestclassifier__max_depth':        [None, 5, 10],
    'randomforestclassifier__min_samples_leaf': [1, 5],
}

search = GridSearchCV(pipe, param_grid, cv=5, scoring='f1',
                      n_jobs=-1, return_train_score=True)
search.fit(X_train, y_train)

print(f"Best CV F1: {search.best_score_:.3f}")
print(f"Test F1:    {search.score(X_test, y_test):.3f}")
```

<p class="muted">The test set is touched once — the <strong>last line</strong>.</p>

---

## Read cv_results_, not just best_params_

```python
import pandas as pd
results = pd.DataFrame(search.cv_results_)
cols = ['mean_train_score', 'mean_test_score', 'std_test_score',
        'param_randomforestclassifier__max_depth']
print(results[cols].sort_values('mean_test_score',
                                ascending=False).head())
```

The **std** tells you whether the winner is stable.

---

## Grid, random, halving

<div class="cols-3">
<div class="callout">
<h3>Grid</h3>
<p>Small space — full coverage.</p>
</div>
<div class="callout">
<h3>Randomized</h3>
<p>Large or continuous space — sample n configs.</p>
</div>
<div class="callout">
<h3>Halving</h3>
<p>Many configs — eliminate losers early.</p>
</div>
</div>

**Grid** for 1–2 knobs. **Randomized** past 3–4.

---

## The metric is the search's objective

**The wrong metric is silent.** The code runs, the search finishes — the answer is wrong for your problem.

</div>

Whatever you pass to `scoring=` is what wins.

---

## scoring= strings

**Regression**   `r2` `neg_mean_squared_error` `neg_root_mean_squared_error` `neg_mean_absolute_error`

**Binary**   `accuracy` `f1` `roc_auc` `average_precision`

**Multi-class**   `f1_macro` `f1_weighted` `balanced_accuracy`

</div>

<p class="muted">Loss metrics are negated — the <code>neg_</code> convention. Backup slide explains.</p>

---

## Multi-metric evaluation

```python
search = GridSearchCV(
    pipe, param_grid, cv=5,
    scoring={'f1': 'f1', 'roc_auc': 'roc_auc'},
    refit='f1',
)
```

Optimise on F1 — *report* ROC-AUC too. Same selection rule.

---

## The pattern, end to end

<p class="kicker">The pattern · memorise this</p>

1. **Split** off the test set. Lock it.
2. **Build** a `Pipeline` — preprocessing + model
3. **Define** the grid with `stepname__param` keys
4. **GridSearchCV** with the right splitter and metric
5. **Inspect** `cv_results_` — not just `best_params_`
6. **Score** on the test set — once

---

## Pitfalls

---

## Leakage inside CV

<p class="kicker">Pitfall 1 · The L3 rule, per fold</p>

<div class="cols">
<div class="callout warn">
<h3>Wrong</h3>
<p>Scale the training set, <em>then</em> cross-validate — the scaler saw the validation folds.</p>
</div>
<div class="callout">
<h3>Right</h3>
<p>Scaler <strong>inside the pipeline</strong> — refit on each fold's training data.</p>
</div>
</div>

---

## The winner's curse

<p class="kicker">Pitfall 2 · Optimism from the best CV score</p>

<div class="cols">
<div class="stat stat--word stat--red">
<span class="stat-value">best_score_</span>
<span class="stat-label">optimistic — a max<br>over many tries</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">score(X_test)</span>
<span class="stat-label">honest — touched once</span>
</div>
</div>

---

## Too many tries

<p class="kicker">Pitfall 3 · One of 50 workflows will look great by chance</p>

<ul class="checklist">
<li class="no">The winner sits at the <strong>edge of the grid</strong></li>
<li class="no">The best score's std is tiny next to the gaps between configs</li>
<li class="no">A different <code>random_state</code> crowns a different winner</li>
</ul>

Defence: focused grid, **simpler on ties**, re-run with another seed.

---

## Nested CV

<p class="kicker">The rigorous upgrade</p>

```python
from sklearn.model_selection import cross_val_score, StratifiedKFold

inner = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)
outer = StratifiedKFold(n_splits=5, shuffle=True, random_state=1)

search = GridSearchCV(pipe, param_grid, cv=inner, scoring='f1')
nested = cross_val_score(search, X_train, y_train,
                         cv=outer, scoring='f1')
```

The outer loop scores the **whole procedure — tuning included.**

<p class="muted">For coursework, GridSearchCV + a held-out test set is enough.</p>

---

## Worked example — three families

---

## The protocol

<p class="kicker">load_wine() · same splits · same metric · same rule</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">178</span>
<span class="stat-label">samples</span>
</div>
<div class="stat stat--green">
<span class="stat-value">13</span>
<span class="stat-label">chemical features</span>
</div>
<div class="stat">
<span class="stat-value">3</span>
<span class="stat-label">classes · f1_macro</span>
</div>
</div>

All three families compete under the **same rule** — that's what makes it fair.

---

## Step 1 — Lock the test set

```python
from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split

X, y = load_wine(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
```

80/20, stratified, seeded — **locked.**

---

## Family 1 — Logistic regression

<p class="kicker">Family 1 · Logistic regression</p>

```python
from sklearn.linear_model import LogisticRegression

pipe_lr = make_pipeline(StandardScaler(),
                        LogisticRegression(max_iter=5000))
grid_lr = {'logisticregression__C': [0.01, 0.1, 1.0, 10.0]}

search_lr = GridSearchCV(pipe_lr, grid_lr, cv=5, scoring='f1_macro')
search_lr.fit(X_train, y_train)
```

Best C = 0.1 · CV f1_macro = **0.993 ± 0.015**

---

## The U-curve, in the wild

![CV f1_macro vs C for the wine logistic search](../figures/wine-c-curve.svg)

Your grid **samples a curve** — steep rise, then a plateau.

---

## Family 2 — k-NN

<p class="kicker">Family 2 · k-NN</p>

```python
from sklearn.neighbors import KNeighborsClassifier

pipe_knn = make_pipeline(StandardScaler(), KNeighborsClassifier())
grid_knn = {'kneighborsclassifier__n_neighbors': [3, 5, 11, 21, 41]}

search_knn = GridSearchCV(pipe_knn, grid_knn, cv=5,
                          scoring='f1_macro')
search_knn.fit(X_train, y_train)
```

Best k = 41 · CV f1_macro = **0.974 ± 0.025**

<p class="muted">k = 41 is the <strong>edge of the grid</strong> — Pitfall 3 says: extend it.</p>

---

## Family 3 — Random forest

<p class="kicker">Family 3 · Random forest</p>

```python
from sklearn.ensemble import RandomForestClassifier

pipe_rf = make_pipeline(RandomForestClassifier(random_state=42))
grid_rf = {
    'randomforestclassifier__n_estimators': [100, 300],
    'randomforestclassifier__max_depth':    [None, 5, 10],
}

search_rf = GridSearchCV(pipe_rf, grid_rf, cv=5,
                         scoring='f1_macro', n_jobs=-1)
search_rf.fit(X_train, y_train)
```

CV f1_macro = **0.986 ± 0.027** — no scaling needed, same protocol.

---

## Honest comparison

<p class="kicker">One loop · one table</p>

```python
for name, s in [('Logistic', search_lr),
                ('k-NN',     search_knn),
                ('RF',       search_rf)]:
    cv = s.best_score_
    cv_std = s.cv_results_['std_test_score'][s.best_index_]
    test = s.score(X_test, y_test)
    print(f"{name:9s} CV = {cv:.3f} ± {cv_std:.3f}   Test = {test:.3f}")
```

The number is not the story — the **spread across folds** is.

---

## The numbers

<p class="kicker">Test f1_macro · CV mean ± std underneath</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">1.000</span>
<span class="stat-label">Logistic · CV 0.993 ± 0.015</span>
</div>
<div class="stat">
<span class="stat-value">0.923</span>
<span class="stat-label">k-NN · CV 0.974 ± 0.025</span>
</div>
<div class="stat stat--green">
<span class="stat-value">1.000</span>
<span class="stat-label">RF · CV 0.986 ± 0.027</span>
</div>
</div>

---

## Which one would you ship?

<ul class="checklist">
<li>Logistic and RF <strong>tie</strong> within their CV std → ship the <strong>simpler</strong> one: logistic</li>
<li class="no">k-NN's winner sits at the <strong>grid edge</strong> → extend before judging it</li>
<li class="no">Test ≪ CV for a winner → suspect the <strong>winner's curse</strong></li>
</ul>

---

## Report and ship

---

## What to report

<p class="kicker">Your project · all six, every family</p>

- The **pipeline** — preprocessing + model
- The **CV protocol** — splitter, k, seed, stratification
- The **metric** — and why it matches the problem
- The **grid** searched
- The chosen config's CV score — **mean ± std**
- The **test score** — reported once

</div>

---

A reproducible **selection rule** beats a "best model" claim.

The rule survives a data update. The number does not.

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Protocol</span>
<span class="stat-label">same data, same splits,<br>same metric</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">Search</span>
<span class="stat-label">GridSearchCV — and read<br>cv_results_</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Once</span>
<span class="stat-label">the test set, exactly</span>
</div>
</div>

---

## Before Lecture 8

- **Run** today's wine workflow on your machine
- **Apply** the protocol to your dataset — at least two families

</div>

All the code is on the course site.

---

## Questions

Backup slides below — press down ↓

## Leave-one-out CV

<p class="kicker">Resampling · the k = n limit</p>

- Each fold leaves out a **single observation**
- Lowest bias, highest variance, slowest — $n$ fits

For everything but tiny samples, **k = 5 or 10 is the right answer.**

## Repeated k-fold

<p class="kicker">Resampling · tighter estimates</p>

```python
from sklearn.model_selection import RepeatedStratifiedKFold

cv = RepeatedStratifiedKFold(n_splits=5, n_repeats=10,
                             random_state=0)
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring='f1')
```

Averages over multiple shuffles — a tighter estimate at 10× the cost.

## Calibration in model selection

<p class="kicker">Beyond one metric · when probabilities matter</p>

```python
from sklearn.metrics import brier_score_loss
y_prob = search.best_estimator_.predict_proba(X_test)[:, 1]
print(f"Brier: {brier_score_loss(y_test, y_prob):.3f}")
```

Evaluate **calibration** alongside discrimination (AUC, F1).

## Why neg_mean_squared_error?

<p class="kicker">Convention · why the score is negative</p>

```python
search = GridSearchCV(pipe, grid, cv=5,
                      scoring='neg_mean_squared_error')
print(f"Best CV MSE: {-search.best_score_:.3f}")
```

scikit-learn **maximises** — so loss metrics are negated.

## Permutation importance

<p class="kicker">After selection · interpreting the winner</p>

```python
from sklearn.inspection import permutation_importance

result = permutation_importance(
    search.best_estimator_, X_test, y_test,
    n_repeats=20, random_state=0, scoring='f1_macro'
)
```

<p class="muted">Model-agnostic, and more honest than <code>.feature_importances_</code> for trees.</p>

---

## What's next

**Lecture 8:** Project showcase (optional)

- Volunteers present their project to the class
- Feedback from me, ideas from your classmates
- Not presenting? Come anyway — half the value is in the audience