Unsupervised Learning and PCA

<p class="kicker">MST0052 · Lecture 9 · Fall 2026</p>

## Unsupervised Learning<br>and PCA

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L4–L7 chased a **target**. Today there is no target — only **structure**.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

No target — the question changes from *how wrong?* to **what structure?**

PCA is the workhorse answer. One method, done right.

---

## Today's plan

<p class="kicker">90 minutes · no labels · one workhorse method</p>

- What changes when there is **no target**
- PCA — the **idea**, then the **objective**
- PCA **in practice** — pipeline, scaling, reading the output
- **Worked example:** digits, ending in a CV-chosen `n_components`

---

## No labels

---

## The pivot

<div class="cols">
<div class="callout">
<h3>Supervised — until now</h3>
<p>(X, y) — minimise error against y. A score to chase.</p>
</div>
<div class="callout">
<h3>Unsupervised — today</h3>
<p>X only. No loss function, no test accuracy.</p>
</div>
</div>

---

## Two goals

<div class="cols">
<div class="callout">
<h3>Dimensionality reduction</h3>
<p>Compress many correlated columns into few informative ones — today.</p>
</div>
<div class="callout">
<h3>Clustering</h3>
<p>Group similar rows without defining "similar" — Lecture 13.</p>
</div>
</div>

---

## Structure without labels

<div class="cols-3">
<div class="callout">
<h3>Surveys</h3>
<p>200 questions — maybe 5 underlying attitudes.</p>
</div>
<div class="callout">
<h3>Behaviour logs</h3>
<p>No segments defined — yet.</p>
</div>
<div class="callout">
<h3>Images</h3>
<p>Most pixels move together.</p>
</div>
</div>

Fewer **effective dimensions** than raw columns.

---

## Your project

<p class="kicker">The L1 rule · support, not headline</p>

<ul class="checklist">
<li>PCA as a <strong>preprocessing step</strong> before your headline model</li>
<li>Clusters as <strong>features</strong> for a downstream model</li>
<li class="no">A pure clustering study with no predictive target</li>
</ul>

---

## The idea

---

## The motivating problem

30 features — many correlated: height and weight, income and rent.

> What is the smallest set of *new* features that keeps most of what the 30 carry?

---

## The geometric picture

![Correlated 2D cloud with the PC1 and PC2 directions](../figures/pca-2d-rotation.svg)

The data *wants* to be one-dimensional — PCA finds the diagonal and calls it **PC1**.

---

## The first principal component

**PC1** = the direction of **maximum variance**.

- A **unit vector** — a *combination* of the features, not one of them
- Project each row onto it → one number per row: its **PC1 score**

---

## The rest of the components

- **PC2** — next-largest variance, **orthogonal** to PC1
- **PC3** — next, orthogonal to both. And so on.

$p$ features → $p$ components. You keep $k \ll p$.

---

## What PCA optimises

$$\max\ \text{Var}(\text{projection of the data onto } w)$$

subject to: $w$ is a **unit vector**, orthogonal to all previous components.

Under the hood: an eigendecomposition of the covariance matrix.

---

## What PCA is — and isn't

<div class="cols">
<div>
<ul class="checklist">
<li>A <strong>rotation</strong> of feature space</li>
<li><strong>Compression</strong> of correlated features</li>
<li>Unsupervised — $y$ is never used</li>
</ul>
</div>
<div>
<ul class="checklist">
<li class="no">Selecting "the best" original features</li>
<li class="no">Discovering causal structure</li>
<li class="no">Optimised for prediction</li>
<li class="no">Able to capture curved structure</li>
</ul>
</div>
</div>

---

## PCA in practice

---

## The workflow

<p class="kicker">The pattern · memorise this</p>

1. **Centre** — PCA does it for you
2. **Scale** — *you* do it, with `StandardScaler`
3. **Fit** on training data, inside a `Pipeline`
4. **Transform** → component scores
5. **Feed** the scores to a downstream model — optional

---

**Scale first.** PCA maximises variance — and variance has units.

Metres to millimetres: a million times the variance, identical information.

---

## In code, inside a pipeline

```python
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(StandardScaler(), PCA(n_components=2))
X_pca = pipe.fit_transform(X_train)   # shape: (n_samples, 2)
```

<p class="muted">PCA learns its rotation from the data — inside a <code>Pipeline</code> it is refit on each CV fold. No leakage.</p>

---

## Read the variance

```python
import numpy as np

pca = pipe.named_steps['pca']
print(pca.explained_variance_ratio_)         # per component
print(np.cumsum(pca.explained_variance_ratio_))  # first k together
```

The cumsum is the **most useful diagnostic** PCA gives you.

---

## Choosing `n_components`

<div class="cols-3">
<div class="callout">
<h3>Visualise</h3>
<p>2 or 3 — human eyes are the constraint.</p>
</div>
<div class="callout">
<h3>Variance target</h3>
<p><code>PCA(n_components=0.9)</code> keeps just enough for 90%.</p>
</div>
<div class="callout">
<h3>Let CV choose</h3>
<p>The downstream CV score decides — L7's rule.</p>
</div>
</div>

Pick the rule that matches the question — never a number out of thin air.

---

## The scree plot

![Scree plot for the wine dataset with the elbow marked](../figures/scree-plot.svg)

Look for the **elbow** — after it, a component buys little.

---

## Cumulative variance

![Cumulative explained variance for wine with the 90% line](../figures/cumulative-variance.svg)

The 0.9 line makes "how many components?" a **visual** question — wine: k = 8.

---

## Loadings

```python
pca.components_        # shape: (n_components, n_features)
pca.components_[0]     # PC1's recipe — one weight per feature
```

Large loadings on a **few** features → you *might* informally name the axis.

---

## Pitfalls

---

## Linear only

<p class="kicker">Pitfall 1 · A rotation cannot bend</p>

<div class="cols">
<div class="callout">
<h3>What PCA finds</h3>
<p>Linear combinations — straight directions through the cloud.</p>
</div>
<div class="callout warn">
<h3>What it misses</h3>
<p>Curves, rings, manifolds — distorted or hidden.</p>
</div>
</div>

Nonlinear escape hatches: kernel PCA, t-SNE, UMAP — in the backups.

---

## Components have no names

<p class="kicker">Pitfall 2 · Naming is a story you impose</p>

<div class="cols">
<div class="callout">
<h3>Strong, sparse loadings</h3>
<p>Calling PC1 a "size" axis is defensible.</p>
</div>
<div class="callout warn">
<h3>Loadings spread thin</h3>
<p>Naming is wishful thinking. Call it PC1.</p>
</div>
</div>

---

## Variance ≠ predictive signal

<p class="kicker">Pitfall 3 · The one that bites projects</p>

![Constructed dataset where the high-variance direction carries no class signal](../figures/variance-vs-signal.svg)

Drop the low-variance direction and you **destroy the model**.

---

## When PCA is the wrong choice

<p class="kicker">Before you reach for PCA</p>

<ul class="checklist">
<li class="no">Few features — p = 5 is not a dimensionality problem</li>
<li class="no">Tree-based downstream model — trees shrug at correlated inputs</li>
<li class="no">You need the <strong>original features</strong> interpretable</li>
<li class="no">Sparse data — use <code>TruncatedSVD</code> instead (backup)</li>
</ul>

---

## Worked example — digits

---

## The protocol

<p class="kicker">load_digits() · 8×8 images · 80/20 stratified · random_state=42</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">1,797</span>
<span class="stat-label">handwritten digits</span>
</div>
<div class="stat stat--green">
<span class="stat-value">64</span>
<span class="stat-label">pixel features</span>
</div>
<div class="stat">
<span class="stat-value">10</span>
<span class="stat-label">classes · labels colour only</span>
</div>
</div>

PCA never sees $y$ — the labels only colour the plots.

---

## Step 1 — Fit PCA, read the cumsum

<p class="kicker">Step 1 · Fit PCA, read the cumsum</p>

```python
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

pca_pipe = make_pipeline(StandardScaler(), PCA())
pca_pipe.fit(X_train)
cum = np.cumsum(pca_pipe.named_steps['pca'].explained_variance_ratio_)
```

PC1+PC2: **21.7%** — 90% needs **31 of 64** components.

---

## Step 2 — Project to 2D

<p class="kicker">Step 2 · Project to 2D</p>

![Digits on PC1–PC2, coloured by class](../figures/digits-pca-2d.svg)

PCA never saw the labels — **structure shows up anyway**. 0, 4 and 6 form islands; 1, 7 and 9 smear together.

---

## Step 3 — Let CV choose

<p class="kicker">Step 3 · Let CV choose n_components</p>

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

pipe = make_pipeline(StandardScaler(), PCA(),
                     LogisticRegression(max_iter=5000))
grid = {'pca__n_components': [2, 5, 10, 20, 30, 40, 50, 64]}

search = GridSearchCV(pipe, grid, cv=5, scoring='accuracy', n_jobs=-1)
search.fit(X_train, y_train)
print(search.best_params_, f"{search.best_score_:.3f}")
print(f"{search.score(X_test, y_test):.3f}")   # touched once
```

<p class="muted"><code>n_components</code> is a hyperparameter like any other — L7's machinery, unchanged.</p>

---

## Step 4 — Read the numbers

<p class="kicker">Step 4 · Read the numbers</p>

<div class="cols-3">
<div class="stat stat--red">
<span class="stat-value">64</span>
<span class="stat-label">best n_components —<br>the grid edge</span>
</div>
<div class="stat stat--blue">
<span class="stat-value">0.970</span>
<span class="stat-label">CV accuracy · with PCA<br>± 0.008</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.970</span>
<span class="stat-label">CV accuracy · no PCA<br>± 0.008</span>
</div>
</div>

Test: **0.972 both ways** — on digits, PCA bought **no accuracy at all**.

---

## What the experiment shows

<ul class="checklist">
<li class="no">PCA is <strong>not</strong> a free accuracy upgrade — CV said keep everything</li>
<li><code>n_components=40</code> sits <strong>one std</strong> below the best — L7's tie-break rule buys <strong>37% compression</strong> for free</li>
<li>The selection rule was <strong>CV — not the scree plot</strong></li>
</ul>

---

## Wrap-up

---

PCA buys you **fewer features** — it never promises better predictions.

Compression, visualisation, redundancy. Let CV make the call.

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Scale</span>
<span class="stat-label">StandardScaler before<br>PCA — always</span>
</div>
<div class="stat stat--word stat--red">
<span class="stat-value">Variance</span>
<span class="stat-label">is not predictive<br>signal</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Choose</span>
<span class="stat-label">n_components by CV,<br>not by eye</span>
</div>
</div>

---

## Before Lecture 10

- **Run** the digits PCA workflow on your machine
- Correlated features in your project? **Try PCA as preprocessing** — CV `n_components` against a no-PCA baseline
- Read ahead: **L10 is ensembles** — the bias-variance tradeoff returns

</div>

All the code is on the course site.

---

## Questions

Backup slides below — press down ↓

## t-SNE and UMAP

<p class="kicker">Beyond linear · visualisation only</p>

```python
from sklearn.manifold import TSNE
X_tsne = TSNE(n_components=2, perplexity=30).fit_transform(X_train)
```

- **t-SNE** preserves local neighbourhoods; **UMAP** is faster, keeps more global shape
- **Visualisation only** in this course — never features for a headline model

## Kernel PCA

<p class="kicker">Beyond linear · the kernel trick</p>

```python
from sklearn.decomposition import KernelPCA

kpca = KernelPCA(n_components=2, kernel='rbf', gamma=0.1)
X_kpca = kpca.fit_transform(X_train)
```

Costs: $n^2$ compute, no `explained_variance_ratio_`, a `gamma` to tune.

## TruncatedSVD for sparse data

<p class="kicker">Sparse data · TF-IDF</p>

```python
from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=50)
X_svd = svd.fit_transform(X_sparse)
```

PCA's centring **destroys sparsity** — same math, no centring.

## Whitening

<p class="kicker">Option · PCA(whiten=True)</p>

```python
PCA(n_components=10, whiten=True)
```

- Scales each component to **unit variance** after projection
- Cost: forgets which component mattered most

## When the smallest PC carries the signal

<p class="kicker">Pitfall 3 · the construction</p>

- 30 features; the target depends on **PC28** alone
- PC1–PC10 hold **95%** of the variance — none of the signal
- A variance cutoff **destroys the model**; CV keeps PC28

The strongest argument for CV-driven `n_components`.

## Reconstruction

<p class="kicker">Diagnostic · reconstruction error</p>

```python
X_back = pipe.inverse_transform(X_pca)
```

- All components → exact; fewer → **lossy**
- Per-row reconstruction error flags **outliers and unusual cases**

---

## What's next

**Lecture 10:** Ensemble methods

- Decision trees
- Bagging and random forests
- Why combining models works