Clustering – MST0052

<p class="kicker">MST0052 · Lecture 13 · Fall 2026</p>

## Clustering

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L9 found **directions of variance**; today finds **groups of similar rows** — the second half of unsupervised, and the **last new method** of the course.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

K-means will hand you **k groups** on any dataset you give it — whether or not the data **has** any.

Today: how to find groups — and how to check they deserve the name.

---

## Today's plan

<p class="kicker">90 minutes · no labels, no ground truth · the last new method</p>

- Grouping without labels — and the **L1 rule**
- **k-means** — algorithm, initialisation, choosing k
- **Hierarchical** — dendrograms and linkage
- Honest evaluation + **worked example:** wine, labels hidden until the reveal

---

## No labels, again

---

## The pivot, continued

<div class="cols">
<div class="callout">
<h3>L9 — directions</h3>
<p>Compress correlated columns into components.</p>
</div>
<div class="callout">
<h3>Today — groups</h3>
<p>Collect similar rows into clusters.</p>
</div>
</div>

Same X-only world, new question.

---

## Two families today

<div class="cols">
<div class="callout">
<h3>Partitioning — k-means</h3>
<p>Split into a fixed k of disjoint clusters.</p>
</div>
<div class="callout">
<h3>Hierarchical</h3>
<p>A tree of nested merges — every k at once.</p>
</div>
</div>

<p class="muted">Density-based — DBSCAN — in the backups.</p>

---

## Your project

<p class="kicker">The L1 rule · support, not headline</p>

<ul class="checklist">
<li>Exploratory — <strong>are there natural groups</strong> in my data?</li>
<li>Cluster <strong>labels or distances as features</strong> for the headline model</li>
<li>Per-cluster <strong>error analysis</strong> — where does my model fail?</li>
<li class="no">A clustering-only project with <strong>no predictive target</strong></li>
</ul>

---

## When clustering finds nothing

The features carry **no group signal** — or the "clusters" are artefacts of **scaling and feature choice**, and the k changes with the algorithm.

</div>

---

## k-means

---

## The objective

Given $k$: find centroids $\mu\_j$ and assignments that minimise

$$\sum\_{i=1}^{n} \min\_{j} \lVert x\_i - \mu\_j \rVert^2$$

The within-cluster sum of squares — **inertia**.

---

## Lloyd's algorithm

<p class="kicker">Assign · Update · Repeat</p>

![K-means from a bad initialisation to convergence](../figures/kmeans-iterations.svg)

Two steps, repeated — converges to a **local** minimum, not the best one.

---

## k-means++ and n_init

<div class="cols">
<div class="callout">
<h3>k-means++ — default</h3>
<p>Spreads the starting centroids. Most of the protection.</p>
</div>
<div class="callout">
<h3>n_init=10 — default</h3>
<p>Ten fresh starts, keep the best inertia. Free insurance.</p>
</div>
</div>

Leave both alone — never `n_init=1` in a report.

---

## Scale first — again

```python
from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline

pipe = make_pipeline(
    StandardScaler(),
    KMeans(n_clusters=3, n_init=10, random_state=42),
)
labels = pipe.fit_predict(X)
```

Euclidean distance — same rule as k-NN, ridge, PCA, SVM.

---

## What k-means assumes

<div class="cols">
<div class="callout">
<h3>Assumes</h3>
<p>Roughly spherical, similar-sized clusters. k known up front.</p>
</div>
<div class="callout warn">
<h3>Breaks on</h3>
<p>Elongated shapes, rare segments, unknown k.</p>
</div>
</div>

"k-means said three" is not "there are three."

---

## Choosing k — the elbow

![Inertia vs k on the wine dataset, bend at k = 3](../figures/kmeans-elbow.svg)

Diminishing returns after the bend — better at **ruling out** than ruling in.

---

## Choosing k — the silhouette

For each point: own-cluster **cohesion** vs nearest-other-cluster **separation**, in $[-1, 1]$.

```python
from sklearn.metrics import silhouette_score

silhouette_score(X_scaled, labels)
```

Maximise it — among **defensible** options only.

---

## Hierarchical clustering

---

## The agglomerative idea

- Every point starts as **its own cluster**
- Merge the **closest pair** of clusters
- Repeat until one remains

```python
from sklearn.cluster import AgglomerativeClustering

hc = AgglomerativeClustering(n_clusters=3, linkage='ward')
labels = hc.fit_predict(X_scaled)
```

---

## The dendrogram

![Dendrogram with a horizontal cut giving three clusters](../figures/dendrogram-example.svg)

Every horizontal **cut** is a partition — all values of k in one picture.

<p class="muted">scikit-learn clusters; <code>scipy.cluster.hierarchy</code> draws — <code>linkage(X, 'ward')</code> + <code>dendrogram(Z)</code>.</p>

---

## Linkage rules

<p class="kicker">Reference — screenshot this</p>

| Linkage | Cluster distance | Effect |
|---------|------------------|--------|
| `single` | closest pair | stringy chains |
| `complete` | farthest pair | tight, compact |
| `average` | mean over pairs | compromise |
| `ward` | variance increase | k-means-like — **default** |

---

## Four linkages, one dataset

![Single, complete, average and Ward linkage on the same noisy moons](../figures/linkage-comparison.svg)

Single linkage **chains through the noise** — Ward behaves like k-means.

---

## k-means vs hierarchical

<div class="cols">
<div class="callout">
<h3>k-means</h3>
<p>Fast at large n. Needs k up front. Random init. Scatter + silhouette.</p>
</div>
<div class="callout">
<h3>Hierarchical</h3>
<p>O(n²) and up. The cut chooses k. Deterministic. The dendrogram.</p>
</div>
</div>

<p class="muted">Default: k-means with <code>n_init=10</code>. Switch when n is small and the dendrogram is the deliverable.</p>

---

## No ground truth

---

Every clustering metric scores **geometry** — tight and well-separated. No metric can tell you the clusters are **real**.

Silhouette grades the shape of your answer, not its truth.

---

## Internal metrics

<div class="cols-3">
<div class="callout">
<h3>Silhouette</h3>
<p>Cohesion vs separation. Higher better — the default.</p>
</div>
<div class="callout">
<h3>Davies-Bouldin</h3>
<p>Nearest-cluster similarity. Lower better.</p>
</div>
<div class="callout">
<h3>Calinski-Harabasz</h3>
<p>Between- vs within-variance. Higher better.</p>
</div>
</div>

On hard cases they **disagree** — the disagreement is itself the signal.

---

## External metrics

When you **secretly have labels** and want to grade a clustering against them:

```python
from sklearn.metrics import adjusted_rand_score

adjusted_rand_score(y_true, cluster_labels)
```

- **ARI** — partition agreement, chance-adjusted, in $[-1, 1]$
- **NMI** — its information-theoretic cousin, in $[0, 1]$

For **benchmarking** the method — a real project has no clusters to grade against.

---

## Sensitivity checks

<p class="kicker">The minimum bar · if your project clusters</p>

<ul class="checklist">
<li><strong>Reseed</strong> — change <code>random_state</code>. Same clusters?</li>
<li><strong>Drop features</strong> one at a time. Do the groups survive?</li>
<li><strong>Subsample</strong> — cluster on 80%. Same groups?</li>
<li class="no">Reporting a result that survives <strong>none</strong> of these</li>
</ul>

---

## Pitfall 1 — Clusters have no names

<p class="kicker">Pitfall 1 · Naming is a story you impose</p>

<div class="cols">
<div class="callout">
<h3>Checked against the data</h3>
<p>"Cluster 2 is high-alcohol, low-flavanoid" — defensible.</p>
</div>
<div class="callout warn">
<h3>Invented</h3>
<p>"Cluster 2 is the premium segment" — wishful thinking. Call it cluster 2.</p>
</div>
</div>

---

## Pitfall 2 — Trusting silhouette in isolation

<p class="kicker">Pitfall 2 · One number is not evidence</p>

- A **high** silhouette with no domain meaning is suspicious
- A **low** one can still track reality — wine, in twenty minutes
- Silhouette + stability + domain sense — **together**

---

## Pitfall 3 — transform() swaps, it doesn't add

<p class="kicker">Pitfall 3 · KMeans in a pipeline replaces your features</p>

```python
make_pipeline(StandardScaler(), KMeans(3), LogisticRegression())
# LR sees ONLY 3 centroid distances — the 13 features are gone

FeatureUnion([("raw", FunctionTransformer()),
              ("km",  KMeans(3, n_init=10, random_state=42))])
```

`KMeans.transform()` returns **distances to centroids** — and inside a pipeline they *replace* the input. `FeatureUnion` to augment.

---

## Worked example — wine

---

## The protocol

<p class="kicker">load_wine() · scaled · all 178 rows · labels locked away until Step 3</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">178</span>
<span class="stat-label">wines</span>
</div>
<div class="stat stat--green">
<span class="stat-value">13</span>
<span class="stat-label">chemical features</span>
</div>
<div class="stat">
<span class="stat-value">3</span>
<span class="stat-label">classes — hidden</span>
</div>
</div>

No split today — clustering **describes** this dataset. Step 4 puts prediction back inside CV.

---

## Step 1 — Sweep k, read both diagnostics

<p class="kicker">Step 1 · Sweep k, read both diagnostics</p>

```python
for k in range(2, 11):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X_scaled)
    print(k, km.inertia_,
          silhouette_score(X_scaled, km.labels_))
```

Elbow at **3** — silhouette agrees: **0.285** at k = 3, vs 0.259 at k = 2.

<p class="muted">0.285 is <strong>weak</strong> by the >0.5 rule of thumb. Hold that thought.</p>

---

## Step 2 — Break the protection on purpose

<p class="kicker">Step 2 · Single inits across five seeds</p>

```python
for s in [0, 1, 42, 100, 999]:
    km_s = KMeans(n_clusters=3, n_init=1,
                  random_state=s).fit(X_scaled)
    print(adjusted_rand_score(km_ref.labels_, km_s.labels_))
```

<div class="cols-3">
<div class="stat stat--green">
<span class="stat-value">1.000</span>
<span class="stat-label">n_init=10 —<br>all five seeds identical</span>
</div>
<div class="stat stat--blue">
<span class="stat-value">0.982</span>
<span class="stat-label">worst k-means++<br>single init</span>
</div>
<div class="stat stat--red">
<span class="stat-value">0.882</span>
<span class="stat-label">worst plain-random<br>single init</span>
</div>
</div>

---

## Step 3 — The labels come out of the drawer

<p class="kicker">Step 3 · ARI against the hidden classes</p>

```python
from sklearn.metrics import adjusted_rand_score

adjusted_rand_score(y, km.labels_)
```

<div class="cols-3">
<div class="stat stat--yellow">
<span class="stat-value">0.285</span>
<span class="stat-label">silhouette —<br>"weak structure"</span>
</div>
<div class="stat stat--blue">
<span class="stat-value">0.897</span>
<span class="stat-label">ARI · k-means<br>vs truth</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.790</span>
<span class="stat-label">ARI · Ward<br>vs truth</span>
</div>
</div>

Weak geometry, **90% agreement with reality** — the metric graded the shape, not the truth. It cuts both ways.

---

## Step 4 — Back inside the CV protocol

<p class="kicker">Step 4 · Cluster distances as features</p>

```python
augmented = make_pipeline(
    StandardScaler(),
    FeatureUnion([("raw", FunctionTransformer()),
                  ("km",  KMeans(3, n_init=10, random_state=42))]),
    LogisticRegression(max_iter=5000),
)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(augmented, X, y, cv=cv)
```

The Pitfall-3 fix in action — KMeans refit inside **every fold**, no leakage.

---

## Step 4 — Read the numbers

<p class="kicker">CV accuracy · mean ± std</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">0.983</span>
<span class="stat-label">baseline — 13 features<br>± 0.014</span>
</div>
<div class="stat stat--yellow">
<span class="stat-value">0.978</span>
<span class="stat-label">augmented — 13 + 3<br>± 0.021</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.978</span>
<span class="stat-label">distances only — 3 features<br>± 0.027</span>
</div>
</div>

Cluster features bought **nothing** — but three distances nearly match thirteen features. **Compression, not accuracy** — L9's verdict, again.

---

## What goes in the report?

<ul class="checklist">
<li>k = 3 — <strong>elbow + silhouette + stability</strong> all point the same way</li>
<li>Describe clusters <strong>with caveats</strong> — scaled features; Ward agrees at 0.85</li>
<li class="no">Cluster features in the headline model — <strong>no gain to report</strong></li>
<li class="no">"We discovered three wine types" — the cultivars were <strong>known</strong>. ARI 0.897 is recovery, not discovery.</li>
</ul>

---

## Wrap-up

---

A clustering that survives **reseeding, reslicing and rescaling** is a finding. One that doesn't is an **artefact**.

Partition, perturb, support — and the toolbox is complete.

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Partition</span>
<span class="stat-label">you chose k — the split is<br>yours, not the data's</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">Perturb</span>
<span class="stat-label">reseed, drop, subsample —<br>keep only what survives</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Support</span>
<span class="stat-label">cluster features serve the<br>target — never the headline</span>
</div>
</div>

---

## Before Lecture 14

- **Run** the wine example on your machine
- Project: cluster your training data — if groups survive the **sensitivity checks**, test cluster features against a no-cluster baseline under your L7 protocol
- Read ahead: **L14 is neural networks** — a conceptual bridge, not a new project tool

</div>

<p class="muted">That was the <strong>last new method</strong> — the toolbox is complete.</p>

---

## Questions

Backup slides below — press down ↓

## DBSCAN

<p class="kicker">Beyond k-means · density</p>

```python
from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.5, min_samples=5).fit(X_scaled)
labels = db.labels_   # -1 means noise
```

- Clusters of **arbitrary shape**; no k to choose
- Outliers flagged as **noise** instead of forced into a group

## Gaussian mixtures

<p class="kicker">Beyond k-means · soft assignments</p>

```python
from sklearn.mixture import GaussianMixture

gm = GaussianMixture(n_components=3, random_state=42).fit(X_scaled)
probs = gm.predict_proba(X_scaled)
```

Each point gets a **probability** per cluster — and clusters can be **elliptical**, not just spherical.

## MiniBatchKMeans

<p class="kicker">Scale · hundreds of thousands of rows</p>

```python
from sklearn.cluster import MiniBatchKMeans

km = MiniBatchKMeans(n_clusters=10, batch_size=1024,
                     n_init=10, random_state=42).fit(X_scaled)
```

Same algorithm on random **batches** — much faster, slightly noisier centroids.

## Beyond Euclidean

<p class="kicker">Distance metrics · metric=</p>

| Data type | Better metric |
|-----------|---------------|
| Binary features | Hamming, Jaccard |
| Text (TF-IDF) | Cosine |
| Mixed numeric + categorical | Gower |

```python
AgglomerativeClustering(n_clusters=3, metric='cosine',
                        linkage='average')
```

## Cluster visualisation via PCA

<p class="kicker">L9 callback · the standard plot</p>

```python
from sklearn.decomposition import PCA

X_2d = PCA(n_components=2).fit_transform(X_scaled)
plt.scatter(X_2d[:, 0], X_2d[:, 1], c=km.labels_)
```

Project to 2D, colour by cluster — the standard "did it find structure?" plot.

---

## What's next

**Lecture 14:** Neural networks

- From linear models to neural networks
- The conceptual bridge from classical ML
- When deep learning beats classical methods