Support Vector Machines

<p class="kicker">MST0052 · Lecture 11 · Fall 2026</p>

## Support Vector Machines

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L10 **averaged** three hundred trees. Today: a single boundary, **optimised** — the course's most geometric model.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

A forest **averages** three hundred rough boundaries. An SVM **optimises one** — chosen by geometry.

Two design philosophies. One protocol will judge them.

---

## Today's plan

<p class="kicker">90 minutes · one model family · four workflows</p>

- The **maximum-margin** idea — a geometric principle
- **Soft margin** and the `C` parameter
- The **kernel trick** — nonlinear boundaries without nonlinear features
- **Worked example:** three kernels vs L10's **forest**

---

## The maximum margin

---

## Which line?

![Four candidate boundaries, all separating the training data](../figures/svm-candidates.svg)

All of them separate the training data **perfectly**. Training accuracy can't choose.

---

## The maximum margin

![The maximum-margin boundary with support vectors ringed](../figures/svm-margin.svg)

**Margin** = distance to the nearest point on either side. Pick the boundary that **maximises** it.

$$\min \tfrac{1}{2}\lVert w \rVert^2 \quad \text{s.t.} \quad y\_i(w^\top x\_i + b) \geq 1$$

---

## Support vectors

<div class="cols">
<div class="callout">
<h3>On the margin</h3>
<p>The few points that pin the boundary in place.</p>
</div>
<div class="callout">
<h3>Everyone else</h3>
<p>Delete them — nothing moves.</p>
</div>
</div>

The model **is** the support vectors.

---

## The soft margin

---

## Hard margins break

One **outlier** — or one mislabeled row — can swing the whole boundary. Or make separation **impossible**.

</div>

Real data is never perfectly separable. We need to allow **violations**.

---

## Slack variables

$$\min \tfrac{1}{2}\lVert w \rVert^2 + C \sum\_i \xi\_i$$

<div class="cols">
<div class="callout">
<h3>First term</h3>
<p>Wants a wide margin.</p>
</div>
<div class="callout">
<h3>Second term</h3>
<p>Charges for violations.</p>
</div>
</div>

---

## C is the dial

<div class="cols">
<div class="callout">
<h3>Small C</h3>
<p>Violations are cheap → wide margin, smooth. High bias, low variance.</p>
</div>
<div class="callout">
<h3>Large C</h3>
<p>Violations are expensive → narrow margin, contorts. Low bias, high variance.</p>
</div>
</div>

L6's dial in SVM clothing — tuned with CV, L7's machinery.

---

## C, visible

![SVM boundaries for C = 0.1, 1 and 100 on the same data](../figures/svm-soft-margin.svg)

Same data, only **C** changes — watch the margin width and the violation count.

---

## The kernel trick

---

## When no line works

Some datasets cannot be split by a straight line at any cost. Three options:

1. **Give up** — a different family. L10's forest is legitimate.
2. **Hand-build** nonlinear features and hope a line works there.
3. Let the SVM build them **implicitly** — the kernel trick.

---

## The explicit lift

$$\phi(x) = (x\_1^2,\ x\_2^2,\ \sqrt{2} x\_1 x\_2)$$

Rings in 2D become a **plane-separable** cloud in 3D.

The cost: dimensions explode — and you had to **guess** $\phi$.

---

A kernel buys you the **lifted feature space** without ever **computing the lift**.

One kernel evaluation per pair of points — that is the entire price.

---

## Common kernels

<p class="kicker">Reference — screenshot this</p>

| Kernel | $k(x, x')$ | What it buys |
|--------|------------|--------------|
| **Linear** | $x^\top x'$ | a straight boundary — like a regularised logistic |
| **Polynomial** | $(\gamma x^\top x' + r)^d$ | curved boundaries of degree $d$ |
| **RBF (Gaussian)** | $\exp(-\gamma \lVert x - x' \rVert^2)$ | flexible, local boundaries |

<p class="muted">Default: RBF. When p ≫ n, or for text: linear.</p>

---

## Three kernels, one dataset

![Linear, polynomial and RBF boundaries on concentric rings](../figures/svm-kernel-boundaries.svg)

Linear **fails** on rings; polynomial and RBF lift their way out.

---

## gamma — the kernel's reach

![RBF boundaries for gamma = 0.01, 1 and 100](../figures/svm-gamma-effect.svg)

Small gamma → global and smooth. Large gamma → **one bump per training point**.

---

## SVMs in practice

---

## The recipe

<p class="kicker">The pattern · L7 machinery, new model</p>

1. **Split** off the test set
2. **Scale** — mandatory (Pitfall 1)
3. **Pipeline** — `StandardScaler` + `SVC`
4. **Tune** `C` × `gamma` on a 2D log grid
5. **Test** — once

---

## SVC in code

```python
from sklearn.svm import SVC
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

pipe = make_pipeline(
    StandardScaler(),
    SVC(kernel='rbf', C=1.0, gamma='scale'),
)
pipe.fit(X_train, y_train)
```

<p class="muted"><code>gamma='scale'</code> = 1 / (p · Var(X)) — a genuinely sensible default.</p>

---

## Pitfall 1 — Skipping the scaler

<p class="kicker">Pitfall 1 · Kernels live on distances</p>

<div class="cols">
<div class="callout">
<h3>Scaled</h3>
<p>Every feature votes in the kernel.</p>
</div>
<div class="callout warn">
<h3>Raw units</h3>
<p>One wide-range feature owns every distance.</p>
</div>
</div>

Same lesson as k-NN, ridge, PCA — here it is **non-negotiable**.

---

## Pitfall 2 — probability=True

<p class="kicker">Pitfall 2 · SVC has no native probabilities</p>

- `decision_function` gives a **signed distance** — use it for rankings
- `probability=True` bolts on Platt scaling via internal CV — **slow**, often miscalibrated
- **Deprecated in scikit-learn 1.9** — use `CalibratedClassifierCV(SVC(), ensemble=False)`

---

## Pitfall 3 — Tuning C with gamma frozen

<p class="kicker">Pitfall 3 · C and gamma move together</p>

```python
param_grid = {
    'svc__C':     [0.1, 1, 10, 100],
    'svc__gamma': [0.001, 0.01, 0.1, 1],
}
```

Always a **2D grid**, both on a **log scale** — and read the **heatmap**, not just `best_params_`.

---

## Where SVMs stop scaling

<div class="cols">
<div class="callout">
<h3>SVC</h3>
<p>Any kernel — training O(n²) to O(n³). Fine to ~10⁴ rows.</p>
</div>
<div class="callout">
<h3>LinearSVC</h3>
<p>Linear kernel only, different solver — scales far further. First choice for text.</p>
</div>
</div>

<p class="muted">Memory grows with the support-vector count — a large fraction of n on noisy data.</p>

---

## When an SVM is the right call

<ul class="checklist">
<li><strong>Moderate</strong> n — hundreds to tens of thousands of rows</li>
<li><strong>High-dimensional</strong> p relative to n</li>
<li>Text / sparse features — <code>LinearSVC</code></li>
<li class="no">Millions of rows</li>
<li class="no">Calibrated probabilities out of the box</li>
</ul>

---

## Worked example — three kernels vs the forest

---

## The protocol

<p class="kicker">load_breast_cancer() · 80/20 stratified · random_state=42 · 5-fold CV · f1</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">569</span>
<span class="stat-label">rows</span>
</div>
<div class="stat stat--green">
<span class="stat-value">30</span>
<span class="stat-label">numeric features</span>
</div>
<div class="stat">
<span class="stat-value">4</span>
<span class="stat-label">workflows · 3 kernels + forest</span>
</div>
</div>

Same dataset and protocol as L10 — the forest's numbers carry over as the **incumbent**.

---

## Workflow 1 — linear SVM

<p class="kicker">Workflow 1 · One knob</p>

```python
pipe_lin = make_pipeline(StandardScaler(), SVC(kernel='linear'))
grid_lin = {'svc__C': [0.01, 0.1, 1, 10, 100]}

search_lin = GridSearchCV(pipe_lin, grid_lin, cv=cv, scoring='f1')
search_lin.fit(X_train, y_train)
```

Best C = 0.1 — CV f1 **0.981 ± 0.008**. The bar is set high.

---

## Workflow 2 — RBF SVM

<p class="kicker">Workflow 2 · C × gamma, 16 cells</p>

```python
pipe_rbf = make_pipeline(StandardScaler(), SVC(kernel='rbf'))
grid_rbf = {
    'svc__C':     [0.1, 1, 10, 100],
    'svc__gamma': [0.001, 0.01, 0.1, 1],
}

search_rbf = GridSearchCV(pipe_rbf, grid_rbf, cv=cv, scoring='f1')
search_rbf.fit(X_train, y_train)
```

Best C = 10, gamma = 0.01 — CV f1 **0.981 ± 0.012**. **Identical** to linear.

---

## Read the heatmap

![CV f1 over the C × gamma grid](../figures/svm-cv-heatmap.svg)

A **plateau** at small gamma — and a **cliff** at gamma = 1: every C loses ~0.2.

---

## Workflow 3 — polynomial SVM

<p class="kicker">Workflow 3 · Degree 3</p>

```python
pipe_poly = make_pipeline(StandardScaler(),
                          SVC(kernel='poly', degree=3))
grid_poly = {'svc__C': [0.1, 1, 10], 'svc__coef0': [0, 1]}

search_poly = GridSearchCV(pipe_poly, grid_poly, cv=cv, scoring='f1')
search_poly.fit(X_train, y_train)
```

Best C = 1, coef0 = 1 — CV f1 **0.979 ± 0.013**. The coef0 = 0 cells sink to 0.886.

---

## Workflow 4 — the incumbent forest

<p class="kicker">Workflow 4 · L10's champion, unchanged</p>

```python
rf = RandomForestClassifier(n_estimators=300, max_features='sqrt',
                            n_jobs=-1, random_state=42)
scores = cross_val_score(rf, X_train, y_train, cv=cv, scoring='f1')
```

CV f1 **0.972 ± 0.014** — exactly L10's number. No scaler, no kernel, no grid.

---

## Read the numbers

<p class="kicker">Test f1 · CV mean ± std underneath</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">0.986</span>
<span class="stat-label">linear SVM<br>CV 0.981 ± 0.008</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.986</span>
<span class="stat-label">RBF SVM<br>CV 0.981 ± 0.012</span>
</div>
<div class="stat stat--yellow">
<span class="stat-value">0.979</span>
<span class="stat-label">poly SVM<br>CV 0.979 ± 0.013</span>
</div>
</div>

<p class="muted">L10's random forest: test <strong>0.958</strong> · CV 0.972 ± 0.014 — behind by <strong>less than one fold-std</strong> at CV time.</p>

---

## What actually moved the score

<p class="kicker">CV f1 gaps · same grid, three knobs</p>

<div class="cols-3">
<div class="stat stat--green">
<span class="stat-value">0.000</span>
<span class="stat-label">best linear vs best RBF —<br>kernels tie</span>
</div>
<div class="stat stat--yellow">
<span class="stat-value">0.022</span>
<span class="stat-label">C range<br>at the best gamma</span>
</div>
<div class="stat stat--red">
<span class="stat-value">0.21</span>
<span class="stat-label">the gamma = 1 cliff —<br>at every C</span>
</div>
</div>

The dangerous knob isn't the kernel menu — it's an **untuned gamma**.

---

## Which one would you ship?

<ul class="checklist">
<li>The <strong>linear SVM</strong> — ties everything, <strong>one knob</strong>, 51 support vectors, easiest to defend</li>
<li class="no">The poly — same score, the most parameters to justify</li>
<li class="no">A "SVMs beat forests" headline — one dataset, and the CV gap is <strong>inside one std</strong></li>
<li class="no">Touching the test set again — it was spent <strong>once</strong></li>
</ul>

---

## Wrap-up

---

The model is not the data — it is the boundary plus the **few points** that pin it.

Margin for safety, kernel for shape, C and gamma on the dial.

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Margin</span>
<span class="stat-label">the widest safety strip,<br>pinned by support vectors</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">Kernel</span>
<span class="stat-label">the lifted space,<br>never the lift</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Grid</span>
<span class="stat-label">C × gamma together,<br>log-scaled — read the heatmap</span>
</div>
</div>

---

## Before Lecture 12

- **Run** today's four-workflow comparison on your machine
- Project: add an SVM under your **L7 protocol** — `LinearSVC` if text or high-dimensional
- Read ahead: **L12 is gradient boosting**

</div>

<p class="muted">L12 returns to trees: boosting swaps averaging for <strong>sequential correction</strong>.</p>

---

## Questions

Backup slides below — press down ↓

## One-class SVM

<p class="kicker">Novelty detection · OneClassSVM</p>

```python
from sklearn.svm import OneClassSVM

oc = OneClassSVM(kernel='rbf', gamma='scale', nu=0.05)
oc.fit(X_normal)
labels = oc.predict(X_new)   # +1 = normal, -1 = anomaly
```

Fit a boundary around the **normal** class; flag what falls outside.

## NuSVC

<p class="kicker">Same model, different knob · nu</p>

```python
from sklearn.svm import NuSVC

clf = NuSVC(nu=0.3, kernel='rbf', gamma='scale')
```

- `nu` bounds the **violation fraction** from above
- …and the **support-vector fraction** from below

## Support vector regression

<p class="kicker">Regression · epsilon-insensitive loss</p>

```python
from sklearn.svm import SVR

pipe = make_pipeline(StandardScaler(),
                     SVR(kernel='rbf', C=1.0, epsilon=0.1))
```

Errors within $\varepsilon$ cost nothing; outside, they grow linearly.

## Kernel approximation

<p class="kicker">Escape hatch · Nystroem / RBFSampler</p>

```python
from sklearn.kernel_approximation import RBFSampler
from sklearn.svm import LinearSVC

pipe = make_pipeline(
    StandardScaler(),
    RBFSampler(gamma=0.1, n_components=500, random_state=0),
    LinearSVC(C=1.0, max_iter=5000),
)
```

Approximate the RBF features **explicitly** → linear-model cost, SVM-like shape.

## Why SVM probabilities are weird

<p class="kicker">Probabilities · Platt scaling under the hood</p>

1. Cross-validated decision-function scores
2. A logistic regression maps scores → probabilities
3. Refit on the full data with the scaling locked in

Often **miscalibrated** for imbalanced data — and `probability=True` is deprecated in scikit-learn 1.9.

## Multi-class SVMs

<p class="kicker">Multi-class · one-vs-one under the hood</p>

- `SVC` trains one binary model per **pair** of classes — $K(K-1)/2$ of them — and votes
- `LinearSVC` uses **one-vs-rest** instead — faster, linear only

---

## What's next

**Lecture 12:** Gradient boosting

- Boosting — learn sequentially from mistakes
- Gradient boosting machines
- XGBoost and LightGBM