SVMs — the Geometry and the Dual

MST0052 · Lecture 11 supplement · Optional

## SVMs — the Geometry and the Dual

### Vegard H. Larsen — BI Norwegian Business School

---

## What this deck is

- **Optional** — nothing here is required for the project
- Where the **½‖w‖² objective** comes from
- **Proofs** of the two headline claims: support vectors pin the model, kernels need only inner products
- New required material

</div>

---

01

## The margin, measured

---

## Distance to a hyperplane

The boundary is $w^\top x + b = 0$, and $w$ is its **normal vector**:

$$\text{dist}(x\_i) = \frac{|w^\top x\_i + b|}{\lVert w \rVert}$$

Larger $|w^\top x\_i + b|$ → further from the boundary, **in units of** $\lVert w \rVert$.

---

## The picture

![The separating hyperplane, margin lines, normal vector and margin width](../figures/svm-margin-geometry.svg)

---

## A normalisation, then the objective

The scale of $(w, b)$ is **free** — fix it so the closest points satisfy

$$y\_i (w^\top x\_i + b) = 1$$

Then each side's margin is $\frac{1}{\lVert w \rVert}$, total width $\frac{2}{\lVert w \rVert}$ — and

$$\max \frac{2}{\lVert w \rVert} \iff \min \tfrac{1}{2} \lVert w \rVert^2 \quad \text{s.t.} \quad y\_i(w^\top x\_i + b) \geq 1$$

---

02

## Why only support vectors matter

---

## Constrained optimisation → Lagrangian

One multiplier $\alpha\_i \geq 0$ per training point:

$$L(w, b, \alpha) = \tfrac{1}{2}\lVert w \rVert^2 - \sum\_i \alpha\_i \left[ y\_i (w^\top x\_i + b) - 1 \right]$$

Each $\alpha\_i$ prices its constraint: **how much point $i$ pushes on the boundary**.

---

## Setting derivatives to zero

$$\frac{\partial L}{\partial w} = 0 \Longrightarrow w = \sum\_i \alpha\_i y\_i x\_i \qquad \frac{\partial L}{\partial b} = 0 \Longrightarrow \sum\_i \alpha\_i y\_i = 0$$

The boundary's normal is a **weighted sum of training points**.

---

## Complementary slackness

At the optimum, for every point:

$$\alpha\_i \left[ y\_i (w^\top x\_i + b) - 1 \right] = 0$$

- Point **off the margin** → bracket ≠ 0 → $\alpha\_i = 0$ → contributes **nothing** to $w$
- $\alpha\_i > 0$ only where $y\_i(w^\top x\_i + b) = 1$ — **on the margin**

"Delete them — nothing moves" — the main deck's claim, proved.

---

## The dual problem

Substitute $w = \sum\_i \alpha\_i y\_i x\_i$ back in:

$$\max\_{\alpha} \sum\_i \alpha\_i - \tfrac{1}{2} \sum\_i \sum\_j \alpha\_i \alpha\_j y\_i y\_j \left( x\_i^\top x\_j \right)$$

The data enters **only through inner products** $x\_i^\top x\_j$.

---

03

## Kernels, made honest

---

## Swap the inner product

Replace every $x\_i^\top x\_j$ with $k(x\_i, x\_j) = \phi(x\_i)^\top \phi(x\_j)$. Prediction too:

$$f(x) = \sum\_{i \in \text{SV}} \alpha\_i y\_i k(x\_i, x) + b$$

- Train and predict in the lifted space — $\phi$ is **never computed**
- The sum runs over **support vectors only**

---

## The rings example, verified

The main deck's lift: $\phi(x) = (x\_1^2,\ x\_2^2,\ \sqrt{2} x\_1 x\_2)$. Expand the kernel:

$$\left( x^\top z \right)^2 = (x\_1 z\_1 + x\_2 z\_2)^2 = x\_1^2 z\_1^2 + x\_2^2 z\_2^2 + 2 x\_1 z\_1 x\_2 z\_2 = \phi(x)^\top \phi(z)$$

One multiplication squared **equals** the 3-D inner product — that's the whole trick.

---

## The RBF kernel lifts infinitely

$$\exp\left( -\gamma \lVert x - z \rVert^2 \right) = \phi(x)^\top \phi(z) \quad \text{for an } \infty\text{-dimensional } \phi$$

- Expand the exponential as a **power series** → every polynomial degree at once
- Explicitly computing $\phi$: impossible. The kernel value: **one line**

Why "even infinite-dimensional spaces become affordable" — the main deck's pivot, literally.

---

04

## The SVM is a cousin of logistic regression

---

## Slack is a loss function

Optimal slack is $\xi\_i = \max\left(0,\ 1 - y\_i(w^\top x\_i + b)\right)$ — so soft margin is:

$$\min\_{w, b}\ \sum\_i \max\left(0,\ 1 - y\_i(w^\top x\_i + b)\right) + \frac{1}{2C} \lVert w \rVert^2$$

**Loss + penalty** — the same template as ridge (L4) and logistic regression (L5).

---

## Three losses, one picture

![Zero-one, hinge and logistic loss against the signed margin](../figures/hinge-loss.svg)

Both are **convex stand-ins** for the 0-1 loss — hinge goes **exactly flat** past margin 1.

---

## See it in code

```python
svm = SVC(kernel="linear", C=0.1).fit(X_s, y_train)
sv = svm.support_                       # 51 of 455 training rows

svm_sv_only = SVC(kernel="linear", C=0.1).fit(X_s[sv], y_train[sv])

np.abs(svm.coef_ - svm_sv_only.coef_).max()   # 4.6e-04 — solver noise
(svm.predict(X_test_s) == svm_sv_only.predict(X_test_s)).all()  # True
```

Delete the other **404 points**, refit — **identical predictions**.

Pre-tested: <code>scripts/examples/lecture_11_svm_supplement.py</code> — also verifies the rings kernel identity numerically.

---

## What this buys you at the oral exam

- **Why maximum margin** — distance is $\frac{|w^\top x + b|}{\lVert w \rVert}$, normalise, minimise ½‖w‖²
- **Why support vectors** — complementary slackness zeroes everyone else
- **Why kernels work** — the dual only sees inner products
- **How SVMs relate to L5** — hinge vs logistic loss, same penalty

---

## Questions

Backup slides below ↓

## Backup — the soft-margin dual

Slack changes exactly **one thing** in the dual:

$$0 \leq \alpha\_i \leq C$$

- The box **caps any single point's force** on the boundary — even an outlier's
- Margin violators sit at $\alpha\_i = C$; margin points in between; the rest at 0

## Backup — what makes a valid kernel

A function $k$ is a kernel **iff** every Gram matrix $K\_{ij} = k(x\_i, x\_j)$ is **positive semi-definite** (Mercer).

- Sums and products of kernels are kernels — build your own
- The `sigmoid` kernel violates this for some parameters — one reason it's rarely used

---

## What's next

Back to the **main track** — and the family picture so far:

- **L4/L5:** loss + penalty, fitted by formula or descent
- **L11:** same template, a kink in the loss → sparsity → support vectors
- **L12:** boosting — the loss's *gradient* becomes the next tree's target