PCA as an Eigenvalue Problem

<p class="kicker">MST0052 · Lecture 9 supplement · Optional</p>

## PCA as an<br>Eigenvalue Problem

### Vegard H. Larsen — BI Norwegian Business School

---

## What this deck is

- **Optional** — nothing here is required for the project
- The **solution** of L9's one-line optimisation problem
- Why explained-variance ratios are **eigenvalue shares**
- The **link** between PCA, ridge, and the L4/L6 supplements
- New required material

</div>

---

## The objective

---

## Variance of a projection

Center $X$, pick a direction $w$. The projected data is $Xw$, and

$$\mathrm{Var}(Xw) = \frac{1}{n-1} (Xw)^\top (Xw) = w^\top \Sigma w \qquad \Sigma = \frac{X^\top X}{n-1}$$

One matrix, $\Sigma$, holds **every direction's variance** at once.

---

## L9's one-line problem, properly stated

$$\max\_{w}\ w^\top \Sigma w \qquad \text{s.t.} \quad \lVert w \rVert = 1$$

- Without the constraint, doubling $w$ quadruples the "variance" — **scale must be pinned**
- This is the main deck's "maximise the variance of the projection", in symbols

---

## The solution is an eigenvector

---

## One Lagrange multiplier

$$L(w, \lambda) = w^\top \Sigma w - \lambda \left( w^\top w - 1 \right)$$

$$\frac{\partial L}{\partial w} = 2\Sigma w - 2\lambda w = 0 \quad\Longrightarrow\quad \Sigma w = \lambda w$$

The candidates are exactly the **eigenvectors** of $\Sigma$.

---

## Which eigenvector? Read the variance

At any candidate, plug $\Sigma w = \lambda w$ back into the objective:

$$w^\top \Sigma w = w^\top (\lambda w) = \lambda \lVert w \rVert^2 = \lambda$$

- The variance along an eigenvector **is its eigenvalue**
- So the first component is the eigenvector with the **largest** $\lambda$
- Next components: same problem, **orthogonal** to those already chosen → eigenvectors in order

---

## The picture

![A correlated point cloud with both eigenvectors drawn, scaled by their eigenvalues](../figures/pca-eigen-geometry.svg)

---

## Explained variance ratio, demystified

Total variance is the **trace**: $\sum\_j \Sigma\_{jj} = \sum\_j \lambda\_j$. So component $j$ explains

$$\frac{\lambda\_j}{\sum\_k \lambda\_k}$$

The main deck's **cumsum workflow** is a running sum of eigenvalue shares.

---

## What sklearn actually computes

---

## The SVD view

Factor the centered data directly: $X = U S V^\top$. Then

$$\Sigma = \frac{X^\top X}{n-1} = V \frac{S^2}{n-1} V^\top$$

- Columns of $V$ = the **components** · $\lambda\_j = s\_j^2 / (n-1)$
- sklearn's `PCA` runs an **SVD of $X$** — $\Sigma$ is never formed

<p class="muted">Same answer, better numerics — the same story as L4's closed form vs sklearn's solver.</p>

---

## See it in code

```python
cov = X_s.T @ X_s / (n - 1)
eigvals, eigvecs = np.linalg.eigh(cov)        # the long way
pca = PCA(n_components=10).fit(X_s)           # the sklearn way

np.abs(pca.explained_variance_ - eigvals[:10]).max()   # 0.0
(X_s @ eigvecs[:, 0]).var(ddof=1)             # 7.3448
pca.explained_variance_[0]                    # 7.3448  -- lambda_1
```

L9's digits PCA, recomputed by **eigendecomposition** — identical to machine precision.

<p class="muted">Pre-tested: <code>scripts/examples/lecture_09_pca_supplement.py</code> — also checks the directions match up to sign.</p>

---

## One matrix, three lectures

---

## PCA and ridge share their geometry

$X^\top X$ keeps appearing — PCA **diagonalises** the matrix that L4 **inverted**:

- **L4:** OLS needs $(X^\top X)^{-1}$ — near-zero eigenvalues blow it up
- **Ridge:** $+\lambda I$ lifts every eigenvalue — shrinking **low-variance directions hardest**
- **L9:** those low-variance directions are exactly the **trailing principal components**

<p class="muted">Ridge quietly trusts the leading principal components — the L4 and L6 supplements, closed.</p>

---

## Two pitfalls, now mechanical

<div class="cols">
<div class="callout">
<h3>Variance ≠ signal</h3>
<p>The objective never mentions y. Nothing stops the predictive direction from being component 40.</p>
</div>
<div class="callout">
<h3>Scaling is mandatory</h3>
<p>Variance has units. One wide-range feature owns the top eigenvector of an unscaled Σ.</p>
</div>
</div>

The main deck's Pitfalls 3 and 1 — visible **in the formulas**.

---

## Questions

Backup slides below ↓

## Backup — the other definition of PCA

PCA equivalently **minimises reconstruction error**:

$$\min\_{W}\ \lVert X - X W W^\top \rVert^2 \qquad W^\top W = I\_k$$

- Keep $k$ directions, project, map back — the loss is the **discarded eigenvalues** $\sum\_{j>k} \lambda\_j$
- Max-variance and min-error give the **same** $W$

## Backup — why components are orthogonal

- $\Sigma$ is **symmetric** — the spectral theorem applies
- Symmetric matrices have a full set of **orthonormal eigenvectors** with real eigenvalues
- So "uncorrelated components" is not an extra constraint — it comes **free** with the matrix

---

## What's next

Back to the **main track** — the supplement story continues:

- **L10 supplement:** the variance of an *average* — why forests work
- The eigenvalues of $X^\top X$ return whenever a model **inverts, shrinks, or rotates**