Classification Methods

<p class="kicker">MST0052 · Lecture 5 · Fall 2026</p>

## Classification<br>Methods

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L4 predicted **numbers**. Today we predict **categories**.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

The hard part of classification isn't the classifier — it's **proving it works**.

Three baseline methods today. One evaluation discipline.

---

## Today's plan

<p class="kicker">90 minutes · three classifiers · one scoreboard</p>

- What changes when the target is a **category**
- **Logistic regression** · **k-NN** · **naive Bayes**
- The scoreboard: **confusion matrix, precision, recall, F1**
- **Worked example:** breast cancer diagnosis

---

## The classification problem

---

## Regression vs classification

<div class="cols">
<div class="callout">
<h3>Regression</h3>
<p>A continuous number — price, temperature.</p>
</div>
<div class="callout">
<h3>Classification</h3>
<p>A category — churn, spam, malignant.</p>
</div>
</div>

Same **L3 pipeline** — different model, different **metrics**.

Binary today — multi-class is in the backups.

---

## Labels vs probabilities

<div class="cols">
<div class="callout">
<h3>Label</h3>
<p>"This customer will churn."</p>
</div>
<div class="callout">
<h3>Probability</h3>
<p>"This customer has a 73% chance of churning."</p>
</div>
</div>

The probability is the **richer output** — you choose the threshold.

---

## The free baseline

<div class="cols">
<div class="stat stat--red">
<span class="stat-value">70%</span>
<span class="stat-label">always predict "no churn"<br>— zero effort</span>
</div>
<div class="stat stat--green">
<span class="stat-value">?</span>
<span class="stat-label">your model<br>— must beat this</span>
</div>
</div>

Every classifier must beat the **majority class.**

---

## Logistic regression

---

## The model, in one equation

$$\Pr(y = 1 \mid x) = \frac{1}{1 + e^{-(\beta\_0 + x^\top\beta)}}$$

The L4 **weighted sum**, squeezed into [0, 1].

Coefficients live on the log-odds scale — sign and size still read like L4.

---

## The sigmoid

![Sigmoid curve mapping linear scores to probabilities](../figures/sigmoid.svg)

Score 0 → probability 0.5 — the default **threshold**.

---

## Logistic regression in scikit-learn

<p class="kicker">Baseline · same pipeline as L4</p>

```python
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

pipe = make_pipeline(StandardScaler(), LogisticRegression())
pipe.fit(X_train, y_train)

y_pred = pipe.predict(X_test)
y_prob = pipe.predict_proba(X_test)[:, 1]
```

<p class="muted"><code>predict_proba</code> is the output you evaluate with.</p>

---

## When to use logistic regression

<ul class="checklist">
<li><strong>First model</strong> for any binary problem</li>
<li>Transparent, fast, <strong>probabilistic</strong></li>
<li class="no">Strongly <strong>nonlinear</strong> boundaries</li>
</ul>

Always try it first — even if you plan something fancier.

---

## Two more baselines — k-NN & naive Bayes

---

## k-NN — let the neighbours vote

Find the $k$ closest training points. **Majority wins.** No training phase.

<div class="cols">
<div class="callout">
<h3>k = 1</h3>
<p>Memorises — follows noise. High variance.</p>
</div>
<div class="callout">
<h3>k = 50</h3>
<p>Blurs — misses local patterns. High bias.</p>
</div>
</div>

Choose k by cross-validation. (This is L6's bias-variance tradeoff, in one knob.)

---

## k-NN in scikit-learn

<p class="kicker">Tune k · same GridSearchCV as L4</p>

```python
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import GridSearchCV

pipe = make_pipeline(StandardScaler(), KNeighborsClassifier())
param_grid = {'kneighborsclassifier__n_neighbors': [3, 5, 11, 21]}
search = GridSearchCV(pipe, param_grid, cv=5, scoring='f1')
search.fit(X_train, y_train)

print(f"Best k: {search.best_params_}")
```

<p class="muted"><strong>Distances</strong> dominate by scale — <code>StandardScaler</code> is mandatory.</p>

---

## When to use k-NN

- Small datasets with clear cluster structure
- No assumptions about the boundary shape
- High-dimensional data — curse of dimensionality 
- Large datasets — slow at prediction time

</div>

Best role: a **non-parametric sanity check.**

---

## Naive Bayes — the formula

$$P(c \mid x) \propto P(c) \prod\_{j=1}^{p} P(x\_j \mid c)$$

Given the class, each feature **tells its own story** — independently.

"Naive" = the independence assumption. Almost never true; often works anyway.

---

## Naive Bayes in scikit-learn

<p class="kicker">Zero knobs · zero scaling</p>

```python
from sklearn.naive_bayes import GaussianNB

model = GaussianNB()
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_prob = model.predict_proba(X_test)[:, 1]
```

The fastest baseline you can get.

<p class="muted">Text data? <code>MultinomialNB</code> — see the notes.</p>

---

## When to use naive Bayes

- An **instant baseline** for any problem
- **Text** classification — spam, sentiment, topic
- Trusting its raw probabilities — overconfident

</div>

---

## Three shapes, one dataset

![Decision boundaries: logistic regression, k-NN, naive Bayes](../figures/decision-boundary.svg)

Same data, three ways to carve the space — so **which wins?**

---

## The scoreboard — evaluating classifiers

---

**95% accurate** — and it misses every sick patient.

Accuracy collapses when classes are imbalanced.

---

## The confusion matrix

<p class="kicker">Rows = truth · Columns = prediction</p>

<div class="cols">
<div class="callout">
<h3>True positive</h3>
<p>Sick, flagged sick — caught it.</p>
</div>
<div class="callout warn">
<h3>False negative</h3>
<p>Sick, cleared — <strong>the dangerous cell.</strong></p>
</div>
</div>
<div class="cols">
<div class="callout">
<h3>False positive</h3>
<p>Healthy, flagged — a false alarm.</p>
</div>
<div class="callout">
<h3>True negative</h3>
<p>Healthy, cleared — correctly.</p>
</div>
</div>

Every metric is a function of these **four numbers.**

<p class="muted">scikit-learn prints negatives first: <code>[[TN, FP], [FN, TP]]</code>.</p>

---

## Precision and recall

$$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}} \qquad \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$$

Of my alarms, how many were real — of the real ones, how many did I **catch**?

Raise the threshold: recall ↓, precision usually ↑.

---

## F1

$$F\_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$$

**Course default** — unless your problem clearly favours one side.

---

## ROC and AUC

<div class="cols">
<div class="stat stat--green">
<span class="stat-value">1.0</span>
<span class="stat-label">perfect ranking</span>
</div>
<div class="stat stat--red">
<span class="stat-value">0.5</span>
<span class="stat-label">a coin flip</span>
</div>
</div>

AUC: can the model **rank** positives above negatives — across every threshold?

---

## classification_report

```python
from sklearn.metrics import classification_report, confusion_matrix

y_pred = pipe.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
```

Precision, recall, F1 **per class** — print the matrix alongside, always.

---

## Choosing your metric

<p class="kicker">Reference — screenshot this</p>

| Situation | Metric |
|-----------|--------|
| Balanced classes | Accuracy or F1 |
| Imbalanced classes | F1, precision-recall, AUC |
| Cost asymmetry — missing fraud is worse | Favour recall |
| Ranking matters — prioritise likely churners | AUC |

**State your choice. Justify it in the report.**

---

## Worked example — breast cancer

---

## The brief

<p class="kicker">Predict malignant vs benign · binary</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">569</span>
<span class="stat-label">biopsies</span>
</div>
<div class="stat">
<span class="stat-value">30</span>
<span class="stat-label">numeric features</span>
</div>
<div class="stat stat--red">
<span class="stat-value">63%</span>
<span class="stat-label">benign — the free baseline</span>
</div>
</div>

63% accuracy costs nothing — and scores **F1 = 0**, because it never flags a tumour.

---

## Step 1 — Load and split

<p class="kicker">Step 1 · Load & split</p>

```python
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
y = 1 - y  # sklearn codes 1 = benign; flip so 1 = malignant

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
```

<p class="muted">Positive class = the one we must catch. <code>stratify=y</code> keeps the balance in both splits.</p>

---

## Step 2 — Three pipelines

<p class="kicker">Step 2 · Three pipelines</p>

```python
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

models = {
    'Logistic': make_pipeline(StandardScaler(),
                              LogisticRegression(max_iter=5000)),
    'k-NN':     make_pipeline(StandardScaler(),
                              KNeighborsClassifier(n_neighbors=5)),
    'NaiveBayes': GaussianNB(),
}
```

Same data, same splits, same metric — a **fair fight.**

---

## Step 3 — Cross-validate

<p class="kicker">Step 3 · Cross-validate</p>

```python
from sklearn.model_selection import cross_val_score

for name, model in models.items():
    scores = cross_val_score(model, X_train, y_train,
                             cv=5, scoring='f1')
    print(f"{name}: F1 = {scores.mean():.3f} ± {scores.std():.3f}")
```

---

## Step 4 — Compare

<p class="kicker">Step 4 · Compare</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">0.961</span>
<span class="stat-label">Logistic</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.948</span>
<span class="stat-label">k-NN</span>
</div>
<div class="stat">
<span class="stat-value">0.921</span>
<span class="stat-label">Naive Bayes</span>
</div>
</div>

Close — but **logistic leads**, and it's also the most readable.

---

## Step 5 — Inspect

<p class="kicker">Step 5 · The confusion matrix, for real</p>

```python
from sklearn.metrics import classification_report, confusion_matrix

best = models['Logistic']
best.fit(X_train, y_train)
y_pred = best.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
```

How many **malignant cases did we miss**?

---

The winner depends on **your data** — that's why you compare.

Start with logistic. Add challengers. Report honestly.

---

## Wrap-up

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Probabilities</span>
<span class="stat-label">demand predict_proba,<br>not just labels</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">Confusion</span>
<span class="stat-label">four numbers before<br>any metric</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Compare</span>
<span class="stat-label">same splits, same metric,<br>beat the baseline</span>
</div>
</div>

---

## What to report

- Your **metric** — and why it fits the problem
- The **confusion matrix**
- **Baseline** (majority class) vs your model
- The same **resampling rule** for every model

</div>

---

## Before Lecture 6

- Try all three classifiers on **your own dataset**
- Regression project? Use the **L4 pipeline** instead
- Read ahead: L6 is **bias-variance** — the most important idea in the course

</div>

All the code is on the course site.

---

## Questions

Backup slides below — press down ↓

## Class imbalance

<p class="kicker">Start with one parameter</p>

```python
LogisticRegression(class_weight='balanced')
```

One parameter change — before any resampling.

## The precision-recall curve

<p class="kicker">Heavy imbalance · alternative to ROC</p>

```python
from sklearn.metrics import PrecisionRecallDisplay
PrecisionRecallDisplay.from_estimator(pipe, X_test, y_test)
```

More informative than ROC when the positive class is **rare**.

## Multi-class classification

<p class="kicker">Three or more classes · automatic</p>

- scikit-learn handles multi-class **automatically**
- Logistic regression fits all classes **jointly** (multinomial); some models, like SVMs, combine binary ones
- `classification_report` shows metrics **per class**
- CV scoring: `f1_macro` or `f1_weighted`

## Probability calibration

<p class="kicker">When the probability itself matters</p>

```python
from sklearn.calibration import CalibratedClassifierCV

calibrated = CalibratedClassifierCV(GaussianNB(),
                                    method='isotonic', cv=5)
calibrated.fit(X_train, y_train)
y_prob = calibrated.predict_proba(X_test)[:, 1]
```

Does a predicted 0.7 actually mean **70%**?

## The decision threshold

<p class="kicker">0.5 is a default, not a law</p>

```python
y_prob = pipe.predict_proba(X_test)[:, 1]
threshold = 0.3  # lower threshold = more positives
y_pred_custom = (y_prob >= threshold).astype(int)
```

Set it by the **cost of errors** in your problem.

---

## What's next

**Lecture 6:** Bias-variance tradeoff

- Why training error is not enough
- The U-shaped test error curve
- How model complexity relates to generalisation