Neural Networks – MST0052

<p class="kicker">MST0052 · Lecture 14 · Fall 2026</p>

## Neural Networks<br>A Bridge from Classical ML

### Vegard H. Larsen — BI Norwegian Business School

---

## Where we are

L13 **closed the toolbox** — today is the only lecture that adds **no new tool**: a bridge to deep learning, and the case for the project rule.

![Semester timeline with four phases](../figures/semester-timeline.svg)

---

Deep learning is not a different species. It is **logistic regression, stacked**.

Today: the bridge across — and an honest map of where classical ML still wins.

---

## Today's plan

<p class="kicker">90 minutes · no new tool · vocabulary + judgment</p>

- Why this lecture exists — and why the **project ban** stays
- **Logistic regression → neuron → layer → network**
- How networks **learn** — gradient descent and backprop, in plain English
- When **classical ML still wins** — with receipts

---

## A bridge, not a recipe

---

## What today is — and isn't

<ul class="checklist">
<li>Place neural networks in your <strong>mental model</strong> — what they generalise, cost, and buy</li>
<li>Leave able to <strong>read</strong> a deep-learning paper</li>
<li class="no">Train ImageNet or write PyTorch</li>
<li class="no">Derive backprop</li>
</ul>

---

## The project ban, revisited

<div class="cols">
<div class="callout">
<h3>This course</h3>
<p>Validation, leakage, comparison, interpretation — defensible at the oral.</p>
</div>
<div class="callout warn">
<h3>Deep learning</h3>
<p>Often hides exactly those decisions behind library APIs.</p>
</div>
</div>

A **positive design choice** — take the deep-learning course *afterwards*, with these prerequisites.

---

## What you leave with

<div class="cols-3">
<div class="callout">
<h3>A story</h3>
<p>What a network <em>is</em>, in terms you already own.</p>
</div>
<div class="callout">
<h3>A vocabulary</h3>
<p>Epoch, batch, backprop, attention — enough to read on.</p>
</div>
<div class="callout">
<h3>A position</h3>
<p>When classical wins — for the oral, and the next decade.</p>
</div>
</div>

---

## From logistic regression to a network

---

## Logistic regression, restated

From L5:

$$\Pr(y = 1 \mid x) = \sigma(\beta\_0 + x^\top \beta)$$

A **linear combination**, **squashed** through a nonlinearity, fitted by **likelihood**.

That is also a neural network. It has **one neuron**.

---

## A neuron is logistic regression renamed

<p class="kicker">Translation · same math, new words</p>

![Logistic regression equation next to the equivalent single-neuron diagram](../figures/neuron-vs-logistic.svg)

Same equation — the **vocabulary** changed, not the math.

---

## The dictionary

<p class="kicker">Translation · same math, new words</p>

<div class="cols">
<div class="callout">
<h3>Statistics — L4, L5</h3>
<p>β · β₀ · σ · log-likelihood · fitting</p>
</div>
<div class="callout">
<h3>Deep learning</h3>
<p>weights w · bias b · activation · cross-entropy loss · training</p>
</div>
</div>

Knowing the translation is **half the battle**.

---

## Neurons in parallel — a layer

$$\text{layer}(x) = \sigma(W x + b), \qquad W \in \mathbb{R}^{h \times p}$$

**h logistic regressions in parallel** — the output is a new, *learned* representation of the input.

---

## Layers stacked — a network

![A small feedforward network with two hidden layers](../figures/network-stacked-layers.svg)

Compose layers — without the nonlinearity between them, the stack **collapses to one linear map**.

---

## Activations — the shapes

![ReLU, sigmoid and tanh on one panel](../figures/activation-functions.svg)

**ReLU** is the default — cheap, and it doesn't saturate for positive inputs.

---

## Activations — reference

<p class="kicker">Reference — screenshot this</p>

| Function | Formula | Where used |
|----------|---------|------------|
| **ReLU** | $\max(0, z)$ | default for hidden layers |
| **Sigmoid** | $1 / (1 + e^{-z})$ | output — binary classification |
| **Softmax** | $e^{z\_c} / \sum\_j e^{z\_j}$ | output — multi-class |
| **Tanh** | $\tanh(z)$ | older; rarely used now |

---

## How networks learn

---

## The loss is an old friend

<p class="kicker">Translation · same math, new words</p>

<div class="cols">
<div class="callout">
<h3>Regression</h3>
<p>Squared error — L4's loss.</p>
</div>
<div class="callout">
<h3>Classification</h3>
<p>Cross-entropy — the log-loss from L5.</p>
</div>
</div>

Training = minimise average loss. Same idea as ridge — just **1,537 weights** instead of 31.

---

## Gradient descent

$$w \leftarrow w - \eta \cdot \nabla\_w L$$

- The gradient points **uphill** — step the other way
- Repeat, hundreds of thousands of times

$\eta$ is the **learning rate** — L12's word, same meaning.

---

## Backprop = the chain rule, automated

The **chain rule** from calculus, applied layer by layer — from the output back to the input.

</div>

PyTorch, TensorFlow and JAX compute it **automatically** — you never write it.

---

## Minibatches — noise as a feature

<div class="cols">
<div class="callout">
<h3>Full batch</h3>
<p>Exact gradient, slow steps.</p>
</div>
<div class="callout">
<h3>Minibatch — 32 to 256 rows</h3>
<p>Noisy gradient, many more steps — and the noise mildly regularises.</p>
</div>
</div>

---

## Inside .fit() — the training loop

```text
for each epoch:
    shuffle the training set
    for each minibatch:
        compute predictions
        compute loss
        compute gradient via backprop
        update weights with learning rate
    evaluate on validation set
    optionally: save best model, decay learning rate, stop early
```

Every framework's loop — `MLPClassifier` runs this when you call `.fit()`.

---

## What makes deep learning work

---

## Universal approximation, deflated

One hidden layer of sufficient width can approximate **any continuous function**.

"Sufficient width" can be **astronomical** — and the theorem says nothing about **finding** those weights.

</div>

The real reason deep networks work is **depth**.

---

## Depth = feature engineering, automated

<div class="cols-3">
<div class="callout">
<h3>Early layers</h3>
<p>Edges.</p>
</div>
<div class="callout">
<h3>Middle layers</h3>
<p>Motifs.</p>
</div>
<div class="callout">
<h3>Late layers</h3>
<p>Objects.</p>
</div>
</div>

You hand-built features in L3 — a deep net learns them **during training**. The price is data and compute.

---

## The data hunger

<div class="cols-3">
<div class="callout">
<h3>Thousands of rows</h3>
<p>Classical ML — structure is encoded in the model.</p>
</div>
<div class="callout">
<h3>Millions of rows</h3>
<p>Deep learning starts to win.</p>
</div>
<div class="callout">
<h3>Images, text, audio</h3>
<p>Deep learning — the structure is perceptual.</p>
</div>
</div>

---

## Architectures are priors

**Images**   `CNN` — translation invariance

**Sequences**   `RNN` `LSTM` — legacy

**Everything, 2026**   `Transformer` — attention

</div>

The right architecture **shrinks the data hunger**.

---

## The 2026 landscape

**Foundation models** — trained once at internet scale, then **fine-tuned** or **prompted** for specific tasks. LLMs are the visible example; the same recipe runs vision, audio, code and proteins.

</div>

The frontier is **using them well** — not training from scratch.

---

## When classical ML still wins

---

The win comes from the **data type and scale** — never from the algorithm in isolation.

Spreadsheet-shaped data: the toolbox you already have. Millions of images: a different tool.

---

## Tabular data — classical, full stop

<ul class="checklist">
<li>Gradient boosting is <strong>still state of the art</strong> on tabular benchmarks</li>
<li>At a <strong>fraction of the compute</strong></li>
<li>A few hundred–few thousand rows is <strong>not enough</strong> for from-scratch deep learning</li>
<li class="no">"Classical ML = what people did before deep learning"</li>
</ul>

---

## Models you can defend

<div class="cols-3">
<div class="callout">
<h3>Linear</h3>
<p>A coefficient table.</p>
</div>
<div class="callout">
<h3>Ensembles</h3>
<p>Permutation importance.</p>
</div>
<div class="callout warn">
<h3>Networks</h3>
<p>A black box, plus post-hoc tools of uncertain faithfulness.</p>
</div>
</div>

The oral probes classical methods **because there is something to defend**.

---

## The practical boundary

<p class="kicker">The heuristic · default — deviate only with a defence</p>

**If your data fits in a spreadsheet and your features are meaningful, start with gradient boosting.** If your data is images, text or sequences — consider a **pretrained model first**.

</div>

---

## Worked example — the protocol

<p class="kicker">load_breast_cancer() · 80/20 stratified · random_state=42 · 5-fold CV · f1</p>

<div class="cols-3">
<div class="stat stat--blue">
<span class="stat-value">455</span>
<span class="stat-label">training rows</span>
</div>
<div class="stat stat--green">
<span class="stat-value">114</span>
<span class="stat-label">test rows</span>
</div>
<div class="stat">
<span class="stat-value">30</span>
<span class="stat-label">features</span>
</div>
</div>

The **fourth family** onto the L10–L12 plate — same rows, same folds, same metric.

---

## Step 1 — The MLP under the L7 protocol

<p class="kicker">Step 1 · Two hidden layers, scaled, early stopping</p>

```python
mlp = make_pipeline(
    StandardScaler(),
    MLPClassifier(hidden_layer_sizes=(32, 16),
                  activation='relu', max_iter=300,
                  early_stopping=True, random_state=42),
)
scores = cross_val_score(mlp, X_train, y_train, cv=cv, scoring='f1')
```

CV f1 **0.964 ± 0.010** — test **0.933**, **below every incumbent**.

<p class="muted">Single fit: 0.02 s — speed was never the problem here.</p>

---

## Step 2 — Reseed it

<p class="kicker">Step 2 · Five seeds, same spec</p>

<div class="cols-3">
<div class="stat stat--red">
<span class="stat-value">0.039</span>
<span class="stat-label">MLP CV-mean spread<br>across 5 seeds</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.005</span>
<span class="stat-label">forest spread —<br>same seeds</span>
</div>
<div class="stat stat--yellow">
<span class="stat-value">0.076</span>
<span class="stat-label">MLP test-f1 spread<br>0.897–0.973</span>
</div>
</div>

**8× the forest's seed sensitivity** — "more sensitive to seed" is now a number.

---

## Step 3 — The flag that ate the data

<p class="kicker">Step 3 · early_stopping on 455 rows</p>

- `early_stopping=True` carves off **45 rows** as validation → stopped at iteration **25** of 300
- Remove it → converges at 295: CV **0.984 ± 0.007**, test **0.965**
- Seed spread falls from 0.039 to **0.009**

One flag flipped **lose** into **tie**. That fragility *is* the finding.

---

## Step 4 — Read the four families

<p class="kicker">Test f1 · CV mean ± std underneath</p>

<div class="cols-3">
<div class="stat stat--yellow">
<span class="stat-value">0.986</span>
<span class="stat-label">linear SVM (L11)<br>CV 0.981 ± 0.008</span>
</div>
<div class="stat stat--blue">
<span class="stat-value">0.973</span>
<span class="stat-label">HistGBM tuned (L12)<br>CV 0.976 ± 0.012</span>
</div>
<div class="stat stat--green">
<span class="stat-value">0.965</span>
<span class="stat-label">MLP, best config<br>CV 0.984 ± 0.007</span>
</div>
</div>

<p class="muted">Forest (L10): test <strong>0.958</strong> · CV 0.972 ± 0.014. CV gaps sit near one fold-std — but the MLP needed a flag flipped to join the tie. <strong>Champion unchanged.</strong></p>

---

## The verdict

<ul class="checklist">
<li>The best-case MLP <strong>ties</strong> — it never exceeds</li>
<li>Its result hinged on <strong>one flag</strong> the other three families don't have</li>
<li class="no">"Neural networks win tabular" — not on these rows</li>
<li class="no">Changing your project headline — <strong>boosting stays the default</strong></li>
</ul>

---

## Where would a network actually win here?

<p class="kicker">The counterfactual · raw biopsy images</p>

**It wouldn't.**

If the input were the **raw biopsy image** instead of 30 engineered features, a pretrained CNN beats every tabular model — the **data** changed, so the tool changes.

---

## Wrap-up

---

Translate the words, stack the layers — and for data that fits in a spreadsheet, ship the model you can **defend**.

The toolbox closed at L13. Today is why it's enough.

---

## Three words for this lecture

<div class="cols-3">
<div class="stat stat--word stat--blue">
<span class="stat-value">Translate</span>
<span class="stat-label">β→w, σ→activation —<br>same math, new words</span>
</div>
<div class="stat stat--word stat--green">
<span class="stat-value">Stack</span>
<span class="stat-label">layers compose simple maps<br>into learned features</span>
</div>
<div class="stat stat--word">
<span class="stat-value">Defend</span>
<span class="stat-label">pick the model you can defend —<br>on tabular, that's classical</span>
</div>
</div>

---

## Vocabulary you now own

**Training**   `epoch` `batch` `learning rate` `backprop` `early stopping`

**Regularisation**   `dropout` `weight decay`

**Architecture**   `CNN` `attention` `transformer`

**2026**   `foundation model` `fine-tuning`

</div>

Enough to read the literature — success criterion two, met.

---

## Before Lecture 15

- **Finalise model selection** — pick the workflow you'll defend
- Write the **selection rule**, run the test-set evaluation **once**
- Want feedback first? Office hours or email

</div>

<p class="muted">The toolbox is <strong>closed</strong> — L15 is the rehearsal workshop. Bring your project.</p>

---

## Questions

Backup slides below — press down ↓

## A taste of PyTorch

<p class="kicker">Beyond sklearn · the same model, five lines</p>

```python
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(30, 32)
        self.fc2 = nn.Linear(32, 1)

def forward(self, x):
        return self.fc2(torch.relu(self.fc1(x)))
```

Five lines for the architecture; the rest is the training loop from section 03.

## Convolutional networks

<p class="kicker">Architectures · images</p>

- **Translation invariance** — a cat is a cat in any corner
- **Local connectivity** — nearby pixels correlate; distant ones don't
- **Parameter sharing** — one filter slides everywhere → far fewer weights

CNNs dominate vision because they encode these priors **architecturally**.

## Transformers

<p class="kicker">Architectures · attention</p>

```text
attention(Q, K, V) = softmax(Q Kᵀ / √d) V
```

- Every token looks at **every other token**, weighted by similarity
- No fixed receptive field — long-range dependencies for free
- Parallelisable, and **scales beautifully** with data + compute

## Regularisation, translated

<p class="kicker">Translation · the L6 dial in new clothes</p>

| Technique | What it does |
|-----------|--------------|
| **Dropout** | randomly zero neurons during training |
| **Weight decay** | L2 penalty on weights — ridge, renamed |
| **Batch norm** | normalise activations per batch |
| **Early stopping** | stop when validation stalls |
| **Augmentation** | transformed copies of training rows |

## Fine-tune vs from-scratch

<p class="kicker">Foundation models · the decision</p>

```text
Do you have millions of labelled examples
in your task's exact domain?

Yes → train from scratch   (rare)
  No  → fine-tune a foundation model,
        or use its embeddings + a classical model
```

For most real tasks in 2026, the answer is **fine-tune**.

---

## What's next

**Lecture 15:** Project workshop

- Present your project to peers
- Give and receive feedback
- Final preparation before submission