Ensemble Methods

Single decision trees are transparent but overfit easily. This lecture shows how combining many trees into an ensemble dramatically improves performance. We cover decision tree basics (splits, impurity, pruning), then bagging (bootstrap aggregating) and random forests (bagging + random feature selection). Key topics include feature importance, out-of-bag error, and the key hyperparameters to tune.

Optional: technical supplement

For students who want the formula: the variance of an average of B correlated estimators — ρσ² + (1−ρ)σ²/B — derived in one slide and verified empirically. It explains why more trees stop helping (the ρσ² floor), why max_features exists (it attacks ρ, accepting worse individual trees), and why averaging can never reduce bias.

Open the supplementary slides ↗ — optional and self-contained; nothing in it is required for the project.