Assignment 2: Model Selection and Ensembles

This is voluntary, ungraded practice. Nothing is submitted and there is no deadline. Work through it after Lecture 12, when all of the model families below have been introduced.

The task

Compare several model families on the Wisconsin breast-cancer dataset using one defensible selection protocol. You will treat detecting a malignant tumour as the positive class, tune models using training data only, and inspect whether the apparent winner is meaningfully better than the alternatives.

The data ships with scikit-learn, so it does not require a download.

from sklearn.datasets import load_breast_cancer

data = load_breast_cancer(as_frame=True)
X = data.data

# In the original data, malignant is encoded as 0.
# Recode it so the clinically important class is the positive class.
y = (data.target == 0).astype(int)
y.name = "malignant"

Part 1: Define one protocol

Create a stratified 80/20 train-test split with random_state=42. Keep the test set untouched until Part 5.

For all model selection, use the same five-fold StratifiedKFold with shuffling and random_state=42. Use F1 for malignant cases as the primary metric, but also track precision, recall, and ROC AUC.

Before fitting anything, write down:

Why malignant cases should be treated as the positive class
Which error is likely more costly in practice
Why every candidate must use the same folds

Part 2: Establish the incumbent

Fit two reference workflows:

A most-frequent DummyClassifier
Standardisation followed by logistic regression

Use cross-validation on the training data. Record the mean and standard deviation for each metric. The logistic model is your incumbent: a more complex model should offer a convincing reason to replace it.

Part 3: Add three challengers

Construct and tune:

Random forest

Keep n_estimators reasonably large and search a small grid over:

max_depth
min_samples_leaf
max_features

Support vector machine

Use a pipeline with standardisation and SVC. Search a small grid over:

Linear and RBF kernels
C
gamma for the RBF kernel

Gradient boosting

Use HistGradientBoostingClassifier. Search a small grid over:

learning_rate
max_leaf_nodes
l2_regularization

Keep the grids modest. The exercise is about the selection procedure, not consuming as much compute as possible.

Part 4: Compare honestly

Create one comparison table containing, for every candidate:

Best hyperparameters
Mean cross-validation F1
Standard deviation across folds
Mean precision and recall
Approximate fitting time

Then answer:

Is the highest-scoring model ahead by more than ordinary fold-to-fold variation?
Does the ranking change when recall replaces F1?
Which model would you choose if interpretability mattered?
Which model would you choose if missing a malignant case were especially costly?

If two models are effectively tied, prefer the simpler workflow and explain why.

Part 5: Open the test set once

Select one workflow using only the cross-validation evidence. Refit it on all training data and evaluate it once on the held-out test set.

Inspect:

Confusion matrix
Precision, recall, F1, and ROC AUC
The individual false-negative cases

Do not return to model selection after seeing the test score. Explain why doing so would turn the test set into another validation set.

Part 6: Interpret with care

Use permutation importance on the held-out data for the selected model. Compare the result with random-forest impurity importance if you fitted a forest.

Consider:

Do correlated measurements divide or hide importance?
Does importance tell you that a feature causes malignancy?
Would you be comfortable making a clinical decision from this model alone?

Optional extensions

Plot a validation curve for the most influential hyperparameter.
Replace the single split with nested cross-validation and compare the estimated performance.
Tune a decision threshold for higher recall using training data only.
Check calibration and compare raw SVM scores with calibrated probabilities.