Assignment 3: Full Pipeline and Project Synthesis

This is voluntary, ungraded practice. Nothing is submitted and there is no deadline. Its purpose is to rehearse the complete modelling argument you will need for your semester project and oral exam.

Choose your data

The best option is to use your semester-project dataset. If that project is not ready, use the built-in wine-classification data:

from sklearn.datasets import load_wine

data = load_wine(as_frame=True)
X = data.data
y = data.target

For the wine data, treat the task as multi-class classification and use macro F1 as the primary metric.

Part 1: Frame the prediction problem

In no more than five sentences, state:

1. What one row represents
2. What is predicted, and at what point in time
3. Who might use the prediction
4. What a useful prediction would change
5. What the model must not be used to claim

Identify possible leakage before examining model performance. If observations are grouped or ordered in time, explain why an ordinary random split may be inappropriate.

Part 2: Reserve the final test set

Choose a split appropriate to the data:

• Stratified random split for independent classification observations
• Time-based split for genuinely temporal predictions
• Grouped split when several rows belong to the same person, company, product, or event

Set the test data aside. Define one cross-validation strategy and one primary metric for all selection decisions.

Part 3: Build the baseline ladder

Create three levels of evidence:

1. A naive baseline such as the majority class or mean prediction
2. A simple linear or logistic model
3. At least two contrasting model families from the course

Keep all learned preprocessing inside each pipeline. Use the same resampling folds for every candidate. Tune only a few defensible hyperparameters rather than constructing an enormous search.

Build a comparison table with cross-validation mean, standard deviation, and fitting time. Mark candidates whose difference from the best score is small relative to fold-to-fold variation.

Part 4: Test one modelling claim

Choose one substantive question rather than adding complexity for its own sake. Examples:

• Does regularisation materially improve the linear baseline?
• Does a nonlinear model outperform a linear boundary?
• Does PCA provide useful compression without sacrificing predictive performance?
• Do cluster-distance features add information to the supervised model?
• Does careful feature engineering help more than switching algorithms?

State the claim before running the comparison. Change one major thing at a time and use the same validation protocol on both sides.

Part 5: Make and justify the selection

Choose a final model using more than the largest mean score. Discuss:

• Performance and uncertainty
• Error costs
• Interpretability
• Training and prediction cost
• Sensitivity to preprocessing and hyperparameters
• Whether the model is simple enough to maintain and defend

If your chosen model is not the numerical winner, explain the tradeoff explicitly.

Part 6: Evaluate once and audit the errors

Refit the selected workflow on all training data and evaluate it once on the test set.

Go beyond a single aggregate score:

• Inspect the most serious errors
• Compare performance across meaningful subgroups or ranges
• Check whether errors cluster in time or around unusual feature values
• Use an appropriate interpretation method, while avoiding causal claims

Describe one realistic failure mode that the test metric does not capture.

Part 7: Rehearse the oral defence

Answer these questions aloud without looking at your code:

1. Why is this a predictive rather than descriptive problem?
2. What exactly was fitted inside cross-validation?
3. Why is your metric appropriate?
4. What does your baseline establish?
5. Why did you select this model over its closest competitor?
6. What would count as data leakage here?
7. Which result are you least confident in?
8. What would you change with another week of work?

If an answer depends on “the library did it,” return to that part of the workflow until you can explain it in plain language.

Optional stress tests

• Repeat the analysis with several random seeds.
• Compare performance with and without the most important feature.
• Test sensitivity to a different reasonable metric or validation split.
• Deliberately introduce one leakage mistake, measure the inflated score, then fix it.

Useful references

• ISLP — all chapters covered in the course
• scikit-learn User Guide
• Common pitfalls and recommended practices