Assignment 1: Preprocessing and Linear Models

This is voluntary, ungraded practice. Nothing is submitted and there is no deadline. Work through it after Lectures 3—7, ideally in a notebook where you can inspect each intermediate result.

The task

Build and compare classification pipelines that predict whether a Titanic passenger survived. The point is not to obtain the highest possible score. The point is to practise a workflow in which preprocessing, model selection, and evaluation are kept honest.

The dataset is the same one used in Lecture 3. Loading it for the first time requires an internet connection.

import seaborn as sns

df = sns.load_dataset("titanic")
columns = [
    "survived", "pclass", "sex", "age",
    "sibsp", "parch", "fare", "embarked",
]
df = df[columns]

X = df.drop(columns="survived")
y = df["survived"]

Do not use alive, adult_male, class, who, or deck. Some duplicate information already present in the selected features, while alive directly reveals the target.

Part 1: Inspect before modelling

Answer these questions with a small table or a few lines of code:

How many rows and predictors are there?
Which columns contain missing values?
What proportion of passengers survived?
Which columns should be treated as numeric, and which as categorical?
Why would accuracy alone be an incomplete description of performance?

Part 2: Split first

Create a stratified 80/20 train-test split with random_state=42. Set the test data aside until Part 6.

Explain why imputing missing values or scaling the complete dataset before this split would leak information from the test set.

Part 3: Build one reusable preprocessor

Use a ColumnTransformer containing:

A numeric pipeline with median imputation and standardisation
A categorical pipeline with most-frequent imputation and one-hot encoding
handle_unknown="ignore" in the encoder

Keep all preprocessing inside the pipeline. Verify that fit can run even though age and embarked contain missing values.

Part 4: Establish a baseline and challengers

Compare these models:

DummyClassifier(strategy="most_frequent")
Logistic regression
k-nearest neighbours

Wrap every model in the same preprocessing pipeline. On the training data only, use five-fold stratified cross-validation and report the mean and standard deviation of:

Accuracy
F1 score for the survived class

Questions to consider:

Which models clearly beat the dummy baseline?
Are differences between the two real models large relative to fold-to-fold variation?
Why is scaling important for k-NN and useful for penalised logistic regression?

Part 5: Tune without touching the test set

Use GridSearchCV with the same cross-validation folds and F1 as the selection metric.

For logistic regression, try:

{"model__C": [0.01, 0.1, 1, 10, 100]}

For k-NN, try:

{
    "model__n_neighbors": [3, 5, 11, 21, 41],
    "model__weights": ["uniform", "distance"],
}

Plot or tabulate the cross-validation scores across the candidate values. Prefer a simpler setting when several candidates perform essentially the same.

Part 6: Evaluate once

Choose one final workflow using cross-validation results alone. Fit it to all training data, then evaluate it once on the held-out test set.

Inspect:

Accuracy and F1
The confusion matrix
Precision and recall for survivors

Write three short conclusions:

What did the chosen model improve over the baseline?
Which error is more common: predicting survival when the passenger did not survive, or missing a survivor?
What is one important limitation of this analysis?

Optional extensions

Compare L1 and L2 regularisation in logistic regression.
Engineer family_size = sibsp + parch + 1 and test whether it helps under the same validation rule.
Change the selection metric from F1 to recall. Does the selected model or threshold change?
Inspect logistic-regression coefficients after preprocessing. Which interpretations are safe, and which are merely associations?