Question

A classifier is evaluated on previously unseen customers. Each customer has multiple rows. Hyperparameters are compared using cross-validation, then a final score is published. Which protocol prevents both customer leakage and use of the test set for model selection? One answer is expected.

Accepted Answer

First reserve a customer-disjoint final test set; use grouped cross-validation on the remaining customers, fitting every learned preprocessing step only within each training fold; evaluate the selected model once on the final test set.

Answer

Randomly split rows, normalize all data, then choose the best test score.

Answer

Use row-wise cross-validation and only remove the customer identifier column.

Answer

Split by customer, but select hyperparameters using the final test score.

Question from the Python Machine Learning test