Train/Test Splits, Cross-Validation & Leakage

Validation estimates performance on data not used to fit the procedure. The word procedure includes preprocessing, feature selection, hyperparameter search, threshold choice, architecture choice, and every human decision informed by results. Holding out rows only from the final estimator is not enough.

Split according to the deployment claim #

Let training data be sampled from $P_{train}(X,Y)$ and future use from $P_{deploy}(X,Y)$. A test set estimates deployment performance only to the extent that its construction represents the deployment population and dependence structure.

Common split units are:

Stratification preserves class proportions but does not create independence. Randomly splitting visits lets the same patient's history appear on both sides; randomly splitting image crops can put pieces of one original image in train and test.

Training, validation, and test have different jobs #

• Training data fit parameters.
• Validation data choose representations, hyperparameters, models, and thresholds.
• Test data estimate the performance of the entire frozen selection procedure.

Repeatedly checking the test set and changing the model overfits the test set. It becomes another validation set, even if no gradient was computed from it.

Cross-validation rotates the validation role across folds. In $K$-fold CV, each fold is held out once and the $K$ scores are aggregated. This uses limited data efficiently, but the folds must respect grouping and order. Ordinary shuffled $K$-fold is wrong for future forecasting and dependent subjects.

Preprocessing lives inside each fold #

Any transformation that learns from data must be fitted on the fold's training portion only:

flowchart TD
  D["Raw development data"] --> F["Create fold split"]
  F --> T["Fit imputer, scaler, selector and model on fold-train"]
  T --> V["Apply frozen pipeline to fold-validation"]
  V --> S["Aggregate out-of-fold predictions"]

Global standardization leaks validation means and variances. Selecting features using all labels leaks their correlations. Deduplicating after splitting can leave near-copies on both sides. Oversampling before splitting can duplicate minority cases into validation. The safe unit is a pipeline fitted end to end inside each training fold.

Worked example: repeated patient visits #

Suppose 2,000 visits come from 300 patients and the model predicts readmission. A random row split may learn patient-specific disease codes and place earlier visits in training and later visits in test. Its score answers “can we predict another visit for partly known patients?”

If deployment includes new patients, split all visits for a patient together. If the system will be deployed next year, also respect time: train on earlier patients/visits and test on a later period. When both claims matter, use grouped temporal evaluation or separate test sets for each claim.

The label “readmitted within 30 days” also needs a complete 30-day observation window. Rows near the data-extract end have censored labels and cannot be treated as negatives.

Hyperparameter search needs nesting for an unbiased estimate #

Choosing the best of many candidates exploits noise in validation scores. Nested cross-validation separates two loops:

1. Inner folds select hyperparameters.
2. Each outer fold evaluates the complete inner selection process on unseen data.

Nested CV is useful when data are scarce and an unbiased estimate of the selection procedure matters. If a final untouched temporal or external test set exists, development CV can select a model and that test can provide the final estimate.

Scores vary for more than one reason #

Fold-to-fold variation reflects sampling, group composition, temporal change, and sometimes training randomness. Do not report only a mean. Keep out-of-fold predictions to compute uncertainty, inspect segments, and compare models on the same cases. Repeated random CV is not a cure for dependence; it resamples the same mistaken unit more often.

For time series, expanding-window evaluation mirrors retraining on all history; rolling windows mirror finite-memory training. Leave a gap when features near the boundary contain information that overlaps the target horizon.

Leakage patterns to actively test #

• post-outcome fields, resolution codes, or future aggregates;
• target encodings or imputations fitted outside the fold;
• duplicate entities, documents, or media across splits;
• normalization statistics computed from the test population;
• labels inferred from a data source whose presence is itself predictive;
• manual feature choices made after test inspection;
• train/serve skew in event time, defaults, joins, or software versions.

A useful diagnostic is to train a deliberately simple model on suspicious metadata—timestamps, source system, missingness, row order, or identifiers. Implausibly strong performance often reveals pipeline structure rather than real signal.

What a test result can claim #

A high score on an IID random holdout supports only the same-population IID claim. It does not establish temporal robustness, causal benefit, site transfer, subgroup adequacy, or calibrated probabilities. Those need corresponding splits, experiments, and metrics.

The right split is the one that makes cheating impossible in the same way deployment makes it impossible.