StuckOut

ResourcesMachine learning

Data Leakage: Why Your Model Scores 99% and Then Fails

The single most common reason a student model looks brilliant in training and collapses on the marker's data.

6 min read

The symptom

Validation accuracy is suspiciously high, close to training accuracy, and barely moves however you change the model. That is not a good model. That is information from the test set reaching the training set.

The usual cause

Shuffling or augmenting before splitting, scaling on the full dataset before splitting, or duplicate rows appearing on both sides. Split first, then do everything else on the training portion alone.

How to check

Look for near-duplicate rows across the split. Compare the distribution of each feature between train and test. If your accuracy drops sharply when you fix the split, the original number was never real.

How to write it up

Markers reward the student who notices leakage and says so. A section explaining what you found and how you fixed it is worth more than a higher number you cannot defend.

Still not moving?

If this did not unstick you, the problem is probably specific to your code or your data. Send it over and we will look at the actual thing.

Send my requirements →

Related subjects