Same copying, different damage

KDD Cup 99 and UCI Phishing copy about the same share of their test sets from training: 68.4% and 64.4% of test rows are exact copies of a training row. Removing the copies costs logistic regression 17.5 macro-F1 points on the first and 1.5 on the second. The copy rate is not the number to report.

The prediction that failed

Before any model was retrained, the plan committed to a prediction (H2): on the two most-copied benchmarks, removing exact and near copies from the test set would cost the standard baselines at least 5 macro-F1 points. Those two were KDD Cup 99 and UCI Phishing, chosen by copy rate alone.

KDD Cup 99 passed with every model, losing 15.0 to 17.5 points. UCI Phishing failed with every model, losing at most 1.7. H2 is recorded as not supported, and the failure is the more useful half of the result.

What a copy is, in each

UCI Phishing describes each website with a few dozen features, each coded in a handful of levels. With so few possible values, unrelated sites often end up with identical rows. A copy there is a common pattern, and a model that has learned the pattern gets the copy right for the same reason it gets new sites right. Take the copies out and what remains is no harder.

KDD Cup 99's copies are a few traffic patterns repeated hundreds of thousands of times. The rows left after removing them are mostly the rare attacks that a model trained on those floods barely saw. Most of that damage comes from near copies rather than exact ones: 72.6% of logistic regression's gap is still there after the exact copies alone are taken out.

Memorisation, visible directly

On the leaked KDD Cup 99 rows, all four models score 92.4% macro-F1. On an exact copy every model predicts the training label, so every model makes the same mistakes: on the rows whose training copies carry the opposite label. On SMS Spam, the random forest, gradient boosting and the MLP score exactly 100% on the leaked rows in every seed.

How sure

Over five seeds, the 95% interval for logistic regression's KDD Cup 99 gap is 16.9 to 18.2 points. UCI Phishing's gradient-boosting gap of 0.8 has an interval of -0.1 to 1.7, which includes zero.

The weaker model's gap is larger than the stronger model's on every dataset where either reaches a point, which is what the pre-registered rule for H4 asked. Paired by seed, that difference is clearly above zero on 4 of them (KDD Cup 99, NSL-KDD, UNSW-NB15, EMBER 2018); on the two small benchmarks it is within seed noise. The pattern, a leakage gap that shrinks as the model gets stronger, is the one Moinuddin Hasan reported on a small image dataset. Here it holds on four large security benchmarks and cannot be shown on two small ones.

What to report instead

A copy rate says how much of a test set could be inflated, not how much is. The number that answers the question is the score after removing the copies, for the model being reported, next to the score before. That is what leakcheck's clean command and the E2 table produce, and what to ask for next to any 99%.

Every number here is read from site/data.json, which is generated from the ledger: E1 audit, E2 retraining (4 models × 5 seeds) and E2 uncertainty. Full tables in RESULTS.md; the account with its caveats in FINDINGS.md.