How much of a 99% score is the test set copying the training set?
leakcheck finds duplicated rows, contradictory labels, future data and one-column shortcuts across train/test splits, then retrains the standard models without them.
Row numbers from the KDD sample you can load in the scanner.
of KDD Cup 99's test rows have exactly the same features as a training row. Removing leaked rows costs up to 17.5 macro-F1 points.
A high score can be a lookup, not a detector
Security benchmarks make high scores easy: on a random split of CIC-IDS2017, untuned baseline models reach 97.1–99.9% macro-F1. Many benchmarks put identical or near-identical rows on both sides of the train/test split, mix future data into training, or contain a single column that gives the label away.
A model can then score by recognising rows it has already seen. That says little about how it handles traffic or files it has not seen, which is the thing a deployment needs.
Six checks on every split
leakcheck clean removes the leakage, then re-reads its own output and re-checks every row. If anything remains, it fails.
Eight benchmarks, one audit
| Dataset | Domain | Split | Test rows | Exact copies | Near copies (× train baseline) | Best single feature |
|---|---|---|---|---|---|---|
| KDD Cup 99 | Network intrusion | official | 311,029 | 68.4% | 14.8% (×1.37) | src_bytes (93.8%) |
| NSL-KDD | Network intrusion | official | 22,544 | 2.9% | 30.1% (×0.60) | src_bytes (83.0%) |
| UNSW-NB15 | Network intrusion | official | 82,332 | 10.4% | 7.5% (×1.35) | smean (79.1%) |
| CIC-IDS2017 | Network intrusion | random 80/20 | 566,149 | 14.1% | 62.7% (×0.98) | Destination Port (90.2%) |
| EMBER 2018 | Malware (PE files) | official | 200,000 | 0.009% | 20.0% (×0.79) | sec_entropy_max (71.7%) |
| UCI Phishing Websites | Phishing | random 80/20 | 2,211 | 64.4% | 0.0% | SSLfinal_State (90.3%) |
| PhiUSIIL | Phishing URLs | random 80/20 | 47,159 | 0.0% | 0.31% (×1.32) | URLSimilarityIndex 99.6% |
| SMS Spam Collection | Spam | random 80/20 | 1,115 | 11.0% | — | — |
× train baseline above 1 means test rows sit closer to training rows than training rows sit to each other. Best single feature is the balanced accuracy of a small tree on that one column.
Same copying, very different damage
KDD Cup 99 loses 17.5 macro-F1 points when its leaked test rows are removed. UCI Phishing, with as much copying, loses 1.5.
Four untuned baselines × 5 seeds; this chart shows two of them. Over the five seeds the 95% interval for KDD Cup 99's logistic-regression gap is 16.9–18.2 points; small gaps are less certain: UCI Phishing's gradient-boosting gap of 0.8 has an interval of -0.1 to 1.7, which includes zero. All four models, with 95% intervals, are in RESULTS.md.
9 hypotheses, decided by rules fixed before the runs
Each decision rule was committed to PLAN.md before the experiment it judges was run.
Four findings
Copying is not the same as inflation.
UCI Phishing copies about as much as KDD Cup 99 (64.4% vs 68.4% of test rows) but loses only 1.5 macro-F1 points when the copies go, against 17.5 for KDD Cup 99. With 30 features of 2–3 values each, a UCI Phishing copy is just a common pattern a model gets right anyway; KDD Cup 99's copies are memorised exceptions, and 73% of its gap comes from near-copies rather than exact ones.
The control comes back clean.
EMBER 2018's official split is by time. leakcheck finds 0.009% exact copies, 0.0% of training files newer than the test period, and test rows further from training than training rows are from each other (×0.79). The audit does not flag everything.
Too easy to rank models.
Every baseline scores at least 99.99% macro-F1 on PhiUSIIL. Removing the four features that each reach 95% on their own still leaves at least 99.95%. A test that easy cannot tell models apart.
Random splits can hide a collapse.
On CIC-IDS2017 the baselines score 97.1–99.9% macro-F1 on a random split, and 46.2–56.5% when trained on earlier capture days and tested on the last ones, whose attack types (PortScan, DDoS) never appear in training.
Checked against the literature
KDD Cup 99 records / distinct records, against Tables I and II of Tavallaee et al. (2009), the paper that introduced NSL-KDD:
| Slice | Paper | leakcheck | Status |
|---|---|---|---|
| Train · total | 4,898,431 / 1,074,992 | 4,898,431 / 1,074,992 | exact |
| Train · attack | 3,925,650 / 262,178 | 3,925,650 / 262,178 | exact |
| Train · normal | 972,781 / 812,814 | 972,781 / 812,814 | exact |
| Test · total | 311,027 / 77,289 | 311,029 / 77,291 | +2 records |
| Test · attack | 250,436 / 29,378 | 250,436 / 29,378 | exact |
| Test · normal | 60,591 / 47,911 | 60,593 / 47,913 | +2 records |
The two-record difference is in the paper's test-set normal count (60,591); the published file has 60,593.
Reproduce every number
# 1. Install, then fetch every dataset (SHA-256 verified)
> python -m venv .venv
# then activate it: .venv\Scripts\activate (Windows) or source .venv/bin/activate
> python -m pip install -e ".[dev]"
> python scripts/fetch.py
# 2. Validate, audit, retrain, ablate
> python -m experiments.e0_sources && python -m experiments.e0_sources_week4
> python -m experiments.e0_synthetic && python -m experiments.e0_kdd
> python -m experiments.e1_audit
> python -m experiments.e2_retrain kdd99 nslkdd unsw_nb15 cicids2017 ember2018 uci_phishing phiusiil sms_spam
> python -m experiments.e4_ablation phiusiil cicids2017
# 3. Regenerate the report and this page's numbers
> python scripts/report.py
python -m venv .venv # then activate it: .venv\Scripts\activate (Windows) or source .venv/bin/activate python -m pip install -e ".[dev]" python scripts/fetch.py python -m experiments.e0_sources && python -m experiments.e0_sources_week4 python -m experiments.e0_synthetic && python -m experiments.e0_kdd python -m experiments.e1_audit python -m experiments.e2_retrain kdd99 nslkdd unsw_nb15 cicids2017 ember2018 uci_phishing phiusiil sms_spam python -m experiments.e4_ablation phiusiil cicids2017 python scripts/report.py
Limitations
- A copy is not always a bug: real traffic repeats. Both the original and the cleaned score are reported.
- The cleaned test set is harder by construction; the gap measures how much of a score was earned on copies.
- Near-copy thresholds (0.01 RMS, Jaccard 0.8) are fixed choices; a curve and a training-set baseline are reported alongside.
- Baselines are untuned standard models, and training sets are capped at 300,000 rows.
- EMBER is scanned on leakcheck's own numeric view, not the official 2,381-dimension feature vector.
- CIC-IDS2017's time split is confounded with unseen attack types, so H3 was decided on EMBER.