Core terms
Overfitting (backtest overfitting)
Tuning a rule's parameters until it fits the noise of one sample, so its in-sample result overstates what it does on new data.
Intraday rules have many knobs: range length, entry offset, stop, target, time stop, volume threshold, price tier and time of day. Bailey and co-authors (2014) describe how the chance of an overfit backtest rises with the number of configurations tried. In one simulation, fifty rules with no true edge, each tested on 40 trades, produced a best rule averaging about +0.42R per trade in the median run (Illustrative).
Out-of-sample tests
Freeze the rule and its parameters, then test it on data that was not used to build or tune it. Walk-forward testing repeats this through time: tune on months 1–3 and test on month 4, then tune on months 2–4 and test on month 5, and so on. Report how many variants were tried alongside the result, and prefer few parameters.
What traders should verify
Log every variant tried, keep a block of data untouched until the rule is frozen, and report in-sample and out-of-sample results side by side with their trade counts.
Do not conflate
An out-of-sample period reused for repeated adjustments becomes in-sample. A short out-of-sample test has a wide interval, so passing one is weak evidence of an edge, and failing one is not proof that none exists.
Primary source
Primary sourceBailey et al., 2014