In-sample vs out-of-sample decay, explained

In-sample is the period a strategy's rules, parameters, or entry/exit logic were developed and tested against — the data the builder was looking at while deciding what the strategy should do. Given enough tweaking, it's almost always possible to find a rule set that performs beautifully on a specific stretch of history, because there's no penalty in-sample for fitting to noise that happened to be there rather than a pattern that will happen again. This is overfitting, and it's invisible from in-sample numbers alone — an overfitted strategy and a genuinely robust one can produce identical-looking in-sample results.

Out-of-sample is a separate period — ideally trades made after the strategy's rules were already fixed, not retroactively selected — that the strategy had no opportunity to be tuned against. It's the closest thing to a real forward test that a finished trade history can offer. A strategy that was fitted to noise will typically fall apart here, because the specific noise it was fitted to doesn't repeat. A strategy with a genuine, repeatable edge will tend to hold up close to its original performance, because the thing driving its results wasn't a coincidence of one dataset.

This is why out-of-sample decay is treated as the single strongest robustness check there is: it's the one number that's genuinely hard to fake. A seller can select which history to show, tune parameters until the in-sample equity curve looks smooth, and cherry-pick a track record's starting date — all of that shows up in in-sample stats. None of it protects an out-of-sample period that was locked in before the results existed.

The check that matters is the size of the drop in profit factor (or a similar edge measure) moving from in-sample to out-of-sample. A small decay — the strategy performing close to as well, or even better, on unseen data — is the strongest positive signal a track record can offer. A large decay, especially profit factor falling toward or below 1.0, is close to a direct measurement of how much of the original edge was actually curve-fitting.

There's no out-of-sample check that proves a strategy will keep working forever — markets change. What it does prove is that the strategy wasn't only good at describing history it had already seen, which is the specific, common failure mode this check exists to catch.

An out-of-sample comparison requires the seller to have set aside a period in advance and not touched the strategy's rules during it — discipline that costs them nothing to skip and a marketplace has no way to verify after the fact from a single growth chart. It's also the check most likely to produce a result the seller doesn't like. Neither of those is evidence of bad faith on its own, but it's exactly why an independent verification step, rather than trusting the seller's own presentation, is where this check actually gets applied.