Your backtest didn’t lie to you. Your search did.

The chart is telling the truth

Take a strategy that returned 24% a year in a backtest with a Sharpe ratio of 1.8. Assume, generously, that the data is clean, no future information leaked in, and realistic costs were applied. Every number on that report is correct. If you ran that exact rule over that exact history, that is what would have happened.

So when it falls apart live, the instinct is to look for the error. Bad data. A bug in the fill logic. Slippage worse than modeled. Sometimes it is one of those. Usually it is none of them, and the search for a mistake comes up empty, because there was no mistake in the thing you are looking at.

The problem is not in the backtest. It is in the sentence you skipped over on the way to it: "I tried a few things and this one worked."

What "I tried a few things" actually costs

Here is the arithmetic, and it is worth doing slowly because it is the whole argument.

Suppose a completely worthless strategy — coin flips, no edge at all — has a 1-in-20 chance of posting a good-looking backtest by luck. That is a conservative way to think about it, and 1-in-20 is not generous odds.

• Test one worthless strategy: 5% chance you see something that looks good.
• Test 20: about a 64% chance at least one looks good.
• Test 100: 99.4%.
• Test 1,000: it is essentially certain, and you will have several to choose between.

Now notice what that last line means. If you tested a thousand variants of a worthless idea, you do not merely risk finding a false positive — you are guaranteed a portfolio of them, and you get to pick the prettiest. The chart you end up looking at is the best of a thousand draws from noise, and it looks exactly like a chart that came from an edge.

And a thousand variants is not an unusual amount of searching. It is one afternoon with a parameter sweep. Two moving-average lengths, four stop levels, three holding periods, five instruments and two date ranges is 240 combinations, and nobody would describe that as an exhaustive search. They would describe it as "trying a few things".

Why almost nobody counts

Because the count is unflattering, it arrives at the worst moment, and nothing forces you to write it down.

The moment you find a result you like is the moment the trial count matters most, and it is precisely the moment you least want to interrogate it. Everything before the winner feels like process rather than evidence — dead ends, ideas that did not pan out, versions you would not have shipped. It does not feel like data. It is the data.

And the counting is genuinely hard to do honestly after the fact. The parameter sweep you ran on Tuesday and abandoned counts. The idea you dropped because the first two years looked wrong counts. Reading a paper, deciding the effect was real, and testing only that one — that counts too, because somebody else did the searching and you inherited the survivor. Human memory reliably undercounts all of this, in the direction that flatters.

So the number that decides how much the result is worth is the one number the process never records.

What honest looks like instead

The fix is not to search less. Searching is how research works, and a rule against trying things is a rule against doing the job.

The fix is to make the search part of the result. Once you know how many variants were tried, the bar the winner has to clear can be raised to match — and that is a solved problem, not a philosophical one. The deflated Sharpe ratio does exactly this: it takes the observed figure, the length of the sample, the shape of the returns and the number of trials, and returns the probability that the result is more than the best draw from a pile of noisy attempts.

It changes the picture more often than people expect. A strategy showing an in-sample Sharpe ratio above 2 can carry a deflated figure near zero once a large search is accounted for. Both numbers are correct. The first describes the chart; the second describes what the chart is worth.

Three things follow, and they are worth adopting whatever tools you use.

1. Record the trial count as you go, not afterwards. A number reconstructed from memory is a number chosen to be comfortable.
2. Treat a strategy that survived a large search as needing more evidence than one that survived a small search, not less. The size of the hunt is a cost, not an achievement.
3. Hold data back before you start, and do not look at it. Not as a final check you run when the result already looks good — by then the discipline has already failed.

Backtested and hypothetical results are not indicative of future performance. Nothing here is investment advice.

The uncomfortable version

If this argument is right, then most of what circulates as a promising strategy is not a lie and not a mistake. It is an honest measurement of the winner of an uncounted competition, presented by someone who genuinely believes it, because nothing in their process ever told them otherwise.

That is a more troubling situation than fraud, and a more fixable one. Fraud requires catching people. This requires counting.