| Question | Does PBO answer it? |
|---|---|
| Was this backtest selected out of many attempts? | Yes — that is the whole subject |
| How often would that selection have misled me? | Yes |
| Is this specific strategy profitable? | No |
| Will the edge persist in future markets? | No |
| Did information leak backwards in time? | No — see look-ahead bias |
| Were delisted or failed instruments excluded? | No — see survivorship bias |
| Will costs and slippage remove the edge? | No |
How is PBO calculated?
Build the trials matrix
Record the return series of every configuration you tested, not just the one you kept. This step is where most attempts fail: the discarded variants are the data.
Split the sample into blocks
Divide the time axis into an even number of equal, contiguous blocks. Sixteen is the usual choice, which is large enough to give many splits and small enough to stay computable.
Form every symmetric split
Take every combination that assigns half the blocks to in-sample and the other half to out-of-sample. Each combination is one experiment, and the method is symmetric because every block serves in both roles across the set.
Pick the in-sample winner, then look out-of-sample
For each split, select the configuration with the best in-sample performance — imitating exactly what you did when you chose your strategy — and then find where that configuration ranks among all configurations on the held-out half.
Count how often the winner disappointed
PBO is the fraction of splits in which the in-sample winner fell below the median out-of-sample. A PBO of 0.8 means that in four splits out of five, the procedure you used to choose a strategy would have handed you a below-average one.
What does a PBO of 0.8 mean?
It means your selection procedure is mostly picking noise. Four times out of five, the configuration that won in-sample turned out to be a below-median performer on data that played no part in the choice. The strategy you are holding is one draw from that procedure, and the procedure is unreliable.
Read carefully, that is a stronger statement than "this strategy might not work". It says that a strategy chosen this way is more likely than not to be a disappointment, so the fact that yours won tells you almost nothing about it.
Is PBO a property of the strategy or of the search?
Of the search. This is the point most explanations skip, and it changes how the number should be used. You cannot compute a PBO for a single strategy in isolation, because there is no selection to evaluate — the statistic needs the whole set of configurations you chose from.
The practical consequence: two people can hold the identical strategy with the identical returns and correctly report different PBOs, because one found it in eleven attempts and the other in eleven thousand. The strategy is the same. The evidence for it is not.
What counts as a trial?
More than people want it to. Every parameter setting you evaluated counts. So does every variant you looked at, judged bad, and closed without saving. So does every earlier version of the idea, and every rule you added after seeing the chart and then removed. The count is not the number of strategies in your folder; it is the number of times the data influenced a choice.
This is the hardest honest bookkeeping in quantitative research, and it is why the number is more often cited than computed. Manual research effectively cannot count it. A system that generates and grades candidates automatically can, and that is one of the few genuine advantages of running the search inside a harness rather than by hand.
Why does not ordinary cross-validation solve this?
Standard k-fold cross-validation assumes the observations can be shuffled, and financial returns cannot: they are ordered, serially dependent, and clustered in volatility. Shuffling them lets information from the future inform the past, which produces optimistic results rather than honest ones.
A single train-and-test split avoids that but introduces a different problem. It is one experiment, so its answer depends heavily on which slice you happened to hold out — and once you have looked at that held-out slice and gone back to adjust anything, it is not held out any more. The combinatorially symmetric approach behind PBO exists to keep the time ordering intact while running many splits instead of one.
What does PBO not tell you?
It says nothing about whether an effect is economically real, whether it will persist, or whether it survives costs. A low PBO means your selection procedure is sound. It does not mean the market will keep cooperating, and no statistic computed from history can promise that.
It is also blind to problems in the data itself. If prices leaked forward, or the universe quietly excludes companies that went to zero, PBO will be computed faithfully on a corrupted series and report a reassuring number.
Backtested and hypothetical results are not indicative of future performance. Nothing on this page is investment advice.
Common questions
What is a good PBO?
Lower is better and there is no universal threshold, because PBO is a probability and the acceptable level depends on what a false positive costs you. What can be said plainly is that values above 0.5 mean your selection procedure is more likely to mislead you than not, and that a PBO computed from an undercounted set of trials is optimistic by construction.
Who developed the probability of backtest overfitting?
It was introduced by David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jay Zhu, using a method they call combinatorially symmetric cross-validation. The same body of work produced the deflated Sharpe ratio and the minimum track record length.
Can I compute PBO if I only kept the winning strategy?
No. The statistic is computed across the set of configurations you chose from, so discarding the losers destroys the input. If you want PBO later, keep the trials matrix now — the returns of every variant you evaluated, including the ones you dismissed in seconds.
How does PBO differ from the deflated Sharpe ratio?
PBO evaluates the selection procedure: how often does picking the in-sample winner produce a below-median result? The deflated Sharpe ratio evaluates one candidate: given the size of the search, how likely is this particular result to be real? They are usually read together, and they can disagree, which is informative.