Why Most Crypto Backtests Look Better Than They Really Are
A strategy can look excellent in a backtest and still be economically useless.
That is not cynicism. It is the ordinary condition of quantitative research, and it is why experienced researchers distrust a good-looking equity curve until they know how it was produced.
When a backtest disappoints in live conditions, the instinct is to blame the model. Usually the model is not the problem. The measurement is.
Here are five ways a backtest inflates itself, roughly in the order they do the most damage.
Information leakage
The most common form is also the most boring: deciding and transacting on the same bar.
If a signal computed from bar t's close is filled at bar t's close, the close is being used twice — once to make the decision, once to execute it. No live system can do that. In our own engine the entry delay defaults to one bar for exactly this reason: decide on t, execute on t+1.
Leakage also enters through quieter doors. Normalising a feature with statistics drawn from the whole sample. Deciding whether a bar was liquid enough to trade using that same bar's volume, instead of liquidity observable before the trade. Choosing a threshold on data the model will later be scored on.
Each one leaks a little of the future into the past, and the backtest rewards it.
Retrospective selection
Test enough variants and one of them will look good.
If you evaluate a few hundred configurations and report the best one's Sharpe ratio as though it were the only thing you tried, you are not reporting a discovery. You are reporting the maximum of a few hundred draws from a noisy distribution.
Corrections exist. The deflated Sharpe ratio adjusts for how many trials were run. Probability of backtest overfitting asks whether in-sample winners stay winners out of sample. But both depend on an honest trial count, and a trial count is only honest if trials were recorded as they ran rather than remembered afterwards. Hiding attempts inflates the very correction meant to catch you.
If your PBO sits near 0.5, your selection procedure has no predictive content, whatever the headline number says.
Costs
Costs rarely reduce an edge proportionally. They frequently reverse it.
Here is an audit almost nobody runs. Point your engine at a flat price series, force it to open and close a position, and check the accounting. In a market that does not move, a round trip must lose exactly its costs. If net is anything other than minus the sum of the itemised components — fees, spread, slippage, impact, funding — you have a bug, and every result that engine has produced is suspect.
Two details matter more than they look. A slippage floor, so that a suspiciously tight spread estimate cannot produce a free fill. And impact that scales with participation, so that size is never free.
The same candidate can be positive under central cost assumptions and negative under adverse ones. When that happens, the sign of your result is an assumption, not a finding.
Temporal instability
Crypto changes regime, and an aggregate number hides when the performance actually occurred.
In our own campaigns, the recorded reasons candidates failed were dominated not by insignificance but by concentration: results confined to a few months, to one half of the sample, or to a single symbol. The primary recorded cause in the final campaign was, in one word, instability.
The cheapest tests here are the most brutal. Is the result still positive after removing the best trade? The best day, week, month? Are both chronological halves positive? Most quarters? These separate a strategy from an anecdote in about ten lines of code.
And seal a holdout, then honour the seal. A holdout you have looked at is no longer a holdout. Every peek moves information out of it and into your selection process.
Failure disappears
The last one is structural rather than statistical.
Most research keeps the experiment that worked and quietly drops the rest. That is understandable, and it is expensive. The discarded runs hold much of the information: which families of ideas are already exhausted, which failures came from measurement rather than from the market, which assumptions were load-bearing.
A rejected hypothesis is a result. If it is never written down, the next researcher — often you, six months later — pays again to rediscover it.
The point
A backtest is not evidence because it produced a positive number. It becomes evidence when you can explain how that number was produced, and show that it survives realistic costs, an honest trial count, and validation that respects time.
Most of the work in quantitative research is not finding signals. It is building measurement you are willing to be wrong in front of.
This is the research philosophy behind Crypto Futures Quant Lab — a reproducible toolkit for testing, validating and documenting quantitative trading ideas, including the ones that did not work.
https://whop.com/quant-systems-lab/crypto-futures-quant-lab
We will use this space for notes on systematic trading, validation, market structure, probabilistic modelling and research engineering.
Quant Systems Lab — evidence over hype.