Anatomy of an honest backtest

A backtest is a simulation of what would have happened. Every simulation embeds assumptions, and the ones that flatter results are the ones nobody checks. Most strategies that “worked in backtest” and failed live were killed by one of the biases below.

Look-ahead bias

Using information at time $t$ that was not available until later.

• Computing a signal from a bar’s close and filling the trade at that same close. In reality you see the close, then trade at the next available price.
• Normalising features with statistics of the whole dataset (a full-sample mean or max).
• Using revised data (restated earnings, back-adjusted futures prices with future roll information) as if it were known in real time.

Test: shift all your signals one bar later. If performance collapses, the edge probably lived in that bar.

Survivorship bias

Backtesting only on assets that exist today. The coins and stocks that died — delisted, rugged, bankrupt — are missing, and they were disproportionately the losers. A crypto momentum backtest on today’s top 50 coins implicitly knows which coins made it into the top 50.

Fix: use point-in-time universes that include dead assets.

Costs, slippage and fills

• Fees: maker and taker rates differ; use the ones you’ll actually pay.
• Spread and slippage: market orders cross the spread and move the price, more so for size and in thin markets (see market impact).
• Funding and borrow: perps pay or receive funding; short stocks pay borrow.
• Limit-order fills: a limit order touched by price is not necessarily filled. Assume fills only when price trades through your level.

A useful habit: report the strategy’s break-even cost — the per-trade cost at which its profit hits zero. If it is close to the realistic cost, the strategy is not robust.

Data quality

Bad ticks, missing bars, timezone mistakes, and exchange outages produce fake signals and fake profits. Plot the raw data before trusting any result built on it.

The honest backtest checklist

1. Signals use only data available strictly before the decision time.
2. Execution happens at a realistic later price with fees, spread, slippage and funding.
3. The universe is point-in-time, including dead assets.
4. Parameters were chosen on one period and evaluated on another (walk-forward).
5. The number of variants tried is recorded, and the Sharpe is deflated for it.
6. Results are robust to small parameter changes and to a one-bar signal delay.
7. You can explain why the edge should exist, in one sentence, without referring to the backtest.