Hypothesis tests and the t-stat of a strategy

The setup

A hypothesis test asks how surprising your data would be if nothing interesting were going on. For a strategy, “nothing interesting” is the null hypothesis that the true mean return is zero.

With $n$ return observations, sample mean $\hat\mu$ and sample standard deviation $\hat\sigma$ , the t-statistic is

t = \frac{\hat\mu}{\hat\sigma / \sqrt{n}}

If returns were independent and the null true, $t$ would follow (approximately) a standard normal distribution for large $n$ . A $|t|$ above 1.96 happens only 5% of the time by chance, so it is conventionally called significant at the 5% level. The p-value is the probability, under the null, of a result at least this extreme.

The link to the Sharpe ratio

Notice that $\hat\mu/\hat\sigma$ is just the per-period Sharpe ratio. If $\widehat{SR}$ is the annualised Sharpe and you have $Y$ years of data,

t \approx \widehat{SR} \times \sqrt{Y}

How much history do you need?

To reach $t = 2$ :

Annual Sharpe	Years needed
0.5	16
1.0	4
2.0	1
3.0	about 5 months

A modest Sharpe of 0.5 — respectable for a real long-horizon strategy — cannot be distinguished from luck with less than a decade and a half of data. This is why short backtests of slow strategies say almost nothing.

Type I and Type II errors

A Type I error is a false positive: you deploy a strategy that has no edge. The significance level (say 5%) controls how often this happens for a single test.
A Type II error is a false negative: you discard a strategy that works. Its probability falls as data and true effect size grow; one minus it is the test’s power.

Multiple testing

The 5% threshold is for one test. If you try 20 random strategies, you expect one to clear $t = 2$ by luck alone. Try 1,000 parameter combinations and dozens will. Two common corrections:

Bonferroni: with $N$ tests, require $p < 0.05/N$ . Simple and very conservative.
Harvey, Liu and Zhu (2016) argued that, given how many factors have been tested in finance, a new factor should clear roughly $t > 3$ rather than 2.

The full treatment, including the deflated Sharpe ratio, is in overfitting.

Common mistakes

Treating overlapping returns as independent. Daily observations of a 20-day holding-period return share 19 days of data; the effective sample size is about 20 times smaller.
Reading $p = 0.04$ as “96% chance the strategy works”. The p-value is $P(\text{data} \mid \text{null})$ , not $P(\text{null} \mid \text{data})$ — the Bayes confusion again.

The setup

The link to the Sharpe ratio

Type I and Type II errors

Multiple testing

Put it to work