Mean-variance optimisation

The problem

Given expected returns $\mu$ and covariance $\Sigma$ (see linear algebra for portfolios), choose weights $w$ to get the best trade-off between return and variance. One common form maximises

w^\top \mu - \frac{\lambda}{2}\, w^\top \Sigma\, w

where $\lambda$ is risk aversion. Setting the gradient to zero gives the unconstrained solution

w^* = \frac{1}{\lambda}\, \Sigma^{-1} \mu

The efficient frontier

Plot every portfolio by volatility (x) and expected return (y). The upper-left boundary of what is achievable is the efficient frontier: for each level of risk, the highest possible return. Anything below it is dominated.

With a risk-free asset available, the best portfolios all lie on a straight line from the risk-free rate tangent to the frontier. The tangent point is the maximum-Sharpe (tangency) portfolio, and every investor should hold some mix of it and cash. That is the core of the Capital Asset Pricing Model.

Diversification in one formula

For $n$ assets with equal volatility $\sigma$ and the same pairwise correlation $\rho$ , an equal-weighted portfolio has variance

\sigma_p^2 = \sigma^2\left(\frac{1}{n} + \frac{n-1}{n}\,\rho\right) \;\longrightarrow\; \rho\,\sigma^2 \quad (n \to \infty)

Adding assets kills the idiosyncratic part, but the correlated part never diversifies away. With $\rho = 0.3$ , no number of assets gets volatility below about 55% of a single asset’s.

The same idea for strategies

Combining $n$ uncorrelated strategies, each with Sharpe $S$ , gives a portfolio Sharpe of $S\sqrt{n}$ . Four uncorrelated Sharpe-0.7 strategies make a Sharpe-1.4 book. This is why multi-strategy funds exist, and why low correlation is worth more than a slightly higher individual Sharpe.

Why it breaks in practice

The optimiser treats your estimates as truth, and it is an error maximiser: it piles weight into whatever assets have the highest estimated return and the lowest estimated correlation, which are exactly the estimates most likely to be noise. Expected returns are the worst offender; tiny changes in $\mu$ swing $w^*$ wildly.

Practical fixes, roughly in order of how often they are used:

Constrain weights (long-only, position caps, turnover limits).
Shrink the covariance matrix towards a structured target (Ledoit–Wolf).
Shrink expected returns towards a prior, as in the Black–Litterman model, which starts from market-implied returns.
Skip expected returns entirely: minimum-variance, risk parity (equal risk contribution) and hierarchical risk parity use only $\Sigma$ , which is far easier to estimate.

Common mistake

Using a backtest’s in-sample mean returns as $\mu$ . The optimiser will produce a portfolio that looks superb in-sample and disappoints out-of-sample, because it has optimised your estimation error.