Variance for Intuition, CVaR for Optimization

This post gives a high-level presentation of the Variance for Intuition, CVaR for Optimization SSRN article.

Historical yearly return distribution of S&P 500 and the best possible normal calibration.

While everyone understand that investment risk is characterized by large losses or drawdowns, mainstream finance and economics academics still continue to promote mean-variance analysis.

Even Harry Markowitz understood that risk should be measured by the downside, but in 1950’s the computational burden was unimaginably large. Estimating a low-dimensional covariance matrix was considered computationally intensive at that time.

The truth is that mean-variance is mainly used for marketing purposes. It sounds cool when people tell clients that they are using “Noble prize winning theory”. At least when they don’t tell them that it was made for the technology of the 1950’s, and that it focuses on minimizing the upside as much as the downside.

Due to the above business and academic vested interests, many attempts are made at justifying mean-variance despite its obvious deficiencies. These are, however, not based on valid scientific arguments.

Some people also simply don’t like to admit that what they have been doing for a very long time is in fact not that beneficial for generating good risk-adjusted returns.

However, as an ambitious investment manager yourself, you have to use the best methods that maximize the probability of you being successful.

You are not suddenly going to become a billionaire hedge fund manager using the same technologies that are easily available to anyone who has some knowledge of Python and statistics.

What mean-variance is good for

As argued in the Variance for Intuition, CVaR for Optimization article¹, mean-variance can be used to build investment intuition, because the risk can be easily decomposed into standard deviations and correlations.

The overall conclusions and rules of thumb are likely to hold for better investment risk measures like the Conditional Value-at-Risk (CVaR). This argument is based on the fact that CVaR is a coherent risk measure and can therefore be decomposed similarly to variance. The expressions are just more general, see the article.

The issue is that mean-variance is usually not promoted as something that is only suitable for building investment intuition in the idealized, albeit highly unrealistic, case. It is generally promoted as a state-of-the-art method for generating better risk-adjusted returns.

However, in reality mean-variance is probably a recipe for disaster. Think about the fact that you are assuming that the historically observed left tails are not there, and that the dependencies are constant and linear, no matter if we are in a significant risk off scenario or a calm market:

The next generation CVaR alternative

As thoroughly presented in the Portfolio Construction and Risk Management book², there exist a practically feasible alternative which handles all of the main issues of the variance-based approach.

Instead of relying on the highly oversimplified elliptical distribution assumption, as mean-variance does, we can work with fully general investment distributions represented by the joint scenario matrix R and associated joint scenario probability vector p:

\(R=\left(\begin{array}{cccc} R_{1,1} & R_{1,2} & \cdots & R_{1,I}\\ R_{2,1} & R_{2,2} & \cdots & R_{2,I}\\ \vdots & \vdots & \ddots & \vdots\\ R_{S,1} & R_{S,2} & \cdots & R_{S,I} \end{array}\right)\quad p=\left(\begin{array}{c} p_{1}\\ p_{2}\\ \vdots\\ p_{S} \end{array}\right).\)

We then focus on minimizing the CVaR for these fully general distributions:

\(e^{\star}=\text{mean-CVaR}\left(e,R,p,\mathcal{E}\right).\)

The above market representation even allows us to perform fully general views and stress-testing using the Sequential Entropy Pooling method³.

Note also that we are optimizing over portfolio exposures e instead of weights. This allows us to handle derivatives in an elegant way⁴ and even handle fully general portfolio optimization parameter uncertainty⁵.

The catch

The new methods are naturally significantly harder to implement fast and stable versions of. In fact, fully general CVaR optimization is still considered to be too hard to solve for most people.

However, practically feasible algorithms exist that speed up the CVaR optimization computation time significantly, see the accompanying code to the article.

It is even possible to solve CVaR optimization problems subject to risk targets and tracking error constraints as presented in Section 6.3 of the Portfolio Construction and Risk Management book.

Exclusive case studies

Paid subscribers have access to an expanding collection of exclusive case studies going through the theory and methods presented above. As a free subscriber, you can claim a one-time access to a paid post.

Consider a paid subscription to lock in the current subscription price.

1. Variance for Intuition, CVaR for Optimization SSRN article: https://ssrn.com/abstract=4034316

2. Portfolio Construction and Risk Management Book post: https://antonvorobets.substack.com/p/pcrm-book

3. Sequential Entropy Pooling post: https://antonvorobets.substack.com/p/sequential-entropy-pooling

4. Derivatives Portfolio Management Framework video post: https://antonvorobets.substack.com/p/5-derivatives-portfolio-management

5. Portfolio Optimization and Parameter Uncertainty video post: https://antonvorobets.substack.com/p/9-portfolio-optimization-and-parameter-uncertainty