My crystal ball is better than yours: superforecasting

The inspiration for this post is a book I read recently called ‘Superforecasting: the art and science of prediction’ by Philip Tetlock and Dan Gardner. It’s a good read with quite a few interesting insights. The possible applications of these insights to investing really struck a chord with me.

The book is based on two big pieces of research Philip Tetlock was involved in.

The first was an experiment, where a large number of experts of various types were asked to predict the probability of certain events occurring (often these related to complex geopolitical issues). The headline finding was crudely characterised as the average forecaster is about as talented as a chimpanzee with a dart. However, more importantly, the study found differences between between two groups of forecasters: ‘big ideas’ idealogically driven people versus pragmatists more prone to change their minds in response to new information. Not surprisingly, the pragmatists were a lot better.

The second was a forecasting competition run over several years following up on the question of why some forecasters do better than others. It found that some people were consistently very substantially better forecasters than others (the ‘superforecasters’) and identified what it was about their approach that made them so much better.

Investing is essentially a game of prediction, except instead of predicting an event you are predicting the movement of a share price. Consequently all of the attributes and approaches relevant to forecasting I believe are also relevant to investing.

Understand probability and uncertainty

The more forecasters embraced probabilistic thinking (i.e. appropriately characterising uncertainty according to probabilities) the better they performed.

People are predisposed to deal with uncertainty without nuance – our natural settings have evolved to be just ‘no, yes and maybe’. Assessing probabilities beyond that requires active reasoning with some analytical structure. However, what many of us do is instead create stories that turn probability into certainty. In investing this implies believing in a story that suggests our investment choices are more or less certain to succeed. After the fact we then confirm this feeling of certainty by rationalising the stories in hindsight. Either the story was correct and all the underlying reasoning too, or we were wrong but should have known it, had we not overlooked something that must have been obvious at the time. Needless to say this type of thinking leads to serious behavioural errors, like confirmation bias, as to achieve certainty we are biased to look for the information that supports our story and dismiss information that goes against it.

For investing, the starting point is to acknowledge how much uncertainty there is. It comes from various sources:

  • As investors the information we use for our analysis of a business is very limited. The resulting informational gaps create uncertainty.
  • Businesses are complex. Lots of individual factors drive their success. Even if we knew them inside out predicting the outcome of all of these factors in combination is very difficult.
  • There are unknown unknowns in the form of unforeseeable events, which may affect specific firms or the entire industry.
  • There are macroeconomic factors, such as interest rates or consumer confidence, which can materially impact performance.
  • Market risk or investor confidence can mean a share price falls even when everything goes well for the business.

While individually it is possible to think about many of these factors, the sheer volume of them and complexity of the interactions makes prediction very difficult. At best we can predict with a high degree of uncertainty. However, uncertainty isn’t really a barrier to outperformance if approached correctly. Understanding whether outperformance is 55% likely to happen rather than 45% likely can result in a huge edge with enough repetition.

So how does an investor properly account for uncertainty into their approach to investing? It’s not really possible to measure directly so isn’t this all a bit academic?

As a starting point, I think it’s important to recognise that the possibility that your analysis is wrong or missing something is high. Employ a trading strategy that incorporates the probability that you might be wrong on occasion and minimises the loss from these occasions, while maximising the gain when you are right. To do this it’s both more practical and psychologically preferable to assume that ‘the market is always right’ rather than try to prove it wrong with your own bottom-up valuation analysis. This analysis is invariably much more incomplete than you think and leads to a psychological attachment and a false sense of certainty. In my experience a profit warning or collapsing share price is more likely to be a sign you are wrong about the prospects of the business, than a temporary opportunity to get an even better deal.

Outside view then inside view

As well as recognising the possibility of being wrong, accounting for uncertainty is obviously relevant when deciding what investments to buy in the first place. Better forecasters tended to employ a particular approach to breaking down a prediction problem. To predict the likelihood of an event occurring, better forecasters would always first take the ‘outside view’ to determine the base rate of that type of event happening, before then taking the ‘inside view’, making adjustments to the base rate for the specific factors. For example, to work out the probability of a certain conflict occurring (within a particular time frame), good forecasters would first look at how often that sort of conflict had occurred in the past, resisting the temptation to leap into the facts of the specific political situation at the time.

I think there is an obvious read-across to investing. It suggests it might be better to prioritise using a more top down statistical approach, before looking at the detail of a specific business. The key priority should be to get the ‘base rate’ probabilities onside. Focusing on detailed stock specific research without having a good idea of the base rate for that type of investment seems an elementary mistake that many investors make. Why even consider types of investment with a poor base rate, like blue-sky story stocks, businesses operating in very competitive markets or capital intensive businesses laden with debt?

A top down ‘outside view-first’ approach can of course be much more sophisticated than simply avoiding dubious types of investments. I think the way to take the ‘outside view’ in investing is to use ‘factors’, the characteristics of an investment that make it statistically more likely to outperform in general, before making adjustments for the specific investment. Focusing on high level factors also gives a more robust framework to try to prioritise the aspects of the investment that are more important and avoid getting sucked into irrelevant detail. My approach is heavily shaped by this idea, first filtering based on quality factors to create watchlist, and then ranking this watchlist by other factors to guide my attention towards those shares that should be statistically more likely to succeed.

When the facts change, I change my mind

Better forecasters were more prepared to challenge their own thinking and become detached from it. They would more frequently update their predictions and were able to adjust in response to news appropriately, neither under or over reacting.

I think another advantage of the top down approach is that it is more conducive to psychological detachment than getting to know an individual business in too much detail. But even if you are not too attached to a particular idea it can still be a challenge to know how to respond to news.

In the stock market the two main sources of news are from a company’s RNS and the movement of the share price. It is important to adapt to these signal swiftly but appropriately. This is not straightforward but I try to do this in the following ways:

  • Monitor news for my portfolio and watchlist fairly frequently. Often investors are advised not to do this to avoid the risk of making behavioural errors. This is good advice for many but to be a better investor, unsurprisingly it pays to pay attention.
  • To avoid under or over reacting to news, I try to reflect how it adjusts the ranking of shares according to factors on my watchlist. I update these scores regularly to help shape my attitude to my investments and watchlist in an objective way on an ongoing basis.
  • I try to size positions according to to probability of success. When I get better than expected news my confidence in a position improves and I add to it. When I get worse than expected news I see how it affects the investment case and then, as my investment case is typically predicated on good news, generally I sell.

Forecast, measure, revise

The biggest predictor of a forecaster’s success was a positive attitude to self-improvement. To improve at forecasting it is key to measure the success of your process, learn from mistakes and improve. I think the same is true for investing.

One of the challenges in investing is that it is hard to know whether outperformance or underperformance is due to luck or skill. On top of this even if you do measure your performance effectively it can be hard to understand the underlying drivers on it. Because of these challenges, there are limitations to how effectively you can measure success and refine your process accordingly. I think it is still important to try to identify what works so you can do more of it and less of what doesn’t.

Benchmarking relative performance is a useful way to measure success. While you have to take benchmarking with a pinch of salt, as even good strategies may underperform over certain time periods, I believe these time periods should not be very long. Or to put it another way, choosing a strategy that targets success over a shorter time frame gives you the very significant advantage of being able to measure performance and seek to improve it (as well as being more lucrative if it works of course).

It can also be useful to use benchmarking to focus on where your strategy might be going wrong. To do this I benchmark against similar versions of my strategy with variations in approach, trying to break down the return from the various components of my strategy: focusing on quality business; ranking my watchlist according to quality, value and momentum; and finally the execution of my trades.

As well as benchmarking it’s also key to look at past mistakes to learn lessons from them. I do this in two ways:

  • Look in detail at profit warnings to see if there was anything that could have been predicted in advance.
  • Look at how my previous sells have performed to see what I might have been wrong to sell or whether I might be overtrading.

In doing this, I need to be careful not to over-attribute lessons to single mistakes. What I am trying to do is pick up on any patterns or mistakes that are more systematic to adjust my strategy accordingly. This becomes more valuable when done over a longer time period.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s