The Big Lesson

When you make a big claim, you have to be careful. This is the lesson hardest to learn. I am still learning it. My stock market education helped me a lot. The one thing it always taught me was to be ready for a surprise. It happened again today, as markets got Trumped. The idea of frequent outliers is hard to grasp because humans herd. It gives us comfort to herd but that’s the only way the society can function. It has to form clusters and then burst them. The only way for life to continue is by surprises. The role of uncertainty is so critical when it comes to system functioning. This is why sticking your neck out is a dangerous way to live. Nobody can tell you this better than stock market forecasters. The pundit to disrepute journey is very uplifting and humbling. Anyway, a failed forecast is good for system building as it forces us to go back to the drawing board and look into our systems. Hence there is a positive flip side to every surprise.

Limited System

My father showed me an article about Nate Silver in an Indian Newspaper. Nate was the king as he correctly predicted the outcome of 50 out of 50 states during the US presential election in 2012. Well! as you can expect FiveThirtyEight got the attention, the book, “The Signal and the Noise” became a hit. It was a probabilistically a “Poor Get Richer” event. From nowhere or somewhere on the scene to be at top of the scene.

The methodology is always good and the book was an interesting read too, but again interesting reads and forecasting accuracy are two things. One of the questions I asked myself then was howcome sports and political forecasting could not be extended to stock market predictions. How can you excel in the business of prediction without extending the system to other domains? This is an ingrained problem not just with polls but also with our stock market systems. They cannot be extended to politics and sports. It’s this system limitation why our anticipatory systems are weak and can either be used for politics or for earthquakes, not for both.


The three components of a natural system encompass the rich, the poor and the rest. A natural system cannot be conceived to just resolve the problem of the Rich get Richer (RGR). It has to resolve the other scenarios that accompany the RGR problem, i.e. why Rich get Poor (RGP), Poor get Poorer (PGP) and Poor get Richer (PGR)?

Hence any group (assets, variables, polls etc.) can be seen as made of the Rich, the Poor, and Rest. This is the only way we can explain the complete problem. A substantial part of the three classifications gets transformed into each other. The blue being the initial all Rich, the grey being the Poor, and the brown being the all Rest. The transitions explain how a part of the Rich transform into the Rest while a part of the Rich Stay Rich. How the Poor stop staying Poor and start moving towards Richness.

The Rest acts as a buffer staying broadly where it is over different periods. A three bin probabilistic classification finds it easy to explain the problem probabilistically. How many Rich will stay Richer in the future?

Missing the PGR estimate

Let’s see what’s wrong with Nate’s probability model. It just looked at RGR estimates and ignored the PGR all together. The probabilistic system did not account for aspects of a natural system that bring in uncertainty. For RGR estimation the models should account for PGR.

Trump had a lower RGR estimate compared to Hillary (Nate’s estimates). He was an outsider, he lacked diplomacy, the news spin was against him etc. Hillary had a higher and favorable RGR. It was the lack of appreciation of PGR, which messed up the calculations. Accounting for RGR and PGR together could have reduced the error and hence the shock.

Assuming equal PGR = 0.5 for both Candidates

Just let’s assume the PGR was 0.5 for both Hillary and Trump. The aggregate odds would have still changed. RGR was 0.7 for Hillary (Nate’s estimates) and RGR was 0.3 for Trump (Nate’s estimates). Now add 0.5 PGR to each of the individual RGR estimates. This will give a new probability.

(RGR+PGR)/2 = Actual Probability
(0.7+0.5)/2 = 0.6 for Hillary


(RGR+PGR)/2 = Actual Probability
(0.3+0.5)/2 = 0.4 for Trump

The shock suddenly became less as the expectation changed from 0.7-0.3 to 0.6-0.4 in favour of Hillary.

Why PGR = 0.6 for Trump?

Considering Trump was the underdog here, especially if you keep giving him lower chances of success in RGR. He had to have a relatively higher PGR compared to Hillary. So if we bump the 0.5 PGR to 0.6 PGR for Trump, the average probabilities dramatically change.

(RGR+PGR)/2 = Actual Probability
(0.7+0.4)/2=0.55 for Hillary


(RGR+PGR)/2 = Actual Probability
(0.3+0.6)/2=0.45 for Trump

This meant a cliffhanger, which is what happened.

This is what we call data universality. A probabilistic system that is holistic and is domain agnostic. Because uncertainty is everywhere, be it sports, markets, earthquakes or politics.


[1] Pal, M. “How Physics Solved Your Wealth Problem!,” SSRN, 2016
[2] Silver, N. “The Signal and The Noise”, Pengin, 2012