Should You Consider Fate when Planning Ahead?

I was recently asked on Quora whether there is some kind of a grand scheme to things: a destiny that we all share, a guiding hand that acts according to some kind of moral rules.

This is a great question, and one that we’re all worried about. While there’s no way to know for sure, the evidence points against this kind of fate-biased thinking – as a forecasting experiment funded by the US Department of Defense recently showed.

In 2011, the US Department of Defense began funding an unusual project: the Good Judgement Project. In this project, led by Philip E. Tetlock, Barbara Mellers and Don Moore, people were asked to volunteer their time and rate the chance of occurence for certain events. Overall, thousands of people took part in the exercise, and answered hundreds of questions over a time period of two years. Their answers were checked constantly, as soon as the events actually occurred.

After two years, the directors of the project identified a sub-type of people they called Superforecasters. These top forecasters were doing so well, that their predictions were 30% more accurate than those of intelligence officials who had access to highly classified information!

(and yes, for the statistics-lovers among us: the researchers absolutely did run statistical tests that showed the chances of those people being accidentally so accurate were miniscule. The superforecasters kept doing well, over and over again)

Once the researchers identified this subset of people, they began analyzing their personalities and methods of thinking. You can read about it in some of the papers about the research (attached at the end of this answer), as well as in the great book – Superforecasting: the Art and Science of Prediction. For this answer, the important thing to note is that those superforecasters were also tested for what I call “the fate bias”.

Neither one seems to work. Sorry ’bout that.

The Fate Bias

There’s no denying that most people believe in fate of some sort: a guiding hand that makes everything happen for a reason, in accordance with some grand scheme or moral rules. This tendency seems to manifest itself most strongly in children, and in God-believers (84.8 percent of whom believe in fate), but even 54.3 percent of atheists believe in fate.

It’s obvious why we want to believe in fate. It gives our woes, and the sufferings of others, a special meaning. It justifies our pains, and makes us think that “it’s all for a reason”. Our belief in fate helps us deal with bereavement and with physical and mental pain.

But it also makes us lousy forecasters.


Fate is Incompatible with Accurate Forecasting

In the Good Judgement Project, the researchers ran tests on the participants to check for their belief in fate. They found out that the superforecasters utterly rejected fate. Even more significantly, the better an individual was at forecasting, the more inclined he was to reject fate. And the more he rejected fate, the more accurate he was at forecasting the future.


Fate is Incompatible with the Evidence

And so, it seems that fate is simply incompatible with the evidence. People who try to predict the occurrence of events in a ‘fateful’ way, as if they obeying a certain guiding hand, are prone to failure. On the other hand, those who believe there is no ‘higher order to things’ and plan accordingly, turn out to be usually right.

Does that mean there is no such thing as fate, or a grand scheme? Of course not. We can never disprove the existence of such a ‘grand plan’. What we can say with some certainty, however, is that human beings who claim to know what that plan actually is, seem to be constantly wrong – whereas those who don’t bother explaining things via fate, find out that reality agrees with them time and time again.

So there may be a grand plan. We may be in a movie, or God may be looking down on us from up above. But if that’s the case, it’s a god we don’t understand, and the plan – if there actually is one – is completely undecipherable to us. As Neil Gaiman and the late Terry Pratchett beautifully wrote –

God does not play dice with the universe; He plays an ineffable game of His own devising… an obscure and complex version of poker in a pitch-dark room, with blank cards, for infinite stakes, with a Dealer who won’t tell you the rules, and who smiles all the time.

And if that’s the case, I’d rather just say outloud – “I don’t believe in fate”, and plan and invest accordingly.

You’ll simply have better success that way. And when the universe is cheating at poker with blank cards, Heaven knows you need all the help you can get.



For further reading, here are links to some interesting papers about the Good Judgement Project and the insights derived from it –

Bringing probability judgments into policy debates via forecasting tournaments

Superforecasting: How to Upgrade Your Company’s Judgment

Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions

Psychological Strategies for Winning a Geopolitical Forecasting Tournament

Rethinking the training of intelligence analysts


Failures in Foresight, Part II: The Failure of the Paradigm

I often imagine myself meeting James Clark Maxwell, one of the greatest physicists in the history of the Earth, and the one indirectly responsible for almost all the machinery we’re using today – from radio to television sets and even power plants. He was recognized as a genius in his own time, and became a professor at the age of 25 years old. His research resulted in Maxwell’s Equations, which describe the connection between electric and magnetic fields. Every electronic device in existence today, and practically all the power stations transmitting electricity to billions of souls worldwide – they all owe their existence to Maxwell’s genius.

And yet when I approach that towering intellectual of the 19th century in my imagination, and try to tell him about all that has transpired in the 20th century, I find that he does not believe me. That is quite unseemly of him, seeing as he is a figment of my imagination, but when I devote some more thought to the issue, I realize that he has no reason to accept any word that I say. Why should he?

At first I decide to go cautiously with the old boy, and tell him about the X-rays – whose discovery was made in 1895, just 26 years after Maxwell’s death. “Are you talking of light that can go through the human body and chart all the bones in the way?” he asks me incredulously. “That’s impossible!”

And indeed, there is no scientific school in 1879 – Maxwell’s death date – that can support the idea of X-rays.

I decide to jump ahead and skip the theory of relativity, and instead tell him about the atom bomb that demolished Nagasaki and Hiroshima. “Are you trying to tell me that just by banging together two pieces of that chemical which you call Uranium 235, I can release enough energy to level an entire town?” he scoffs. “How gullible do you think I am?”

And once again, I find that I cannot fault him for disbelieving my claims. According to all the scientific knowledge from the 19th century, energy cannot come from nowhere. Maxwell, for all his genius, does not believe me, and could not have forecast these advancements when he was alive. Indeed, no logical forecasters from the 19th century would have made these predictions about the future, since they suffered from the Failure of the Paradigm.

Scientific Paradigms

A paradigm, according to Wikipedia, is “a distinct set of concepts or thought patterns”. In this definition one could include theories and even research methods. More to the point, a paradigm describes what can and cannot happen. It sets the boundaries of belief for us, and any forecast that falls outside of these boundaries requires the forecaster to come up with extremely strong evidence to justify it.

Up to our modern times and the advent of science, paradigms changed in a snail-like pace. People in the medieval times largely figured that their children would live and die the same way as they themselves did, as would their grandchildren and grand-grandchildren, up to the day of rapture. But then Science came, with thousands of scientists researching the movement of the planets, the workings of the human body – and the connections between the two. And as they uncovered the mysteries of the universe and the laws that govern our bodies, our planets and our minds, paradigms began to change, and the impossible became possible and plausible.

The discovery of the X-rays is just one example of an unexpected shift in paradigms. Other such shifts include –

Using nuclear energy in reactors and in bombs

Lord Rutherford – the “father of nuclear physics” in the beginning of the 20th century, often denigrated the idea that the energy existing in matter would be utilized by mankind, and yet one year after his death, the fission of the uranium nucleus was discovered.


According to the legend, the great experimental physicist Michael Faraday was paid a visit by governmental representatives back in the 19th century. Faraday showed the delegation his clunky and primitive electric motors – the first of their kind. The representatives were far from impressed, and one of them asked “what could possibly be the use for such toys?” Faraday’s answer (which is probably more urban myth than fact) was simple – “what use is a newborn baby?”

Today, our entire economy and life are based on electronics and on the power obtained from electric power plants – all of them based on Faraday’s innovations, and completely unexpected at his time.

Induced Pluripotent Stem Cells

This paradigm shift has happened just nine years ago. It was believed that biological cells, once they mature, can never ‘go back’ and become young again. Shinya Yamanaka other researchers have turned that belief on its head in 2006, by genetically engineering mature human cells back into youth, turning them into stem cells. That discovery has earned Yamanaka his 2012 Nobel prize.

Plugs everywhere. You can blame Maxwell for this one.
Plugs everywhere. You can blame Maxwell and Faraday for this one.

How Paradigms Advance

It is most illuminating to see how computers have advanced throughout the 20th century, and have constantly shifted from one paradigm to the other along the years. From 1900 to the 1930s, computers were electromechanical in nature: slow and cumbersome constructs with electric switches. As technology progressed and new scientific discoveries were made, computers progressed to using electric relay technology, and then to vacuum tubes.

Computing power increases exponentially as paradigms change. Source: Ray Kurzweil's The Singularity is Near
Computing power increases exponentially as paradigms change. Source: Ray Kurzweil’s The Singularity is Near

One of the first and best known computers based on vacuum tubes technology is the ENIAC (Electronic Numerical Integrator and Computer), which weighed 30 tons and used 200 kilowatts of electricity. It could perform 5,000 calculations a second – a task which every smartphone today exceeds without breaking a sweat… since the smartphones are based on new paradigms of transistors and integrated circuits.

At each point in time, if you were to ask most computer scientists whether computers could progress much beyond their current state of the art, the answer would’ve been negative. If the scientists and engineers working on the ENIAC were told about a smartphone, they would’ve been completely baffled. “How can you put so many vacuum tubes into one device?” they would’ve asked. “and where’s the energy to operate them all going to come from? This ‘smartphone’ idea is utter nonsense!”

And indeed, one cannot build a smartphone with vacuum tubes. The entire computing paradigm needed to change in order for this new technology to appear on the world’s stage.

The Implications

What does the Failure of the Paradigm mean? Essentially what it means is that we cannot reliably forecast a future that is distant enough for a paradigm shift to occur. Once the paradigm changes, all previous limitations and boundaries are absolved, and what happens next is up to grabs.

This insight may sound gloomy, since it makes clear that reliable forecasts are impossible to make a decade or two into the future. And yet, now that we understand our limitations we can consider ways to circumvent them. The solutions I’ll propose for the Failure of the Paradigm are not as comforting as the mythical idea that we can know the future, but if you want to be better prepared for the next paradigm, you should consider employing them.

And now - for the solutions!
And now – for the solutions!

Solutions for the Failure of the Paradigm

First Solution: Invent the New Paradigm Yourself

The first solution is quite simple: invent the new paradigm yourself, and thus be the one standing on top when the new paradigm takes hold. The only problem is, nobody is quite certain what the next paradigm is going to be. This is the reason why we see the industry giants of today – Google, Facebook, and others – buying companies left-and-right. They’re purchasing drone companies, robotics companies, A.I. companies, and any other idea that looks as if it has a chance to grow into a new and successful paradigm a decade from now. They’re spreading and diversifying their investments, since if even one of these investments leads into the new paradigm, they will be the Big Winners.

Of course, this solution can only work for you if you’re an industry giant, with enough money to spare on many futile directions. If you’re a smaller company, you might consider the second solution instead.

Second Solution: Utilize New Paradigms Quickly

The famous entrepreneur Peter Diamandis often encourages executives to invite small teams of millennials into their factories and companies, and asking them to actively come up with ideas to disrupt the current workings of the company. The millennials – people between 20 to 30 years old – are less bound by ancient paradigms than the people currently working in most companies. Instead, they are living the new paradigms of social media, internet everywhere, constant surveillance and loss of privacy, etc. They can utilize and deploy the new paradigms rapidly, in a way that makes the old paradigms seem antique and useless.

This solution, then, helps executives circumvent the Failure of the Paradigm by adapting to new paradigms as quickly as possible.

Third Solution: Forecast Often, and Read Widely

One of the rules for effective Forecasting, as noted futurist Paul Saffo wrote in Harvard Business Review in 2007, is to forecast often. The proficient forecaster needs to be constantly on the alert for new discoveries and breakthroughs in science and technology – and be prepared to suggest new forecasts accordingly.

The reason behind this rule is that new paradigms rarely (if ever) appear out of the blue. There are always telltale signs, which are called Weak Signals in foresight slang. Such weak signals can be uncovered by searching for new patents, reading Scientific American, Science and Nature to find out about new discoveries, and generally browsing through the New York Times every morning. By so doing, one can be certain to have better hunch about the oncoming of a new paradigm.

Fourth Solution: Read Science Fiction

You knew that one was coming, didn’t you? And for a good reason, too. Many science fiction novels are based on some kind of a paradigm shift occurring, that forces the world to adapt to it. Sometimes it’s the creation of the World Wide Web (which William Gibson speculated about in his science fiction works), or rockets being sent to the moon (As was the case in Jules Verne’s book – “From the Earth to the Moon”), or even dealing with cloning, genetic engineering and bringing back extinct species, as in Michael Crichton’s Jurassic Park.

Science fiction writers consider the possible paradigm shifts and analyze their consequences and implications for the world. Gibson and other science fiction writers understood that if the World Wide Web will be created, then we’ll have to deal with cyber-hackers, with cloud computing, and with mass-democratization of information. In short, they forecast the implications of the new paradigm shift.

Science fiction does not provide us with a solid forecast for the future, then, but it helps us open our minds and escape the Failure of the Paradigm by considering many potential new paradigms at the same time. While there is no research to support this claim, I truly believe that avid science fiction readers are better prepared for new paradigms than everyone else, as they’ve already lived those new paradigms in their minds.

Fifth Solution: Become a Believer

When trying to look far into the future, don’t focus on the obstacles of the present paradigm. Rather if you constantly see that similar obstacles have been overcome in the past (as happened with computers), there is a good reason to assume that the current obstacles will be defeated as well, and a new paradigm will shine through. Therefore, you have to believe that mankind will keep on finding solutions and developing new paradigms. The forecaster is forced, in short, to become a believer.

Obviously, this is one of the toughest solutions to implement for us as rational human beings. It also requires us to look carefully at each technological field in order to understand the nature of the obstacles, and how long will it take (according to the trends from the past) to come up with a new paradigm to overcome them. Once the forecaster identifies these parameters, he can be more secure in his belief that new paradigms will be discovered and established.

Sixth Solution: Beware of Experts

This is more of an admonishment than an actual solution, but is true all the same. Beware of experts! Experts are people whose knowledge was developed during the previous paradigm, or at best during the current one. They often have a hard time translating their knowledge into useful insights about the next paradigm. While they can highlight all the difficulties existing in the current paradigm, it is up to you to consider how in touch those experts are with the next potential paradigms, and whether or not to listen to their advice. That’s what Arthur C. Clarke’s first law is all about –

“When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.”


The Failure of the Paradigm is a daunting one, since it means we can never forecast the future as reliably as we would like to. Nonetheless, business people today can employ the above solutions to be better prepared for the next paradigm, whatever it turns out to be.

Of all the proposed solutions to the Failure of the Paradigm, I like the fourth one the best: read science fiction. It’s a cheap solution that also brings much enjoyment to one’s life. In fact, when I consult for industrial firms, I often hire science fiction writers to write stories about the possible future of the company in light of a few potential paradigms. The resulting stories are read avidly by many of the employees in the company, and in many cases show the executives just how unprepared they are for these new paradigms.