The Hidden Order Behind Randomness

Some events feel almost impossibly unlikely. You move thousands of kilometres away from where you grew up, walk into a restaurant in another country, and suddenly see someone you knew fifteen years ago. Your first reaction is usually, “What are the chances?” If you calculate it naively, it seems absurd. There are billions of people, thousands of cities and countless places where both of you could have been instead.

But that is probably the wrong probability to calculate.

The better question is not, “What is the probability of randomly choosing this person from everyone on Earth?” It is, “Given everything I know about both of us, how likely was it that our paths would eventually cross?” That small change in the question is the heart of Bayesian thinking.

Bayesian probability is basically about updating probabilities when new information becomes available. Two people are not randomly thrown into the world like independent particles. Perhaps they grew up in the same city, belong to the same generation, studied similar subjects, work in similar industries and immigrated to countries with similar opportunities. They may also have overlapping friends, interests, restaurants, conferences and neighbourhoods. Once these hidden conditions are included, something that originally looked like a one-in-a-million coincidence may still be unlikely, but it is no longer mysterious.

We make this mistake all the time because we see the final event but not the structure that produced it. Someone receives an incredible career opportunity and we notice the email, not the ten years of relationships that made somebody think of them. A company suddenly takes off and we notice the revenue curve, not the technological, cultural and economic changes that accumulated underneath it. Two people independently have the same idea and we call it strange, while ignoring the fact that both were exposed to the same changes in technology and society.

Sometimes randomness is simply what complexity looks like from far away.

This way of thinking becomes even more interesting when combined with the idea of equilibrium. In game theory, a Nash equilibrium describes a situation where each participant is responding to what everyone else is doing, and nobody can improve their position simply by changing their own strategy while everyone else stays the same. The equilibrium does not have to be good or fair. It simply means that the incentives of the participants are keeping the system where it is.

A company may have a terrible internal process that everyone complains about, yet nobody fixes it. At first this looks irrational. But perhaps replacing it requires one team to do six months of painful work while everyone else receives the benefit. Managers may be rewarded for delivering this quarter rather than improving something two years from now, and nobody wants to take responsibility for a risky migration. What initially looked like stupidity starts looking more like an equilibrium created by incentives.

Markets, careers, companies and social systems often behave in similar ways. When one side becomes unusually attractive, that change creates forces that push in the other direction. Very high profits attract competitors. A shortage raises prices and encourages new supply. Expensive labour gives companies a stronger reason to automate. A successful investment strategy attracts more people until the opportunity becomes harder to exploit.

This does not mean that the world automatically returns to some fair balance. Some bad equilibria can survive for decades, and some imbalances can become worse before they improve. The useful idea is simply to ask what counterforces are being created by the situation itself.

That question also helps protect us from our own biases. Humans are extremely good at inventing stories after something happens. A market falls and within minutes there are explanations everywhere. A startup succeeds and suddenly every decision its founders made appears brilliant. A startup fails and those same decisions suddenly look obviously foolish. Once we know the ending, our minds rewrite the beginning.

The opposite mistake is calling everything luck. One extreme sees a meaningful cause behind everything, while the other sees randomness everywhere. A better habit is to ask, “What mechanism could have produced this outcome?” Perhaps luck played a role, but perhaps there were also incentives, feedback loops, hidden information and conditions that we simply could not see.

There is, however, one important complication to this worldview: quantum physics.

In everyday life, randomness often comes from missing information. Think about throwing a die. The result looks random because we do not know the exact force of the throw, the rotation of the die, the air movement and the way it will hit the table. If somehow we knew all of those details perfectly, classical physics suggests that the result could, in principle, be predicted. The randomness comes from our ignorance.

Quantum mechanics introduced a stranger possibility. At very small scales, nature may contain events where knowing everything we are allowed to know still does not tell us exactly what will happen next. Physics may only give us probabilities.

For a long time, scientists debated whether there might simply be some hidden information underneath quantum mechanics that we had not discovered yet. Experiments inspired by physicist John Bell made this idea much harder to maintain in its simplest form. They showed that nature does not behave as though particles simply carry ordinary hidden instructions that determine every outcome locally in advance.

The important point does not require knowing quantum mathematics. It is simply that there may be two very different types of randomness: randomness because we do not know enough, and randomness that may actually be built into nature itself.

Modern quantum technology makes this more than a philosophical argument. Researchers can now use quantum experiments to generate randomness with stronger guarantees that the results were not simply produced from a secret predetermined list. As quantum systems improve, we are becoming better at experimentally testing where unpredictability comes from rather than treating it only as a theoretical debate.

For me, this changes the conclusion in an important way. I would not say that everything happens for a reason, because that statement goes further than we can justify. I would say that when something looks random, it is usually worth searching for the mechanism before deciding that no mechanism exists.

That leads to a useful way of thinking about decisions. First ask what hidden conditions might change the probability of what you are seeing. Then ask what incentives and counterforces are keeping the system in its current state. Finally, ask how much uncertainty remains even after you understand those forces.

This does not let us predict everything, but it changes the quality of the questions we ask. Instead of saying, “This happened randomly,” we start asking what produced it. Instead of assuming a trend will continue forever, we ask what pressure the trend itself is creating. Instead of becoming certain because we found a convincing story, we leave room for uncertainty.

The world may contain genuine randomness, but a surprising amount of what looks random is simply a system whose variables we cannot yet see. Learning to search for those variables, the incentives behind them and the forces pushing the system toward its next state is often what helps us understand not only what might happen, but what to do, when to do it and how confident we should be.