We need uncertainty
Of all notions in science, randomness has always intrigued me. Much of our reality is shaped by probabilities according to quantum mechanics. But for me probability theory often feels like a fantasy theory, as if it was originating from a brilliant science fiction book.
The reason for this is, unlike other natural phenomena, probability (and statistics in general) cannot be directly observed in a single instance. Probability theory is deducted by observations over a large number of instances.
In addition, humans cannot reproduce a truly random process. Our computers work as a set of deterministic instructions. Everything inside a computer is either a 1 or o, not a 50% chance of being a 1 and a 50% chance of being a 0.
We can borrow random processes from nature, such as the frequency of rain in a location, to generate random numbers but we cannot artificially create them.
Randomness generates uncertainty and not everyone shares my excitement about randomness because of this. In fact, I see more and more evidence that we are in denial about uncertainty being fundamental to our lives. We try to ditch uncertainty in so many facets of life:
Planning down to the day and hours in the productivity religion (I have been guilty of this!)
Feeling safe in the default career work path and considering alternative paths as risky
Believing too strongly in our view of the world as if it is a simple aggregation of deterministic causes and effects
Embracing black box AI models to not have to make ambiguous decisions (e.g. credit scoring models for loan applications that decide the financial fate of thousands or AI generated ratings on the risk of repeat offending used by US judges when sentencing people)
So let me try to convince you that we need uncertainty and it may be the key to navigating life.
The case for injecting some randomness
To solve some optimization problems, an algorithm called hill-climbing is sometimes used in computer science. When following this algorithm, a computer calculates a solution, makes a small adjustment compared the new solution to the previous one and discards whichever one is worse. It repeats this process several times until it lands on a solution that is better than any adjacent solution.
This algorithm is very effective if the problem has a single optimal solution. However, if the problem has multiple unrelated solutions, hill climbing will likely get you stuck on a solution that is decent but not the best.
We also employ this type of optimization in our lives. For example when I was stuck in a job I did not like, I thought my options consisted of trying to find another job in the same field. Often, new job offers had similar or worse conditions (especially if the new employer knows you are desperate to move!). So I ended-up doing nothing and stuck at this local optimal point.
It turns out that to escape this suboptimal solution, you need to inject random irrational moves. A modified hill-climbing technique called the Metropolis algorithm incorporates random chance so that the computer accepts worse solutions short term to avoid getting stuck in the local solution. By allowing suboptimal steps, the algorithm can have the opportunity to explore more possible solution paths and discover the most optimal solution.
In real life, we often climb the wrong hills. We often choose to climb hills which everyone else is climbing: well trodden career paths, for example. These hills are often not the highest ones - whether from a materialistic or self-fulfillment perspective.
But unless we get out of these paths and take the time to experiment by injecting some randomness, we cannot open ourselves to more optimal solutions. Often it is the case, that you need to descent from the small hill you are climbing to climb the higher one.
The case for injecting even more randomness
In How to Live, Derek Sivers takes the argument above to the extreme advocating for a fulfilling life built entirely on randomness. Sivers argues that the world and your life is already significantly affected by randomness. By acknowledging this and making decisions with randomness, your life won't be materially more "risky".
I believe that Sivers' point about our lives being already ruled by randomness becomes clearer whenever we or someone close to us face death. We are reminded it could all stop from one minute to the next before our planned departures.
Sivers argues that making random decisions allows you to encounter valuable experiences that you would have never intentionally chosen. This has the potential to transform you: you will no longer base your identity or self-worth on your career or the way you dress since you didn't chose them.
In some ways, this can lead to a smaller ego. You stop worrying about whether you are making the optimal choice at every step and the decision to step out of a well-known and safe path may also feel less serious or grave. Instead, you are free to live life wholly in the present enjoying its randomness rather than being shaped by it.
In fairness, I do not believe Sivers advocates to make random decisions all the time. His purpose is to show that there isn't a single philosophy or approach to life than can possibly capture all of its complexity. Sometimes making what seems like contradictory steps on the surface is what is called for.
Balanced options to use randomness
In between the incremental randomness injection of the Metropolis algorithm and the life guided by as much random choices as possible, there is a balance which we can strike.
Option 1: Experiment randomly but select deliberately
This path has often led to ground-breaking discoveries in science. Accidents in lab experiments often show scientists some information which they were not aware of. Then, they deliberately pick this new set of information and explore more deeply.
In our lives, this means increasing our exposure to events that could potential change our lives and being active in networks which facilitate new connections. The more "accidents" can happen, the more interesting results you could collect and the more deep-dives you can explore.
Option 2: Reduce randomness over time
As you embark on a new path, it makes sense to increase the number of random choices to learn as much as possible. When you become more familiar with a path, it can be optimal to reduce randomness and stick to a more clear set of choices.
If you want to write online but have no clue what could be both interesting for you and for potential readers, experimenting with as many topics as possible makes sense. Once you become better at communicating ideas, you may want to stick to topics where your writing excels particularly.
Now, if you want to take another path such as creating video content, you could again increase randomness and try out different formats before narrowing your set of style and platform choices.
In summary, while uncertainty may feel uncomfortable and scary, it seems to be fundamental to our universe. Even in deterministic fields such as computer science, randomness is sometimes needed to come to an optimal solution. Moreover, one can argue that our lives are already influenced to a large extent by randomness. There is a benefit then to intentionally embrace it rather than being shaped by it. And since randomness feels still somewhat mystic as if coming from a fantasy world, wouldn't you want to be a wizard who harnesses uncertainty to create, build and explore?