The Creative Search Space

Published Categorized as Articles

This is a summary of the book, ‘Why Greatness Cannot Be Planned – The Myth of the Objective’ by Kenneth Stanley and Joel Lehman, with my own interpretation converted into heuristics that we can use.

Some context

“We can reliably find something amazing. We just cannot say what that something is!”

To achieve anything, there are a number of ‘steps’ or decisions between where we are and the goal. For most goals or problems, the space isn’t too large and we can generally rely on heuristics to guide our decision making. We don’t need to think through every possible way of achieving a basic goal in order to achieve it. 

For slightly more complex problems, we tend rely on objectives often because they feel reassuring. It provides some focus towards the goal – a governing constraint – and they’re often effective for that. The constraint converges the realm of possibility towards that goal. But these objectives often only work when the solution is known and proximate – and by this we mean, the known space between where you are and what you want to achieve is just a few (known) stepping stones. 

However, to achieve anything truly remarkable, great or ambitious, the authors of the book argue that this cannot be done with objectives. In fact, objectives will be a hinderance since they blind us to the greatness we may otherwise achieve. Within the search space the greatness is probably not the immediate objective. The reality is – as they illustrate with examples from inventions to music – that you cannot know what the greatness is beforehand. It is more than a a few stepping stones away. Almost no invention had the actual invention in mind as the original intention, it was usually something else. Furthermore: the invention required the amalgamation of many previous inventions to have taken place first. 

So “Creation and discovery happen within a space of possibilities that contains stepping stones leading from one space to another”.

Here are a few heuristics that that help us to consider how greatness can be achieved. Note that there is a huge overlap with John Vervaeke’s work (Awakening from the Meaning Crisis), and interestingly the authors come to similar conclusions – but starting from a different perspective: with AI and algorithms.


Follow your own instincts and interests

So if you aren’t following an objective, what do you follow? You follow your instinct. Humans are very curious about things that are novel and interesting. If something is interesting, it opens up many new possibilities (increasing the search space). Hence it opens further stepping stones. It’s important to trust this instinct even if you don’t know where it will take you (mainly because you cannot know anyway). Within a community of searching, it’s important that you also follow your own interests (to maximise the divergence as a whole) and allow others to follow theirs, because, at some point you will pick up where others left off!

Pick up where others left off

Many creations were only possible by piggy-backing off other discoveries that came beforehand – often with a radically different intention. A microwave could only be invented after the radio, but the radio never had a microwave in mind. Had someone intended to invent a mircowave, it wouldn’t have been possible, since there were too many invisible stepping sones to achieve the objective. Greatness is the combination of many minds with many different interests and often the breakthrough is by chance, so you need to be prepared.

Be Prepared

Breakthroughs are serendipitous, but they were rarely ‘accidents’, the inventors were already searching for something novel or something interesting and then they spotted it! ‘In the fields of observation, chance favours only the prepared mind’. So the clue here is about novelty. Being prepared here is being open-minded and willing to pursue a different route. If you are too focussed on the objective (which is often the case in a business context, you may not have the bandwidth to pursue the new novelty).

Enjoy and trust the novelty

Objectives can be outperformed by a more clever type of ‘objective ignorance’. It seems surprising but a robot trained for seeking novel spaces in a maze outperformed a robot that was trained to get out of the maze (i.e. the robot that wasn’t looking for the exit, found it, but the robot that was looking for the exit often got stuck). The search is a kind of endless game of ‘hot or cold’ that kids play, with the gradient we want being towards increasing novelty; and novelty providing ever more branches and divergence which lead onto possibilities we couldn’t imagine or foresee. Thus “Creativity is a kind of search”.

Be a treasure hunter / opportunistic explorer

When an opportunity arises grab it! Too often the objective is a deception to this incredible new opportunity – especially when we consider how greatness ‘cannot be planned’ with objectives. Therefore the treasure hunters follow their instinct, and change tack if something novel and interesting appears which will be greater than a ‘pre-planned objective’. Again: following objectives creates a kind of blindness, so look for opportunities not objectives.

Share in the community: collect, filter, share

Allow others to pick up where you left off, in the same way you picked up from where others left off. Create opportunities to somehow collect ‘stepping stones’ in a shared search space. Find some way of filtering the stepping stones (ideally not objective based, so perhaps based on novelty or interestingness with a particular shared interest or theme) and then some way to share widely and discuss the insights. We already have this in many other domains: radio shows about specific music genres or science conferences that share latest research about a particular field with others – find the others! The methods they use can be easily copied.

And now, I’ll leave this following quote as a final thought: 

“Non-objective search is the true source that gives much value in our lives”