The Problem With Patterns
Humans are natural pattern recognizers. Whether, as in prehistoric times, we were recognizing danger in a telltale rustle of the bushes or skimming a page of letters and numbers today, we use patterns to derive meaning without having to do a more detailed inspection.
Futurist and entrepreneur Ray Kurzweil considers pattern recognition so important that in his recent book, How to Create a Mind, he argued that pattern recognition and intelligence are essentially the same thing. Expertise, in essence, is the familiarity of patterns of a specific field.
Today, machines are learning to recognize patterns as well. Marketers analyze patterns to target their messages and IBM’s Watson can detect patterns embedded in millions of documents in fields such as medicine. However, there’s a problem with patterns. Just as they can uncover hidden meaning, they can also make us see things that aren’t really there.
The Law Of Entropy And The Need To See Order
Most people learn in high school science class about the law of entropy—the universe tends toward disorder. Just as our bedrooms get messy over time if we don’t straighten them up and gases tend to disperse across a room, we expect most things in this world to be random instead of ordered.
A related concept is that of cause and effect. We expect meaning in the patterns we see because, in a random universe, it takes energy to create order. So when we see a particular pattern, we expect that through investigation we can identify the force that caused it. That’s how we learn new things.
Yet as Aristotle pointed out, this line of thinking leaves us with a conundrum. If everything has a cause, then those causes must have causes themselves, resulting in an infinite regress. Therefore, there must be a first cause—an unmoved mover—at the beginning of it all. This cosmological argument logically leads to the idea that every effect must have a cause.
But taking the concept too far can lead us into trouble. Just because a pattern exists, doesn’t mean that the cause of that pattern is important or meaningful.
The Bible Code
One particularly famous exercise in pattern recognition is the Bible Code, made popular by Michael Drosnin’s bestselling book, which uses pattern recognition to unlock prophecies and other messages from an analysis of the sacred text.
In Drosnin’s version of the technique, he takes fifth letter from each sentence to reveal the hidden messages. By doing so he identified passages that predicted the assassination of Yitzhak Rabin, the rise of Hitler and the moon landing, as well as natural disasters like earthquakes.
Once you get it into your head that every cause is important and meaningful, then obscure patterns become cryptic signs of prophecy. So if we find a pattern of R-A-B-I-N or H-I-T-L-E-R in an ancient text like the bible, we see indications of a deeper force at work.
Surely, such intricate order cannot be a random accident. Or can it?
Ramsey’s Formula For Order From Disorder
In the early 20th century, Cambridge was a great center of human thought. From Bertrand Russell and Ludwig Wittgenstein to John Maynard Keynes and G. H. Hardy and even literary lights like E.M. Forster and Virginia Woolf, virtually every field of human endeavor was being advanced there. Yet none shone brighter than Frank Ramsey.
Although he died from from liver problems at the tender age of 26, Ramsey left behind a profound legacy. Primarily a mathematician, he collaborated with Keynes in economics and with Russell and Wittgenstein in mathematical logic, producing important work in each area. Paul Samuelson called his work in economics “great legacies.”
Yet perhaps Ramsey’s greatest accomplishment was to create a new field of mathematics called Ramsey Theory. What Ramsey proved was that order is the inevitable result of a large amount of random trials. In other words, given a large document like the Bible, there will inevitably be random samples that spell out specific messages.
A related idea is the infinite monkey theorem, which states that if you had an infinite number of monkeys banging away on an infinite number of typewriters for an infinite amount of time, they would, in time, produce the complete works of Shakespeare or Tolstoy or any other literary masterpiece. There’s a fine line between creation and curation.
Incidentally, patterns similar to those found in The Bible Code have also been found in Hebrew translations of the collected works of Shakespeare and War and Peace.
The Search For Patterns
G. H. Hardy once wrote that, “a mathematician, like a painter or poet, is a maker of patterns. If his patterns are more permanent than theirs, it is because they are made with ideas.” Today, as we increasingly live in a world of bits rather than a world of atoms, designing patterns is how we create value.
Yet there is a substantive difference between patterns that we consider to be preordained and those that are to be discovered. If you believe that the patterns of the past determine our future, then you will cling to them dearly. On the other hand, if you believe that the most important patterns are those we have yet to uncover, then the future has no bounds.
Advancement is the discovery of new patterns. Darwin’s theory paved the way for lifesaving drugs, just as Einstein made much of our modern technology possible and the Human Genome Project is advancing miracle cures today. The Bible Code, on the other hand, may nourish the spirit of some, but will neither feed, nor clothe, nor cure anyone.
And that’s the problem with patterns. The human mind is incapable of swallowing them whole, so we curate them instead. Inevitably, what we recognize is our own image. If you seek knowledge that you already believe you possess, then that’s often the most you will ever find.
– Greg
Interesting post, thank you. Which is worse though, finding patterns and insights not there, or never looking at the data? I have seen both, but it is an open mind that is most needed.
Good question. To be honest, I think both are essentially the same. Some have a pattern in their head and go to the data looking to confirm it and others have a pattern in their head and don’t bother. The virtue of the latter approach is that it saves time. It’s one thing to fool yourself, but to actually work at it seems especially foolish:-)
Another nice essay Greg – thanks.
By the way – apparently a website is a requirement for sending a reply.
Is that intentional?
Not required. It’s optional.
Thanks Edward.