The Magical Mystery Tour

Heuristic today but gone tomorrow. Pattern recognition, or more correctly both true and false pattern recognition is a huge part of what makes us human. Literally immersed in input data from our universe we have evolved methods to segment, synthesize and analyze data sets which allows us to create heuristic patterns (short cuts and approximations) in our brains that we can use to deal with future scenarios more effectively and efficiently. This is an important trait considering the enormous amount of data that streams past us each day. if we can't filter and compact it into some kind of useful patterns, we would be in trouble. The shear size of the data set that just one of us is exposed to each day is amazing. And it is a drop in the cosmic ocean to what constitutes the data within the known universe. I'll circle back to that fact at the end.

A simple example of synthesizing data into useful patterns from many a childhood is the data combination of glowing red round objects, stove, heat, equals burned fingers and pain. Therefore we create a heuristic that prevents us from repeating that same mistake twice in most cases. These patterns are also flexible which allows us to get more mileage out of them. We don't necessarily need a separate blue flame heuristic to avoid getting our fingers singed. This is both good and bad depending on the quality or truth of the pattern. Some heuristics can best be labeled as instinctive behaviors no doubt refined over eons of natural selection. If you don't believe that humans exhibit instinctive behavior you've obviously never been to a singles bar, college commons, high school dance, etc.

Humans have been engaged in both practical and fanciful pattern recognition since before recorded history. We see patterns everywhere. Where they don't exist we make them up. Look at the night sky and you might see a hunter. Or you might just see a set of stars that are separated by hundreds of light years from any plane that our ancestors imagined containing poor Orion. If any messages from the aggregate that is Orion are being sent to any of our Houses even at the speed of light it's going to take a lot longer than the writers of horoscopes are likely to be willing to wait. Every rock, hill, mountain or valley has been assigned some extra-geologic significance at one time or another. Other than a few places like Easter Island or Mount Rushmore there's not much moi to it...

Heuristics and pattern recognition are good things and allow for all sorts of learning and problem solving based on prior experience. They are also potentially dangerous for one simple reason - the method by which we create and retain learned patterns is not required to be based in reality. It's based in feedback experience and repetition. Now in some cases that's fine and dandy such as our hot stove example. The burned finger heuristic reflects the basic physics of heat conduction and rapid oxidation to a high order. But a heuristic need not be grounded in objective reality for it to be adopted and reinforced over time. It just depends upon the criteria we establish to satisfy the conditions for the pattern. The more nebulous or coincidental the easier it is to satisfy the pattern and yell "see, I told you!". In fact as time and positive feedback continue, patterns and associations can actually become more and more resistant to change even in light of strong objective evidence. Not only that, but there is evidence that the more abstract the pattern, the more resistant it is to correction if false. It shouldn't take most of us long to reject a belief in a cold flame heuristic around the stove. But convincing people that fortune cookies really can't predict the future is tougher.

In other words we pattern recognizers often recognize patterns that we create in our minds but which may or may not be true objective patterns within the universe. And once we grab onto one we are loath to discard it. One of the best studied is the fascination with numerology. Large strings of numbers can be 'found' to contain all sorts of interesting patterns that say pretty much whatever the seeker is looking for. If not the Da Vinci Code would be less popular. This rose colored glass aspect of finding corroborating information in large sets of data is called confirmation bias. We establish a pattern of interest and then seek it out in new data sets. We may find the pattern but this has absolutely no bearing on whether the pattern represents anything more than coincident data in large data sets. It is a huge problem in human cognition and one of the aspects that the scientific method specifically combats. Our tendency is to accept patterns and their labels without performing the sort of constant process improvement that might result in our rejection of the pattern. Finding a coincident pattern of data is not the same thing as finding a true association and light years from proving true causation.

Addition: I'm adding this section in light of some comments to the original post for clarification.

Does this mean that only true cause and effect patterns have any validity outside of the arts? Not at all. There are many many instances where useful patterns can be inferred, but I believe this is key - they are logically inferred from other true cause and effect patterns. Epidemiology provides a great example (and one often butchered by lawyers). Take the example of a smokers and exposure to asbestos. We know from true cause and effect studies that smoking and asbestos individually can cause lung cancer. The mechanisms are well established and beyond reproach to anyone but a payed corporate apologist. We also know from studies that the combined risk is about 80x the baseline risk when the two are combined. From these known patterns we can infer that a new individual with mesothelioma who was exposed to asbestos probably got it from the asbestos even though we can't prove it 100%. This is a logical inference that makes perfect sense to anyone but a corporate liability attorney.

How is it possible for people to find these patterns unless they really exist? That is a common retort to any question about debatable patterns. A twisted metaphor may help. Remember the old saw about a huge number of monkeys banging away on typewriters? The story goes that given enough time one of the monkeys will randomly type out the works of Shakespeare. Maybe, maybe not. But one thing is for sure - a human could look at all that typing and would for sure be able to find the works of Shakespeare on those pages and believe that there was some hidden meaning in it... This in large part is how psychics and other con artists make money. Supply a vague enough data set to a hungry mind and the mark will fill in the missing pieces and swear that you are a genius.

False heuristics are aided by a couple of other things as well. The universe supplies us with an almost infinite signal to noise problem similar to the task of finding an FM signal is a sea of noise. Pattern recognition, be it radio or human decision-making is challenged by the pattern recognition system's ability to pick the pattern out of the ocean of other data signals that are present as well. All that data has some meaning yes, but it rarely means what you think it means. For example, part of that static on your TV when tuned to an off air station is actually an echo of the Big Bang, but that doesn't mean that ET is calling you. While we're on that example we have confidence in that statement because this static is a unidirectional signal of precisely the right frequency and power that would be expected if it was steadily losing energy over the last 13 plus billion years.

Radio and TV signals have an easier time plucking useful data out of the ether because we have well defined parameters for the source pattern that aids us in finding the useful bits amongst the static and we know where to look for them. We are certain of what we are looking for. Much of human life lacks such well defined source patterns and we are free to make them up as we go along. As long as the pattern doesn't run headlong into the laws of physics we generally get a buy so false heuristics can flourish. The optimist's assertion that he can fly may be encouraged by the projectile motion he experiences while leaping form the roof of the skyscraper but eventually that 9.8 meter per second per second acceleration of Earth's gravity will close the loop on that particular false belief. Though others might watch the spectacle and be certain that he did not believe enough in his ability to fly and therefore failed (otherwise known as the Christian Scientist model of reality). Such is the power of abstract belief.

There are patterns in the din that are truths - gravity, thermodynamics, kinetic energy transfer, etc. Truths must by definition involve some universal data sets with demonstrable and repeatable cause and effect. But for every truth there are huge numbers of coincident data points that only gain significance when they are gathered into the net of a human's falsely created pattern. The universe appears nonplused by our beliefs one way or the other so most of the time it doesn't rub our noses in it when we are wrong. That doesn't mean that these false patterns have worth just that they have no impact on objective reality so other than us, no one or thing really cares.

For example, the facts of evolution are not warped by the disbelief of humans. We can't wish it away no matter how hard we try. All we can do is rob our society of using these facts to better understand our place in the cosmos and develop more rational ways to manage both the present and our future based upon a more truthful understanding of our actual nature. Better to cover our ears like Miracle Max and yell, "I'm not Listening!" We prefer our stories to the facts so we lose the opportunity to learn from reality. The stories may provide comfort but are not likely to be of much help when next they run up against the true nature of the universe.

With the enormity of the data stream that passes by our senses on a given day it is no wonder that amongst all the coincident data, that one can create associations to almost anything you can imagine. In reality, the amount of coincident data is orders of magnitude greater than the truly associated data which is vastly greater than the true cause and effect relationships. swimming around in the data. Creating heuristics involves establishing some set of data that is labeled as a pattern and linked to some effect. Again, notice that none of this requires that the pattern be truthful in any way. That is really key. As long as a pattern can be reinforced through experience, it will tend to thrive. So if we erroneously create a pattern recognition target the next time we will have an easier time seeing this pattern through a run away positive feedback loop. The more we look for the pattern the more likely we are able to see it and the stronger the urge to find the pattern and the greater is our belief in the validity of the pattern - regardless of any true cause and effect relationships. I leave the reader to consider examples of this phenomenon.

Contributing to all of this is probably the evolutionary history of problem solving. In the animal kingdom we see large numbers of examples of the most rudimentary problem solving relationships: danger avoidance, food gathering, and finding a suitable mate. That's where we came from so much of our more complex cognition must have developed from traits that controlled those basic building blocks. On this site I have repeated described modern man as descendants of those who ran. It's an apt description. Recall the old 'Lions in the tall grass' scenario. When dealing with a camouflaged and stealthy predator having an over developed trait to see facial patterns in all sorts of places along the savanna would probably have been a good thing to pass along to your offspring. Running away from any potentially dangerous pattern of visual inputs might mean that you dropped a meal unnecessarily from time to time but were invited to dinner far less often. What about the budding scientist in the group who insisted on examining true cause and effect? Lunch! This is a somewhat playful supposition of course but it does illustrate the fact that a case can be made for the selective advantage of reacting to perceived patterns absent constant corroboration or complete data. A process that would make us somewhat vulnerable to leaping to false conclusions. We congratulate ourselves for running away and convince ourselves that another lion was avoided whether or not one was present. The pattern is reinforced absent any kind of confirmation. In the case of crouching lions discretion is the better part of valor so this kind of avoidance behavior becomes well ingrained and there may be little selection pressure to be more analytical than running like hell. As long as the pattern is perceived as being valuable it will be retained.

There are many facets to human cognitive bias but our denial of negative data is key. Negative data is that which essentially rules out a certain set of conditions by its presence. It's hugely leveraged in good clinical diagnoses for example. In cause and effect analysis, negative data is usually the most predictive. Human cognition does not naturally accept this fact, riddled with confirmation bias as we are. Confirmation bias by definition is cherry picking of data to support a foregone conclusion. Pesky negative data is ignored. Look around at all the examples where this is alive and well. Again the huge data sets we encounter each day are rich grounds for cherry picking in support of all manner of questionable patterns. Some have argued that might not emphasis on negative data create another kind of bias? Not really because any viable model or pattern that truly represents reality must be able to account for the existence of this negative data. if not, the model is at best incomplete and at worst just plain wrong. There are instances of erroneous data of course but we still must account for even these errors if we want to be complete. Most people are satisfied with a much lower level of validation. If the model can't account for the negatives it does not constitute a false rejection of the positive elements since the model is false to begin with. How can one assess the positivity of data in a false model?

Where does that leave us? Unfortunately to most of us humans cause and effect means that I will assign a cause to what ever gives me the desired effect. Does this mean that people are seeing things that don't exist? Depends on how you define the problem. The raw data probably exists as long as the person is sane. What may or may not exist is any true significance to a specific pattern of data that has been created by an individual to label part of their individual experience. The fact that one 'sees' a pattern is far from proof that such a pattern actually exists in objective reality. A classic example is paranoia. One may see patterns of persecution where none actually exist. The paranoid may be actually interacting with others but the paranoid model projects false meanings on the raw data and infers motives that likely are pure fabrication. Yes, the individual did meet with the paranoid on such and such a date but was not involved in a galactic conspiracy to harvest his brain. It take much more than a personal sense that there is truth in these kinds of assertions. Human senses are extremely easy to confound. True relationships should be observable and predictive. And not retrospectively where confirmation bias can make almost any set of observations look interesting. With the ability to think abstractly the individual can seek refuge from the objective pattern recognition police in all sorts of out of the way places inaccessible to cause and effect analysis. By definition such patterns within the data sets of the universe must include intersections with imagined sets of data outside of this universe's data sets. We aren't talking about data within the universe such as molecular genetics which was unknown to Darwin. We are talking about data which is outside of our ability to detect it. That is a very convenient hiding place. Once that happens logic breaks down unless some objective way to incorporate the metaphysical data can be reliably and reproducibly identified. Those seeking refuge in extensions to the known or knowable will often assert that objective reality is just one way to determine truth. or that the scientific method is limited in some way that prevents it from assessing alternative realities. It can be argued that this is magical thinking seeking magical patterns within magical data sets. The existence of some reality beyond that which we can evaluate is a huge stretch that requires more than vigorous assertion to be convincing. So what is the moral of this story? We know that humans are vulnerable to all sorts of cognitive biases that result in our seeing patterns in data where no truth really exists. When this is put into the context of the immensity of the data sets in this universe it must make one pause - Logical assessment of the enormity of the ever expanding data set that comprises this universe and our human propensity for false pattern recognition suggests that the failure to identify relevant explanatory patterns within the natural confines of the universe is more likely due to faulty or inadequate human pattern recognition than any missing data or requirement for super-universal data sets.

NOTE: please forgive the unforgivable pun. Well, actually 2. And an ever so slight modification to a cultural icon.