New multi-sensory Illusions from CalTech

First, let’s go directly to the new illusions. Once you have personally experienced it we can then dig down into what is going on here.

Just play the clip and follow the instructions as it proceeds …

The Explanation

The Caltech researchers have developed these two new illusions to reveal that your senses can influence each other—in particular, how sound can give rise to visual illusions. These illusions occur so quickly that they illustrate a phenomenon called postdiction (as opposed to prediction) in which a stimulus that occurs later can retroactively affect our perceptions of an earlier event.

That is quite frankly amazing …

• Your brain has received visual information
• A little bit later it also receives audio information. This audio information is then used to alter your perception of the visual information as it struggles to make sense of conflicting signals

This is called postdictive processing. Your brain is cued to expect something, then when something conflicts with that expectation, it will edit what actually happened so that what it expects is perceived to have happened even though it did not actually happen.

There is a paper on this

Titled “What you saw is what you will hear: Two new illusions with audiovisual postdictive effects“, it was published in the Open Access journal PLOS on 3rd October.

Postdictive processing has been previously demonstrated before for a specific sense. What is new here is that they have demonstrated this happening across both our audio and visual senses, indicating the interplay between the two. This is the new bit, hence this is essentially why they published.

Their abstract explains …

We have discovered two related multimodal illusions in which audition instigates postdictive changes in visual perception. In the first illusion (called the “Illusory Audiovisual Rabbit”), the location of an illusory flash is influenced by an auditory beep-flash pair that follows the perceived illusory flash. In the second illusion (called the “Invisible Audiovisual Rabbit”), a beep-flash pair following a real flash suppresses the perception of the earlier flash. Thus, we showed experimentally that these two effects are influenced significantly by postdiction.

What scientific value does this have?

OK, so they have found a couple of interesting illusions. Beyond the pure entertainment, what value is this?

What is really of interest here is to gain an understanding of how the brain works. One means for gaining insights is via illusions such as this because they give researchers a window to see what is going on inside our heads …

“Illusions are a really interesting window into the brain,” study first author Noelle Stiles, from the University of Southern California, said in a statement. “By investigating illusions, we can study the brain’s decision-making process. For example, how does the brain determine reality with information from multiple senses that is at times noisy and conflicting? The brain uses assumptions about the environment to solve this problem. When these assumptions happen to be wrong, illusions can occur as the brain tries to make the best sense of a confusing situation. We can use these illusions to unveil the underlying inferences that the brain makes.”

… and so the insight that these new illusions give is this …

These new additions to auditory-visual interactions indicate a spatio-temporally fine-tuned coupling of the senses to generate perception.

Further Details directly from their published paper

This next bit gets a bit technical, so unless interested in such details, then you can probably stop reading here.

Predictive models and illusions have largely dominated multisensory interactions; they include effects such as speeded audio-visual cuing, temporal ventriloquism, the McGurk Effect auditory-visual language illusion, and Maximum Likelihood Estimation (MLE) in multisensory integration [20–27]. The brain seems to be using prior information stored about each modality, as well as direct attention, to integrate multimodal cues and effectively predict future stimuli. The AV rabbit is not fully explained by these predictive models of audiovisual interaction. In particular, our control experiments indicate that the sensory information presented before the illusory flash is not sufficient to move the illusory flash toward the final flash location. However, the necessity of postdiction to explain the illusory flash location, does not mean that temporal ventriloquism, sensory uncertainty, and MLE do not contribute to generating the illusory flash, binding the auditory-visual stimuli, and generating the perceived audiovisual timing. In effect, the AV rabbit is an excellent example of a multimodal illusion that relies both on predictive processing, as well postdictive processing within the sensory regions of the brain (as indicated in experiments 1.2, 1.3, and 2.2). Our main claim is that in addition to predictive processing, postdiction is also necessary to fully explain this illusion.

Neuroscience research has begun to investigate the emerging area of multimodal postdiction, where later stimuli can affect prior stimuli. In fact, a study showed that the flash-lag effect, which is modeled as postdictive (at least in some theories), can occur crossmodally, or audio-visually [10]. The AV Rabbit Illusions add to this growing body of postdictive effects bridging the senses, and point to a new mechanism for sensory combination within short perceptual time scales. As the number of instances of crossmodal postdiction increases, the neural processing underlying postdictive interactions may become more clear, and thereby broaden the types of processing within the sensory regions of the brain. Furthermore, such crossmodal cases may further bridge the gap between lower-level, unimodal sensory postdiction and higher-level, long-term, cognitive postdiction [1, 26]. In addition, one may speculate that such postdictive processes are universal across various time-scales and various parts/spatial scales of the brain, and they evolved due to a biological advantage of saving memory and yet having predictive power for the future.