Open research questions in Multisensory perception and integration

What the literature leaves open

• Solomon SS, Tang H, Sussman E, Kohn A (2021) Limited evidence for sensory prediction error responses in visual cortex of macaques and humans. The insula remains relatively poorly understood, but is thought to be involved in detecting salient events and coordinat- ing large-scale behavioral networks (Menon and Uddin, 2010).

  Human primary auditory cortex and insula encode perceptual decisions, not stimulus features · 2026 · DOI

• Although we were able to conduct a large number of experiments to test our hypotheses in ARTICLE IN PRESS detail, our study suffers from several limitations. We aimed at investigating the subjective experiences associated with retro-perception by using ratings for visibility and confidence on different features of a single stimulus. Our choice of instructions for visibility ratings in Experiment 2 was coarse and may not have allowed individuals to distinguish between features: indeed, both scales shared the instruction to use visibilities bigger than zero as soon as they had a feeling that they might have detected something. Future work will be needed to better characterize participants’ subjective experience at each level independently. As highlighted in studies on sound induced flash illusions, metacognitive awareness can be much more subtle and difficult to reveal than first order effects on behavior, especially in rapid multisensory integration situations67,68. In our case, null-effects for subjective visibility could be specifically reassessed in a dedicated protocol. We provided instructions with a focus on distinguishing between the presence and absence of the stimulus, which may not have been suited to explore subtle differences in the perceptual quality of experiences of word-meaning and word-form. Another major limitation of the present study is that, although we could make sure that partial awareness could not account for the cueing effect we observed, it is also clear that partial awareness did occur across all conditions, in the form of catching a glimpse of a few letters and using this information to respond. This limitation is inherent to our protocol: a balance must be struck between the intensity of visual masking and the unmasking potential of the cue. Indeed, semantic processing of masked words is highly sensitive to masking strength35 and decays rapidly ARTICLE IN PRESS over time31,34. It would however be possible to circumvent this limitation, as masking is not the only method for producing unconscious stimuli: for example, diverting attention away from the target by using the attentional blink69 could shed a complementary light on the results presented here. These results bring important evidence for current debates about the role of sensory processing in conscious perception. They seem to challenge theories postulating that conscious perception is linked with the establishment of local loops within sensory cortices10,12, since visual masking typically disrupts these loops, and hence should definitively preclude conscious perception thereafter30,35,70.

  Consciously detecting and recognizing a past visual word after its sensory trace is gone · 2026 · DOI

• Scientific Journal of Computer Science 2026, 2, 2 https://journal.futuristech.co.id/index.php/sjcs Donatus et al, A Structured Survey of Attention Mechanisms in Audio-Visual Fusion: Architectures, Challenges, and Evaluation Frameworks 241 Table 2. Inclusion and Exclusion Criteria.

  A Structured Survey of Attention Mechanisms in Audio-Visual Fusion: Architectures, Challenges, and Evaluation Frameworks · 2026 · DOI

• It is further suggested that more than 2 levels of difficulty of the n-back task should be considered for further investigation of the relative sensitivity of different DRT stimuli modalities.

  Sensitivity evaluation of the visual, tactile, and auditory detection response task method while driving · 2016 · DOI

• Based on these findings we argue that one of the possible explanations for finding different and sometimes contradictory results in the literature regarding processing and representations of animates and inanimates in the brain could lie in the variability of selected items within each of the categories, that is, homogeneity of the categories.

  A unitary account of conceptual representations of animate/inanimate categories · 2010 · DOI

• Very few investigations have used sequential intermodal presentation (SIP) with amodal relations, and none have used SIP procedures with arbitrary relations.

  Intermodal matching of vision and audition in infancy: A proposal for a new taxonomy · 2007 · DOI

• The specificity of the auditory cortical suppression to self-produced speech, a prediction derived from the posited operation of a precise forward model system, has not been established.

  Fine‐tuning of auditory cortex during speech production · 2005 · DOI

• A second experiment with 8 subjects demonstrated that the effects of attention are limited to the auditory nerve component when using 8000 Hz stimuli but are significant for both the auditory nerve and collicular components when a mid‐frequency (2000 Hz) tone pip is used.

  Human Auditory Attention: The Olivocochlear Bundle May Function as a Peripheral Filter · 1980 · DOI

• In addition to the lack of a more adequate sampling base in terms of number and type of emotions and the problems associated with using role-played rather than real emotions (although the present samples were drawn from a coherent context of a play rather than asking actors for isolated portrayals), the present correlational approach is rather inconclusive.

  Minimal cues in the vocal communication of affect: Judging emotions from content-masked speech · 1972 · DOI

• The author suggests that, should such a study be under- taken, a taxonomy limited to the relatively isolated and de- A PROPOSED TAXONOMY OF PERCEPTION : 4X~ tachable auditory sense might lend itself most readily to un- equivocal interpretation. The third analysis yielded equivocal results because of the lack of an appropriate analytical rotation procedure.

  The perceptual-motor domain and a proposed taxonomy of perception · 1970 · DOI

• Whether they would be less able performers in our echo perception tasks than the original test group of six remains to be seen as logistics problems have so far prevented this testing. On the basis of this limited data it cannot, of course, be concluded that a form of exceptional percep tual development has been identified by this unusual auditory task.

  Perceptual Enhancement in the Early Blind? · 1969 · DOI

• I n sequential processing, the rate of encoding sensory information is limited by the processing rate and the short-term memory capacity of the operator-the length of time the operator can hold the auditory signals in short-term storage prior to encoding. The rate of operating on events in long-term memory would be limited by the remaining free time available when the operator is not encoding sensory or short-term memory events. Information theory and stimulus-independent thought 429 I n conclusion, the unequivocal evidence for the linear regression of stimulus- independent thought on task-information rate supports the original proposition that, as available operator capacity decreases, there is a corresponding decrement in the rate of carrying out the elementary cognitive operations or transformation which precede a subject’s awareness of a thought or image.

  INFORMATION THEORY AND STIMULUS‐INDEPENDENT THOUGHT · 1968 · DOI

• The relevance of all these to the communication of each school COMMENT AND CRITIQUE : 8~ PHOTO 9 PHOTO "i0 subject to each school audience remains to be established.

  Perceptual principles and audiovisual practices · 1964 · DOI

• Unfortunately, its usefulness is limited by the variability each individual manifests from day to day. Assuming that a neural scan operates a t about 10 cycles per second, it was predicted that ( a ) a circular pattern expanding or contracting a t about ten cycles per second would cause perceptual distortion, and ( b ) the perception of motion in such a display would be limited by the velocity of the assumed scan.

  The Temporal Coding of Sensory Data: An Investigation of Two Theories · 1960 · DOI

• , simply are not understood in terms of any comprehensive, Downloaded by [University of Illinois at Urbana-Champaign] at 04:40 12 March 2015 RUSSELL B. T h e extent to which learning theory and personality theory can be formulated and studied scientifically would seem to be at present limited by a lack of understanding and of concise, compre- hensive specification of those aspects (contents, elements, “gestalts,” etc.

  Three Studies Relating Internal Sensory Experiences and Feeling Reactions · 1959 · DOI

• EDWARDS 329 limited data available at the present time greatly limits the possibility of I t is of course apparent that with one exception there is generalizations. T h e one exception is not accounted for except presumably be- cause of insufficient data.

  Accuracy of Auditory Depth Perception · 1955 · DOI

• However, the assumption is warranted that in normal color vision, the visual image is the result of various fusion processes involving all retinal stimuli, and hence, that the normal person receives the same stimuli as does the color blind, but in addition, other stimuli which actually dominate the entire situation and determine the nature and the character of the response. However, a general statement is warranted that the experimental experience was entered into as a totally new experience and the extent and effects of this the light of general clinical in can only be speculated upon understandings. the con- clusion is warranted, namely, that the findings demonstrate clearly that cortical processes and activities can play a highly significant r6le in color vision and can effect results similar if not identical with those based primarily upon peripheral activity and conditions.

  The Induction of Color Blindness by a Technique of Hypnotic Suggestion · 1939 · DOI

• However, despite the vast literature on the detrimental effects of processing speech-in-noise, little is known about the time-course of these effects.

  Bouncing back: Transient, but not long-lasting, effects of background disturbances on speech processing · 2026 · DOI

• Although humans and animals excel at interpreting target sounds in competing noise, sound recognition abilities can vary widely due to statistical characteristics of the interfering background.

  Noise statistics drive background- and channel-specific interference in auditory midbrain representations of speech · 2026 · DOI

• Abstract Perceiving success in group conversations is central to social life, yet little is known about the extent to which the conversation success perceived by conversational partners aligns with that perceived by third-party observers.

  Sound is essential for observers to accurately assess unsuccessful conversations · 2026 · DOI

• The principles governing how repetition may underpin this process are not well understood, particularly for the noisy, approximate repetition typical of biological environments.

  Computational Principles of Auditory Object Formation Are Revealed by Repetition Coherence · 2026 · DOI

• Iconicity is widespread in natural languages, whether signed or spoken, but its instantiation across different domains of meaning has not been systematically studied.

  Domain-specificity in phonological and acoustic characteristics of pseudoword iconicity across multiple domains · 2026 · DOI

• However, whether patients’ confidence is only increased during such erroneous perceptions and whether perceptual and metacognitive decisions arise from supramodal mechanisms across sensory modalities remains unknown.

  Patients with psychotic disorders exhibit different audio-visual perceptual decision biases and metacognitive abilities · 2023 · DOI

• What remains unknown is the mechanism by which these sounds influence infant cognition, and how this initially broader set of privileged sounds narrows to only human speech between 4 and 6 months.

  Developmental changes in auditory‐evoked neural activity underlie infants’ links between language and cognition · 2021 · DOI

• However, only a few studies have questioned whether and how impressions formed via one source can be updated due to incremental information gathered from the alternative source.

  Cross‐modal impression updating: Dynamic impression updating from face to voice and the other way around · 2021 · DOI

Most-cited papers in Multisensory perception and integration

• Informational and energetic masking effects in the perception of two simultaneous talkers · The Journal of the Acoustical Society of America · 2001 · 750 citations
• Informational and energetic masking effects in the perception of multiple simultaneous talkers · The Journal of the Acoustical Society of America · 2001 · 529 citations
• Evaluating the neurophysiological evidence for predictive processing as a model of perception · Annals of the New York Academy of Sciences · 2020 · 297 citations
• Senses of place: architectural design for the multisensory mind · Cognitive Research Principles and Implications · 2020 · 231 citations
• Fine‐tuning of auditory cortex during speech production · Psychophysiology · 2005 · 227 citations
• THE LEGACY OF GUTTMAN AND KALISH (1956): 25 YEARS OF RESEARCH ON STIMULUS GENERALIZATION · Journal of the Experimental Analysis of Behavior · 1981 · 219 citations
• Emotional Responses to Multisensory Environmental Stimuli · SAGE Open · 2016 · 171 citations
• A central capacity limit to the simultaneous storage of visual and auditory arrays in working memory. · Journal of Experimental Psychology General · 2007 · 164 citations
• McGurk effect in non-English listeners: Few visual effects for Japanese subjects hearing Japanese syllables of high auditory intelligibility · The Journal of the Acoustical Society of America · 1991 · 150 citations
• Human Auditory Attention: The Olivocochlear Bundle May Function as a Peripheral Filter · Psychophysiology · 1980 · 150 citations

Most recent work

• Hearing sounds when the eyes move: A case study implicating the tensor tympani in eye movement-related peripheral auditory activity · bioRxiv · 2026
• Domain-specificity in phonological and acoustic characteristics of pseudoword iconicity across multiple domains · bioRxiv · 2026
• Higher-order crossmodal representations in the auditory cortex of deaf and hearing individuals · bioRxiv · 2026
• The addressee’s face as a cue for gesture production · Language Cognition and Neuroscience · 2026
• Quantifying emotion-dependent brain–eye interactions during audiovisual emotional stimulation · Frontiers in Systems Neuroscience · 2026
• Saccade-synchronized alpha rhythms predict strength of memory trace of stimulus orientation · BMC Biology · 2026
• Developmental changes in audio-visual speech integration during the first year of life in infants at elevated and typical likelihood of autism · PLOS One · 2026
• Integrating multiple sensory modalities during dyadic interactions drives self-other distinction at the behavioral and electrocortical level. · PubMed · 2026
• Auditory hemispheric lateralization for speech and melody: from cortex to subcortex · JoCN Forum · 2026
• Factors underlying flicker-induced time dilation: Temporal frequency, semantic content, and subjective saliency of visual stimuli · Journal of Vision · 2026

Related topics in Psychology