Temporal and Semantic Effects on Multisensory Integration

TL;DR

This study investigates how temporal synchrony and semantic similarity influence multisensory integration, revealing distinct and overlapping neural mechanisms for processing these factors during perception.

Contribution

It provides new insights into the neural substrates involved in temporal and semantic multisensory integration, highlighting their distinct and shared pathways.

Findings

Temporal structure engages right frontal cortex.

Semantic incongruence activates left frontal regions.

Both incongruences involve temporal, occipital, and lingual cortices.

Abstract

How do we integrate modality-specific perceptual information arising from the same physical event into a coherent percept? One possibility is that observers rely on information across perceptual modalities that shares temporal structure and/or semantic associations. To explore the contributions of these two factors in multisensory integration, we manipulated the temporal and semantic relationships between auditory and visual information produced by real-world events, such as paper tearing or cards being shuffled. We identified distinct neural substrates for integration based on temporal structure as compared to integration based on event semantics. Semantically incongruent events recruited left frontal regions, while temporally asynchronous events recruited right frontal cortices. At the same time, both forms of incongruence recruited subregions in the temporal, occipital, and lingual cortices. Finally, events that were both temporally and semantically congruent modulated activity in the parahippocampus and anterior temporal lobe. Taken together, these results indicate that low-level perceptual properties such as temporal synchrony and high-level knowledge such as semantics play a role in our coherent perceptual experience of physical events.