Multisensory Integration and Sensory Substitution Across Vision, Audition, and Haptics: Answering the What, Which, and When in Study Protocols

TL;DR

This paper reviews the complexities of multisensory integration involving three or more senses, emphasizing factors like timing and congruence, and proposes protocols for studying these interactions effectively.

Contribution

It provides a comprehensive analysis of multisensory integration factors and introduces study protocols for exploring three or more sensory modalities.

Findings

Identifies key factors affecting multisensory integration.

Highlights challenges in studying three or more modalities.

Proposes protocols for effective MSI research.

Abstract

We experience the world through multiple senses that work together to create a cohesive perception, whether in daily life or immersive technologies. Understanding this multisensory integration (MSI) requires examining the interactions between sensory modalities, each with unique temporal dynamics and characteristics. While most research focuses on unimodal or bimodal cues, the integration of three or more modalities remains underexplored. MSI studies must account for factors like cross-modal correspondence, congruence, cognitive load, and stimulus timing, which become increasingly complex as modalities multiply. This article examines these key factors and how they can be applied to 8 design effective MSI study protocols.