A Vision for Multisensory Intelligence: Sensing, Science, and Synergy

TL;DR

This paper envisions a future of multisensory AI that integrates diverse human senses and signals, advancing sensing, science, and synergy to enhance human-AI interaction and understanding.

Contribution

It proposes a comprehensive research framework for multisensory AI, emphasizing sensing, scientific modeling, and synergy, with practical projects and resources from MIT Media Lab.

Findings

New multisensory sensing technologies

Unified models for multimodal interactions

Technical challenges in multisensory integration

Abstract

Our experience of the world is multisensory, spanning a synthesis of language, sight, sound, touch, taste, and smell. Yet, artificial intelligence has primarily advanced in digital modalities like text, vision, and audio. This paper outlines a research vision for multisensory artificial intelligence over the next decade. This new set of technologies can change how humans and AI experience and interact with one another, by connecting AI to the human senses and a rich spectrum of signals from physiological and tactile cues on the body, to physical and social signals in homes, cities, and the environment. We outline how this field must advance through three interrelated themes of sensing, science, and synergy. Firstly, research in sensing should extend how AI captures the world in richer ways beyond the digital medium. Secondly, developing a principled science for quantifying multimodal heterogeneity and interactions, developing unified modeling architectures and representations, and understanding cross-modal transfer. Finally, we present new technical challenges to learn synergy between modalities and between humans and AI, covering multisensory integration, alignment, reasoning, generation, generalization, and experience. Accompanying this vision paper are a series of projects, resources, and demos of latest advances from the Multisensory Intelligence group at the MIT Media Lab, see https://mit-mi.github.io/.