Multimodal Systems: Taxonomy, Methods, and Challenges

TL;DR

This paper provides an overview of multimodal systems, discussing their history, advantages, modalities, input modeling, fusion, data collection, and challenges, highlighting their potential to surpass unimodal systems in human-computer interaction.

Contribution

It offers a comprehensive taxonomy and analysis of multimodal systems, emphasizing recent advancements, common modalities, and research challenges in the field.

Findings

Multimodal systems improve task completion rates.

They reduce errors compared to unimodal systems.

Speech and gestures are the most common input modalities.

Abstract

Naturally, humans use multiple modalities to convey information. The modalities are processed both sequentially and in parallel for communication in the human brain, this changes when humans interact with computers. Empowering computers with the capability to process input multimodally is a major domain of investigation in Human-Computer Interaction (HCI). The advancement in technology (powerful mobile devices, advanced sensors, new ways of output, etc.) has opened up new gateways for researchers to design systems that allow multimodal interaction. It is a matter of time when the multimodal inputs will overtake the traditional ways of interactions. The paper provides an introduction to the domain of multimodal systems, explains a brief history, describes advantages of multimodal systems over unimodal systems, and discusses various modalities. The input modeling, fusion, and data collection were discussed. Finally, the challenges in the multimodal systems research were listed. The analysis of the literature showed that multimodal interface systems improve the task completion rate and reduce the errors compared to unimodal systems. The commonly used inputs for multimodal interaction are speech and gestures. In the case of multimodal inputs, late integration of input modalities is preferred by researchers because it allows easy update of modalities and corresponding vocabularies.