A Review on Methods and Applications in Multimodal Deep Learning

TL;DR

This paper reviews recent advancements in multimodal deep learning, analyzing various modalities like image, text, and audio, and discusses baseline methods, applications, challenges, and future research directions.

Contribution

It provides a comprehensive taxonomy of multimodal deep learning methods and a detailed analysis of recent developments from 2017 to 2021.

Findings

Detailed taxonomy of multimodal methods

Analysis of recent advancements (2017-2021)

Identification of key challenges and future directions

Abstract

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning (MMDL) is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of the baseline approaches and an in-depth study of recent advancements during the last five years (2017 to 2021) in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning methods is proposed, elaborating on different applications in more depth. Lastly, main issues are highlighted separately for each domain, along with their possible future research directions.