Dense Multimodal Fusion for Hierarchically Joint Representation

TL;DR

This paper introduces Dense Multimodal Fusion (DMF), a hierarchical feature integration method that stacks shared layers between modality-specific networks, capturing correlations at multiple levels for improved multimodal learning.

Contribution

It proposes a novel dense fusion approach that captures hierarchical correlations across modalities, leading to faster convergence and better performance.

Findings

Improved performance on audiovisual speech recognition

Enhanced cross-modal retrieval accuracy

Lower training loss and faster convergence

Abstract

Multiple modalities can provide more valuable information than single one by describing the same contents in various ways. Hence, it is highly expected to learn effective joint representation by fusing the features of different modalities. However, previous methods mainly focus on fusing the shallow features or high-level representations generated by unimodal deep networks, which only capture part of the hierarchical correlations across modalities. In this paper, we propose to densely integrate the representations by greedily stacking multiple shared layers between different modality-specific networks, which is named as Dense Multimodal Fusion (DMF). The joint representations in different shared layers can capture the correlations in different levels, and the connection between shared layers also provides an efficient way to learn the dependence among hierarchical correlations. These two properties jointly contribute to the multiple learning paths in DMF, which results in faster convergence, lower training loss, and better performance. We evaluate our model on three typical multimodal learning tasks, including audiovisual speech recognition, cross-modal retrieval, and multimodal classification. The noticeable performance in the experiments demonstrates that our model can learn more effective joint representation.