XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification

TL;DR

XFlow introduces cross-modal deep neural networks that enable data exchange between audio and visual streams, improving audiovisual classification by exploiting correlations and achieving state-of-the-art results on multiple datasets.

Contribution

The paper presents novel cross-modality dataflow architectures and extends cross-connections to non-compatible data, enhancing multimodal learning capabilities.

Findings

Models outperform baselines by up to 11.5%

Achieve state-of-the-art results on AVletters, CUAVE, and Digits datasets

Learn interpretable features that improve discrimination ability

Abstract

In recent years, there have been numerous developments towards solving multimodal tasks, aiming to learn a stronger representation than through a single modality. Certain aspects of the data can be particularly useful in this case - for example, correlations in the space or time domain across modalities - but should be wisely exploited in order to benefit from their full predictive potential. We propose two deep learning architectures with multimodal cross-connections that allow for dataflow between several feature extractors (XFlow). Our models derive more interpretable features and achieve better performances than models which do not exchange representations, usefully exploiting correlations between audio and visual data, which have a different dimensionality and are nontrivially exchangeable. Our work improves on existing multimodal deep learning algorithms in two essential ways: (1) it presents a novel method for performing cross-modality (before features are learned from individual modalities) and (2) extends the previously proposed cross-connections which only transfer information between streams that process compatible data. Illustrating some of the representations learned by the connections, we analyse their contribution to the increase in discrimination ability and reveal their compatibility with a lip-reading network intermediate representation. We provide the research community with Digits, a new dataset consisting of three data types extracted from videos of people saying the digits 0-9. Results show that both cross-modal architectures outperform their baselines (by up to 11.5%) when evaluated on the AVletters, CUAVE and Digits datasets, achieving state-of-the-art results.