Gated Multimodal Units for Information Fusion

TL;DR

This paper introduces the Gated Multimodal Unit (GMU), a neural network component that effectively combines multiple data modalities, demonstrated on movie genre classification, outperforming existing fusion methods.

Contribution

The paper proposes the GMU model for multimodal data fusion and introduces the large MM-IMDb dataset for movie genre prediction.

Findings

GMU improves macro F-score over single-modality models

GMU outperforms other fusion strategies including mixture of experts

The MM-IMDb dataset is the largest publicly available multimodal dataset for this task

Abstract

This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.