Towards Good Practices for Multi-modal Fusion in Large-scale Video Classification

TL;DR

This paper introduces multi-modal factorized bilinear pooling (MFB) for fusing visual and audio features, significantly improving large-scale video classification performance on the Youtube-8M v2 dataset.

Contribution

The paper proposes a novel MFB method for multimodal fusion, outperforming simple fusion techniques in large-scale video classification tasks.

Findings

MFB outperforms simple fusion methods on Youtube-8M v2 dataset

Multimodal fusion with MFB improves classification accuracy

Experimental results validate the effectiveness of MFB in large-scale settings

Abstract

Leveraging both visual frames and audio has been experimentally proven effective to improve large-scale video classification. Previous research on video classification mainly focuses on the analysis of visual content among extracted video frames and their temporal feature aggregation. In contrast, multimodal data fusion is achieved by simple operators like average and concatenation. Inspired by the success of bilinear pooling in the visual and language fusion, we introduce multi-modal factorized bilinear pooling (MFB) to fuse visual and audio representations. We combine MFB with different video-level features and explore its effectiveness in video classification. Experimental results on the challenging Youtube-8M v2 dataset demonstrate that MFB significantly outperforms simple fusion methods in large-scale video classification.