Multi-modal Aggregation for Video Classification

TL;DR

This paper introduces a multi-modal aggregation approach for large-scale video classification, leveraging visual, motion, and audio data, achieving top performance by emphasizing temporal-spatial features from 3D convolutions.

Contribution

The paper presents a novel multi-modal aggregation method that effectively combines various modalities, especially highlighting the impact of 3D convolution-based temporal-spatial features.

Findings

Achieved a mAP of 0.8741 on the LSVC2017 test set.

Temporal-spatial features from 3D convolution significantly improved performance.

Ensemble modeling contributed to top-ranking results.

Abstract

In this paper, we present a solution to Large-Scale Video Classification Challenge (LSVC2017) [1] that ranked the 1st place. We focused on a variety of modalities that cover visual, motion and audio. Also, we visualized the aggregation process to better understand how each modality takes effect. Among the extracted modalities, we found Temporal-Spatial features calculated by 3D convolution quite promising that greatly improved the performance. We attained the official metric mAP 0.8741 on the testing set with the ensemble model.