Exploiting Spatial-Temporal Modelling and Multi-Modal Fusion for Human Action Recognition

TL;DR

This paper introduces a novel spatial-temporal network (StNet) and an improved multi-modal fusion strategy (iTXN) for human action recognition, achieving top performance on the Kinetics-600 challenge.

Contribution

It proposes StNet for enhanced joint spatial-temporal video modeling and an improved fusion method (iTXN) for integrating multiple modalities, advancing state-of-the-art results.

Findings

StNet RGB model achieves 78.99% top-1 accuracy.

Multi-modal iTXN reaches 82.35% accuracy.

Ensemble method achieves 85.0% top-1 accuracy, ranking first.

Abstract

In this report, our approach to tackling the task of ActivityNet 2018 Kinetics-600 challenge is described in detail. Though spatial-temporal modelling methods, which adopt either such end-to-end framework as I3D \cite{i3d} or two-stage frameworks (i.e., CNN+RNN), have been proposed in existing state-of-the-arts for this task, video modelling is far from being well solved. In this challenge, we propose spatial-temporal network (StNet) for better joint spatial-temporal modelling and comprehensively video understanding. Besides, given that multi-modal information is contained in video source, we manage to integrate both early-fusion and later-fusion strategy of multi-modal information via our proposed improved temporal Xception network (iTXN) for video understanding. Our StNet RGB single model achieves 78.99\% top-1 precision in the Kinetics-600 validation set and that of our improved temporal Xception network which integrates RGB, flow and audio modalities is up to 82.35\%. After model ensemble, we achieve top-1 precision as high as 85.0\% on the validation set and rank No.1 among all submissions.