Correlation Net: Spatiotemporal multimodal deep learning for action recognition

TL;DR

This paper introduces a correlation network that captures spatiotemporal multimodal correlations to improve action recognition accuracy in videos, addressing fusion and overfitting issues in deep CNNs.

Contribution

It proposes a novel correlation network with Shannon fusion that complements existing CNN-based action recognition methods.

Findings

Enhanced recognition accuracy on UCF-101 and HMDB-51 datasets.

Correlation network effectively captures long-range spatiotemporal correlations.

Complemented existing multimodal fusion approaches.

Abstract

This paper describes a network that captures multimodal correlations over arbitrary timestamps. The proposed scheme operates as a complementary, extended network over a multimodal convolutional neural network (CNN). Spatial and temporal streams are required for action recognition by a deep CNN, but overfitting reduction and fusing these two streams remain open problems. The existing fusion approach averages the two streams. Here we propose a correlation network with a Shannon fusion for learning a pre-trained CNN. A Long-range video may consist of spatiotemporal correlations over arbitrary times, which can be captured by forming the correlation network from simple fully connected layers. This approach was found to complement the existing network fusion methods. The importance of multimodal correlation is validated in comparison experiments on the UCF-101 and HMDB-51 datasets. The multimodal correlation enhanced the accuracy of the video recognition results.