Real-time Action Recognition with Enhanced Motion Vector CNNs

TL;DR

This paper proposes a real-time action recognition method using motion vector CNNs, which are faster than optical flow-based methods, by transferring knowledge from optical flow CNNs to improve accuracy.

Contribution

It introduces a novel knowledge transfer approach to enhance motion vector CNNs, enabling real-time action recognition with competitive accuracy.

Findings

Achieves 390.7 frames per second, 27 times faster than traditional methods.

Maintains recognition performance comparable to state-of-the-art.

Uses knowledge transfer strategies to compensate for motion vector limitations.

Abstract

The deep two-stream architecture exhibited excellent performance on video based action recognition. The most computationally expensive step in this approach comes from the calculation of optical flow which prevents it to be real-time. This paper accelerates this architecture by replacing optical flow with motion vector which can be obtained directly from compressed videos without extra calculation. However, motion vector lacks fine structures, and contains noisy and inaccurate motion patterns, leading to the evident degradation of recognition performance. Our key insight for relieving this problem is that optical flow and motion vector are inherent correlated. Transferring the knowledge learned with optical flow CNN to motion vector CNN can significantly boost the performance of the latter. Specifically, we introduce three strategies for this, initialization transfer, supervision transfer and their combination. Experimental results show that our method achieves comparable recognition performance to the state-of-the-art, while our method can process 390.7 frames per second, which is 27 times faster than the original two-stream method.