Depth2Action: Exploring Embedded Depth for Large-Scale Action Recognition

TL;DR

This paper introduces Depth2Action, a novel framework that leverages estimated depth information from videos to improve large-scale human action recognition, achieving state-of-the-art results by integrating depth cues with appearance and motion data.

Contribution

The paper presents a new depth-based approach with innovative normalization and motion mapping techniques, enhancing action recognition performance on large-scale video benchmarks.

Findings

Depth2Action achieves state-of-the-art accuracy.

Depth cues are complementary to appearance and motion.

Proposed methods improve temporal consistency in depth estimation.

Abstract

This paper performs the first investigation into depth for large-scale human action recognition in video where the depth cues are estimated from the videos themselves. We develop a new framework called depth2action and experiment thoroughly into how best to incorporate the depth information. We introduce spatio-temporal depth normalization (STDN) to enforce temporal consistency in our estimated depth sequences. We also propose modified depth motion maps (MDMM) to capture the subtle temporal changes in depth. These two components significantly improve the action recognition performance. We evaluate our depth2action framework on three large-scale action recognition video benchmarks. Our model achieves state-of-the-art performance when combined with appearance and motion information thus demonstrating that depth2action is indeed complementary to existing approaches.