Dynamic Appearance: A Video Representation for Action Recognition with Joint Training

TL;DR

This paper introduces Dynamic Appearance, a novel video representation capturing motion-related features by filtering static appearance, and proposes Pixel-Wise Temporal Projection integrated with CNNs or Transformers for improved action recognition.

Contribution

The paper presents the concept of Dynamic Appearance and the Pixel-Wise Temporal Projection method, enabling efficient motion feature extraction and end-to-end training for action recognition.

Findings

Effective motion feature extraction demonstrated on multiple benchmarks.

Improved action recognition accuracy over baseline models.

End-to-end training framework enhances model performance.

Abstract

Static appearance of video may impede the ability of a deep neural network to learn motion-relevant features in video action recognition. In this paper, we introduce a new concept, Dynamic Appearance (DA), summarizing the appearance information relating to movement in a video while filtering out the static information considered unrelated to motion. We consider distilling the dynamic appearance from raw video data as a means of efficient video understanding. To this end, we propose the Pixel-Wise Temporal Projection (PWTP), which projects the static appearance of a video into a subspace within its original vector space, while the dynamic appearance is encoded in the projection residual describing a special motion pattern. Moreover, we integrate the PWTP module with a CNN or Transformer into an end-to-end training framework, which is optimized by utilizing multi-objective optimization algorithms. We provide extensive experimental results on four action recognition benchmarks: Kinetics400, Something-Something V1, UCF101 and HMDB51.