Differentiable Frequency-based Disentanglement for Aerial Video Action Recognition

TL;DR

This paper introduces a novel differentiable frequency-based method for disentangling static and dynamic features in UAV videos to improve human activity recognition, achieving significant performance gains over existing methods.

Contribution

The paper proposes a differentiable frequency mask prior and a specialized cost function to enhance feature disentanglement and frame selection in UAV video action recognition.

Findings

Achieved 5.72% - 13.00% improvement over state-of-the-art methods.

Demonstrated 14.28% - 38.05% improvement over baseline models.

Validated on UAV Human and NEC Drone datasets.

Abstract

We present a learning algorithm for human activity recognition in videos. Our approach is designed for UAV videos, which are mainly acquired from obliquely placed dynamic cameras that contain a human actor along with background motion. Typically, the human actors occupy less than one-tenth of the spatial resolution. Our approach simultaneously harnesses the benefits of frequency domain representations, a classical analysis tool in signal processing, and data driven neural networks. We build a differentiable static-dynamic frequency mask prior to model the salient static and dynamic pixels in the video, crucial for the underlying task of action recognition. We use this differentiable mask prior to enable the neural network to intrinsically learn disentangled feature representations via an identity loss function. Our formulation empowers the network to inherently compute disentangled salient features within its layers. Further, we propose a cost-function encapsulating temporal relevance and spatial content to sample the most important frame within uniformly spaced video segments. We conduct extensive experiments on the UAV Human dataset and the NEC Drone dataset and demonstrate relative improvements of 5.72% - 13.00% over the state-of-the-art and 14.28% - 38.05% over the corresponding baseline model.