Frequency Selective Augmentation for Video Representation Learning

TL;DR

FreqAug is a novel frequency domain augmentation technique that enhances self-supervised video learning by encouraging models to focus on dynamic features, leading to improved performance across multiple downstream tasks.

Contribution

The paper introduces FreqAug, a frequency-based augmentation method that reduces static information bias in video representations, applicable across various self-supervised learning frameworks.

Findings

Consistent performance improvements on five action recognition tasks.

Enhanced focus on dynamic features in learned representations.

Effective across multiple self-supervised frameworks.

Abstract

Recent self-supervised video representation learning methods focus on maximizing the similarity between multiple augmented views from the same video and largely rely on the quality of generated views. However, most existing methods lack a mechanism to prevent representation learning from bias towards static information in the video. In this paper, we propose frequency augmentation (FreqAug), a spatio-temporal data augmentation method in the frequency domain for video representation learning. FreqAug stochastically removes specific frequency components from the video so that learned representation captures essential features more from the remaining information for various downstream tasks. Specifically, FreqAug pushes the model to focus more on dynamic features rather than static features in the video via dropping spatial or temporal low-frequency components. To verify the generality of the proposed method, we experiment with FreqAug on multiple self-supervised learning frameworks along with standard augmentations. Transferring the improved representation to five video action recognition and two temporal action localization downstream tasks shows consistent improvements over baselines.