Aligning Correlation Information for Domain Adaptation in Action Recognition

TL;DR

This paper introduces ACAN, a novel adversarial network that aligns correlation features for effective domain adaptation in video action recognition, and presents a new dataset with larger domain shifts to evaluate performance.

Contribution

The paper proposes ACAN, a new method for aligning correlation features in video domain adaptation, and introduces the HMDB-ARID dataset with significant domain shifts.

Findings

ACAN achieves state-of-the-art results on existing video DA datasets.

ACAN effectively reduces correlation discrepancies across domains.

The HMDB-ARID dataset provides a challenging benchmark for dark video classification.

Abstract

Domain adaptation (DA) approaches address domain shift and enable networks to be applied to different scenarios. Although various image DA approaches have been proposed in recent years, there is limited research towards video DA. This is partly due to the complexity in adapting the different modalities of features in videos, which includes the correlation features extracted as long-term dependencies of pixels across spatiotemporal dimensions. The correlation features are highly associated with action classes and proven their effectiveness in accurate video feature extraction through the supervised action recognition task. Yet correlation features of the same action would differ across domains due to domain shift. Therefore we propose a novel Adversarial Correlation Adaptation Network (ACAN) to align action videos by aligning pixel correlations. ACAN aims to minimize the distribution of correlation information, termed as Pixel Correlation Discrepancy (PCD). Additionally, video DA research is also limited by the lack of cross-domain video datasets with larger domain shifts. We, therefore, introduce a novel HMDB-ARID dataset with a larger domain shift caused by a larger statistical difference between domains. This dataset is built in an effort to leverage current datasets for dark video classification. Empirical results demonstrate the state-of-the-art performance of our proposed ACAN for both existing and the new video DA datasets.