Human-Centric Transformer for Domain Adaptive Action Recognition

TL;DR

This paper introduces HCTransformer, a human-centric transformer model that improves domain adaptive action recognition by explicitly focusing on human cues and human-context interactions, leading to state-of-the-art results.

Contribution

The paper proposes a novel human-centric learning paradigm with a decoupled transformer architecture to better exploit human cues for domain adaptation in action recognition.

Findings

Achieves state-of-the-art performance on three benchmark datasets.

Effectively preserves human cues during domain-invariant feature learning.

Demonstrates the importance of human-centric cues in cross-domain action recognition.

Abstract

We study the domain adaptation task for action recognition, namely domain adaptive action recognition, which aims to effectively transfer action recognition power from a label-sufficient source domain to a label-free target domain. Since actions are performed by humans, it is crucial to exploit human cues in videos when recognizing actions across domains. However, existing methods are prone to losing human cues but prefer to exploit the correlation between non-human contexts and associated actions for recognition, and the contexts of interest agnostic to actions would reduce recognition performance in the target domain. To overcome this problem, we focus on uncovering human-centric action cues for domain adaptive action recognition, and our conception is to investigate two aspects of human-centric action cues, namely human cues and human-context interaction cues. Accordingly, our proposed Human-Centric Transformer (HCTransformer) develops a decoupled human-centric learning paradigm to explicitly concentrate on human-centric action cues in domain-variant video feature learning. Our HCTransformer first conducts human-aware temporal modeling by a human encoder, aiming to avoid a loss of human cues during domain-invariant video feature learning. Then, by a Transformer-like architecture, HCTransformer exploits domain-invariant and action-correlated contexts by a context encoder, and further models domain-invariant interaction between humans and action-correlated contexts. We conduct extensive experiments on three benchmarks, namely UCF-HMDB, Kinetics-NecDrone and EPIC-Kitchens-UDA, and the state-of-the-art performance demonstrates the effectiveness of our proposed HCTransformer.