Learnable Motion-Focused Tokenization for Effective and Efficient Video Unsupervised Domain Adaptation

TL;DR

This paper introduces LMFT, a learnable tokenization method that focuses on motion-rich regions in videos to improve unsupervised domain adaptation for action recognition, achieving state-of-the-art results efficiently.

Contribution

The paper proposes a novel learnable motion-focused tokenization approach that discards background tokens, enhancing VUDA performance and computational efficiency.

Findings

Achieves state-of-the-art performance on three VUDA benchmarks.

Reduces computational overhead significantly.

Effectively discards background tokens, focusing on motion-rich regions.

Abstract

Video Unsupervised Domain Adaptation (VUDA) poses a significant challenge in action recognition, requiring the adaptation of a model from a labeled source domain to an unlabeled target domain. Despite recent advances, existing VUDA methods often fall short of fully supervised performance, a key reason being the prevalence of static and uninformative backgrounds that exacerbate domain shifts. Additionally, prior approaches largely overlook computational efficiency, limiting real-world adoption. To address these issues, we propose Learnable Motion-Focused Tokenization (LMFT) for VUDA. LMFT tokenizes video frames into patch tokens and learns to discard low-motion, redundant tokens, primarily corresponding to background regions, while retaining motion-rich, action-relevant tokens for adaptation. Extensive experiments on three standard VUDA benchmarks across 21 domain adaptation settings show that our VUDA framework with LMFT achieves state-of-the-art performance while significantly reducing computational overhead. LMFT thus enables VUDA that is both effective and computationally efficient.