Modality-Collaborative Low-Rank Decomposers for Few-Shot Video Domain Adaptation

TL;DR

This paper introduces a novel framework called MC-LRD for few-shot video domain adaptation, effectively decomposing and aligning multimodal features to improve cross-domain generalization in videos.

Contribution

We propose Modality-Collaborative Low-Rank Decomposers (MC-LRD), a new method that decomposes modality-specific and shared features with domain-aware alignment for few-shot video adaptation.

Findings

Significant performance improvements on three benchmarks.

Effective decomposition of modality-unique and shared features.

Enhanced domain alignment through cross-domain activation consistency.

Abstract

In this paper, we study the challenging task of Few-Shot Video Domain Adaptation (FSVDA). The multimodal nature of videos introduces unique challenges, necessitating the simultaneous consideration of both domain alignment and modality collaboration in a few-shot scenario, which is ignored in previous literature. We observe that, under the influence of domain shift, the generalization performance on the target domain of each individual modality, as well as that of fused multimodal features, is constrained. Because each modality is comprised of coupled features with multiple components that exhibit different domain shifts. This variability increases the complexity of domain adaptation, thereby reducing the effectiveness of multimodal feature integration. To address these challenges, we introduce a novel framework of Modality-Collaborative LowRank Decomposers (MC-LRD) to decompose modality-unique and modality-shared features with different domain shift levels from each modality that are more friendly for domain alignment. The MC-LRD comprises multiple decomposers for each modality and Multimodal Decomposition Routers (MDR). Each decomposer has progressively shared parameters across different modalities. The MDR is leveraged to selectively activate the decomposers to produce modality-unique and modality-shared features. To ensure efficient decomposition, we apply orthogonal decorrelation constraints separately to decomposers and subrouters, enhancing their diversity. Furthermore, we propose a cross-domain activation consistency loss to guarantee that target and source samples of the same category exhibit consistent activation preferences of the decomposers, thereby facilitating domain alignment. Extensive experimental results on three public benchmarks demonstrate that our model achieves significant improvements over existing methods.