Calibrating Class Weights with Multi-Modal Information for Partial Video Domain Adaptation

TL;DR

This paper introduces MCAN, a novel method for partial video domain adaptation that leverages multi-modal features and a class weight calibration technique to improve cross-domain video classification accuracy.

Contribution

The paper proposes MCAN, which enhances feature extraction with multi-modal data and introduces a class weight calibration method to reduce negative transfer in PVDA.

Findings

MCAN outperforms state-of-the-art PVDA methods on benchmark datasets.

Multi-modal features improve robustness against domain shifts.

Calibration effectively reduces negative transfer caused by incorrect class weights.

Abstract

Assuming the source label space subsumes the target one, Partial Video Domain Adaptation (PVDA) is a more general and practical scenario for cross-domain video classification problems. The key challenge of PVDA is to mitigate the negative transfer caused by the source-only outlier classes. To tackle this challenge, a crucial step is to aggregate target predictions to assign class weights by up-weighing target classes and down-weighing outlier classes. However, the incorrect predictions of class weights can mislead the network and lead to negative transfer. Previous works improve the class weight accuracy by utilizing temporal features and attention mechanisms, but these methods may fall short when trying to generate accurate class weight when domain shifts are significant, as in most real-world scenarios. To deal with these challenges, we propose the Multi-modality Cluster-calibrated partial Adversarial Network (MCAN). MCAN enhances video feature extraction with multi-modal features from multiple temporal scales to form more robust overall features. It utilizes a novel class weight calibration method to alleviate the negative transfer caused by incorrect class weights. The calibration method tries to identify and weigh correct and incorrect predictions using distributional information implied by unsupervised clustering. Extensive experiments are conducted on prevailing PVDA benchmarks, and the proposed MCAN achieves significant improvements when compared to state-of-the-art PVDA methods.