Object-based (yet Class-agnostic) Video Domain Adaptation

TL;DR

This paper introduces ODAPT, a simple object-based framework for video domain adaptation that leverages sparse, class-agnostic object annotations to improve action recognition across different environments.

Contribution

The paper proposes ODAPT, a novel, general framework for video domain adaptation that effectively utilizes sparse, class-agnostic object annotations, enhancing existing methods.

Findings

Achieves +6.5 accuracy improvement across kitchens in Epic-Kitchens.

Attains +3.1 accuracy boost between Epic-Kitchens and EGTEA datasets.

Enhances performance when combined with existing unsupervised methods.

Abstract

Existing video-based action recognition systems typically require dense annotation and struggle in environments when there is significant distribution shift relative to the training data. Current methods for video domain adaptation typically fine-tune the model using fully annotated data on a subset of target domain data or align the representation of the two domains using bootstrapping or adversarial learning. Inspired by the pivotal role of objects in recent supervised object-centric action recognition models, we present Object-based (yet Class-agnostic) Video Domain Adaptation (ODAPT), a simple yet effective framework for adapting the existing action recognition systems to new domains by utilizing a sparse set of frames with class-agnostic object annotations in a target domain. Our model achieves a +6.5 increase when adapting across kitchens in Epic-Kitchens and a +3.1 increase adapting between Epic-Kitchens and the EGTEA dataset. ODAPT is a general framework that can also be combined with previous unsupervised methods, offering a +5.0 boost when combined with the self-supervised multi-modal method MMSADA and a +1.7 boost when added to the adversarial-based method TA$^3$N on Epic-Kitchens.