Simplifying Open-Set Video Domain Adaptation with Contrastive Learning

TL;DR

This paper introduces a contrastive learning framework for open-set video domain adaptation, effectively distinguishing shared classes from unknown ones by leveraging temporal information in videos.

Contribution

It proposes a unified contrastive learning approach with a novel temporal loss to improve feature discrimination and separation in open-set video domain adaptation.

Findings

Effective separation of known and unknown classes demonstrated

Improved clustering of video features using temporal contrastive loss

Outperforms prior methods on multiple benchmarks

Abstract

In an effort to reduce annotation costs in action recognition, unsupervised video domain adaptation methods have been proposed that aim to adapt a predictive model from a labelled dataset (i.e., source domain) to an unlabelled dataset (i.e., target domain). In this work we address a more realistic scenario, called open-set video domain adaptation (OUVDA), where the target dataset contains "unknown" semantic categories that are not shared with the source. The challenge lies in aligning the shared classes of the two domains while separating the shared classes from the unknown ones. In this work we propose to address OUVDA with an unified contrastive learning framework that learns discriminative and well-clustered features. We also propose a video-oriented temporal contrastive loss that enables our method to better cluster the feature space by exploiting the freely available temporal information in video data. We show that discriminative feature space facilitates better separation of the unknown classes, and thereby allows us to use a simple similarity based score to identify them. We conduct thorough experimental evaluation on multiple OUVDA benchmarks and show the effectiveness of our proposed method against the prior art.