Distributional Shift Adaptation using Domain-Specific Features

TL;DR

This paper introduces a domain-specific feature-based method for adapting machine learning models to out-of-distribution data, leveraging confident predictions to improve performance significantly over state-of-the-art approaches.

Contribution

It proposes a novel approach that uses correlations in both invariant and domain-specific features to adapt models to new target domains effectively.

Findings

Performance improved by approximately 10-20% over SOTA methods.

The approach effectively leverages confident predictions for domain adaptation.

Empirical results on benchmark datasets validate the method's effectiveness.

Abstract

Machine learning algorithms typically assume that the training and test samples come from the same distributions, i.e., in-distribution. However, in open-world scenarios, streaming big data can be Out-Of-Distribution (OOD), rendering these algorithms ineffective. Prior solutions to the OOD challenge seek to identify invariant features across different training domains. The underlying assumption is that these invariant features should also work reasonably well in the unlabeled target domain. By contrast, this work is interested in the domain-specific features that include both invariant features and features unique to the target domain. We propose a simple yet effective approach that relies on correlations in general regardless of whether the features are invariant or not. Our approach uses the most confidently predicted samples identified by an OOD base model (teacher model) to train a new model (student model) that effectively adapts to the target domain. Empirical evaluations on benchmark datasets show that the performance is improved over the SOTA by ~10-20%