Label Alignment Regularization for Distribution Shift

TL;DR

This paper introduces a novel regularization technique for unsupervised domain adaptation that leverages label alignment properties and singular vector analysis to improve classifier alignment with target domain data, outperforming traditional methods.

Contribution

The paper proposes a new regularization method based on label alignment and singular vectors, removing the need for joint risk assumptions and enhancing domain adaptation performance.

Findings

Improved accuracy on MNIST-USPS domain adaptation

Enhanced performance in cross-lingual sentiment analysis

Theoretical validation of the method's alignment properties

Abstract

Recent work has highlighted the label alignment property (LAP) in supervised learning, where the vector of all labels in the dataset is mostly in the span of the top few singular vectors of the data matrix. Drawing inspiration from this observation, we propose a regularization method for unsupervised domain adaptation that encourages alignment between the predictions in the target domain and its top singular vectors. Unlike conventional domain adaptation approaches that focus on regularizing representations, we instead regularize the classifier to align with the unsupervised target data, guided by the LAP in both the source and target domains. Theoretical analysis demonstrates that, under certain assumptions, our solution resides within the span of the top right singular vectors of the target domain data and aligns with the optimal solution. By removing the reliance on the commonly used optimal joint risk assumption found in classic domain adaptation theory, we showcase the effectiveness of our method on addressing problems where traditional domain adaptation methods often fall short due to high joint error. Additionally, we report improved performance over domain adaptation baselines in well-known tasks such as MNIST-USPS domain adaptation and cross-lingual sentiment analysis.