Theory-inspired Label Shift Adaptation via Aligned Distribution Mixture

TL;DR

This paper introduces the Aligned Distribution Mixture framework for label shift adaptation, improving classifier training by theoretically justified distribution blending and coupling strategies, with applications including COVID-19 diagnosis.

Contribution

It proposes the concept of aligned distribution mixture, providing theoretical optimality and generalization bounds, and enhances existing label shift methods with a novel one-step approach and bi-level optimization.

Findings

Theoretical analysis shows the superiority of aligned distribution mixture over direct blending.

Enhanced label shift methods achieve better adaptation performance.

Experimental results confirm effectiveness in COVID-19 diagnosis applications.

Abstract

As a prominent challenge in addressing real-world issues within a dynamic environment, label shift, which refers to the learning setting where the source (training) and target (testing) label distributions do not match, has recently received increasing attention. Existing label shift methods solely use unlabeled target samples to estimate the target label distribution, and do not involve them during the classifier training, resulting in suboptimal utilization of available information. One common solution is to directly blend the source and target distributions during the training of the target classifier. However, we illustrate the theoretical deviation and limitations of the direct distribution mixture in the label shift setting. To tackle this crucial yet unexplored issue, we introduce the concept of aligned distribution mixture, showcasing its theoretical optimality and generalization error bounds. By incorporating insights from generalization theory, we propose an innovative label shift framework named as Aligned Distribution Mixture (ADM). Within this framework, we enhance four typical label shift methods by introducing modifications to the classifier training process. Furthermore, we also propose a one-step approach that incorporates a pioneering coupling weight estimation strategy. Considering the distinctiveness of the proposed one-step approach, we develop an efficient bi-level optimization strategy. Experimental results demonstrate the effectiveness of our approaches, together with their effectiveness in COVID-19 diagnosis applications.