Transductive Model Selection under Prior Probability Shift

TL;DR

This paper introduces a transductive model selection method tailored for prior probability shift scenarios, enabling hyperparameter tuning directly on unlabelled data at application time, which improves performance in anti-causal learning contexts.

Contribution

It proposes a novel transductive hyperparameter optimization method specifically designed for prior probability shift, bypassing traditional cross-validation on labelled data.

Findings

Method improves model selection accuracy under prior probability shift.

Experimental results demonstrate enhanced performance over traditional approaches.

Applicable to anti-causal learning problems with dataset shift.

Abstract

Transductive learning is a supervised machine learning task in which, unlike in traditional inductive learning, the unlabelled data that require labelling are a finite set and are available at training time. Similarly to inductive learning contexts, transductive learning contexts may be affected by dataset shift, i.e., may be such that the IID assumption does not hold. We here propose a method, tailored to transductive classification contexts, for performing model selection (i.e., hyperparameter optimisation) when the data exhibit prior probability shift, an important type of dataset shift typical of anti-causal learning problems. In our proposed method the hyperparameters can be optimised directly on the unlabelled data to which the trained classifier must be applied; this is unlike traditional model selection methods, that are based on performing cross-validation on the labelled training data. We provide experimental results that show the benefits brought about by our method.