Kernel Robust Bias-Aware Prediction under Covariate Shift

TL;DR

This paper introduces a kernel-based robust bias-aware prediction method designed to handle covariate shift, providing improved performance and theoretical guarantees over existing approaches in biased and natural shift scenarios.

Contribution

It extends the representer theorem to the RBA setting and develops a kernel-based approach with consistency guarantees for covariate shift adaptation.

Findings

Outperforms competing methods on synthetic biased datasets

Demonstrates effectiveness on datasets with natural covariate shift

Provides theoretical consistency guarantees for the proposed method

Abstract

Under covariate shift, training (source) data and testing (target) data differ in input space distribution, but share the same conditional label distribution. This poses a challenging machine learning task. Robust Bias-Aware (RBA) prediction provides the conditional label distribution that is robust to the worstcase logarithmic loss for the target distribution while matching feature expectation constraints from the source distribution. However, employing RBA with insufficient feature constraints may result in high certainty predictions for much of the source data, while leaving too much uncertainty for target data predictions. To overcome this issue, we extend the representer theorem to the RBA setting, enabling minimization of regularized expected target risk by a reweighted kernel expectation under the source distribution. By applying kernel methods, we establish consistency guarantees and demonstrate better performance of the RBA classifier than competing methods on synthetically biased UCI datasets as well as datasets that have natural covariate shift.