Distributionally Robust Post-hoc Classifiers under Prior Shifts

TL;DR

This paper introduces a lightweight post-hoc method for adjusting pre-trained classifiers to improve robustness against distribution shifts caused by changes in class or group priors, with theoretical guarantees and empirical validation.

Contribution

It proposes a novel, constrained optimization-based post-hoc scaling approach to enhance distributional robustness of classifiers under prior shifts.

Findings

Method improves robustness to prior shifts.

Provable guarantees for the proposed approach.

Empirical results demonstrate effectiveness.

Abstract

The generalization ability of machine learning models degrades significantly when the test distribution shifts away from the training distribution. We investigate the problem of training models that are robust to shifts caused by changes in the distribution of class-priors or group-priors. The presence of skewed training priors can often lead to the models overfitting to spurious features. Unlike existing methods, which optimize for either the worst or the average performance over classes or groups, our work is motivated by the need for finer control over the robustness properties of the model. We present an extremely lightweight post-hoc approach that performs scaling adjustments to predictions from a pre-trained model, with the goal of minimizing a distributionally robust loss around a chosen target distribution. These adjustments are computed by solving a constrained optimization problem on a validation set and applied to the model during test time. Our constrained optimization objective is inspired by a natural notion of robustness to controlled distribution shifts. Our method comes with provable guarantees and empirically makes a strong case for distributional robust post-hoc classifiers. An empirical implementation is available at https://github.com/weijiaheng/Drops.