Combining Diverse Feature Priors

TL;DR

This paper investigates how combining models trained with diverse feature priors can reduce failure overlap and improve generalization, especially when jointly trained on unlabeled data, leading to more robust models.

Contribution

It introduces a framework for leveraging diverse feature priors as different data perspectives and demonstrates their combined benefits for model robustness and generalization.

Findings

Models with diverse feature priors have less overlapping failure modes.

Joint training on unlabeled data improves model correction and robustness.

Combining priors enhances resilience to spurious correlations.

Abstract

To improve model generalization, model designers often restrict the features that their models use, either implicitly or explicitly. In this work, we explore the design space of leveraging such feature priors by viewing them as distinct perspectives on the data. Specifically, we find that models trained with diverse sets of feature priors have less overlapping failure modes, and can thus be combined more effectively. Moreover, we demonstrate that jointly training such models on additional (unlabeled) data allows them to correct each other's mistakes, which, in turn, leads to better generalization and resilience to spurious correlations. Code available at https://github.com/MadryLab/copriors