Predicting is not Understanding: Recognizing and Addressing Underspecification in Machine Learning

TL;DR

This paper formalizes underspecification in machine learning, demonstrating how multiple models with similar in-domain accuracy can differ in out-of-distribution performance, and proposes a method to identify and improve models' OOD robustness.

Contribution

It introduces a formal framework for underspecification and a novel method to discover and incorporate meaningful features that enhance OOD performance.

Findings

Models with different functions can have similar in-domain accuracy.

Discovered features improve out-of-distribution generalization.

In-domain accuracy alone is insufficient for reliable model selection.

Abstract

Machine learning (ML) models are typically optimized for their accuracy on a given dataset. However, this predictive criterion rarely captures all desirable properties of a model, in particular how well it matches a domain expert's understanding of a task. Underspecification refers to the existence of multiple models that are indistinguishable in their in-domain accuracy, even though they differ in other desirable properties such as out-of-distribution (OOD) performance. Identifying these situations is critical for assessing the reliability of ML models. We formalize the concept of underspecification and propose a method to identify and partially address it. We train multiple models with an independence constraint that forces them to implement different functions. They discover predictive features that are otherwise ignored by standard empirical risk minimization (ERM), which we then distill into a global model with superior OOD performance. Importantly, we constrain the models to align with the data manifold to ensure that they discover meaningful features. We demonstrate the method on multiple datasets in computer vision (collages, WILDS-Camelyon17, GQA) and discuss general implications of underspecification. Most notably, in-domain performance cannot serve for OOD model selection without additional assumptions.