Model Patching: Closing the Subgroup Performance Gap with Data Augmentation

TL;DR

This paper introduces model patching, a two-stage data augmentation framework that improves classifier robustness by reducing subgroup performance disparities, demonstrated on multiple datasets including skin cancer classification.

Contribution

The paper proposes a novel model patching framework using CycleGAN-based augmentations and a subgroup regularizer to enhance model invariance and robustness.

Findings

Up to 33% reduction in robust error on benchmark datasets.

Effective patching of a skin cancer classifier with spurious features.

Demonstrated improvement in subgroup performance consistency.

Abstract

Classifiers in machine learning are often brittle when deployed. Particularly concerning are models with inconsistent performance on specific subgroups of a class, e.g., exhibiting disparities in skin cancer classification in the presence or absence of a spurious bandage. To mitigate these performance differences, we introduce model patching, a two-stage framework for improving robustness that encourages the model to be invariant to subgroup differences, and focus on class information shared by subgroups. Model patching first models subgroup features within a class and learns semantic transformations between them, and then trains a classifier with data augmentations that deliberately manipulate subgroup features. We instantiate model patching with CAMEL, which (1) uses a CycleGAN to learn the intra-class, inter-subgroup augmentations, and (2) balances subgroup performance using a theoretically-motivated subgroup consistency regularizer, accompanied by a new robust objective. We demonstrate CAMEL's effectiveness on 3 benchmark datasets, with reductions in robust error of up to 33% relative to the best baseline. Lastly, CAMEL successfully patches a model that fails due to spurious features on a real-world skin cancer dataset.