Focusing Image Generation to Mitigate Spurious Correlations

TL;DR

This paper introduces SCGS, a data augmentation technique that reduces spurious correlations in image classification by generating new training data focused on incorrect attention regions, improving model robustness.

Contribution

The paper presents a novel, label-free data augmentation method that mitigates spurious correlations by guiding image synthesis based on classifier attention analysis.

Findings

Reduces reliance on spurious background features in classifiers

Improves classifier robustness across multiple domain datasets

Does not require explicit spurious attribute labels

Abstract

Instance features in images exhibit spurious correlations with background features, affecting the training process of deep neural classifiers. This leads to insufficient attention to instance features by the classifier, resulting in erroneous classification outcomes. In this paper, we propose a data augmentation method called Spurious Correlations Guided Synthesis (SCGS) that mitigates spurious correlations through image generation model. This approach does not require expensive spurious attribute (group) labels for the training data and can be widely applied to other debiasing methods. Specifically, SCGS first identifies the incorrect attention regions of a pre-trained classifier on the training images, and then uses an image generation model to generate new training data based on these incorrect attended regions. SCGS increases the diversity and scale of the dataset to reduce the impact of spurious correlations on classifiers. Changes in the classifier's attention regions and experimental results on three different domain datasets demonstrate that this method is effective in reducing the classifier's reliance on spurious correlations.