Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks

TL;DR

This paper introduces a diffusion-based ensemble diversification method that generates synthetic counterfactuals to reduce shortcut learning in visual tasks, leading to more robust models without extra data collection.

Contribution

It leverages the inherent ability of Diffusion Probabilistic Models to represent multiple cues independently, promoting diversity and mitigating shortcut reliance in models.

Findings

Diffusion-guided diversification reduces shortcut cues influence.

Achieves ensemble diversity comparable to data-intensive methods.

Improves model robustness in underspecified visual tasks.

Abstract

Spurious correlations in the data, where multiple cues are predictive of the target labels, often lead to shortcut learning phenomena, where a model may rely on erroneous, easy-to-learn, cues while ignoring reliable ones. In this work, we propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs). We discover that DPMs have the inherent capability to represent multiple visual cues independently, even when they are largely correlated in the training data. We leverage this characteristic to encourage model diversity and empirically show the efficacy of the approach with respect to several diversification objectives. We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.