Superclass-Guided Representation Disentanglement for Spurious Correlation Mitigation

TL;DR

This paper introduces a superclass-guided disentanglement method that improves robustness to spurious correlations in domain generalization tasks without requiring group annotations, leveraging vision-language models for supervision.

Contribution

It proposes a novel approach using superclasses and vision-language models for spurious correlation mitigation, eliminating the need for explicit group annotations.

Findings

Significant performance improvements over baselines.

Robustness to complex group structures.

Effective feature disentanglement demonstrated.

Abstract

To enhance group robustness to spurious correlations, prior work often relies on auxiliary group annotations and assumes identical sets of groups across training and test domains. To overcome these limitations, we propose to leverage superclasses -- categories that lie higher in the semantic hierarchy than the task's actual labels -- as a more intrinsic signal than group labels for discerning spurious correlations. Our model incorporates superclass guidance from a pretrained vision-language model via gradient-based attention alignment, and then integrates feature disentanglement with a theoretically supported minimax-optimal feature-usage strategy. As a result, our approach attains robustness to more complex group structures and spurious correlations, without the need to annotate any training samples. Experiments across diverse domain generalization tasks show that our method significantly outperforms strong baselines and goes well beyond the vision-language model's guidance, with clear improvements in both quantitative metrics and qualitative visualizations.