eXplaining to Learn (eX2L): Regularization Using Contrastive Visual Explanation Pairs for Distribution Shifts

TL;DR

eX2L introduces an interpretable regularization method that uses contrastive visual explanation pairs to improve robustness against distribution shifts by decorrelating confounding features.

Contribution

It presents a novel explanation-based framework that explicitly decouples label and nuisance attributes during training to enhance distributional robustness.

Findings

eX2L achieves 82.24% average accuracy on the Spawrious benchmark.

It outperforms state-of-the-art methods by 5.49% in average accuracy.

eX2L improves worst-group accuracy by 10.90%.

Abstract

Despite extensive research into mitigating distribution shifts, many existing algorithms yield inconsistent performance, often failing to outperform baseline Empirical Risk Minimization (ERM) across diverse scenarios. Furthermore, high algorithmic complexity frequently limits interpretability and offers only an indirect means of addressing spurious correlations. We propose eXplaining to Learn (eX2L): an interpretable, explanation-based framework that decorrelates confounding features from a classifier's latent representations during training. eX2L achieves this by penalizing the similarity between Grad-CAM activation maps generated by a primary label classifier and those from a concurrently trained confounder classifier. On the rigorous Spawrious Many-to-Many Hard Challenge benchmark, eX2L achieves an average accuracy (AA) of 82.24% +/- 3.87% and a worst-group accuracy (WGA) of 66.31% +/- 8.73%, outperforming the current state-of-the-art (SOTA) by 5.49% and 10.90%, respectively. Beyond its competitive performance, eX2L demonstrates that functional domain invariance can be achieved by explicitly decoupling label and nuisance attributes at the group level.