Deep Attention Reweighting: Post-Hoc Attention-Based Feature Aggregation in CNNs for Disentangling Core and Spurious Features under Spurious Correlations

TL;DR

This paper introduces Deep Attention Reweighting (DAR), a post-hoc attention-based method that replaces GAP in CNNs to better disentangle core features from spurious correlations, improving generalization.

Contribution

The paper proposes DAR, an attention mechanism that enhances feature disentanglement in CNNs by addressing GAP-induced entanglement, outperforming previous methods like DFR.

Findings

DAR consistently outperforms DFR across datasets and metrics.

Attention-based aggregation reduces reliance on spurious features.

Replacing GAP with DAR improves model generalization.

Abstract

Convolutional Neural Networks (CNNs) often exploit spurious correlations in datasets, learning superficially predictive yet causally irrelevant features, leading to poor generalization and fairness issues. Deep Feature Reweighting (DFR) is a post-hoc technique that reduces a trained model's reliance on spurious correlations by retraining its classification head on a target dataset. However, we show that DFR is fundamentally constrained by operating on entangled features, limiting its ability to amplify the core features while simultaneously suppressing the spurious ones. We trace this entanglement to the ubiquitous Global Average Pooling (GAP) layer, which indiscriminately collapses spatially distinct core and spurious features into a single representation. To address this, we propose Deep Attention Reweighting (DAR), a post-hoc attention-based aggregation module that replaces GAP and is retrained jointly with the classification head. DAR computes an adaptive weighting of spatial locations across feature maps, enabling selective suppression of spurious features before the collapse into entangled features. Across various datasets, metrics, and ablations, DAR consistently outperforms DFR, demonstrating that our attention-based aggregation mitigates GAP-induced entanglement and reduces spurious reliance.