SLIM: Spuriousness Mitigation with Minimal Human Annotations

TL;DR

SLIM is a cost-effective method that uses minimal human annotations to reduce spurious correlations in deep learning models, improving robustness without extensive labeling.

Contribution

SLIM introduces a novel attention labeling mechanism and a human-in-the-loop protocol to efficiently mitigate spurious features with minimal annotations.

Findings

SLIM requires human input for less than 3% of instances.

SLIM outperforms or matches leading methods in key benchmarks.

SLIM reduces annotation costs while maintaining high model robustness.

Abstract

Recent studies highlight that deep learning models often learn spurious features mistakenly linked to labels, compromising their reliability in real-world scenarios where such correlations do not hold. Despite the increasing research effort, existing solutions often face two main challenges: they either demand substantial annotations of spurious attributes, or they yield less competitive outcomes with expensive training when additional annotations are absent. In this paper, we introduce SLIM, a cost-effective and performance-targeted approach to reducing spurious correlations in deep learning. Our method leverages a human-in-the-loop protocol featuring a novel attention labeling mechanism with a constructed attention representation space. SLIM significantly reduces the need for exhaustive additional labeling, requiring human input for fewer than 3% of instances. By prioritizing data quality over complicated training strategies, SLIM curates a smaller yet more feature-balanced data subset, fostering the development of spuriousness-robust models. Experimental validations across key benchmarks demonstrate that SLIM competes with or exceeds the performance of leading methods while significantly reducing costs. The SLIM framework thus presents a promising path for developing reliable models more efficiently. Our code is available in https://github.com/xiweix/SLIM.git/.