Mitigating Shortcut Learning with InterpoLated Learning

TL;DR

InterpoLated Learning (InterpoLL) is a novel method that interpolates representations to reduce shortcut reliance, improving minority group generalization in NLP tasks without sacrificing overall accuracy.

Contribution

This paper introduces InterpoLated Learning, a representation interpolation technique that mitigates shortcuts and enhances minority group generalization across various NLP architectures.

Findings

InterpoLL outperforms ERM and existing shortcut mitigation methods.

It improves minority generalization without harming majority accuracy.

Effective across multiple NLP model architectures.

Abstract

Empirical risk minimization (ERM) incentivizes models to exploit shortcuts, i.e., spurious correlations between input attributes and labels that are prevalent in the majority of the training data but unrelated to the task at hand. This reliance hinders generalization on minority examples, where such correlations do not hold. Existing shortcut mitigation approaches are model-specific, difficult to tune, computationally expensive, and fail to improve learned representations. To address these issues, we propose InterpoLated Learning (InterpoLL) which interpolates the representations of majority examples to include features from intra-class minority examples with shortcut-mitigating patterns. This weakens shortcut influence, enabling models to acquire features predictive across both minority and majority examples. Experimental results on multiple natural language understanding tasks demonstrate that InterpoLL improves minority generalization over both ERM and state-of-the-art shortcut mitigation methods, without compromising accuracy on majority examples. Notably, these gains persist across encoder, encoder-decoder, and decoder-only architectures, demonstrating the method's broad applicability.