Overcoming Shortcut Learning in a Target Domain by Generalizing Basic Visual Factors from a Source Domain

TL;DR

This paper introduces a method to reduce shortcut learning in neural networks by augmenting training data with a source domain that promotes learning independent visual factors, improving generalization.

Contribution

The paper proposes a novel data augmentation approach using a source domain to encourage learning of independent visual factors, enhancing compositional generalization.

Findings

Leveraging source domain data effectively mitigates shortcut learning.

Promoting factor independence improves model focus on predictive features.

The approach enhances generalization in both synthetic and real-world domains.

Abstract

Shortcut learning occurs when a deep neural network overly relies on spurious correlations in the training dataset in order to solve downstream tasks. Prior works have shown how this impairs the compositional generalization capability of deep learning models. To address this problem, we propose a novel approach to mitigate shortcut learning in uncontrolled target domains. Our approach extends the training set with an additional dataset (the source domain), which is specifically designed to facilitate learning independent representations of basic visual factors. We benchmark our idea on synthetic target domains where we explicitly control shortcut opportunities as well as real-world target domains. Furthermore, we analyze the effect of different specifications of the source domain and the network architecture on compositional generalization. Our main finding is that leveraging data from a source domain is an effective way to mitigate shortcut learning. By promoting independence across different factors of variation in the learned representations, networks can learn to consider only predictive factors and ignore potential shortcut factors during inference.