Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning

TL;DR

This paper investigates how contrastive vision-language models often learn shortcuts instead of comprehensive representations, introduces synthetic shortcuts to evaluate this issue, and proposes methods to mitigate shortcut learning, highlighting ongoing challenges.

Contribution

The paper introduces a synthetic shortcut framework for evaluating and reducing shortcut learning in contrastive vision-language models, revealing limitations of current training methods.

Findings

Contrastive models often learn shortcuts rather than full representations.

Synthetic shortcuts can be injected to evaluate shortcut learning.

Proposed methods partially reduce shortcut reliance.

Abstract

Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each caption containing both information shared among all captions and unique information per caption about the scene depicted in the image. In such cases, it is unclear whether contrastive losses are sufficient for learning task-optimal representations that contain all the information provided by the captions or whether the contrastive learning setup encourages the learning of a simple shortcut that minimizes contrastive loss. We introduce synthetic shortcuts for vision-language: a training and evaluation framework where we inject synthetic shortcuts into image-text data. We show that contrastive VLMs trained from scratch or fine-tuned with data containing these synthetic shortcuts mainly learn features that represent the shortcut. Hence, contrastive losses are not sufficient to learn task-optimal representations, i.e., representations that contain all task-relevant information shared between the image and associated captions. We examine two methods to reduce shortcut learning in our training and evaluation framework: (i) latent target decoding and (ii) implicit feature modification. We show empirically that both methods improve performance on the evaluation task, but only partly reduce shortcut learning when training and evaluating with our shortcut learning framework. Hence, we show the difficulty and challenge of our shortcut learning framework for contrastive vision-language representation learning.