MASS: Overcoming Language Bias in Image-Text Matching

TL;DR

This paper introduces MASS, a framework that reduces language bias in image-text matching models, improving visual accuracy without additional training and maintaining linguistic understanding.

Contribution

MASS is a novel, easily integrable framework that effectively diminishes language bias in visual-language models, enhancing image-text matching performance.

Findings

MASS reduces reliance on language priors in models.

MASS improves visual accuracy in image-text matching.

MASS maintains linguistic compositionality.

Abstract

Pretrained visual-language models have made significant advancements in multimodal tasks, including image-text retrieval. However, a major challenge in image-text matching lies in language bias, where models predominantly rely on language priors and neglect to adequately consider the visual content. We thus present Multimodal ASsociation Score (MASS), a framework that reduces the reliance on language priors for better visual accuracy in image-text matching problems. It can be seamlessly incorporated into existing visual-language models without necessitating additional training. Our experiments have shown that MASS effectively lessens language bias without losing an understanding of linguistic compositionality. Overall, MASS offers a promising solution for enhancing image-text matching performance in visual-language models.