Survey of Social Bias in Vision-Language Models

TL;DR

This survey reviews social bias issues in vision-language models, highlighting their similarities with NLP and CV biases, and offers guidelines for mitigation to promote fairer AI systems.

Contribution

It provides a comprehensive overview of social bias in pre-trained vision-language models and compares it with NLP and CV biases, offering insights and mitigation strategies.

Findings

VL models are susceptible to social bias similar to NLP and CV.

Current understanding of bias in VL models is limited compared to NLP and CV.

Guidelines are proposed for addressing and reducing social bias in multimodal models.

Abstract

In recent years, the rapid advancement of machine learning (ML) models, particularly transformer-based pre-trained models, has revolutionized Natural Language Processing (NLP) and Computer Vision (CV) fields. However, researchers have discovered that these models can inadvertently capture and reinforce social biases present in their training datasets, leading to potential social harms, such as uneven resource allocation and unfair representation of specific social groups. Addressing these biases and ensuring fairness in artificial intelligence (AI) systems has become a critical concern in the ML community. The recent introduction of pre-trained vision-and-language (VL) models in the emerging multimodal field demands attention to the potential social biases present in these models as well. Although VL models are susceptible to social bias, there is a limited understanding compared to the extensive discussions on bias in NLP and CV. This survey aims to provide researchers with a high-level insight into the similarities and differences of social bias studies in pre-trained models across NLP, CV, and VL. By examining these perspectives, the survey aims to offer valuable guidelines on how to approach and mitigate social bias in both unimodal and multimodal settings. The findings and recommendations presented here can benefit the ML community, fostering the development of fairer and non-biased AI models in various applications and research endeavors.