Stereotyping and Bias in the Flickr30K Dataset

TL;DR

This paper investigates biases and stereotypes present in the Flickr30K dataset, revealing that descriptions often include unwarranted inferences influenced by stereotypes, challenging the assumption that descriptions are solely image-based.

Contribution

It identifies specific biases in Flickr30K descriptions and discusses methods to detect and address stereotype-driven content in future image captioning datasets.

Findings

Flickr30K descriptions contain stereotypes and biases.

Descriptions often include unwarranted inferences not solely based on images.

Proposes methods to identify and mitigate biases in datasets.

Abstract

An untested assumption behind the crowdsourced descriptions of the images in the Flickr30K dataset (Young et al., 2014) is that they "focus only on the information that can be obtained from the image alone" (Hodosh et al., 2013, p. 859). This paper presents some evidence against this assumption, and provides a list of biases and unwarranted inferences that can be found in the Flickr30K dataset. Finally, it considers methods to find examples of these, and discusses how we should deal with stereotype-driven descriptions in future applications.