Who's in and who's out? A case study of multimodal CLIP-filtering in DataComp

TL;DR

This paper critically examines how CLIP-based filtering in data collection introduces biases, excludes marginalized groups, and fails to effectively filter explicit or copyrighted content, highlighting the need for improved dataset curation practices.

Contribution

It provides an empirical analysis of CLIP-filtering biases in DataComp, revealing exclusion of marginalized groups and shortcomings in content filtering methods.

Findings

CLIP-filtering disproportionately excludes data related to marginalized groups.

Filtering amplifies existing underrepresentation of certain demographics.

NSFW filter fails to remove explicit content effectively.

Abstract

As training datasets become increasingly drawn from unstructured, uncontrolled environments such as the web, researchers and industry practitioners have increasingly relied upon data filtering techniques to "filter out the noise" of web-scraped data. While datasets have been widely shown to reflect the biases and values of their creators, in this paper we contribute to an emerging body of research that assesses the filters used to create these datasets. We show that image-text data filtering also has biases and is value-laden, encoding specific notions of what is counted as "high-quality" data. In our work, we audit a standard approach of image-text CLIP-filtering on the academic benchmark DataComp's CommonPool by analyzing discrepancies of filtering through various annotation techniques across multiple modalities of image, text, and website source. We find that data relating to several imputed demographic groups -- such as LGBTQ+ people, older women, and younger men -- are associated with higher rates of exclusion. Moreover, we demonstrate cases of exclusion amplification: not only are certain marginalized groups already underrepresented in the unfiltered data, but CLIP-filtering excludes data from these groups at higher rates. The data-filtering step in the machine learning pipeline can therefore exacerbate representation disparities already present in the data-gathering step, especially when existing filters are designed to optimize a specifically-chosen downstream performance metric like zero-shot image classification accuracy. Finally, we show that the NSFW filter fails to remove sexually-explicit content from CommonPool, and that CLIP-filtering includes several categories of copyrighted content at high rates. Our conclusions point to a need for fundamental changes in dataset creation and filtering practices.