From Global to Local: Social Bias Transfer in CLIP
Ryan Ramos, Yusuke Hirota, Yuta Nakashima, Noa Garcia

TL;DR
This paper empirically investigates how social biases in CLIP models transfer to downstream tasks, revealing that bias measurement depends on data subsets and that biases tend to diminish after adaptation, highlighting challenges in bias transfer analysis.
Contribution
It provides a comprehensive empirical analysis of bias transfer in CLIP models, examining bias measurement variability and the convergence of representations during downstream adaptation.
Findings
Bias measurement varies with data subset used
Difficulty in establishing consistent bias transfer trends
Representation spaces tend to converge during downstream adaptation
Abstract
The recycling of contrastive language-image pre-trained (CLIP) models as backbones for a large number of downstream tasks calls for a thorough analysis of their transferability implications, especially their well-documented reproduction of social biases and human stereotypes. How do such biases, learned during pre-training, propagate to downstream applications like visual question answering or image captioning? Do they transfer at all? We investigate this phenomenon, referred to as bias transfer in prior literature, through a comprehensive empirical analysis. Firstly, we examine how pre-training bias varies between global and local views of data, finding that bias measurement is highly dependent on the subset of data on which it is computed. Secondly, we analyze correlations between biases in the pre-trained models and the downstream tasks across varying levels of pre-training bias,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
