You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Models

TL;DR

This paper investigates how quantization, a common model compression technique, affects social biases in vision-language models, revealing that it does not systematically increase or decrease biases across different models.

Contribution

It provides the first comprehensive analysis showing that quantization does not have a consistent impact on social biases in vision-language models, contrasting prior unimodal model findings.

Findings

Quantization does not systematically amplify biases.

Bias magnitude and direction remain stable across quantized models.

Extensive evaluation across multiple datasets and models supports these conclusions.

Abstract

We study the impact of a standard practice in compressing foundation vision-language models - quantization - on the models' ability to produce socially-fair outputs. In contrast to prior findings with unimodal models that compression consistently amplifies social biases, our extensive evaluation of four quantization settings across three datasets and three CLIP variants yields a surprising result: while individual models demonstrate bias, we find no consistent change in bias magnitude or direction across a population of compressed models due to quantization.