PRSM: A Measure to Evaluate CLIP's Robustness Against Paraphrases

TL;DR

This paper introduces PRSM, a new metric to evaluate CLIP's robustness to paraphrasing, revealing variability in stability across paraphrasing strategies and demographic biases.

Contribution

The paper proposes PRSM, a novel measure for assessing CLIP's sensitivity to paraphrased inputs, and empirically analyzes its robustness and bias using the Social Counterfactuals dataset.

Findings

Robustness varies across different paraphrasing strategies.

Subtle differences in robustness are observed between gender-associated queries.

CLIP's stability is influenced by paraphrasing and demographic factors.

Abstract

Contrastive Language-Image Pre-training (CLIP) is a widely used multimodal model that aligns text and image representations through large-scale training. While it performs strongly on zero-shot and few-shot tasks, its robustness to linguistic variation, particularly paraphrasing, remains underexplored. Paraphrase robustness is essential for reliable deployment, especially in socially sensitive contexts where inconsistent representations can amplify demographic biases. In this paper, we introduce the Paraphrase Ranking Stability Metric (PRSM), a novel measure for quantifying CLIP's sensitivity to paraphrased queries. Using the Social Counterfactuals dataset, a benchmark designed to reveal social and demographic biases, we empirically assess CLIP's stability under paraphrastic variation, examine the interaction between paraphrase robustness and gender, and discuss implications for fairness and equitable deployment of multimodal systems. Our analysis reveals that robustness varies across paraphrasing strategies, with subtle yet consistent differences observed between male- and female-associated queries.