SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentation

TL;DR

This paper introduces SegDebias, a test-time bias mitigation method for CLIP models that uses segmentation to remove confounding visual features without requiring additional training or bias annotations.

Contribution

SegDebias is a novel test-time bias mitigation technique that leverages segmentation to improve robustness of CLIP models without extra training or bias labels.

Findings

Outperforms existing test-time debiasing methods on Waterbirds and CelebA.

Effectively removes bias signals from confounding regions.

Enhances group robustness and attention localization.

Abstract

Vision language models such as CLIP have shown remarkable performance in zero shot classification, but remain susceptible to spurious correlations, where irrelevant visual features influence predictions. Existing debiasing methods often require access to training data and explicit group labels to perform fine-tuning or adjust embeddings, which limits their practicality in real-world settings. Test-time methods attempt to avoid this constraint, but many still depend on prior knowledge of dataset specific biases, limiting their generalizability in open set settings. In this work, we propose a test-time debiasing method for ViT based CLIP models that requires no additional training or assumptions of bias annotations. Our approach uses a pretrained segmentation model to isolate the target visual attribute, then adjusts the non target regions so that their embeddings are uniformly similar to all class specific text prompts. This procedure removes unintended bias signals from confounding visual regions while preserving the target attribute. Experiments on Waterbirds and CelebA show that our method outperforms existing test-time debiasing approaches in both group robustness metrics and Attention IoU. These results demonstrate the effectiveness of segmentation guided interventions for scalable and annotation free bias mitigation in vision language models.