How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositions

TL;DR

This paper investigates how semantic binding in prompts influences bias in text-to-image models, introduces a bias adherence score, and proposes a token decoupling method to improve bias mitigation without harming semantic integrity.

Contribution

It reveals the impact of semantic associations on bias amplification, introduces a bias adherence score, and presents a training-free token decoupling framework for better bias control.

Findings

Bias can be amplified by semantic associations in prompts.

Token decoupling improves debiasing by over 10%.

Current methods struggle to reduce bias without affecting semantics.

Abstract

Text-to-image generative models often exhibit bias related to sensitive attributes. However, current research tends to focus narrowly on single-object prompts with limited contextual diversity. In reality, each object or attribute within a prompt can contribute to bias. For example, the prompt "an assistant wearing a pink hat" may reflect female-inclined biases associated with a pink hat. The neglected joint effects of the semantic binding in the prompts cause significant failures in current debiasing approaches. This work initiates a preliminary investigation on how bias manifests under semantic binding, where contextual associations between objects and attributes influence generative outcomes. We demonstrate that the underlying bias distribution can be amplified based on these associations. Therefore, we introduce a bias adherence score that quantifies how specific object-attribute bindings activate bias. To delve deeper, we develop a training-free context-bias control framework to explore how token decoupling can facilitate the debiasing of semantic bindings. This framework achieves over 10% debiasing improvement in compositional generation tasks. Our analysis of bias scores across various attribute-object bindings and token decorrelation highlights a fundamental challenge: reducing bias without disrupting essential semantic relationships. These findings expose critical limitations in current debiasing approaches when applied to semantically bound contexts, underscoring the need to reassess prevailing bias mitigation strategies.