Embedding Arithmetic: A Lightweight, Tuning-Free Framework for Post-hoc Bias Mitigation in Text-to-Image Models

TL;DR

This paper presents Embedding Arithmetic, a tuning-free inference method for mitigating social bias in text-to-image models, preserving prompt semantics and visual context while offering controllable bias reduction.

Contribution

It introduces a novel embedding arithmetic approach that corrects bias without altering model weights, datasets, or prompts, and proposes a new metric for semantic preservation.

Findings

Embedding space forms an entangled manifold rather than a grid of concepts.

The method outperforms baselines in diversity and concept coherence.

Proposes the Concept Coherence Score (CCS) for robust semantic evaluation.

Abstract

Modern text-to-image (T2I) models amplify harmful societal biases, challenging their ethical deployment. We introduce an inference-time method that reliably mitigates social bias while keeping prompt semantics and visual context (background, layout, and style) intact. This ensures context persistency and provides a controllable parameter to adjust mitigation strength, giving practitioners fine-grained control over fairness-coherence trade-offs. Using Embedding Arithmetic, we analyze how bias is structured in the embedding space and correct it without altering model weights, prompts, or datasets. Experiments on FLUX 1.0-Dev and Stable Diffusion 3.5-Large show that the conditional embedding space forms a complex, entangled manifold rather than a grid of disentangled concepts. To rigorously assess semantic preservation beyond the circularity and bias limitations of of CLIP scores, we propose the Concept Coherence Score (CCS). Evaluated against this robust metric, our lightweight, tuning-free method significantly outperforms existing baselines in improving diversity while maintaining high concept coherence, effectively resolving the critical fairness-coherence trade-off. By characterizing how models represent social concepts, we establish geometric understanding of latent space as a principled path toward more transparent, controllable, and fair image generation.