INFELM: In-depth Fairness Evaluation of Large Text-To-Image Models

TL;DR

INFELM is a comprehensive framework for evaluating fairness in large text-to-image models, addressing biases and content alignment with improved classifiers and bias-sensitive metrics, revealing fairness issues in current models.

Contribution

The paper introduces INFELM, a novel, in-depth fairness evaluation framework with advanced classifiers and bias metrics for large text-to-image models, filling gaps in existing assessment methods.

Findings

Existing models often fail fairness criteria.

Representation bias exceeds alignment errors.

Biases are prominent across social domains.

Abstract

The rapid development of large language models (LLMs) and large vision models (LVMs) have propelled the evolution of multi-modal AI systems, which have demonstrated the remarkable potential for industrial applications by emulating human-like cognition. However, they also pose significant ethical challenges, including amplifying harmful content and reinforcing societal biases. For instance, biases in some industrial image generation models highlighted the urgent need for robust fairness assessments. Most existing evaluation frameworks focus on the comprehensiveness of various aspects of the models, but they exhibit critical limitations, including insufficient attention to content generation alignment and social bias-sensitive domains. More importantly, their reliance on pixel-detection techniques is prone to inaccuracies. To address these issues, this paper presents INFELM, an in-depth fairness evaluation on widely-used text-to-image models. Our key contributions are: (1) an advanced skintone classifier incorporating facial topology and refined skin pixel representation to enhance classification precision by at least 16.04%, (2) a bias-sensitive content alignment measurement for understanding societal impacts, (3) a generalizable representation bias evaluation for diverse demographic groups, and (4) extensive experiments analyzing large-scale text-to-image model outputs across six social-bias-sensitive domains. We find that existing models in the study generally do not meet the empirical fairness criteria, and representation bias is generally more pronounced than alignment errors. INFELM establishes a robust benchmark for fairness assessment, supporting the development of multi-modal AI systems that align with ethical and human-centric principles.