Improving Physical Object State Representation in Text-to-Image Generative Systems

TL;DR

This paper enhances text-to-image models' ability to accurately depict object states by generating synthetic data, fine-tuning models, and evaluating improvements, leading to significant accuracy gains.

Contribution

The work introduces an automatic data generation pipeline and fine-tuning approach to improve object state representation in text-to-image models.

Findings

8+% average improvement on GenAI-Bench dataset

24+% average improvement on curated prompts

Enhanced alignment of images with object states

Abstract

Current text-to-image generative models struggle to accurately represent object states (e.g., "a table without a bottle," "an empty tumbler"). In this work, we first design a fully-automatic pipeline to generate high-quality synthetic data that accurately captures objects in varied states. Next, we fine-tune several open-source text-to-image models on this synthetic data. We evaluate the performance of the fine-tuned models by quantifying the alignment of the generated images to their prompts using GPT4o-mini, and achieve an average absolute improvement of 8+% across four models on the public GenAI-Bench dataset. We also curate a collection of 200 prompts with a specific focus on common objects in various physical states. We demonstrate a significant improvement of an average of 24+% over the baseline on this dataset. We release all evaluation prompts and code.