Is the Elephant Flying? Resolving Ambiguities in Text-to-Image Generative Models

TL;DR

This paper addresses ambiguities in text-to-image generative models by creating a benchmark dataset and proposing a clarification framework, significantly improving the faithfulness of generated images to user intent.

Contribution

The work introduces a new benchmark dataset for ambiguities in text-to-image models and a framework that solicits user clarifications to resolve these ambiguities.

Findings

The framework improves image faithfulness in ambiguous prompts.

Automatic and human evaluations confirm the effectiveness of the approach.

The dataset covers diverse ambiguity types in text-to-image generation.

Abstract

Natural language often contains ambiguities that can lead to misinterpretation and miscommunication. While humans can handle ambiguities effectively by asking clarifying questions and/or relying on contextual cues and common-sense knowledge, resolving ambiguities can be notoriously hard for machines. In this work, we study ambiguities that arise in text-to-image generative models. We curate a benchmark dataset covering different types of ambiguities that occur in these systems. We then propose a framework to mitigate ambiguities in the prompts given to the systems by soliciting clarifications from the user. Through automatic and human evaluations, we show the effectiveness of our framework in generating more faithful images aligned with human intention in the presence of ambiguities.