Testing Relational Understanding in Text-Guided Image Generation

TL;DR

This paper systematically evaluates DALL-E 2's ability to generate images that accurately depict basic physical and social relations, revealing significant gaps in relational understanding compared to human judgments.

Contribution

It provides the first comprehensive empirical assessment of a state-of-the-art text-guided image model's relational reasoning capabilities using human judgments.

Findings

Only about 22% of images matched relation prompts.

Current models struggle with basic relations involving objects and agents.

Analysis suggests potential improvements inspired by biological intelligence.

Abstract

Relations are basic building blocks of human cognition. Classic and recent work suggests that many relations are early developing, and quickly perceived. Machine models that aspire to human-level perception and reasoning should reflect the ability to recognize and reason generatively about relations. We report a systematic empirical examination of a recent text-guided image generation model (DALL-E 2), using a set of 15 basic physical and social relations studied or proposed in the literature, and judgements from human participants (N = 169). Overall, we find that only ~22% of images matched basic relation prompts. Based on a quantitative examination of people's judgments, we suggest that current image generation models do not yet have a grasp of even basic relations involving simple objects and agents. We examine reasons for model successes and failures, and suggest possible improvements based on computations observed in biological intelligence.