Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization

TL;DR

Rhet2Pix introduces a multi-step policy optimization framework with a two-layer diffusion module to improve rhetorical text-to-image generation, capturing hidden semantic meanings beyond literal interpretations.

Contribution

The paper presents Rhet2Pix, a novel multi-layer diffusion-based approach that effectively generates images from rhetorical language by incrementally elaborating prompts and optimizing diffusion trajectories.

Findings

Outperforms SOTA models like GPT-4o and Grok-3 in qualitative evaluations.

Effectively captures semantic richness in rhetorical prompts.

Mitigates reward sparsity in diffusion-based image generation.

Abstract

Generating images from rhetorical languages remains a critical challenge for text-to-image models. Even state-of-the-art (SOTA) multimodal large language models (MLLM) fail to generate images based on the hidden meaning inherent in rhetorical language--despite such content being readily mappable to visual representations by humans. A key limitation is that current models emphasize object-level word embedding alignment, causing metaphorical expressions to steer image generation towards their literal visuals and overlook the intended semantic meaning. To address this, we propose Rhet2Pix, a framework that formulates rhetorical text-to-image generation as a multi-step policy optimization problem, incorporating a two-layer MDP diffusion module. In the outer layer, Rhet2Pix converts the input prompt into incrementally elaborated sub-sentences and executes corresponding image-generation actions, constructing semantically richer visuals. In the inner layer, Rhet2Pix mitigates reward sparsity during image generation by discounting the final reward and optimizing every adjacent action pair along the diffusion denoising trajectory. Extensive experiments demonstrate the effectiveness of Rhet2Pix in rhetorical text-to-image generation. Our model outperforms SOTA MLLMs such as GPT-4o, Grok-3 and leading academic baselines across both qualitative and quantitative evaluations. The code and dataset used in this work are publicly available.