StruVis: Enhancing Reasoning-based Text-to-Image Generation via Thinking with Structured Vision

TL;DR

StruVis introduces a text-based structured visual reasoning framework that improves reasoning-based text-to-image generation without intermediate image generation, enhancing performance across multiple benchmarks.

Contribution

StruVis is a novel, generator-agnostic framework that employs text-based visual representations to enhance reasoning in T2I generation, avoiding costly image generation steps.

Findings

Achieves 4.61% improvement on T2I-ReasonBench

Achieves 4% improvement on WISE

Enhances reasoning-based T2I performance significantly

Abstract

Reasoning-based text-to-image (T2I) generation requires models to interpret complex prompts accurately. Existing reasoning frameworks can be broadly categorized into two types: (1) Text-Only Reasoning, which is computationally efficient but lacks access to visual context, often resulting in the omission of critical spatial and visual elements; and (2) Text-Image Interleaved Reasoning, which leverages a T2I generator to provide visual references during the reasoning process. While this approach enhances visual grounding, it incurs substantial computational costs and constrains the reasoning capacity of MLLMs to the representational limitations of the generator. To this end, we propose StruVis, a novel framework that enhances T2I generation through Thinking with Structured Vision. Instead of relying on intermediate image generation, StruVis employs text-based structured visual representations as intermediate reasoning states, thereby enabling the MLLM to effectively "perceive" visual structure within a purely text-based reasoning process. Powered by this, the reasoning potential for T2I generation of the MLLM is unlocked through structured-vision-guided reasoning. Additionally, as a generator-agnostic reasoning framework, our proposed StruVis can be seamlessly integrated with diverse T2I generators and efficiently enhance their performance in reasoning-based T2I generation. Extensive experiments demonstrate that StruVis achieves significant performance improvements on reasoning-based T2I benchmarks, e.g., a 4.61% gain on T2I-ReasonBench and a 4% gain on WISE.