Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation

TL;DR

This paper introduces TwiG, a novel framework that interleaves textual reasoning during visual content generation, leading to more context-aware and semantically rich images through dynamic, on-the-fly multimodal interaction.

Contribution

It presents the first interleaved approach enabling textual reasoning to guide and reflect during visual generation, with strategies including zero-shot prompting, supervised fine-tuning, and reinforcement learning.

Findings

Interleaved reasoning improves visual output quality.

Multiple strategies demonstrate the framework's versatility.

Code will be publicly available for further research.

Abstract

Recent advances in visual generation have increasingly explored the integration of reasoning capabilities. They incorporate textual reasoning, i.e., think, either before (as pre-planning) or after (as post-refinement) the generation process, yet they lack on-the-fly multimodal interaction during the generation itself. In this preliminary study, we introduce Thinking-while-Generating (TwiG), the first interleaved framework that enables co-evolving textual reasoning throughout the visual generation process. As visual content is progressively generating, textual reasoning is interleaved to both guide upcoming local regions and reflect on previously synthesized ones. This dynamic interplay produces more context-aware and semantically rich visual outputs. To unveil the potential of this framework, we investigate three candidate strategies, zero-shot prompting, supervised fine-tuning (SFT) on our curated TwiG-50K dataset, and reinforcement learning (RL) via a customized TwiG-GRPO strategy, each offering unique insights into the dynamics of interleaved reasoning. We hope this work inspires further research into interleaving textual reasoning for enhanced visual generation. Code will be released at: https://github.com/ZiyuGuo99/Thinking-while-Generating.