The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation

TL;DR

This paper presents RFNet, a training-free diffusion model approach that uses LLMs to improve scene generation from complex, artistic, or specialized prompts, validated by a new benchmark and extensive evaluations.

Contribution

Introduction of RFNet, a novel training-free diffusion model method that leverages LLMs for enhanced scene generation from complex prompts.

Findings

RFNet outperforms existing methods in human evaluations.

RFNet achieves higher GPT-based compositional assessment scores.

RFNet effectively handles complex and imaginative prompts.

Abstract

In spite of recent advancements in text-to-image generation, limitations persist in handling complex and imaginative prompts due to the restricted diversity and complexity of training data. This work explores how diffusion models can generate images from prompts requiring artistic creativity or specialized knowledge. We introduce the Realistic-Fantasy Benchmark (RFBench), a novel evaluation framework blending realistic and fantastical scenarios. To address these challenges, we propose the Realistic-Fantasy Network (RFNet), a training-free approach integrating diffusion models with LLMs. Extensive human evaluations and GPT-based compositional assessments demonstrate our approach's superiority over state-of-the-art methods. Our code and dataset is available at https://leo81005.github.io/Reality-and-Fantasy/.