Interleaved Scene Graphs for Interleaved Text-and-Image Generation Assessment

TL;DR

This paper introduces ISG, a comprehensive evaluation framework and benchmark for interleaved text-and-image generation, revealing current models' limitations and proposing a baseline approach to improve performance.

Contribution

The paper presents ISG, a multi-level evaluation framework, a new benchmark dataset ISG-Bench, and a baseline agent for interleaved text-and-image generation tasks.

Findings

Unified models perform poorly on interleaved generation.

Compositional approaches improve performance by 111%.

Baseline agent improves results by 122%.

Abstract

Many real-world user queries (e.g. "How do to make egg fried rice?") could benefit from systems capable of generating responses with both textual steps with accompanying images, similar to a cookbook. Models designed to generate interleaved text and images face challenges in ensuring consistency within and across these modalities. To address these challenges, we present ISG, a comprehensive evaluation framework for interleaved text-and-image generation. ISG leverages a scene graph structure to capture relationships between text and image blocks, evaluating responses on four levels of granularity: holistic, structural, block-level, and image-specific. This multi-tiered evaluation allows for a nuanced assessment of consistency, coherence, and accuracy, and provides interpretable question-answer feedback. In conjunction with ISG, we introduce a benchmark, ISG-Bench, encompassing 1,150 samples across 8 categories and 21 subcategories. This benchmark dataset includes complex language-vision dependencies and golden answers to evaluate models effectively on vision-centric tasks such as style transfer, a challenging area for current models. Using ISG-Bench, we demonstrate that recent unified vision-language models perform poorly on generating interleaved content. While compositional approaches that combine separate language and image models show a 111% improvement over unified models at the holistic level, their performance remains suboptimal at both block and image levels. To facilitate future work, we develop ISG-Agent, a baseline agent employing a "plan-execute-refine" pipeline to invoke tools, achieving a 122% performance improvement.