Interactive Visual Assessment for Text-to-Image Generation Models

TL;DR

DyEval is an interactive, LLM-powered framework that enhances the evaluation of text-to-image models by enabling dynamic, collaborative, and interpretable assessment, uncovering complex failure patterns more effectively.

Contribution

The paper introduces DyEval, a novel interactive assessment framework that adaptively probes models and provides interpretability, surpassing traditional static evaluation methods.

Findings

Identifies up to 2.56 times more failures than conventional methods.

Uncovers complex failure patterns like pronoun and cultural context issues.

Demonstrates effectiveness through qualitative and quantitative experiments.

Abstract

Visual generation models have achieved remarkable progress in computer graphics applications but still face significant challenges in real-world deployment. Current assessment approaches for visual generation tasks typically follow an isolated three-phase framework: test input collection, model output generation, and user assessment. These fashions suffer from fixed coverage, evolving difficulty, and data leakage risks, limiting their effectiveness in comprehensively evaluating increasingly complex generation models. To address these limitations, we propose DyEval, an LLM-powered dynamic interactive visual assessment framework that facilitates collaborative evaluation between humans and generative models for text-to-image systems. DyEval features an intuitive visual interface that enables users to interactively explore and analyze model behaviors, while adaptively generating hierarchical, fine-grained, and diverse textual inputs to continuously probe the capability boundaries of the models based on their feedback. Additionally, to provide interpretable analysis for users to further improve tested models, we develop a contextual reflection module that mines failure triggers of test inputs and reflects model potential failure patterns supporting in-depth analysis using the logical reasoning ability of LLM. Qualitative and quantitative experiments demonstrate that DyEval can effectively help users identify max up to 2.56 times generation failures than conventional methods, and uncover complex and rare failure patterns, such as issues with pronoun generation and specific cultural context generation. Our framework provides valuable insights for improving generative models and has broad implications for advancing the reliability and capabilities of visual generation systems across various domains.