Let's ViCE! Mimicking Human Cognitive Behavior in Image Generation Evaluation

TL;DR

This paper introduces ViCE, an automated evaluation framework that mimics human cognitive behavior by combining LLMs and VQA to assess the quality and prompt adherence of generated images.

Contribution

The paper proposes a novel automated evaluation method for image generation quality that replicates human cognition, integrating LLMs and VQA in a unified process.

Findings

Preliminary results show promising correlation with human judgment.

ViCE effectively formulates verification questions based on visual concepts.

The method opens new avenues for automatic image evaluation.

Abstract

Research in Image Generation has recently made significant progress, particularly boosted by the introduction of Vision-Language models which are able to produce high-quality visual content based on textual inputs. Despite ongoing advancements in terms of generation quality and realism, no methodical frameworks have been defined yet to quantitatively measure the quality of the generated content and the adherence with the prompted requests: so far, only human-based evaluations have been adopted for quality satisfaction and for comparing different generative methods. We introduce a novel automated method for Visual Concept Evaluation (ViCE), i.e. to assess consistency between a generated/edited image and the corresponding prompt/instructions, with a process inspired by the human cognitive behaviour. ViCE combines the strengths of Large Language Models (LLMs) and Visual Question Answering (VQA) into a unified pipeline, aiming to replicate the human cognitive process in quality assessment. This method outlines visual concepts, formulates image-specific verification questions, utilizes the Q&A system to investigate the image, and scores the combined outcome. Although this brave new hypothesis of mimicking humans in the image evaluation process is in its preliminary assessment stage, results are promising and open the door to a new form of automatic evaluation which could have significant impact as the image generation or the image target editing tasks become more and more sophisticated.