Beyond the Pixels: VLM-based Evaluation of Identity Preservation in Reference-Guided Synthesis

TL;DR

This paper presents Beyond the Pixels, a hierarchical VLM-based evaluation framework that improves identity preservation assessment in generative models by decomposing features and guiding structured reasoning, validated against human judgments.

Contribution

It introduces a novel hierarchical evaluation method that enhances fine-grained identity assessment and reduces hallucinations in VLM-based metrics for generative models.

Findings

Strong alignment with human judgments on identity consistency

Effective decomposition of identity features improves evaluation accuracy

New benchmark with diverse and challenging image-prompt pairs

Abstract

Evaluating identity preservation in generative models remains a critical yet unresolved challenge. Existing metrics rely on global embeddings or coarse VLM prompting, failing to capture fine-grained identity changes and providing limited diagnostic insight. We introduce Beyond the Pixels, a hierarchical evaluation framework that decomposes identity assessment into feature-level transformations. Our approach guides VLMs through structured reasoning by (1) hierarchically decomposing subjects into (type, style) -> attribute -> feature decision tree, and (2) prompting for concrete transformations rather than abstract similarity scores. This decomposition grounds VLM analysis in verifiable visual evidence, reducing hallucinations and improving consistency. We validate our framework across four state-of-the-art generative models, demonstrating strong alignment with human judgments in measuring identity consistency. Additionally, we introduce a new benchmark specifically designed to stress-test generative models. It comprises 1,078 image-prompt pairs spanning diverse subject types, including underrepresented categories such as anthropomorphic and animated characters, and captures an average of six to seven transformation axes per prompt.