GenArena: How Can We Achieve Human-Aligned Evaluation for Visual Generation Tasks?

TL;DR

GenArena introduces a pairwise comparison framework for evaluating visual generation models, significantly improving alignment with human perception and outperforming traditional pointwise scoring methods.

Contribution

The paper proposes a novel pairwise evaluation paradigm that enhances stability and human alignment, outperforming existing scoring standards in visual generation assessment.

Findings

Pairwise evaluation outperforms pointwise scoring in stability and human alignment.

Open-source models can surpass proprietary models using GenArena.

Evaluation accuracy improves by over 20%, with a Spearman correlation of 0.86.

Abstract

The rapid advancement of visual generation models has outpaced traditional evaluation approaches, necessitating the adoption of Vision-Language Models as surrogate judges. In this work, we systematically investigate the reliability of the prevailing absolute pointwise scoring standard, across a wide spectrum of visual generation tasks. Our analysis reveals that this paradigm is limited due to stochastic inconsistency and poor alignment with human perception. To resolve these limitations, we introduce GenArena, a unified evaluation framework that leverages a pairwise comparison paradigm to ensure stable and human-aligned evaluation. Crucially, our experiments uncover a transformative finding that simply adopting this pairwise protocol enables off-the-shelf open-source models to outperform top-tier proprietary models. Notably, our method boosts evaluation accuracy by over 20% and achieves a Spearman correlation of 0.86 with the authoritative LMArena leaderboard, drastically surpassing the 0.36 correlation of pointwise methods. Based on GenArena, we benchmark state-of-the-art visual generation models across diverse tasks, providing the community with a rigorous and automated evaluation standard for visual generation.