FRABench and UFEval: Unified Fine-grained Evaluation with Task and Aspect Generalization

TL;DR

This paper introduces UFEval, a unified evaluator for multimodal models that generalizes across tasks and aspects, supported by FRABench, a large-scale dataset with hierarchical aspect annotations.

Contribution

The paper presents the first unified fine-grained evaluator for multiple multimodal tasks, along with a comprehensive dataset, FRABench, enabling aspect and task generalization.

Findings

UFEval can generalize to unseen aspects through aspect-specific learning.

Joint training on multiple tasks improves evaluation performance.

FRABench provides extensive annotations for diverse multimodal evaluation.

Abstract

Evaluating open-ended outputs of Multimodal Large Language Models has become a bottleneck as model capabilities, task diversity, and modality rapidly expand. Existing ``MLLM-as-a-Judge'' evaluators, though promising, remain constrained to specific tasks and aspects. In this paper, we argue that, on one hand, based on the interconnected nature of aspects, learning specific aspects can generalize to unseen aspects; on the other hand, jointly learning to assess multiple visual aspects and tasks may foster a synergistic effect. To this end, we propose UFEval, the first unified fine-grained evaluator with task and aspect generalization for four evaluation tasks -- Natural Language Generation, Image Understanding, Image Generation, and Interleaved Text-and-Image Generation. However, training such a unified evaluator is hindered by the lack of a large-scale, multi-modal, and aspect-level resource. To address this gap, we introduce FRABench, a comprehensive fine-grained evaluation dataset. Specifically, (1) We first construct a hierarchical aspect taxonomy encompassing 112 distinct aspects across the aforementioned four tasks. (2) Based on this taxonomy, we create FRABench, comprising 60.4k pairwise samples with 325k evaluation labels obtained from a combination of human and GPT-4o annotations. (3) Finally, leveraging FRABench, we develop UFEval, a unified fine-grained evaluator. Experiments show that learning on specific aspects enables UFEval to generalize to unseen aspects, and joint learning to assess diverse visual tasks and aspects can lead to substantial mutual benefits.