UmniBench: Unified Understand and Generation Model Oriented Omni-dimensional Benchmark

TL;DR

UmniBench is a comprehensive benchmark designed to evaluate unified multimodal models across understanding, generation, and editing tasks within a single, omni-dimensional framework, covering diverse domains and concepts.

Contribution

This paper introduces UmniBench, the first benchmark for holistic evaluation of UMMs, enabling simultaneous assessment of multiple abilities and detailed analysis.

Findings

Benchmarking 24 models reveals varied strengths and weaknesses.

UmniBench provides fine-grained, domain-specific insights.

Unified evaluation correlates understanding, generation, and editing performance.

Abstract

Unifying multimodal understanding and generation has shown impressive capabilities in cutting-edge proprietary systems. However, evaluations of unified multimodal models (UMMs) remain decoupled, assessing their understanding and generation abilities separately with corresponding datasets. To address this, we propose UmniBench, a benchmark tailored for UMMs with omni-dimensional evaluation. First, UmniBench can assess the understanding, generation, and editing ability within a single evaluation process. Based on human-examined prompts and QA pairs, UmniBench leverages UMM itself to evaluate its generation and editing ability with its understanding ability. This simple but effective paradigm allows comprehensive evaluation of UMMs. Second, UmniBench covers 13 major domains and more than 200 concepts, ensuring a thorough inspection of UMMs. Moreover, UmniBench can also decouple and separately evaluate understanding, generation, and editing abilities, providing a fine-grained assessment. Based on UmniBench, we benchmark 24 popular models, including both UMMs and single-ability large models. We hope this benchmark provides a more comprehensive and objective view of unified models and logistical support for improving the performance of the community model.