UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models

TL;DR

This paper introduces UNO-Bench, a comprehensive benchmark for evaluating the relationship and capabilities of uni-modal and omni-modal models across diverse tasks, revealing insights into their compositional performance and potential bottlenecks.

Contribution

The paper presents UNO-Bench, a unified benchmark with new datasets and evaluation methods for assessing uni-modal and omni-modal models, facilitating understanding of their compositional abilities.

Findings

Omni-modal performance acts as a bottleneck in weak models.

Strong models show synergistic improvements with omni-modal capabilities.

The benchmark covers 44 task types and 5 modality combinations.

Abstract

Multimodal Large Languages models have been progressing from uni-modal understanding toward unifying visual, audio and language modalities, collectively termed omni models. However, the correlation between uni-modal and omni-modal remains unclear, which requires comprehensive evaluation to drive omni model's intelligence evolution. In this work, we introduce a novel, high-quality, and UNified Omni model benchmark, UNO-Bench. This benchmark is designed to effectively evaluate both UNi-modal and Omni-modal capabilities under a unified ability taxonomy, spanning 44 task types and 5 modality combinations. It includes 1250 human curated samples for omni-modal with 98% cross-modality solvability, and 2480 enhanced uni-modal samples. The human-generated dataset is well-suited to real-world scenarios, particularly within the Chinese context, whereas the automatically compressed dataset offers a 90% increase in speed and maintains 98% consistency across 18 public benchmarks. In addition to traditional multi-choice questions, we propose an innovative multi-step open-ended question format to assess complex reasoning. A general scoring model is incorporated, supporting 6 question types for automated evaluation with 95% accuracy. Experimental result shows the Compositional Law between omni-modal and uni-modal performance and the omni-modal capability manifests as a bottleneck effect on weak models, while exhibiting synergistic promotion on strong models.