OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities

TL;DR

OmnixR is a comprehensive evaluation suite designed to benchmark state-of-the-art omni-modality language models, assessing their reasoning capabilities across multiple modalities like text, vision, and audio in synthetic and real-world scenarios.

Contribution

The paper introduces OmnixR, the first benchmark to evaluate multi-modal reasoning across diverse modalities, addressing limitations of existing single or dual-modality benchmarks.

Findings

State-of-the-art OLMs struggle with multi-modal reasoning tasks.

OmnixR reveals significant gaps in current models' cross-modal understanding.

Analysis highlights challenges in omni-modal AI alignment.

Abstract

We introduce OmnixR, an evaluation suite designed to benchmark SoTA Omni-modality Language Models, such as GPT-4o and Gemini. Evaluating OLMs, which integrate multiple modalities such as text, vision, and audio, presents unique challenges. Particularly, the user message might often consist of multiple modalities, such that OLMs have to establish holistic understanding and reasoning across modalities to accomplish the task. Existing benchmarks are limited to single modality or dual-modality tasks, overlooking comprehensive multi-modal assessments of model reasoning. To address this, OmnixR offers two evaluation variants: (1)synthetic subset: a synthetic dataset generated automatically by translating text into multiple modalities--audio, images, video, and hybrids (Omnify). (2)realistic subset: a real-world dataset, manually curated and annotated by experts, for evaluating cross-modal reasoning in natural settings. OmnixR presents a unique evaluation towards assessing OLMs over a diverse mix of modalities, such as a question that involves video, audio, and text, providing a rigorous cross-modal reasoning testbed unlike any existing benchmarks. Our experiments find that all state-of-the-art OLMs struggle with OmnixR questions that require integrating information from multiple modalities to answer. Further analysis highlights differences in reasoning behavior, underscoring the challenges of omni-modal AI alignment.