Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models

TL;DR

This paper introduces XTC-Bench, a new evaluation framework for measuring semantic consistency across tasks in unified multimodal models, revealing that high task performance does not guarantee aligned representations.

Contribution

The paper proposes a scene-graph-grounded benchmarking method and a fine-grained metric to assess cross-task semantic alignment in uMMs, addressing a gap in current evaluation protocols.

Findings

High performance in generation or understanding does not ensure strong cross-task alignment.

Model architecture and how learning objectives are coupled influence consistency more than unification.

XTC-Bench is reproducible, model-agnostic, and helps diagnose representation misalignment.

Abstract

Unified Multimodal Models (uMMs) aim to support both visual understanding and visual generation within a shared representation. However, existing evaluation protocols assess these two capabilities independently and do not examine whether they are semantically aligned. As a result, it remains unclear whether current uMMs learn coherent unified representations that remain consistent across tasks given a visual concept. We introduce XTC-Bench, a scene-graph-grounded evaluation framework that measures cross-task visual semantic consistency. By deriving both generation prompts and understanding queries from a structured scene graph, our framework enables fact-level alignment analysis across objects, attributes, and relations. We propose Continuous Cross-Task Agreement (CCTA), a fine-grained metric that quantifies semantic agreement between generation and understanding over matched atomic facts, isolating internal consistency from standalone task accuracy. Extensive experiments on eight open-source and one commercial unified models reveal that high generation or understanding performance does not imply strong cross-task alignment, and architectural analysis shows consistency is governed by how tightly learning objectives are coupled across modalities, not by architectural unification alone. XTC-Bench provides a reproducible and model-agnostic framework for diagnosing representation-level misalignment, offering a concrete direction for advancing unified multimodal modeling beyond isolated task performance.