Are Any-to-Any Models More Consistent Across Modality Transfers Than Specialists?

TL;DR

This paper evaluates whether any-to-any generative models outperform specialized models in cross-modal coherence, finding limited consistency in pointwise tests but some weak structured equivariance signals.

Contribution

Introduces ACON, a new dataset and evaluation framework for assessing cross-modal consistency in unified generative models.

Findings

No significant advantage of any-to-any models in cyclic consistency

Weak equivariance signals suggest some structured cross-modal coherence

Structured analysis of latent space reveals potential for improved consistency

Abstract

Any-to-any generative models aim to enable seamless interpretation and generation across multiple modalities within a unified framework, yet their ability to preserve relationships across modalities remains uncertain. Do unified models truly achieve cross-modal coherence, or is this coherence merely perceived? To explore this, we introduce ACON, a dataset of 1,000 images (500 newly contributed) paired with captions, editing instructions, and Q&A pairs to evaluate cross-modal transfers rigorously. Using three consistency criteria-cyclic consistency, forward equivariance, and conjugated equivariance-our experiments reveal that any-to-any models do not consistently demonstrate greater cross-modal consistency than specialized models in pointwise evaluations such as cyclic consistency. However, equivariance evaluations uncover weak but observable consistency through structured analyses of the intermediate latent space enabled by multiple editing operations. We release our code and data at https://github.com/JiwanChung/ACON.