Is Extending Modality The Right Path Towards Omni-Modality?

TL;DR

This paper critically examines whether extending modality in language models is an effective path toward true omni-modality, analyzing trade-offs, model merging, and generalization through extensive experiments.

Contribution

It provides an empirical analysis of modality extension techniques, assessing their impact on language abilities and the potential of model merging for omni-modality.

Findings

Modality extension may compromise core language skills.

Model merging can help achieve omni-modality.

Omni-modality extension does not always improve knowledge sharing.

Abstract

Omni-modal language models (OLMs) aim to integrate and reason over diverse input modalities--such as text, images, video, and audio--while maintaining strong language capabilities. Despite recent advancements, existing models, especially open-source ones, remain far from true omni-modality, struggling to generalize beyond the specific modality pairs they are trained on or to achieve strong performance when processing multi-modal inputs. We study the effect of extending modality, the dominant technique for training multimodal models, where an off-the-shelf language model is fine-tuned on target-domain and language data. Specifically, we investigate three key questions: (1) Does modality extension compromise core language abilities? (2) Can model merging effectively integrate independently fine-tuned modality-specific models to achieve omni-modality? (3) Does omni-modality extension lead to better knowledge sharing and generalization compared to sequential extension? Through extensive experiments, we analyze these trade-offs and provide insights into the feasibility of achieving true omni-modality using current approaches.