Modeling Cross-vision Synergy for Unified Large Vision Model

TL;DR

PolyV is a unified large vision model that leverages architectural design and training strategies to enable cross-vision synergy, improving reasoning across images, videos, and 3D data beyond existing models.

Contribution

The paper introduces PolyV, a novel unified LVM with a sparse Mixture-of-Experts architecture and a synergy-aware training paradigm for cross-vision reasoning.

Findings

PolyV outperforms existing models by over 10% on 10 benchmarks.

The architecture enables modality-specific expertise with bidirectional interaction.

Synergy-aware training enhances reasoning across visual modalities.

Abstract

Recent advances in large vision models (LVMs) have shifted from modality-specific designs toward unified architectures that jointly process images, videos, and 3D data. However, existing unified LVMs primarily pursue functional integration, while overlooking the deeper goal of cross-vision synergy: the ability to reason over complementary priors across visual modalities. To address this, we present PolyV, a unified LVM that achieves cross-vision synergy at both the architectural and training levels. Architecturally, PolyV adopts a sparse Mixture-of-Experts LVM coordinated by a dynamic modality router, allowing each expert to specialize in modality-specific priors while enabling bidirectional interaction and mutual refinement across modalities. Training-wise, a synergy-aware paradigm combines modality-specific pretraining with coarse-to-fine synergy tuning via knowledge distillation and object-/relation-level alignment. Extensive experiments on 10 benchmarks spanning image, video, and 3D understanding, including synergy-focused datasets requiring spatial or temporal priors, demonstrate that PolyV consistently outperforms existing models, achieving over 10% average improvement over its backbone. Overall, PolyV establishes a unified framework for synesthetic visual reasoning, advancing toward truly synergistic LVMs. Project page: https://sqwu.top/PolyV.