SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement

TL;DR

SAILViT introduces a gradual feature refinement approach for Vision Transformers to enhance robustness and generalizability in multimodal large language models, leading to significant performance improvements across various tasks.

Contribution

The paper proposes SAILViT, a novel ViT architecture with coarse-to-fine feature alignment and knowledge infusion, addressing challenges in multimodal training with LLMs.

Findings

SAILViT improves robustness across different model sizes and architectures.

Enhanced performance on the OpenCompass benchmark across multiple downstream tasks.

Thorough empirical analysis confirms generalizability and effectiveness.

Abstract

Vision Transformers (ViTs) are essential as foundation backbones in establishing the visual comprehension capabilities of Multimodal Large Language Models (MLLMs). Although most ViTs achieve impressive performance through image-text pair-based contrastive learning or self-supervised mechanisms, they struggle to engage in connector-based co-training directly with LLMs due to potential parameter initialization conflicts and modality semantic gaps. To address the above challenges, this paper proposes SAILViT, a gradual feature learning-enhanced ViT for facilitating MLLMs to break through performance bottlenecks in complex multimodal interactions. SAILViT achieves coarse-to-fine-grained feature alignment and world knowledge infusion with gradual feature refinement, which better serves target training demands. We perform thorough empirical analyses to confirm the powerful robustness and generalizability of SAILViT across different dimensions, including parameter sizes, model architectures, training strategies, and data scales. Equipped with SAILViT, existing MLLMs show significant and consistent performance improvements on the OpenCompass benchmark across extensive downstream tasks. SAILViT series models are released at https://huggingface.co/BytedanceDouyinContent.