VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization

TL;DR

VersaViT introduces a multi-task framework to enhance vision encoders in multimodal large language models, enabling them to perform well on both high-level reasoning and dense vision tasks.

Contribution

The paper proposes VersaViT, a novel multi-task post-training method that improves vision backbones for diverse vision tasks within MLLMs.

Findings

Improved performance on dense prediction tasks.

Versatile backbone suitable for reasoning and pixel-level understanding.

Effective multi-task optimization framework.

Abstract

Multimodal Large Language Models (MLLMs) have recently achieved remarkable success in visual-language understanding, demonstrating superior high-level semantic alignment within their vision encoders. An important question thus arises: Can these encoders serve as versatile vision backbones, capable of reliably performing classic vision-centric tasks as well? To address the question, we make the following contributions: (i) we identify that the vision encoders within MLLMs exhibit deficiencies in their dense feature representations, as evidenced by their suboptimal performance on dense prediction tasks (e.g., semantic segmentation, depth estimation); (ii) we propose VersaViT, a well-rounded vision transformer that instantiates a novel multi-task framework for collaborative post-training. This framework facilitates the optimization of the vision backbone via lightweight task heads with multi-granularity supervision; (iii) extensive experiments across various downstream tasks demonstrate the effectiveness of our method, yielding a versatile vision backbone suited for both language-mediated reasoning and pixel-level understanding.