Rethinking Visual Information Processing in Multimodal LLMs

TL;DR

This paper introduces LLaViT, a novel approach where large language models are extended to serve as vision encoders, significantly improving multimodal vision-language task performance by integrating visual processing directly into the LLM.

Contribution

LLaViT enables LLMs to function as vision encoders through three key modifications, enhancing multimodal integration and outperforming existing methods like LLaVA.

Findings

LLaViT outperforms LLaVA on multiple benchmarks.

LLaViT surpasses larger models with fewer parameters.

Bidirectional attention improves visual feature integration.

Abstract

Despite the remarkable success of the LLaVA architecture for vision-language tasks, its design inherently struggles to effectively integrate visual features due to the inherent mismatch between text and vision modalities. We tackle this issue from a novel perspective in which the LLM not only serves as a language model but also a powerful vision encoder. To this end, we present LLaViT - Large Language Models as extended Vision Transformers - which enables the LLM to simultaneously function as a vision encoder through three key modifications: (1) learning separate QKV projections for vision modality, (2) enabling bidirectional attention on visual tokens, and (3) incorporating both global and local visual representations. Through extensive controlled experiments on a wide range of LLMs, we demonstrate that LLaViT significantly outperforms the baseline LLaVA method on a multitude of benchmarks, even surpassing models with double its parameter count, establishing a more effective approach to vision-language modeling.