Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation

TL;DR

This paper introduces VisPer-LM, a novel method that enhances multimodal large language models by distilling visual perception knowledge from expert encoders into the LLM's hidden states, improving spatial reasoning and visual understanding.

Contribution

It proposes a coupled optimization pretraining approach that infuses visual perception knowledge into LLMs, outperforming existing baselines in multimodal tasks.

Findings

VisPer-LM outperforms baselines in various benchmarks.

Embedding optimization improves visual representation quality.

Achieves up to 8.7% improvement on Depth task in CV-Bench.

Abstract

In recent times, the standard practice for developing MLLMs is to feed features from vision encoder(s) into the LLM and train with natural language supervision. This approach often causes models to lean towards language comprehension and undermine the rich visual perception signals present in the data, which are critical for tasks involving spatial reasoning in the domain of embodied AI and robotics. Is it possible to optimize both at the same time? In this work, we propose VisPer-LM, the first approach that infuses visual perception knowledge from expert vision encoders into the LLM's (of an MLLM) hidden representations. We start by investigating MLLMs trained solely with natural language supervision and identify a positive correlation between the quality of visual representations within these models and their downstream performance. Given this insight, we formulate the objective during the pretraining stage in MLLMs as a coupled optimization of predictive visual embedding and next (text) token prediction. Moreover, through extensive probing, we observe improved visual representation quality due to embedding optimization, underscoring the effectiveness of our probing setup. We demonstrate that our VisPer-LM outperforms the single and multi-encoder baselines, proving our approach's superiority over explicitly feeding the corresponding features to the LLM. In particular, VisPer-LM boosts performance by an average margin of up to 2.5% on various benchmarks, with a notable improvement of 8.7% on the Depth task in CV-Bench.