BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models

TL;DR

BASIC introduces a novel approach for visual alignment in multimodal large language models by directly supervising visual embeddings, leading to significant performance improvements without extra annotations.

Contribution

The paper proposes a new method called BASIC that refines visual embeddings through direct supervision, enhancing alignment and understanding in MLLMs.

Findings

Improves MLLM performance across multiple benchmarks.

Enhances semantic matching of visual embeddings.

Does not require additional supervisory models or annotations.

Abstract

Mainstream Multimodal Large Language Models (MLLMs) achieve visual understanding by using a vision projector to bridge well-pretrained vision encoders and large language models (LLMs). The inherent gap between visual and textual modalities makes the embeddings from the vision projector critical for visual comprehension. However, current alignment approaches treat visual embeddings as contextual cues and merely apply auto-regressive supervision to textual outputs, neglecting the necessity of introducing equivalent direct visual supervision, which hinders the potential finer alignment of visual embeddings. In this paper, based on our analysis of the refinement process of visual embeddings in the LLM's shallow layers, we propose BASIC, a method that utilizes refined visual embeddings within the LLM as supervision to directly guide the projector in generating initial visual embeddings. Specifically, the guidance is conducted from two perspectives: (i) optimizing embedding directions by reducing angles between initial and supervisory embeddings in semantic space; (ii) improving semantic matching by minimizing disparities between the logit distributions of both visual embeddings. Without additional supervisory models or artificial annotations, BASIC significantly improves the performance of MLLMs across a wide range of benchmarks, demonstrating the effectiveness of our introduced direct visual supervision.