TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings

TL;DR

TG-LLaVA introduces a novel approach to improve vision-language models by guiding the vision encoder with text through learnable latent embeddings, enhancing feature extraction and answer accuracy without extra training data.

Contribution

The paper proposes a new text-guided optimization method for vision encoders using learnable latent embeddings, offering an orthogonal improvement to existing VLMs.

Findings

Improves baseline performance without additional training data.

Enhances the vision encoder's ability to extract text-related features.

Consistently outperforms other methods across various datasets and settings.

Abstract

Currently, inspired by the success of vision-language models (VLMs), an increasing number of researchers are focusing on improving VLMs and have achieved promising results. However, most existing methods concentrate on optimizing the connector and enhancing the language model component, while neglecting improvements to the vision encoder itself. In contrast, we propose Text Guided LLaVA (TG-LLaVA) in this paper, which optimizes VLMs by guiding the vision encoder with text, offering a new and orthogonal optimization direction. Specifically, inspired by the purpose-driven logic inherent in human behavior, we use learnable latent embeddings as a bridge to analyze textual instruction and add the analysis results to the vision encoder as guidance, refining it. Subsequently, another set of latent embeddings extracts additional detailed text-guided information from high-resolution local patches as auxiliary information. Finally, with the guidance of text, the vision encoder can extract text-related features, similar to how humans focus on the most relevant parts of an image when considering a question. This results in generating better answers. Experiments on various datasets validate the effectiveness of the proposed method. Remarkably, without the need for additional training data, our propsoed method can bring more benefits to the baseline (LLaVA-1.5) compared with other concurrent methods. Furthermore, the proposed method consistently brings improvement in different settings.