Decoupling Vision and Language: Codebook Anchored Visual Adaptation

TL;DR

This paper introduces CRAFT, a lightweight, decoupled fine-tuning method for vision encoders in LVLMs that improves domain-specific visual task performance by anchoring representations to a stable codebook, without altering the language model.

Contribution

CRAFT is the first approach to decouple vision encoder fine-tuning from the language model using a discrete codebook, enabling flexible domain adaptation across different LVLM architectures.

Findings

Achieves an average of 13.51% performance gain across 10 benchmarks.

Maintains the linguistic capabilities of the original language models.

Outperforms existing continuous token-based adaptation methods.

Abstract

Large Vision-Language Models (LVLMs) use their vision encoders to translate images into representations for downstream reasoning, but the encoders often underperform in domain-specific visual tasks such as medical image diagnosis or fine-grained classification, where representation errors can cascade through the language model, leading to incorrect responses. Existing adaptation methods modify the continuous feature interface between encoder and language model through projector tuning or other parameter-efficient updates, which still couples the two components and requires re-alignment whenever the encoder changes. We introduce CRAFT (Codebook RegulAted Fine-Tuning), a lightweight method that fine-tunes the encoder using a discrete codebook that anchors visual representations to a stable token space, achieving domain adaptation without modifying other parts of the model. This decoupled design allows the adapted encoder to seamlessly boost the performance of LVLMs with different language architectures, as long as they share the same codebook. Empirically, CRAFT achieves an average gain of 13.51% across 10 domain-specific benchmarks such as VQARAD and PlantVillage, while preserving the LLM's linguistic capabilities and outperforming peer methods that operate on continuous tokens.