DeAR: Fine-Grained VLM Adaptation by Decomposing Attention Head Roles

TL;DR

DeAR introduces a fine-grained approach to adapt vision-language models by decomposing attention head roles, improving task-specific adaptation while preserving zero-shot generalization.

Contribution

The paper proposes a novel method that classifies attention heads into functional roles and controls their interactions, enhancing VLM adaptation without sacrificing generalization.

Findings

Outperforms previous methods on fifteen datasets

Balances task adaptation and zero-shot generalization effectively

Classifies attention heads into Attribute, Generalization, and Mixed roles

Abstract

Prompt learning is a dominant paradigm for adapting pre-trained Vision-Language Models (VLMs) to downstream tasks. However, existing methods often rely on a simplistic, layer-centric view, assuming shallow layers capture general features while deep layers handle task-specific knowledge. This assumption results in uncontrolled interactions between learnable tokens and original tokens. Task-specific knowledge could degrades the model's core generalization and creates a trade-off between task adaptation and the preservation of zero-shot generalization. To address this, we challenge the layer-centric view and propose \textbf{DeAR}, a framework that achieves fine-grained VLM adaptation by \textbf{De}composing \textbf{A}ttention head \textbf{R}oles. We posit that the functional specialization within VLMs occurs not between layers, but at the finer-grained level of individual attention heads in the deeper layers. Based on this insight, we introduce a novel metric, Concept Entropy, to systematically classify attention heads into distinct functional roles: \textit{Attribute}, \textit{Generalization}, and \textit{Mixed}. Guided by these roles, we introduce specialized attribute tokens and a Role-Based Attention Mask mechanism to precisely control information flow, ensuring generalization heads remain isolated from task-specific knowledge. We further incorporate a Task-Adaptive Fusion Strategy for inference. Extensive experiments on fifteen datasets show that DeAR achieves a strong balance between task adaptation and generalization, outperforming previous methods across various tasks.