GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph

TL;DR

GraphAdapter introduces a dual knowledge graph approach to enhance vision-language model tuning by explicitly modeling inter-class relationships across visual and textual modalities, leading to improved performance on multiple benchmarks.

Contribution

The paper proposes GraphAdapter, an adapter-style tuning method that models dual-modality knowledge graphs to better exploit inter-class relationships in vision-language models.

Findings

Significantly outperforms previous adapter-based methods on 11 benchmarks.

Effectively leverages inter-class relationships across modalities.

Enhances classifier performance with dual knowledge graph modeling.

Abstract

Adapter-style efficient transfer learning (ETL) has shown excellent performance in the tuning of vision-language models (VLMs) under the low-data regime, where only a few additional parameters are introduced to excavate the task-specific knowledge based on the general and powerful representation of VLMs. However, most adapter-style works face two limitations: (i) modeling task-specific knowledge with a single modality only; and (ii) overlooking the exploitation of the inter-class relationships in downstream tasks, thereby leading to sub-optimal solutions. To mitigate that, we propose an effective adapter-style tuning strategy, dubbed GraphAdapter, which performs the textual adapter by explicitly modeling the dual-modality structure knowledge (i.e., the correlation of different semantics/classes in textual and visual modalities) with a dual knowledge graph. In particular, the dual knowledge graph is established with two sub-graphs, i.e., a textual knowledge sub-graph, and a visual knowledge sub-graph, where the nodes and edges represent the semantics/classes and their correlations in two modalities, respectively. This enables the textual feature of each prompt to leverage the task-specific structure knowledge from both textual and visual modalities, yielding a more effective classifier for downstream tasks. Extensive experimental results on 11 benchmark datasets reveal that our GraphAdapter significantly outperforms previous adapter-based methods. The code will be released at https://github.com/lixinustc/GraphAdapter