Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

TL;DR

This paper introduces a multi-modal, graph-based, parameter-efficient fine-tuning method that leverages structural knowledge and regularization to improve performance across various datasets.

Contribution

It presents a novel multi-modal fine-tuning approach using graph neural networks and EWC regularization, addressing modality limitations and knowledge utilization in downstream tasks.

Findings

Improved accuracy on OxfordPets, Flowers102, and Food101 datasets.

Effective integration of multi-modal features via graph construction.

Mitigated forgetting with EWC regularization.

Abstract

With the advent of the era of foundation models, pre-training and fine-tuning have become common paradigms. Recently, parameter-efficient fine-tuning has garnered widespread attention due to its better balance between the number of learnable parameters and performance. However, some current parameter-efficient fine-tuning methods only model a single modality and lack the utilization of structural knowledge in downstream tasks. To address this issue, this paper proposes a multi-modal parameter-efficient fine-tuning method based on graph networks. Each image is fed into a multi-modal large language model (MLLM) to generate a text description. The image and its corresponding text description are then processed by a frozen image encoder and text encoder to generate image features and text features, respectively. A graph is constructed based on the similarity of the multi-modal feature nodes, and knowledge and relationships relevant to these features are extracted from each node. Additionally, Elastic Weight Consolidation (EWC) regularization is incorporated into the loss function to mitigate the problem of forgetting during task learning. The proposed model achieves test accuracies on the OxfordPets, Flowers102, and Food101 datasets that improve by 4.45%, 2.92%, and 0.23%, respectively. The code is available at https://github.com/yunche0/GA-Net/tree/master.