Fine-Tuning Graph Neural Networks via Graph Topology induced Optimal Transport

TL;DR

This paper introduces GTOT-Tuning, a novel graph topology-based optimal transport framework for fine-tuning GNNs that effectively preserves graph structure invariances and improves downstream task performance.

Contribution

The paper proposes a new OT-based fine-tuning method that leverages graph topology to enhance knowledge transfer and model performance in GNNs.

Findings

Achieves state-of-the-art results on eight downstream tasks.

Effectively preserves graph structure invariances during fine-tuning.

Reduces redundant transport procedures for efficient knowledge transfer.

Abstract

Recently, the pretrain-finetuning paradigm has attracted tons of attention in graph learning community due to its power of alleviating the lack of labels problem in many real-world applications. Current studies use existing techniques, such as weight constraint, representation constraint, which are derived from images or text data, to transfer the invariant knowledge from the pre-train stage to fine-tuning stage. However, these methods failed to preserve invariances from graph structure and Graph Neural Network (GNN) style models. In this paper, we present a novel optimal transport-based fine-tuning framework called GTOT-Tuning, namely, Graph Topology induced Optimal Transport fine-Tuning, for GNN style backbones. GTOT-Tuning is required to utilize the property of graph data to enhance the preservation of representation produced by fine-tuned networks. Toward this goal, we formulate graph local knowledge transfer as an Optimal Transport (OT) problem with a structural prior and construct the GTOT regularizer to constrain the fine-tuned model behaviors. By using the adjacency relationship amongst nodes, the GTOT regularizer achieves node-level optimal transport procedures and reduces redundant transport procedures, resulting in efficient knowledge transfer from the pre-trained models. We evaluate GTOT-Tuning on eight downstream tasks with various GNN backbones and demonstrate that it achieves state-of-the-art fine-tuning performance for GNNs.