Adaptive Transfer Learning on Graph Neural Networks

TL;DR

This paper introduces an adaptive transfer learning framework for GNNs that dynamically combines auxiliary self-supervised tasks with target tasks during fine-tuning, significantly enhancing performance across multiple downstream tasks.

Contribution

It proposes a novel adaptive auxiliary loss weighting model using meta-learning to better transfer knowledge from self-supervised tasks to downstream GNN tasks.

Findings

Improved performance on multiple downstream tasks.

Effective combination of auxiliary and target tasks.

Versatility across various transfer learning approaches.

Abstract

Graph neural networks (GNNs) is widely used to learn a powerful representation of graph-structured data. Recent work demonstrates that transferring knowledge from self-supervised tasks to downstream tasks could further improve graph representation. However, there is an inherent gap between self-supervised tasks and downstream tasks in terms of optimization objective and training data. Conventional pre-training methods may be not effective enough on knowledge transfer since they do not make any adaptation for downstream tasks. To solve such problems, we propose a new transfer learning paradigm on GNNs which could effectively leverage self-supervised tasks as auxiliary tasks to help the target task. Our methods would adaptively select and combine different auxiliary tasks with the target task in the fine-tuning stage. We design an adaptive auxiliary loss weighting model to learn the weights of auxiliary tasks by quantifying the consistency between auxiliary tasks and the target task. In addition, we learn the weighting model through meta-learning. Our methods can be applied to various transfer learning approaches, it performs well not only in multi-task learning but also in pre-training and fine-tuning. Comprehensive experiments on multiple downstream tasks demonstrate that the proposed methods can effectively combine auxiliary tasks with the target task and significantly improve the performance compared to state-of-the-art methods.