GLEMOS: Benchmark for Instantaneous Graph Learning Model Selection

TL;DR

GLEMOS is a comprehensive benchmark environment that evaluates and facilitates the rapid selection of graph learning models across various tasks, datasets, and settings, addressing the challenge of model choice in graph learning.

Contribution

It introduces GLEMOS, a benchmark with extensive data and evaluation settings for instantaneously selecting effective graph learning models, and is designed for easy extension and future research.

Findings

Existing model selection methods have limitations in different settings.

GLEMOS provides performance data for 366 models on 457 graphs.

The benchmark highlights directions for improving model selection techniques.

Abstract

The choice of a graph learning (GL) model (i.e., a GL algorithm and its hyperparameter settings) has a significant impact on the performance of downstream tasks. However, selecting the right GL model becomes increasingly difficult and time consuming as more and more GL models are developed. Accordingly, it is of great significance and practical value to equip users of GL with the ability to perform a near-instantaneous selection of an effective GL model without manual intervention. Despite the recent attempts to tackle this important problem, there has been no comprehensive benchmark environment to evaluate the performance of GL model selection methods. To bridge this gap, we present GLEMOS in this work, a comprehensive benchmark for instantaneous GL model selection that makes the following contributions. (i) GLEMOS provides extensive benchmark data for fundamental GL tasks, i.e., link prediction and node classification, including the performances of 366 models on 457 graphs on these tasks. (ii) GLEMOS designs multiple evaluation settings, and assesses how effectively representative model selection techniques perform in these different settings. (iii) GLEMOS is designed to be easily extended with new models, new graphs, and new performance records. (iv) Based on the experimental results, we discuss the limitations of existing approaches and highlight future research directions. To promote research on this significant problem, we make the benchmark data and code publicly available at https://github.com/facebookresearch/glemos.