Bridging the Gap of AutoGraph between Academia and Industry: Analysing AutoGraph Challenge at KDD Cup 2020

TL;DR

This paper analyzes the differences between academic and industrial approaches to automated graph neural networks, highlighting gaps in scope, effectiveness, and efficiency through the AutoGraph Challenge at KDD Cup 2020.

Contribution

It provides a comprehensive comparison of academic and industrial AutoGraph solutions, revealing key differences and quantifying their performance and resource utilization.

Findings

Industrial solutions outperform academic ones in scope and effectiveness.

Academic AutoML solutions achieve 97.3% of industrial accuracy on average.

Academic solutions are more resource-efficient, requiring fewer GPU hours and parameters.

Abstract

Graph structured data is ubiquitous in daily life and scientific areas and has attracted increasing attention. Graph Neural Networks (GNNs) have been proved to be effective in modeling graph structured data and many variants of GNN architectures have been proposed. However, much human effort is often needed to tune the architecture depending on different datasets. Researchers naturally adopt Automated Machine Learning on Graph Learning, aiming to reduce the human effort and achieve generally top-performing GNNs, but their methods focus more on the architecture search. To understand GNN practitioners' automated solutions, we organized AutoGraph Challenge at KDD Cup 2020, emphasizing on automated graph neural networks for node classification. We received top solutions especially from industrial tech companies like Meituan, Alibaba and Twitter, which are already open sourced on Github. After detailed comparisons with solutions from academia, we quantify the gaps between academia and industry on modeling scope, effectiveness and efficiency, and show that (1) academia AutoML for Graph solutions focus on GNN architecture search while industrial solutions, especially the winning ones in the KDD Cup, tend to obtain an overall solution (2) by neural architecture search only, academia solutions achieve on average 97.3% accuracy of industrial solutions (3) academia solutions are cheap to obtain with several GPU hours while industrial solutions take a few months' labors. Academic solutions also contain much fewer parameters.