A Process for the Evaluation of Node Embedding Methods in the Context of Node Classification

TL;DR

This paper introduces a standardized, fair evaluation process for node embedding methods in node classification tasks, enabling reproducible comparisons and revealing that lower-dimensional embeddings with proper hyperparameters can perform well.

Contribution

The paper develops a reproducible evaluation process for node embeddings, facilitating fair comparisons and providing insights into hyperparameter effects on performance.

Findings

Lower-dimensional embeddings can perform well with proper hyperparameters.

Multiple hyperparameter settings yield similar performance, reducing the need for extensive tuning.

The evaluation process supports reproducible and fair comparison of node embedding methods.

Abstract

Node embedding methods find latent lower-dimensional representations which are used as features in machine learning models. In the last few years, these methods have become extremely popular as a replacement for manual feature engineering. Since authors use various approaches for the evaluation of node embedding methods, existing studies can rarely be efficiently and accurately compared. We address this issue by developing a process for a fair and objective evaluation of node embedding procedures w.r.t. node classification. This process supports researchers and practitioners to compare new and existing methods in a reproducible way. We apply this process to four popular node embedding methods and make valuable observations. With an appropriate combination of hyperparameters, good performance can be achieved even with embeddings of lower dimensions, which is positive for the run times of the downstream machine learning task and the embedding algorithm. Multiple hyperparameter combinations yield similar performance. Thus, no extensive, time-consuming search is required to achieve reasonable performance in most cases.