Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study

TL;DR

This paper systematically evaluates techniques for diversifying GNN experts in ensembles, demonstrating how different strategies impact performance and providing insights for designing effective MoE frameworks on graph data.

Contribution

It is the first comprehensive empirical study of expert diversification methods for GNN ensembles, analyzing 20 strategies across numerous benchmarks.

Findings

Diversification techniques vary in effectiveness for GNN ensembles.

Certain strategies significantly improve ensemble performance.

Insights into training maximally diverse GNN experts are provided.

Abstract

Graph Neural Networks (GNNs) have become essential tools for learning on relational data, yet the performance of a single GNN is often limited by the heterogeneity present in real-world graphs. Recent advances in Mixture-of-Experts (MoE) frameworks demonstrate that assembling multiple, explicitly diverse GNNs with distinct generalization patterns can significantly improve performance. In this work, we present the first systematic empirical study of expert-level diversification techniques for GNN ensembles. Evaluating 20 diversification strategies -- including random re-initialization, hyperparameter tuning, architectural variation, directionality modeling, and training data partitioning -- across 14 node classification benchmarks, we construct and analyze over 200 ensemble variants. Our comprehensive evaluation examines each technique in terms of expert diversity, complementarity, and ensemble performance. We also uncovers mechanistic insights into training maximally diverse experts. These findings provide actionable guidance for expert training and the design of effective MoE frameworks on graph data. Our code is available at https://github.com/Hydrapse/bench-gnn-diversification.