Dynamic Mixture-of-Experts for Incremental Graph Learning

TL;DR

This paper introduces DyMoE, a dynamic mixture-of-experts approach for incremental graph learning that mitigates catastrophic forgetting and improves accuracy by selectively activating relevant experts, with reduced computational costs.

Contribution

The paper proposes a novel DyMoE GNN layer with specialized experts and a regularization loss, plus a sparse MoE technique to efficiently handle growing data over time.

Findings

Achieved 4.92% relative accuracy increase over baselines.

Effectively mitigates catastrophic forgetting in incremental graph learning.

Reduces computation time using sparse MoE with top-k experts.

Abstract

Graph incremental learning is a learning paradigm that aims to adapt trained models to continuously incremented graphs and data over time without the need for retraining on the full dataset. However, regular graph machine learning methods suffer from catastrophic forgetting when applied to incremental learning settings, where previously learned knowledge is overridden by new knowledge. Previous approaches have tried to address this by treating the previously trained model as an inseparable unit and using techniques to maintain old behaviors while learning new knowledge. These approaches, however, do not account for the fact that previously acquired knowledge at different timestamps contributes differently to learning new tasks. Some prior patterns can be transferred to help learn new data, while others may deviate from the new data distribution and be detrimental. To address this, we propose a dynamic mixture-of-experts (DyMoE) approach for incremental learning. Specifically, a DyMoE GNN layer adds new expert networks specialized in modeling the incoming data blocks. We design a customized regularization loss that utilizes data sequence information so existing experts can maintain their ability to solve old tasks while helping the new expert learn the new data effectively. As the number of data blocks grows over time, the computational cost of the full mixture-of-experts (MoE) model increases. To address this, we introduce a sparse MoE approach, where only the top-$k$ most relevant experts make predictions, significantly reducing the computation time. Our model achieved 4.92\% relative accuracy increase compared to the best baselines on class incremental learning, showing the model's exceptional power.