GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism

TL;DR

GRAPHMOE introduces a self-rethinking mechanism with recurrent routing to interconnect experts in MoE networks, significantly improving language model reasoning and achieving state-of-the-art results.

Contribution

The paper proposes GRAPHMOE, a novel MoE architecture with a self-rethinking mechanism and recurrent routing, enhancing cognitive depth and reasoning in language models.

Findings

Outperforms existing LoRA-based models on benchmark datasets

Achieves state-of-the-art performance in language modeling tasks

Introduces a recurrent routing strategy that enhances expert interaction

Abstract

Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo GraphMoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models.