OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning

TL;DR

This paper introduces OMoE, an orthogonal training method for mixture-of-experts models that enhances diversity among experts, leading to improved performance and efficiency in parameter-efficient fine-tuning tasks.

Contribution

The paper proposes OMoE, a novel orthogonal finetuning approach that promotes expert diversity in MoE models without increasing memory or computational costs.

Findings

OMoE improves model performance on commonsense reasoning benchmarks.

It reduces the number of experts needed for comparable or better results.

The method maintains the learning objective while enforcing expert diversity.

Abstract

Building mixture-of-experts (MoE) architecture for Low-rank adaptation (LoRA) is emerging as a potential direction in parameter-efficient fine-tuning (PEFT) for its modular design and remarkable performance. However, simply stacking the number of experts cannot guarantee significant improvement. In this work, we first conduct qualitative analysis to indicate that experts collapse to similar representations in vanilla MoE, limiting the capacity of modular design and computational efficiency. Ulteriorly, Our analysis reveals that the performance of previous MoE variants maybe limited by a lack of diversity among experts. Motivated by these findings, we propose Orthogonal Mixture-of-Experts (OMoE), a resource-efficient MoE variant that trains experts in an orthogonal manner to promote diversity. In OMoE, a Gram-Schmidt process is leveraged to enforce that the experts' representations lie within the Stiefel manifold. By applying orthogonal constraints directly to the architecture, OMoE keeps the learning objective unchanged, without compromising optimality. Our method is simple and alleviates memory bottlenecks, as it incurs minimal experts compared to vanilla MoE models. Experiments on diverse commonsense reasoning benchmarks demonstrate that OMoE can consistently achieve stable and efficient performance improvement when compared with the state-of-the-art methods while significantly reducing the number of required experts.