Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Function

TL;DR

This paper analyzes the benefits and limitations of sigmoid gating in mixture-of-experts models, proposing modifications to improve convergence and sample complexity, especially in classification tasks.

Contribution

It provides a comprehensive theoretical analysis of sigmoid gates in MoE, introduces a Euclidean scoring method, and addresses convergence and sample complexity issues.

Findings

Sigmoid gate has lower sample complexity than softmax for parameter estimation.

Incorporating a temperature parameter can cause exponential sample complexity.

Replacing the inner product score with a Euclidean score improves sample complexity to polynomial order.

Abstract

The sigmoid gate in mixture-of-experts (MoE) models has been empirically shown to outperform the softmax gate across several tasks, ranging from approximating feed-forward networks to language modeling. Additionally, recent efforts have demonstrated that the sigmoid gate is provably more sample-efficient than its softmax counterpart under regression settings. Nevertheless, there are three notable concerns that have not been addressed in the literature, namely (i) the benefits of the sigmoid gate have not been established under classification settings; (ii) existing sigmoid-gated MoE models may not converge to their ground-truth; and (iii) the effects of a temperature parameter in the sigmoid gate remain theoretically underexplored. To tackle these open problems, we perform a comprehensive analysis of multinomial logistic MoE equipped with a modified sigmoid gate to ensure model convergence. Our results indicate that the sigmoid gate exhibits a lower sample complexity than the softmax gate for both parameter and expert estimation. Furthermore, we find that incorporating a temperature into the sigmoid gate leads to a sample complexity of exponential order due to an intrinsic interaction between the temperature and gating parameters. To overcome this issue, we propose replacing the vanilla inner product score in the gating function with a Euclidean score that effectively removes that interaction, thereby substantially improving the sample complexity to a polynomial order.