On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Experts

TL;DR

This paper analyzes the statistical properties of parameter estimation in a softmax-contaminated mixture of experts model, revealing how overlapping knowledge affects estimability and establishing minimax optimal rates under certain conditions.

Contribution

It introduces a novel notion of distinguishability to characterize parameter estimability and derives minimax optimal convergence rates for the model's parameters.

Findings

Establishes minimax optimal estimation rates under model distinguishability.

Shows slower rates when the prompt overlaps with the pre-trained model.

Empirically validates theoretical results through numerical experiments.

Abstract

The softmax-contaminated mixture of experts (MoE) model is deployed when a large-scale pre-trained model, which plays the role of a fixed expert, is fine-tuned for learning downstream tasks by including a new contamination part, or prompt, functioning as a new, trainable expert. Despite its popularity and relevance, the theoretical properties of the softmax-contaminated MoE have remained unexplored in the literature. In the paper, we study the convergence rates of the maximum likelihood estimator of gating and prompt parameters in order to gain insights into the statistical properties and potential challenges of fine-tuning with a new prompt. We find that the estimability of these parameters is compromised when the prompt acquires overlapping knowledge with the pre-trained model, in the sense that we make precise by formulating a novel analytic notion of distinguishability. Under distinguishability of the pre-trained and prompt models, we derive minimax optimal estimation rates for all the gating and prompt parameters. By contrast, when the distinguishability condition is violated, these estimation rates become significantly slower due to their dependence on the prompt convergence rate to the pre-trained model. Finally, we empirically corroborate our theoretical findings through several numerical experiments.