Revisiting Incremental Stochastic Majorization-Minimization Algorithms with Applications to Mixture of Experts

TL;DR

This paper introduces a generalized incremental stochastic Majorization-Minimization algorithm that relaxes EM constraints, providing theoretical guarantees and outperforming standard optimizers in mixture of experts models, with applications to real-world data.

Contribution

It develops a flexible incremental stochastic MM algorithm with convergence guarantees, extending EM applicability and demonstrating superior empirical performance.

Findings

Algorithm converges to stationary points with vanishing gradients.

Outperforms stochastic gradient descent, RMSProp, Adam, and second-order methods.

Effective on real-world bioinformatics and ecophysiological datasets.

Abstract

Processing high-volume, streaming data is increasingly common in modern statistics and machine learning, where batch-mode algorithms are often impractical because they require repeated passes over the full dataset. This has motivated incremental stochastic estimation methods, including the incremental stochastic Expectation-Maximization (EM) algorithm formulated via stochastic approximation. In this work, we revisit and analyze an incremental stochastic variant of the Majorization-Minimization (MM) algorithm, which generalizes incremental stochastic EM as a special case. Our approach relaxes key EM requirements, such as explicit latent-variable representations, enabling broader applicability and greater algorithmic flexibility. We establish theoretical guarantees for the incremental stochastic MM algorithm, proving consistency in the sense that the iterates converge to a stationary point characterized by a vanishing gradient of the objective. We demonstrate these advantages on a softmax-gated mixture of experts (MoE) regression problem, for which no stochastic EM algorithm is available. Empirically, our method consistently outperforms widely used stochastic optimizers, including stochastic gradient descent, root mean square propagation, adaptive moment estimation, and second-order clipped stochastic optimization. These results support the development of new incremental stochastic algorithms, given the central role of softmax-gated MoE architectures in contemporary deep neural networks for heterogeneous data modeling. Beyond synthetic experiments, we also validate practical effectiveness on two real-world datasets, including a bioinformatics study of dent maize genotypes under drought stress that integrates high-dimensional proteomics with ecophysiological traits, where incremental stochastic MM yields stable gains in predictive performance.