Momentum SVGD-EM for Accelerated Maximum Marginal Likelihood Estimation

TL;DR

This paper introduces Momentum SVGD-EM, an accelerated algorithm combining Stein variational gradient descent and Nesterov momentum to improve the efficiency of maximum marginal likelihood estimation, especially in high-dimensional problems.

Contribution

It proposes a novel accelerated algorithm, Momentum SVGD-EM, that enhances convergence speed of particle-based MMLE methods using Nesterov acceleration.

Findings

Accelerates convergence across various tasks.

Effective in both low- and high-dimensional settings.

Demonstrates improved iteration efficiency.

Abstract

Maximum marginal likelihood estimation (MMLE) can be formulated as the optimization of a free energy functional. From this viewpoint, the Expectation-Maximisation (EM) algorithm admits a natural interpretation as a coordinate descent method over the joint space of model parameters and probability measures. Recently, a significant body of work has adopted this perspective, leading to interacting particle algorithms for MMLE. In this paper, we propose an accelerated version of one such procedure, based on Stein variational gradient descent (SVGD), by introducing Nesterov acceleration in both the parameter updates and in the space of probability measures. The resulting method, termed Momentum SVGD-EM, consistently accelerates convergence in terms of required iterations across various tasks of increasing difficulty, demonstrating effectiveness in both low- and high-dimensional settings.