Dynamic Learning Rate Decay for Stochastic Variational Inference

TL;DR

This paper introduces a novel adaptive learning rate decay method for Stochastic Variational Inference that reduces sensitivity to initial learning rate choices and improves convergence by monitoring parameter oscillations.

Contribution

The authors propose a new decay strategy based on variational parameter oscillations, enhancing existing adaptive methods with minimal additional memory and computation.

Findings

Reduces sensitivity to initial learning rate settings.

Improves convergence stability in variational inference.

Compatible with other adaptive learning rate algorithms.

Abstract

Like many optimization algorithms, Stochastic Variational Inference (SVI) is sensitive to the choice of the learning rate. If the learning rate is too small, the optimization process may be slow, and the algorithm might get stuck in local optima. On the other hand, if the learning rate is too large, the algorithm may oscillate or diverge, failing to converge to a solution. Adaptive learning rate methods such as Adam, AdaMax, Adagrad, or RMSprop automatically adjust the learning rate based on the history of gradients. Nevertheless, if the base learning rate is too large, the variational parameters might still oscillate around the optimal solution. With learning rate schedules, the learning rate can be reduced gradually to mitigate this problem. However, the amount at which the learning rate should be decreased in each iteration is not known a priori, which can significantly impact the performance of the optimization. In this work, we propose a method to decay the learning rate based on the history of the variational parameters. We use an empirical measure to quantify the amount of oscillations against the progress of the variational parameters to adapt the learning rate. The approach requires little memory and is computationally efficient. We demonstrate in various numerical examples that our method reduces the sensitivity of the optimization performance to the learning rate and that it can also be used in combination with other adaptive learning rate methods.