On stochastic optimization methods for Monte Carlo least-squares problems

TL;DR

This paper introduces novel stochastic optimization methods for Monte Carlo least-squares problems, demonstrating faster convergence by increasing sample precision and hybrid strategies, with improvements from Gauss-Newton preconditioning.

Contribution

It proposes increasing precision strategies and hybrid methods for Monte Carlo least-squares, achieving faster convergence than traditional stochastic gradient approaches.

Findings

Increasing sample size per iteration improves asymptotic convergence.

Hybrid methods combining increasing and constant precision enhance pre-asymptotic performance.

Gauss-Newton preconditioning significantly accelerates convergence in some problems.

Abstract

This work presents stochastic optimization methods targeted at least-squares problems involving Monte Carlo integration. While the most common approach to solving these problems is to apply stochastic gradient descent (SGD) or similar methods such as AdaGrad and Adam, which involve estimating a stochastic gradient from a small number of Monte Carlo samples computed at each iteration, we show that for this category of problems it is possible to achieve faster asymptotic convergence rates using an increasing number of samples per iteration instead, a strategy we call increasing precision (IP). We then improve pre-asymptotic convergence by introducing a hybrid approach that combines the qualities of increasing precision and otherwise "constant" precision, resulting in methods such as the IP-SGD hybrid and IP-AdaGrad hybrid, essentially by modifying their gradient estimators to have an equivalent effect to increasing precision. Finally, we observe that, in some problems, incorporating a Gauss-Newton preconditioner to the IP-SGD hybrid method can provide much better convergence than employing a Quasi-Newton approach or covariance-preconditioning as in AdaGrad or Adam.