Self-tuned mirror descent schemes for smooth and nonsmooth high-dimensional stochastic optimization

TL;DR

This paper introduces self-tuned randomized block coordinate stochastic mirror descent methods for high-dimensional stochastic optimization, achieving optimal convergence rates and robustness without prior parameter knowledge.

Contribution

It develops self-tuned stepsize rules for RBSMD methods applicable to both smooth and nonsmooth problems, ensuring convergence and minimized mean squared error.

Findings

Self-tuned schemes outperform standard methods in convergence speed.

The methods are robust to initial stepsize and problem parameter choices.

Numerical experiments on SVM models validate theoretical advantages.

Abstract

We consider randomized block coordinate stochastic mirror descent (RBSMD) methods for solving high-dimensional stochastic optimization problems with strongly convex objective functions. Our goal is to develop RBSMD schemes that achieve a rate of convergence with a minimum constant factor with respect to the choice of the stepsize sequence. To this end, we consider both subgradient and gradient RBSMD methods addressing nonsmooth and smooth problems, respectively. For each scheme, (i) we develop self-tuned stepsize rules characterized in terms of problem parameters and algorithm settings; (ii) we show that the non-averaging iterate generated by the underlying RBSMD method converges to the optimal solution both in an almost sure and a mean sense; (iii) we show that the mean squared error is minimized. When problem parameters are unknown, we develop a unifying self-tuned update rule that can be applied in both subgradient and gradient SMD methods, and show that for any arbitrary and small enough initial stepsize, a suitably defined error bound is minimized. We provide constant factor comparisons with standard SMD and RBSMD methods. Our numerical experiments performed on an SVM model display that the self-tuned schemes are significantly robust with respect to the choice of problem parameters, and the initial stepsize.