Nearly Optimal Robust Method for Convex Compositional Problems with Heavy-Tailed Noise

TL;DR

This paper introduces a robust stochastic algorithm for convex compositional problems with heavy-tailed noise, achieving nearly optimal high-probability convergence bounds without strong convexity or smoothness assumptions.

Contribution

It develops the first single-trial algorithm with nearly optimal sub-Gaussian confidence bounds for heavy-tailed compositional problems, overcoming limitations of previous boosting methods.

Findings

Achieves nearly optimal high-probability convergence bounds.

Handles heavy-tailed noise with only second-order moment assumptions.

First to establish sub-Gaussian confidence bounds for such problems.

Abstract

In this paper, we propose robust stochastic algorithms for solving convex compositional problems of the form $f(\E_\xi g(\cdot; \xi)) + r(\cdot)$ by establishing {\bf sub-Gaussian confidence bounds} under weak assumptions about the tails of noise distribution, i.e., {\bf heavy-tailed noise} with bounded second-order moments. One can achieve this goal by using an existing boosting strategy that boosts a low probability convergence result into a high probability result. However, piecing together existing results for solving compositional problems suffers from several drawbacks: (i) the boosting technique requires strong convexity of the objective; (ii) it requires a separate algorithm to handle non-smooth $r$; (iii) it also suffers from an additional polylogarithmic factor of the condition number. To address these issues, we directly develop a single-trial stochastic algorithm for minimizing optimal strongly convex compositional objectives, which has a nearly optimal high probability convergence result matching the lower bound of stochastic strongly convex optimization up to a logarithmic factor. To the best of our knowledge, this is the first work that establishes nearly optimal sub-Gaussian confidence bounds for compositional problems under heavy-tailed assumptions.