Oblivious Stochastic Composite Optimization

TL;DR

This paper introduces three oblivious stochastic optimization algorithms that do not require prior knowledge of problem parameters, improving robustness and efficiency in large-scale convex optimization tasks.

Contribution

The paper develops three new oblivious stochastic algorithms combining mirror descent and dual averaging, applicable to various convex optimization settings without prior parameter knowledge.

Findings

Algorithms converge without prior knowledge of parameters

Effective in large-scale semidefinite programming

Show improved robustness and efficiency

Abstract

In stochastic convex optimization problems, most existing adaptive methods rely on prior knowledge about the diameter bound $D$ when the smoothness or the Lipschitz constant is unknown. This often significantly affects performance as only a rough approximation of $D$ is usually known in practice. Here, we bypass this limitation by combining mirror descent with dual averaging techniques and we show that, under oblivious step-sizes regime, our algorithms converge without any prior knowledge on the parameters of the problem. We introduce three oblivious stochastic algorithms to address different settings. The first algorithm is designed for objectives in relative scale, the second one is an accelerated version tailored for smooth objectives, whereas the last one is for relatively-smooth objectives. All three algorithms work without prior knowledge of the diameter of the feasible set, the Lipschitz constant or smoothness of the objective function. We use these results to revisit the problem of solving large-scale semidefinite programs using randomized first-order methods and stochastic smoothing. We extend our framework to relative scale and demonstrate the efficiency and robustness of our methods on large-scale semidefinite programs.