Ellipsoid method for convex stochastic optimization in small dimension

TL;DR

This paper introduces an ellipsoid method with minibatching for convex stochastic optimization that converges linearly without requiring smoothness or strong convexity, especially effective in small dimensions.

Contribution

The paper proposes a novel ellipsoid-based algorithm with minibatching for convex stochastic optimization, achieving linear convergence and enabling efficient parallelization.

Findings

Converges linearly without smoothness or strong convexity assumptions.

Requires minibatch size proportional to the inverse square of the desired precision.

Suitable for small-dimensional problems due to quadratic complexity in dimension.

Abstract

The article considers minimization of the expectation of convex function. Problems of this type often arise in machine learning and a number of other applications. In practice, stochastic gradient descent (SGD) and similar procedures are often used to solve such problems. We propose to use the ellipsoid method with minibatching, which converges linearly and hence requires significantly less iterations than SGD. This is verified by our experiments, which are publicly available. The algorithm does not require neither smoothness nor strong convexity of target function to achieve linear convergence. We prove that the method arrives at approximate solution with given probability when using minibatches of size proportional to the desired precision to the power -2. This enables efficient parallel execution of the algorithm, whereas possibilities for batch parallelization of SGD are rather limited. Despite fast convergence, ellipsoid method can result in a greater total number of calls to oracle than SGD, which works decently with small batches. Complexity is quadratic in dimension of the problem, hence the method is suitable for relatively small dimensionalities.