MISO is Making a Comeback With Better Proofs and Rates

TL;DR

This paper revitalizes the MISO stochastic variance reduction method by introducing a parameter-free variant with improved theoretical guarantees, including nonconvex convergence, and demonstrates its practical competitiveness through experiments.

Contribution

The paper introduces a new, parameter-free variant of MISO, extends its analysis to nonconvex settings, and provides minibatching bounds for practical efficiency.

Findings

MISO can be effectively used without strong convexity assumptions.

The new variant outperforms SAGA and SVRG on some real datasets.

Minibatching bounds enable linear speedup in practice.

Abstract

MISO, also known as Finito, was one of the first stochastic variance reduced methods discovered, yet its popularity is fairly low. Its initial analysis was significantly limited by the so-called Big Data assumption. Although the assumption was lifted in subsequent work using negative momentum, this introduced a new parameter and required knowledge of strong convexity and smoothness constants, which is rarely possible in practice. We rehabilitate the method by introducing a new variant that needs only smoothness constant and does not have any extra parameters. Furthermore, when removing the strong convexity constant from the stepsize, we present a new analysis of the method, which no longer uses the assumption that every component is strongly convex. This allows us to also obtain so far unknown nonconvex convergence of MISO. To make the proposed method efficient in practice, we derive minibatching bounds with arbitrary uniform sampling that lead to linear speedup when the expected minibatch size is in a certain range. Our numerical experiments show that MISO is a serious competitor to SAGA and SVRG and sometimes outperforms them on real datasets.