MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization

TL;DR

MURANA is a versatile stochastic optimization framework that unifies various variance reduction techniques, supporting sparse gradients and communication-efficient updates for large-scale and distributed problems.

Contribution

It introduces a general stochastic framework that encompasses many existing variance reduction methods and enables new algorithm designs with reduced computation and communication.

Findings

Supports sparse gradient activation

Reduces communication load via compression

Unifies multiple variance reduction strategies

Abstract

We propose a generic variance-reduced algorithm, which we call MUltiple RANdomized Algorithm (MURANA), for minimizing a sum of several smooth functions plus a regularizer, in a sequential or distributed manner. Our method is formulated with general stochastic operators, which allow us to model various strategies for reducing the computational complexity. For example, MURANA supports sparse activation of the gradients, and also reduction of the communication load via compression of the update vectors. This versatility allows MURANA to cover many existing randomization mechanisms within a unified framework, which also makes it possible to design new methods as special cases.