CMA Light: a novel Minibatch Algorithm for large-scale non convex finite sum optimization

TL;DR

CMA Light is a new minibatch gradient algorithm designed for large-scale non-convex finite sum optimization, offering global convergence and reduced computational effort compared to existing methods, with promising results on image classification.

Contribution

It introduces CMA Light, a globally convergent minibatch algorithm that removes the need for full objective evaluations per iteration, improving efficiency in large-scale non-convex optimization.

Findings

Requires less computational effort than state-of-the-art optimizers

Proven to be globally convergent under mild assumptions

Shows promising early results on large-scale image classification

Abstract

The supervised training of a deep neural network on a given dataset consists in the unconstrained minimization of the finite sum of continuously differentiable functions, commonly referred to as loss with respect to the samples. These functions depend on the network parameters and most of the times are non-convex. We develop CMA Light, a globally convergent mini-batch gradient method to tackle this problem. We consider the recently introduced Controlled Minibatch Algorithm (CMA) framework and we overcome its main bottleneck, removing the need for at least one evaluation of the whole objective function per iteration. We prove globally convergence of CMA Light under mild assumptions and we discuss extensive computational results on the same experimental test-bed used for CMA, showing that CMA Light requires less computational effort than most of the state-of-the-art optimizers. Eventually, we present early results on a large-scale Image Classification task.