SDCA without Duality

TL;DR

This paper extends the Stochastic Dual Coordinate Ascent method to handle non-convex loss functions, demonstrating linear convergence when the expected loss remains convex, thus broadening its applicability.

Contribution

It introduces a variant of SDCA for non-convex losses and proves its linear convergence under convex expected loss conditions.

Findings

Linear convergence rate achieved for non-convex losses

SDCA variant applicable beyond convex loss functions

Theoretical guarantees established for the new method

Abstract

Stochastic Dual Coordinate Ascent is a popular method for solving regularized loss minimization for the case of convex losses. In this paper we show how a variant of SDCA can be applied for non-convex losses. We prove linear convergence rate even if individual loss functions are non-convex as long as the expected loss is convex.