Dual Free Adaptive Mini-batch SDCA for Empirical Risk Minimization

TL;DR

This paper introduces a dual free adaptive mini-batch SDCA algorithm with non-uniform coordinate selection and adaptive probabilities, improving efficiency for empirical risk minimization.

Contribution

It extends dual free SDCA by incorporating adaptive, non-uniform coordinate sampling and an efficient mini-batch generation method, enhancing optimization performance.

Findings

Demonstrates improved convergence efficiency through adaptive sampling.

Shows effectiveness of the method with numerical experiments.

Achieves faster empirical risk minimization compared to existing methods.

Abstract

In this paper we develop dual free mini-batch SDCA with adaptive probabilities for regularized empirical risk minimization. This work is motivated by recent work of Shai Shalev-Shwartz on dual free SDCA method, however, we allow a non-uniform selection of "dual" coordinates in SDCA. Moreover, the probability can change over time, making it more efficient than fix uniform or non-uniform selection. We also propose an efficient procedure to generate a random non-uniform mini-batch through iterative process. The work is concluded with multiple numerical experiments to show the efficiency of proposed algorithms.