Mini-Batch Primal and Dual Methods for SVMs

TL;DR

This paper investigates how mini-batch techniques affect the efficiency of stochastic optimization methods for SVMs, revealing that data spectral norm governs parallelization speedup and proposing new mini-batched SDCA variants.

Contribution

It introduces novel mini-batched SDCA variants and links parallelization speedup to the spectral norm of data for both primal and dual SVM optimization methods.

Findings

Spectral norm controls parallelization speedup for primal and dual methods.

New mini-batched SDCA variants with theoretical guarantees.

Optimization guarantees are based on the original hinge-loss primal problem.

Abstract

We address the issue of using mini-batches in stochastic optimization of SVMs. We show that the same quantity, the spectral norm of the data, controls the parallelization speedup obtained for both primal stochastic subgradient descent (SGD) and stochastic dual coordinate ascent (SCDA) methods and use it to derive novel variants of mini-batched SDCA. Our guarantees for both methods are expressed in terms of the original nonsmooth primal problem based on the hinge-loss.