Linear convergence of SDCA in statistical estimation

TL;DR

This paper proves that SDCA converges linearly for a broad class of statistical models under mild conditions, even without strong convexity, and introduces a dual-free variant applicable to general regularizers.

Contribution

It establishes linear convergence of SDCA under restricted strong convexity without requiring strong convexity, and derives a dual-free version for general regularization.

Findings

SDCA converges linearly under mild conditions.

Applicable to models like Lasso, logistic regression, and SCAD.

Introduces a dual-free SDCA variant.

Abstract

In this paper, we consider stochastic dual coordinate (SDCA) {\em without} strongly convex assumption or convex assumption. We show that SDCA converges linearly under mild conditions termed restricted strong convexity. This covers a wide array of popular statistical models including Lasso, group Lasso, and logistic regression with $\ell_1$ regularization, corrected Lasso and linear regression with SCAD regularizer. This significantly improves previous convergence results on SDCA for problems that are not strongly convex. As a by product, we derive a dual free form of SDCA that can handle general regularization term, which is of interest by itself.