Stochastic dual averaging methods using variance reduction techniques for regularized empirical risk minimization problems

TL;DR

This paper introduces two stochastic dual averaging algorithms with variance reduction for regularized empirical risk minimization, producing sparser solutions with strong theoretical convergence guarantees.

Contribution

The paper presents novel stochastic dual averaging methods that do not require averaging past solutions, leading to sparser, more interpretable models with optimal convergence rates.

Findings

Achieves the best known convergence rates among nonaccelerated stochastic methods.

Produces sparser solutions compared to existing methods.

Supports both strongly and non-strongly convex regularizers.

Abstract

We consider a composite convex minimization problem associated with regularized empirical risk minimization, which often arises in machine learning. We propose two new stochastic gradient methods that are based on stochastic dual averaging method with variance reduction. Our methods generate a sparser solution than the existing methods because we do not need to take the average of the history of the solutions. This is favorable in terms of both interpretability and generalization. Moreover, our methods have theoretical support for both a strongly and a non-strongly convex regularizer and achieve the best known convergence rates among existing nonaccelerated stochastic gradient methods.