Randomized Smoothing SVRG for Large-scale Nonsmooth Convex Optimization

TL;DR

This paper introduces a new randomized smoothing SVRG algorithm that efficiently solves large-scale nonsmooth convex optimization problems with improved convergence rates, applicable to machine learning tasks like ranking.

Contribution

The paper presents a novel algorithm achieving robust linear convergence for nonsmooth convex optimization without requiring extra error bounds or strong convexity.

Findings

Achieves superior time and gradient complexity compared to existing methods.

Demonstrates effective performance on large-scale ranking problems.

Provides theoretical analysis confirming linear convergence rate.

Abstract

In this paper, we consider the problem of minimizing the average of a large number of nonsmooth and convex functions. Such problems often arise in typical machine learning problems as empirical risk minimization, but are computationally very challenging. We develop and analyze a new algorithm that achieves robust linear convergence rate, and both its time complexity and gradient complexity are superior than state-of-art nonsmooth algorithms and subgradient-based schemes. Besides, our algorithm works without any extra error bound conditions on the objective function as well as the common strongly-convex condition. We show that our algorithm has wide applications in optimization and machine learning problems, and demonstrate experimentally that it performs well on a large-scale ranking problem.