Distributed Nonconvex Optimization for Sparse Representation

TL;DR

This paper introduces a unified distributed algorithmic framework for nonconvex sparse representation problems, capable of handling large-scale data across various network structures without restrictive gradient assumptions.

Contribution

It presents the first distributed method with convergence guarantees for nonconvex bi-criteria optimization problems in statistical learning and related fields.

Findings

Algorithm converges to d-stationary solutions asymptotically.

Applicable to arbitrary networks with time-varying connectivity.

Does not require bounded subgradient assumption.

Abstract

We consider a non-convex constrained Lagrangian formulation of a fundamental bi-criteria optimization problem for variable selection in statistical learning; the two criteria are a smooth (possibly) nonconvex loss function, measuring the fitness of the model to data, and the latter function is a difference-of-convex (DC) regularization, employed to promote some extra structure on the solution, like sparsity. This general class of nonconvex problems arises in many big-data applications, from statistical machine learning to physical sciences and engineering. We develop the first unified distributed algorithmic framework for these problems and establish its asymptotic convergence to d-stationary solutions. Two key features of the method are: i) it can be implemented on arbitrary networks (digraphs) with (possibly) time-varying connectivity; and ii) it does not require the restrictive assumption that the (sub)gradient of the objective function is bounded, which enlarges significantly the class of statistical learning problems that can be solved with convergence guarantees.