Per-Block-Convex Data Modeling by Accelerated Stochastic Approximation

TL;DR

This paper introduces an online, modular stochastic approximation algorithm for large-scale, per-block-convex, non-smooth optimization problems with provable convergence, applicable to various data modeling tasks.

Contribution

It develops a novel accelerated stochastic approximation method for non-convex, per-block-convex problems, with linear complexity and convergence guarantees without strict assumptions.

Findings

Proven quadratic convergence rate to an accumulation point.

Superior performance over block coordinate descent in experiments.

Reduced complexity compared to ADMM in numerical tests.

Abstract

Applications involving dictionary learning, non-negative matrix factorization, subspace clustering, and parallel factor tensor decomposition tasks motivate well algorithms for per-block-convex and non-smooth optimization problems. By leveraging the stochastic approximation paradigm and first-order acceleration schemes, this paper develops an online and modular learning algorithm for a large class of non-convex data models, where convexity is manifested only per-block of variables whenever the rest of them are held fixed. The advocated algorithm incurs computational complexity that scales linearly with the number of unknowns. Under minimal assumptions on the cost functions of the composite optimization task, without bounding constraints on the optimization variables, or any explicit information on bounds of Lipschitz coefficients, the expected cost evaluated online at the resultant iterates is provably convergent with quadratic rate to an accumulation point of the (per-block) minima, while subgradients of the expected cost asymptotically vanish in the mean-squared sense. The merits of the general approach are demonstrated in two online learning setups: (i) Robust linear regression using a sparsity-cognizant total least-squares criterion; and (ii) semi-supervised dictionary learning for network-wide link load tracking and imputation with missing entries. Numerical tests on synthetic and real data highlight the potential of the proposed framework for streaming data analytics by demonstrating superior performance over block coordinate descent, and reduced complexity relative to the popular alternating-direction method of multipliers.