Block Coordinate Descent Methods for Structured Nonconvex Optimization with Nonseparable Constraints: Optimality Conditions and Global Convergence

TL;DR

This paper introduces a novel Block Coordinate Descent method for structured nonconvex optimization with nonseparable constraints, providing new optimality conditions and proving global convergence under mild assumptions, with practical applications in machine learning.

Contribution

The paper develops BCD algorithms for nonconvex problems with nonseparable constraints, establishing stronger optimality criteria and convergence guarantees compared to traditional methods.

Findings

BCD methods achieve Q-linear convergence under Luo-Tseng error bounds.

Experiments show superior objective values over existing methods.

Application to real-world data demonstrates practical effectiveness.

Abstract

Coordinate descent algorithms are widely used in machine learning and large-scale data analysis due to their strong optimality guarantees and impressive empirical performance in solving non-convex problems. In this work, we introduce Block Coordinate Descent (BCD) method for structured nonconvex optimization with nonseparable constraints. Unlike traditional large-scale Coordinate Descent (CD) approaches, we do not assume the constraints are separable. Instead, we account for the possibility of nonlinear coupling among them. By leveraging the inherent problem structure, we propose new CD methods to tackle this specific challenge. Under the relatively mild condition of locally bounded non-convexity, we demonstrate that achieving coordinate-wise stationary points offer a stronger optimality criterion compared to standard critical points. Furthermore, under the Luo-Tseng error bound conditions, our BCD methods exhibit Q-linear convergence to coordinate-wise stationary points or critical points. To demonstrate the practical utility of our methods, we apply them to various machine learning and signal processing models. We also provide the geometry analysis for the models. Experiments on real-world data consistently demonstrate the superior objective values of our approaches compared to existing methods.