Beyond Convexity: Proximal-Perturbed Lagrangian Methods for Efficient Functional Constrained Optimization

TL;DR

This paper introduces a novel primal-dual algorithmic framework based on Proximal-Perturbed Augmented Lagrangian for efficiently solving non-convex functional constrained optimization problems, with theoretical guarantees and superior empirical performance.

Contribution

It develops a new Proximal-Perturbed Augmented Lagrangian approach for non-convex problems, providing simple algorithms with convergence guarantees and improved computational efficiency.

Findings

Non-asymptotic iteration complexity of (1/psilon^2)

Algorithm does not require continuous penalty parameter adjustment

Experimental results outperform existing regularization-based methods

Abstract

Non-convex functional constrained optimization problems have gained substantial attention in machine learning and data science, addressing broad requirements that typically go beyond the often performance-centric objectives. An influential class of algorithms for functional constrained problems is the class of primal-dual methods which has been extensively analyzed for convex problems. Nonetheless, the investigation of their efficacy for non-convex problems is under-explored. This paper develops a primal-dual algorithmic framework for solving such non-convex problems. This framework is built upon a novel form of the Lagrangian function, termed the {\em Proximal-Perturbed Augmented Lagrangian}, which enables the development of simple first-order algorithms that converge to a stationary solution under mild conditions. Notably, we study this framework under both non-smoothness and smoothness of the constraint function and provide three key contributions: (i) a simple algorithm that does not require the continuous adjustment of the penalty parameter; (ii) a non-asymptotic iteration complexity of $\widetilde{\mathcal{O}}(1/\epsilon^2)$; and (iii) extensive experimental results demonstrating the effectiveness of the proposed framework in terms of computational cost and performance, outperforming related approaches that use regularization (penalization) techniques and/or standard Lagrangian relaxation across diverse non-convex problems.