Provably Training Overparameterized Neural Network Classifiers with Non-convex Constraints

TL;DR

This paper demonstrates that overparameterized neural networks can be effectively trained to satisfy non-convex constraints with near-optimal solutions using project stochastic gradient descent, supported by no-regret online learning analysis.

Contribution

It provides the first theoretical guarantee for training overparameterized neural networks under non-convex constraints, extending beyond simple models.

Findings

Neural networks can achieve near-optimal solutions with non-convex constraints.

The approach uses project stochastic gradient descent with theoretical guarantees.

No-regret analysis of online learning for neural networks is developed.

Abstract

Training a classifier under non-convex constraints has gotten increasing attention in the machine learning community thanks to its wide range of applications such as algorithmic fairness and class-imbalanced classification. However, several recent works addressing non-convex constraints have only focused on simple models such as logistic regression or support vector machines. Neural networks, one of the most popular models for classification nowadays, are precluded and lack theoretical guarantees. In this work, we show that overparameterized neural networks could achieve a near-optimal and near-feasible solution of non-convex constrained optimization problems via the project stochastic gradient descent. Our key ingredient is the no-regret analysis of online learning for neural networks in the overparameterization regime, which may be of independent interest in online learning applications.