An Analysis of Regularized Approaches for Constrained Machine Learning

TL;DR

This paper analyzes regularization methods for constrained machine learning, demonstrating their limitations in guaranteeing all optimal solutions, especially in non-convex scenarios, and discusses the balance between accuracy and constraint satisfaction.

Contribution

It provides a formal analysis showing regularization approaches may fail to find all optimal solutions in constrained ML, particularly in non-convex cases.

Findings

Regularization cannot guarantee all optimal solutions.

In non-convex problems, some optima are missed by regularization.

Regularization approaches may not always satisfy constraints optimally.

Abstract

Regularization-based approaches for injecting constraints in Machine Learning (ML) were introduced to improve a predictive model via expert knowledge. We tackle the issue of finding the right balance between the loss (the accuracy of the learner) and the regularization term (the degree of constraint satisfaction). The key results of this paper is the formal demonstration that this type of approach cannot guarantee to find all optimal solutions. In particular, in the non-convex case there might be optima for the constrained problem that do not correspond to any multiplier value.