Resilient Constrained Learning

TL;DR

This paper introduces a resilient constrained learning method that adaptively relaxes requirements during training, balancing performance and constraint satisfaction, with theoretical guarantees and practical applications in image classification and federated learning.

Contribution

It proposes a novel adaptive constrained learning approach that relaxes constraints based on their impact, providing theoretical guarantees and demonstrating effectiveness in real-world tasks.

Findings

Effective in image classification with multiple invariances

Applicable to heterogeneous federated learning scenarios

Provides approximation and generalization guarantees

Abstract

When deploying machine learning solutions, they must satisfy multiple requirements beyond accuracy, such as fairness, robustness, or safety. These requirements are imposed during training either implicitly, using penalties, or explicitly, using constrained optimization methods based on Lagrangian duality. Either way, specifying requirements is hindered by the presence of compromises and limited prior knowledge about the data. Furthermore, their impact on performance can often only be evaluated by actually solving the learning problem. This paper presents a constrained learning approach that adapts the requirements while simultaneously solving the learning task. To do so, it relaxes the learning constraints in a way that contemplates how much they affect the task at hand by balancing the performance gains obtained from the relaxation against a user-defined cost of that relaxation. We call this approach resilient constrained learning after the term used to describe ecological systems that adapt to disruptions by modifying their operation. We show conditions under which this balance can be achieved and introduce a practical algorithm to compute it, for which we derive approximation and generalization guarantees. We showcase the advantages of this resilient learning method in image classification tasks involving multiple potential invariances and in heterogeneous federated learning.