HardNet: Hard-Constrained Neural Networks with Universal Approximation Guarantees

TL;DR

HardNet introduces a novel neural network framework that guarantees the satisfaction of multiple input-dependent hard constraints during training, enhancing safety and reliability without sacrificing model capacity.

Contribution

It is the first method enabling efficient, differentiable enforcement of multiple input-dependent inequality constraints in neural networks during end-to-end training.

Findings

HardNet effectively enforces multiple constraints in various applications.

It retains universal approximation capabilities of neural networks.

Demonstrates improved safety and reliability in safety-critical systems.

Abstract

Incorporating prior knowledge or specifications of input-output relationships into machine learning models has attracted significant attention, as it enhances generalization from limited data and yields conforming outputs. However, most existing approaches use soft constraints by penalizing violations through regularization, which offers no guarantee of constraint satisfaction, especially on inputs far from the training distribution--an essential requirement in safety-critical applications. On the other hand, imposing hard constraints on neural networks may hinder their representational power, adversely affecting performance. To address this, we propose HardNet, a practical framework for constructing neural networks that inherently satisfy hard constraints without sacrificing model capacity. Unlike approaches that modify outputs only at inference time, HardNet enables end-to-end training with hard constraint guarantees, leading to improved performance. To the best of our knowledge, HardNet is the first method that enables efficient and differentiable enforcement of more than one input-dependent inequality constraint. It allows unconstrained optimization of the network parameters using standard algorithms by appending a differentiable closed-form enforcement layer to the network's output. Furthermore, we show that HardNet retains neural networks' universal approximation capabilities. We demonstrate its versatility and effectiveness across various applications: learning with piecewise constraints, learning optimization solvers with guaranteed feasibility, and optimizing control policies in safety-critical systems.