Constrained Empirical Risk Minimization: Theory and Practice

TL;DR

This paper introduces a framework for exactly enforcing constraints on deep neural networks by restricting parameters to submanifolds, demonstrated with wavelet CNNs for medical contour prediction.

Contribution

It proposes a novel method for exact constraint enforcement in DNNs by parameter restriction, extending beyond typical soft constraint approaches.

Findings

Exact constraint enforcement is feasible during training.

Wavelet CNNs improve medical contour prediction accuracy.

Framework applies to constraints outside geometric deep learning.

Abstract

Deep Neural Networks (DNNs) are widely used for their ability to effectively approximate large classes of functions. This flexibility, however, makes the strict enforcement of constraints on DNNs an open problem. Here we present a framework that, under mild assumptions, allows the exact enforcement of constraints on parameterized sets of functions such as DNNs. Instead of imposing "soft'' constraints via additional terms in the loss, we restrict (a subset of) the DNN parameters to a submanifold on which the constraints are satisfied exactly throughout the entire training procedure. We focus on constraints that are outside the scope of equivariant networks used in Geometric Deep Learning. As a major example of the framework, we restrict filters of a Convolutional Neural Network (CNN) to be wavelets, and apply these wavelet networks to the task of contour prediction in the medical domain.