Imposing Hard Constraints on Deep Networks: Promises and Limitations

TL;DR

This paper explores the feasibility of imposing hard constraints on deep neural networks, demonstrating computational practicality but revealing limited theoretical advantages over soft constraints.

Contribution

It shows that hard constraints can be computationally feasible in deep networks, but their theoretical benefits are not realized, prompting further research.

Findings

Hard constraints are computationally feasible in deep networks.

Hard constraints do not outperform soft constraints in practice.

Theoretical benefits of hard constraints are not observed in experiments.

Abstract

Imposing constraints on the output of a Deep Neural Net is one way to improve the quality of its predictions while loosening the requirements for labeled training data. Such constraints are usually imposed as soft constraints by adding new terms to the loss function that is minimized during training. An alternative is to impose them as hard constraints, which has a number of theoretical benefits but has not been explored so far due to the perceived intractability of the problem. In this paper, we show that imposing hard constraints can in fact be done in a computationally feasible way and delivers reasonable results. However, the theoretical benefits do not materialize and the resulting technique is no better than existing ones relying on soft constraints. We analyze the reasons for this and hope to spur other researchers into proposing better solutions.