Expressing linear equality constraints in feedforward neural networks

TL;DR

This paper introduces a novel method to impose linear equality constraints in feedforward neural networks by using a saddle-point Lagrangian with auxiliary variables, enabling standard training methods to handle constraints effectively.

Contribution

A new saddle-point Lagrangian formulation with auxiliary predictor variables that simplifies imposing linear equality constraints in neural networks.

Findings

Lagrange parameters interpreted as fixed-weight hidden units

Enables standard optimization despite constraints

Applicable to multi-label classification and autoencoders

Abstract

We seek to impose linear, equality constraints in feedforward neural networks. As top layer predictors are usually nonlinear, this is a difficult task if we seek to deploy standard convex optimization methods and strong duality. To overcome this, we introduce a new saddle-point Lagrangian with auxiliary predictor variables on which constraints are imposed. Elimination of the auxiliary variables leads to a dual minimization problem on the Lagrange multipliers introduced to satisfy the linear constraints. This minimization problem is combined with the standard learning problem on the weight matrices. From this theoretical line of development, we obtain the surprising interpretation of Lagrange parameters as additional, penultimate layer hidden units with fixed weights stemming from the constraints. Consequently, standard minimization approaches can be used despite the inclusion of Lagrange parameters -- a very satisfying, albeit unexpected, discovery. Examples ranging from multi-label classification to constrained autoencoders are envisaged in the future. The code has been made available at https://github.com/anandrajan0/smartalec