Efficient and provably convergent end-to-end training of deep neural networks with linear constraints

TL;DR

This paper introduces a new method for training deep neural networks with linear constraints using a novel HS-Jacobian, enabling efficient backpropagation and convergence guarantees, with demonstrated superior performance across various applications.

Contribution

We develop an HS-Jacobian for projection layers, proving its conservativeness, and integrate it into backpropagation, providing the first convergence guarantees for constrained deep network training.

Findings

HS-Jacobian enables seamless backpropagation with linear constraints.

The proposed Adam-based algorithm converges for linearly constrained networks.

Experiments show superior performance in finance, vision, and architecture design.

Abstract

Training a deep neural network with the outputs of selected layers satisfying linear constraints is required in many contemporary data-driven applications. While this can be achieved by incorporating projection layers into the neural network, its end-to-end training remains challenging due to the lack of rigorous theory and efficient algorithms for backpropagation. A key difficulty in developing the theory and efficient algorithms for backpropagation arose from the nonsmoothness of the solution mapping of the projection layer. To address this bottleneck, we introduce an efficiently computable HS-Jacobian to the projection layer. Importantly, we prove that the HS-Jacobian is a conservative mapping for the projection operator onto the polyhedral set, enabling its seamless integration into the nonsmooth automatic differentiation framework for backpropagation. Therefore, many efficient algorithms, such as Adam, can be applied for end-to-end training of deep neural networks with linear constraints. Particularly, we establish convergence guarantees of the HS-Jacobian based Adam algorithm for training linearly constrained deep neural networks. Extensive experiment results on several important applications, including finance, computer vision, and network architecture design, demonstrate the superior performance of our method compared to other existing popular methods.