Differentiable Projection for Constrained Deep Learning

TL;DR

This paper introduces a differentiable projection layer for deep neural networks that efficiently incorporates prior constraints, improving performance without heavy computation, demonstrated through synthetic and image segmentation experiments.

Contribution

It proposes a novel differentiable projection layer to incorporate constraints into DNNs, replacing traditional KKT-based solutions with a more efficient approach.

Findings

The projection method outperforms baseline methods in experiments.

It effectively incorporates prior knowledge into DNN training.

The approach is computationally efficient and versatile.

Abstract

Deep neural networks (DNNs) have achieved extraordinary performance in solving different tasks in various fields. However, the conventional DNN model is steadily approaching the ground-truth value through loss backpropagation. In some applications, some prior knowledge could be easily obtained, such as constraints which the ground truth observation follows. Here, we try to give a general approach to incorporate information from these constraints to enhance the performance of the DNNs. Theoretically, we could formulate these kinds of problems as constrained optimization problems that KKT conditions could solve. In this paper, we propose to use a differentiable projection layer in DNN instead of directly solving time-consuming KKT conditions. The proposed projection method is differentiable, and no heavy computation is required. Finally, we also conducted some experiments using a randomly generated synthetic dataset and image segmentation task using the PASCAL VOC dataset to evaluate the performance of the proposed projection method. Experimental results show that the projection method is sufficient and outperforms baseline methods.