Learning DNN networks using un-rectifying ReLU with compressed sensing application

TL;DR

This paper introduces a novel un-rectifying ReLU technique that reformulates deep learning as a combinatorial optimization problem, enabling improved compressed sensing recovery with state-of-the-art results on MNIST and natural images.

Contribution

It proposes a data-dependent un-rectifying ReLU method that relaxes discrete activation variables, allowing optimization via constrained real-domain methods and achieving global convergence.

Findings

Achieved state-of-the-art compressed sensing recovery performance.

Demonstrated global convergence potential of the proposed method.

Validated effectiveness on MNIST and natural image datasets.

Abstract

The un-rectifying technique expresses a non-linear point-wise activation function as a data-dependent variable, which means that the activation variable along with its input and output can all be employed in optimization. The ReLU network in this study was un-rectified means that the activation functions could be replaced with data-dependent activation variables in the form of equations and constraints. The discrete nature of activation variables associated with un-rectifying ReLUs allows the reformulation of deep learning problems as problems of combinatorial optimization. However, we demonstrate that the optimal solution to a combinatorial optimization problem can be preserved by relaxing the discrete domains of activation variables to closed intervals. This makes it easier to learn a network using methods developed for real-domain constrained optimization. We also demonstrate that by introducing data-dependent slack variables as constraints, it is possible to optimize a network based on the augmented Lagrangian approach. This means that our method could theoretically achieve global convergence and all limit points are critical points of the learning problem. In experiments, our novel approach to solving the compressed sensing recovery problem achieved state-of-the-art performance when applied to the MNIST database and natural images.