Exploiting Layerwise Convexity of Rectifier Networks with Sign Constrained Weights

TL;DR

This paper introduces sign constrained rectifier networks (SCRNs) with efficient training via MM algorithms, demonstrating their ability to separate disjoint pattern sets and decompose class patterns into convex clusters for better interpretability.

Contribution

The paper proposes a novel SCRN model with sign constraints, proving its capacity for pattern separation and convex clustering, enhancing interpretability and analysis of pattern structures.

Findings

SCRNs can separate any two disjoint pattern sets.

SCRNs decompose class patterns into convex clusters.

Training is efficiently solvable using MM algorithms.

Abstract

By introducing sign constraints on the weights, this paper proposes sign constrained rectifier networks (SCRNs), whose training can be solved efficiently by the well known majorization-minimization (MM) algorithms. We prove that the proposed two-hidden-layer SCRNs, which exhibit negative weights in the second hidden layer and negative weights in the output layer, are capable of separating any two (or more) disjoint pattern sets. Furthermore, the proposed two-hidden-layer SCRNs can decompose the patterns of each class into several clusters so that each cluster is convexly separable from all the patterns from the other classes. This provides a means to learn the pattern structures and analyse the discriminant factors between different classes of patterns.