Neural networks with linear threshold activations: structure and algorithms

TL;DR

This paper characterizes the functions representable by neural networks with linear threshold activations, establishes the necessity of two hidden layers, and introduces a polynomial-time algorithm for global optimization, along with a new neural network class.

Contribution

It provides a precise characterization of linear threshold neural networks, proves the necessity of two hidden layers, and develops a universal polynomial-time algorithm for empirical risk minimization.

Findings

Two hidden layers are necessary and sufficient for representing certain functions.

A polynomial-time algorithm for global ERM optimization in fixed architecture neural networks.

Introduction of shortcut linear threshold networks with desirable theoretical properties.

Abstract

In this article we present new results on neural networks with linear threshold activation functions. We precisely characterize the class of functions that are representable by such neural networks and show that 2 hidden layers are necessary and sufficient to represent any function representable in the class. This is a surprising result in the light of recent exact representability investigations for neural networks using other popular activation functions like rectified linear units (ReLU). We also give precise bounds on the sizes of the neural networks required to represent any function in the class. Finally, we design an algorithm to solve the empirical risk minimization (ERM) problem to global optimality for these neural networks with a fixed architecture. The algorithm's running time is polynomial in the size of the data sample, if the input dimension and the size of the network architecture are considered fixed constants. The algorithm is unique in the sense that it works for any architecture with any number of layers, whereas previous polynomial time globally optimal algorithms work only for very restricted classes of architectures. Using these insights, we propose a new class of neural networks that we call shortcut linear threshold networks. To the best of our knowledge, this way of designing neural networks has not been explored before in the literature. We show that these neural networks have several desirable theoretical properties.