Analytical aspects of non-differentiable neural networks

TL;DR

This paper investigates the expressivity and approximation capabilities of non-differentiable neural networks, proposing regularisation techniques and approximation results for networks with quantized and Heaviside-type activations.

Contribution

It demonstrates that quantized neural networks match the expressivity of deep neural networks and introduces a stochastic regularisation method for differentiable approximation of non-differentiable networks.

Findings

QNNs have the same approximation power as DNNs for Lipschitz functions.

A layer-wise stochastic regularisation technique effectively approximates non-differentiable networks.

Smooth networks can approximate Heaviside-type activation networks under certain conditions.

Abstract

Research in computational deep learning has directed considerable efforts towards hardware-oriented optimisations for deep neural networks, via the simplification of the activation functions, or the quantization of both activations and weights. The resulting non-differentiability (or even discontinuity) of the networks poses some challenging problems, especially in connection with the learning process. In this paper, we address several questions regarding both the expressivity of quantized neural networks and approximation techniques for non-differentiable networks. First, we answer in the affirmative the question of whether QNNs have the same expressivity as DNNs in terms of approximation of Lipschitz functions in the $L^{\infty}$ norm. Then, considering a continuous but not necessarily differentiable network, we describe a layer-wise stochastic regularisation technique to produce differentiable approximations, and we show how this approach to regularisation provides elegant quantitative estimates. Finally, we consider networks defined by means of Heaviside-type activation functions, and prove for them a pointwise approximation result by means of smooth networks under suitable assumptions on the regularised activations.