Approximation of Lipschitz Functions using Deep Spline Neural Networks

TL;DR

This paper introduces learnable spline activation functions with multiple linear regions for Lipschitz-constrained neural networks, demonstrating their optimality and superior expressiveness over ReLU and other activation functions, supported by theoretical proofs and numerical results.

Contribution

It proposes using learnable spline activation functions with at least three linear regions, proving their optimality among Lipschitz activations and their enhanced expressiveness compared to existing methods.

Findings

Spline activations are optimal among Lipschitz functions for approximation.

Spline activations are as expressive as Groupsort for spectral-norm constraints.

Numerical results support theoretical advantages of spline activations.

Abstract

Lipschitz-constrained neural networks have many applications in machine learning. Since designing and training expressive Lipschitz-constrained networks is very challenging, there is a need for improved methods and a better theoretical understanding. Unfortunately, it turns out that ReLU networks have provable disadvantages in this setting. Hence, we propose to use learnable spline activation functions with at least 3 linear regions instead. We prove that this choice is optimal among all component-wise $1$-Lipschitz activation functions in the sense that no other weight constrained architecture can approximate a larger class of functions. Additionally, this choice is at least as expressive as the recently introduced non component-wise Groupsort activation function for spectral-norm-constrained weights. Previously published numerical results support our theoretical findings.