Stable Parametrization of Continuous and Piecewise-Linear Functions

TL;DR

This paper introduces a stable local basis representation for continuous piecewise-linear functions, ensuring a robust and unique parametrization that improves the stability of neural network function mappings.

Contribution

It provides necessary and sufficient conditions for triangulations to form a Riesz basis, estimates the condition number of the representation, and specializes the framework to uniform grids with optimal stability.

Findings

The local hat basis forms a Riesz basis under specific triangulation conditions.

The condition number of the representation can be effectively estimated.

The uniform grid parametrization with shifted box splines is proven to be optimally stable.

Abstract

Rectified-linear-unit (ReLU) neural networks, which play a prominent role in deep learning, generate continuous and piecewise-linear (CPWL) functions. While they provide a powerful parametric representation, the mapping between the parameter and function spaces lacks stability. In this paper, we investigate an alternative representation of CPWL functions that relies on local hat basis functions. It is predicated on the fact that any CPWL function can be specified by a triangulation and its values at the grid points. We give the necessary and sufficient condition on the triangulation (in any number of dimensions) for the hat functions to form a Riesz basis, which ensures that the link between the parameters and the corresponding CPWL function is stable and unique. In addition, we provide an estimate of the $\ell_2\rightarrow L_2$ condition number of this local representation. Finally, as a special case of our framework, we focus on a systematic parametrization of $\mathbb{R}^d$ with control points placed on a uniform grid. In particular, we choose hat basis functions that are shifted replicas of a single linear box spline. In this setting, we prove that our general estimate of the condition number is optimal. We also relate our local representation to a nonlocal one based on shifts of a causal ReLU-like function.