Spline Representation and Redundancies of One-Dimensional ReLU Neural Network Models

TL;DR

This paper explores the structure of one-dimensional ReLU neural networks by linking them to spline functions, revealing parameter redundancies and the capacity to represent arbitrary spline breakpoints.

Contribution

It introduces a recursive algorithm to convert ReLU DNN parameters into spline parameters and characterizes the network's ability to represent splines with prescribed knots.

Findings

ReLU DNN parameters can be transformed into spline parameters.

After removing known redundancies, remaining parameters are independent.

ReLU DNNs with up to three hidden layers can represent splines with arbitrary knots.

Abstract

We analyze the structure of a one-dimensional deep ReLU neural network (ReLU DNN) in comparison to the model of continuous piecewise linear (CPL) spline functions with arbitrary knots. In particular, we give a recursive algorithm to transfer the parameter set determining the ReLU DNN into the parameter set of a CPL spline function. Using this representation, we show that after removing the well-known parameter redundancies of the ReLU DNN, which are caused by the positive scaling property, all remaining parameters are independent. Moreover, we show that the ReLU DNN with one, two or three hidden layers can represent CPL spline functions with $K$ arbitrarily prescribed knots (breakpoints), where $K$ is the number of real parameters determining the normalized ReLU DNN (up to the output layer parameters). Our findings are useful to fix a priori conditions on the ReLU DNN to achieve an output with prescribed breakpoints and function values.