A representer theorem for deep neural networks

TL;DR

This paper introduces a new theoretical framework for deep neural networks, showing that optimal activation functions can be represented as nonuniform linear splines, linking neural network training with sparsity and spline theory.

Contribution

It derives a general representer theorem for deep neural networks, connecting activation functions with splines and sparsity, and proposes a regularization approach for optimizing these functions.

Findings

Optimal activation functions are nonuniform linear splines with adaptive knots.

The framework links neural network training to $ ext{L}_1$ minimization and sparsity.

The scheme is compatible with existing architectures like ReLU and MaxOut.

Abstract

We propose to optimize the activation functions of a deep neural network by adding a corresponding functional regularization to the cost function. We justify the use of a second-order total-variation criterion. This allows us to derive a general representer theorem for deep neural networks that makes a direct connection with splines and sparsity. Specifically, we show that the optimal network configuration can be achieved with activation functions that are nonuniform linear splines with adaptive knots. The bottom line is that the action of each neuron is encoded by a spline whose parameters (including the number of knots) are optimized during the training procedure. The scheme results in a computational structure that is compatible with the existing deep-ReLU, parametric ReLU, APL (adaptive piecewise-linear) and MaxOut architectures. It also suggests novel optimization challenges, while making the link with $\ell_1$ minimization and sparsity-promoting techniques explicit.