A Note on the Representation Power of GHHs

TL;DR

This paper establishes a nearly tight lower bound on the number of nestings needed for generalized hinging hyperplanes to universally represent continuous piecewise linear functions, and discusses limitations of one-hidden-layer neural networks.

Contribution

It proves that n nestings are necessary for GHHs to have universal representation power, refining previous bounds and highlighting neural network limitations.

Findings

n nestings are necessary for GHHs to be universal

One-hidden-layer neural networks lack universal approximation over the entire domain

A key lemma links periodic functions to integrability, with independent interest

Abstract

In this note we prove a sharp lower bound on the necessary number of nestings of nested absolute-value functions of generalized hinging hyperplanes (GHH) to represent arbitrary CPWL functions. Previous upper bound states that $n+1$ nestings is sufficient for GHH to achieve universal representation power, but the corresponding lower bound was unknown. We prove that $n$ nestings is necessary for universal representation power, which provides an almost tight lower bound. We also show that one-hidden-layer neural networks don't have universal approximation power over the whole domain. The analysis is based on a key lemma showing that any finite sum of periodic functions is either non-integrable or the zero function, which might be of independent interest.