Universal Approximation Theorem for Input-Connected Multilayer Perceptrons

TL;DR

This paper introduces the Input-Connected Multilayer Perceptron (IC-MLP), a neural network architecture with direct input connections to hidden neurons, and proves it can universally approximate continuous functions on real intervals and higher-dimensional spaces.

Contribution

It provides the first universal approximation theorem for IC-MLPs, demonstrating their expressive power in both univariate and multivariate settings.

Findings

Deep IC-MLPs can approximate any continuous univariate function with nonlinear activation.

Universal approximation extends to vector inputs and compact subsets of Euclidean space.

Explicit formulas and systematic descriptions of IC-MLPs are provided.

Abstract

We present the Input-Connected Multilayer Perceptron (IC-MLP), a feedforward neural network architecture in which each hidden neuron receives, in addition to the outputs of the preceding layer, a direct affine connection from the raw input. We first study this architecture in the univariate setting and give an explicit and systematic description of IC-MLPs with an arbitrary finite number of hidden layers, including iterated formulas for the network functions. In this setting, we prove a universal approximation theorem showing that deep IC-MLPs can approximate any continuous function on a closed interval of the real line if and only if the activation function is nonlinear. We then extend the analysis to vector-valued inputs and establish a corresponding universal approximation theorem for continuous functions on compact subsets of $\mathbb{R}^n$.