A General Method for Proving Networks Universal Approximation Property

TL;DR

This paper introduces a unified, modular framework for proving the universal approximation property of diverse neural network architectures, simplifying and generalizing previous architecture-specific proofs.

Contribution

It proposes the Universal Approximation Module (UAM) as a fundamental building block, enabling a general proof of universal approximation for any network composed of these modules.

Findings

Unified framework for universal approximation proofs

Any network built from UAMs retains universal approximation

Provides a step-by-step interpretative process for network expressiveness

Abstract

Deep learning architectures are highly diverse. To prove their universal approximation properties, existing works typically rely on model-specific proofs. Generally, they construct a dedicated mathematical formulation for each architecture (e.g., fully connected networks, CNNs, or Transformers) and then prove their universal approximability. However, this approach suffers from two major limitations: first, every newly proposed architecture often requires a completely new proof from scratch; second, these proofs are largely isolated from one another, lacking a common analytical foundation. This not only incurs significant redundancy but also hinders unified theoretical understanding across different network families. To address these issues, this paper proposes a general and modular framework for proving universal approximation. We define a basic building block (comprising one or multiple layers) that possesses the universal approximation property as a Universal Approximation Module (UAM). Under this condition, we show that any deep network composed of such modules inherently retains the universal approximation property. Moreover, the overall approximation process can be interpreted as a progressive refinement across modules. This perspective not only unifies the analysis of diverse architectures but also enables a step-by-step understanding of how expressive power evolves through the network.