Neural Network Approximation: A View from Polytope Decomposition

TL;DR

This paper introduces a polytope decomposition approach to analyze and improve the universal approximation capabilities of ReLU neural networks, emphasizing local regularity and efficiency.

Contribution

It proposes a novel polytope decomposition framework for neural network approximation, incorporating kernel polynomial methods and extending to analytic functions.

Findings

Polytope decomposition enhances approximation efficiency near singular points.

The kernel polynomial method provides explicit universal approximation bounds.

Extension to analytic functions achieves higher approximation rates.

Abstract

Universal approximation theory offers a foundational framework to verify neural network expressiveness, enabling principled utilization in real-world applications. However, most existing theoretical constructions are established by uniformly dividing the input space into tiny hypercubes without considering the local regularity of the target function. In this work, we investigate the universal approximation capabilities of ReLU networks from a view of polytope decomposition, which offers a more realistic and task-oriented approach compared to current methods. To achieve this, we develop an explicit kernel polynomial method to derive an universal approximation of continuous functions, which is characterized not only by the refined Totik-Ditzian-type modulus of continuity, but also by polytopical domain decomposition. Then, a ReLU network is constructed to approximate the kernel polynomial in each subdomain separately. Furthermore, we find that polytope decomposition makes our approximation more efficient and flexible than existing methods in many cases, especially near singular points of the objective function. Lastly, we extend our approach to analytic functions to reach a higher approximation rate.