Extending Universal Approximation Guarantees: A Theoretical Justification for the Continuity of Real-World Learning Tasks

TL;DR

This paper extends universal approximation guarantees by providing conditions under which real-world learning tasks, modeled as conditional expectations of complex transformations, are guaranteed to be continuous, supporting neural network approximation.

Contribution

It introduces verifiable conditions on data-generating processes that ensure the continuity of learning tasks, broadening the theoretical foundation of neural network approximation capabilities.

Findings

Conditions for continuity can be verified using the deterministic map T.

Theoretical justification for the continuity of real-world learning tasks.

Application example with randomized stable matching demonstrates practicality.

Abstract

Universal Approximation Theorems establish the density of various classes of neural network function approximators in $C(K, \mathbb{R}^m)$, where $K \subset \mathbb{R}^n$ is compact. In this paper, we aim to extend these guarantees by establishing conditions on learning tasks that guarantee their continuity. We consider learning tasks given by conditional expectations $x \mapsto \mathrm{E}\left[Y \mid X = x\right]$, where the learning target $Y = f \circ L$ is a potentially pathological transformation of some underlying data-generating process $L$. Under a factorization $L = T \circ W$ for the data-generating process where $T$ is thought of as a deterministic map acting on some random input $W$, we establish conditions (that might be easily verified using knowledge of $T$ alone) that guarantee the continuity of practically \textit{any} derived learning task $x \mapsto \mathrm{E}\left[f \circ L \mid X = x\right]$. We motivate the realism of our conditions using the example of randomized stable matching, thus providing a theoretical justification for the continuity of real-world learning tasks.