Universal Approximation Under Constraints is Possible with Transformers

TL;DR

This paper proves that transformers can universally approximate functions under constraints, ensuring outputs satisfy specific sets, and extends classical theorems to constrained and manifold-valued functions.

Contribution

It introduces a constrained universal approximation theorem for transformers and a deep neural version of Berge's Maximum Theorem, enabling constrained optimization.

Findings

Transformers can exactly encode constraints while approximating functions.

Universal approximation theorem now applies to convex and non-convex constraint sets.

Results include approximation for Riemannian manifold-valued functions with geodesic convexity.

Abstract

Many practical problems need the output of a machine learning model to satisfy a set of constraints, $K$. Nevertheless, there is no known guarantee that classical neural network architectures can exactly encode constraints while simultaneously achieving universality. We provide a quantitative constrained universal approximation theorem which guarantees that for any non-convex compact set $K$ and any continuous function $f:\mathbb{R}^n\rightarrow K$, there is a probabilistic transformer $\hat{F}$ whose randomized outputs all lie in $K$ and whose expected output uniformly approximates $f$. Our second main result is a "deep neural version" of Berge's Maximum Theorem (1963). The result guarantees that given an objective function $L$, a constraint set $K$, and a family of soft constraint sets, there is a probabilistic transformer $\hat{F}$ that approximately minimizes $L$ and whose outputs belong to $K$; moreover, $\hat{F}$ approximately satisfies the soft constraints. Our results imply the first universal approximation theorem for classical transformers with exact convex constraint satisfaction. They also yield that a chart-free universal approximation theorem for Riemannian manifold-valued functions subject to suitable geodesically convex constraints.