Deep Neural Networks as Iterated Function Systems and a Generalization Bound

TL;DR

This paper connects deep neural networks with stochastic iterated function systems, providing a theoretical framework for understanding their stability and generalization, and introduces a new training objective with empirical validation.

Contribution

It establishes a novel link between DNN architectures and IFS theory, enabling rigorous analysis of stability and generalization bounds for generative models.

Findings

Proves existence and uniqueness of invariant measures under certain conditions.

Derives a Wasserstein generalization bound for generative models.

Empirically validates the new training objective on standard datasets.

Abstract

Deep neural networks (DNNs) achieve remarkable performance on a wide range of tasks, yet their mathematical analysis remains fragmented: stability and generalization are typically studied in disparate frameworks and on a case-by-case basis. Architecturally, DNNs rely on the recursive application of parametrized functions, a mechanism that can be unstable and difficult to train, making stability a primary concern. Even when training succeeds, there are few rigorous results on how well such models generalize beyond the observed data, especially in the generative setting. In this work, we leverage the theory of stochastic Iterated Function Systems (IFS) and show that two important deep architectures can be viewed as, or canonically associated with, place-dependent IFS. This connection allows us to import results from random dynamical systems to (i) establish the existence and uniqueness of invariant measures under suitable contractivity assumptions, and (ii) derive a Wasserstein generalization bound for generative modeling. The bound naturally leads to a new training objective that directly controls the collage-type approximation error between the data distribution and its image under the learned transfer operator. We illustrate the theory on a controlled 2D example and empirically evaluate the proposed objective on standard image datasets (MNIST, CelebA, CIFAR-10).