Approximability of deep computations

TL;DR

This paper introduces a formal framework for deep computations, characterizes their computability, and proves the existence of deep equilibria, bridging theoretical foundations with empirical observations in deep learning.

Contribution

It formalizes deep computations and equilibria, providing a rigorous foundation that connects theoretical models with empirical deep learning phenomena.

Findings

Characterization of computable deep computations

Proof of existence of deep equilibria

Framework combining topology, Ramsey theory, and model theory

Abstract

This is the first of a series of papers in which we study deep computations (ultracomputations) and deep iterates, formalizing the ideas of "asymptotic limit" of computations and compositional iterates, respectively. In this first paper of the series, we characterize deep computations that are bona fide computable, and prove the existence of deep equilibria, which hitherto have been found only empirically in deep learning. A subsequent paper will study the complexity of ultracomputations. Our approach adapts and combines technology from topology of function spaces, structural Ramsey theory, topological dynamics, and model theory.