Banach density of generated languages: Dichotomies in topology and dimension

TL;DR

This paper investigates the Banach density of generated languages embedded in multiple dimensions, revealing topological and dimensional dichotomies that influence the density achievable by algorithms.

Contribution

It introduces the use of Banach density in language generation, uncovering new topological and dimensional structures affecting generative models.

Findings

In dimension one, finite Cantor-Bendixson rank allows algorithms to achieve 1/2 Banach density.

Infinite Cantor-Bendixson rank can prevent any positive Banach density from being achieved.

In higher dimensions, Ramsey-theoretic obstacles require nondegeneracy conditions for positive density.

Abstract

The formalism of language generation in the limit studies generative models by requiring an algorithm, given strings from a hidden true language, to eventually generate new valid strings. A core issue is the tension between validity and breadth. Prior work quantified breadth via asymptotic density, where the priority is generating strings early in a natural countable ordering. Here, we study density when the strings are embedded in $d$ dimensions, a ubiquitous structure in current generative models. Our goal is for the generated strings to be dense throughout the embedding. This requires a different measure, the Banach density, which captures whether a set contains large sparse regions. Using Banach density uncovers a rich structure based on dimension and the topology of the language collection. We prove that in dimension one, when the underlying topological space has finite Cantor-Bendixson rank, an algorithm can always generate a subset of the true language with an optimal lower Banach density of 1/2. However, for collections with infinite Cantor-Bendixson rank, there are cases where no algorithm can achieve any positive lower Banach density; the generated set must contain arbitrarily large, sparse regions. This reveals a topological contrast unseen with asymptotic density, where 1/2 is always achievable. We also extend our results to a family of measures interpolating between Banach and asymptotic density. Finally, in dimension $d \geq 2$, our positive result for Banach density encounters a Ramsey-theoretic obstacle regarding two-colored point sets. Overcoming this requires a nondegeneracy condition: the embedding of the true language must be sufficiently represented throughout the full $d$-dimensional space.