On Union-Closedness of Language Generation

TL;DR

This paper explores the limitations of language generation models in the limit, proving that certain classes cannot be combined or boosted, and addressing open questions about the structure and properties of generatable classes.

Contribution

It resolves open questions by showing finite unions of generatable classes may not be generatable and introduces a novel diagonalization method for language generation.

Findings

Finite unions of generatable classes need not be generatable.

There exist classes that are non-uniformly generatable but whose union is non-generatable.

A new diagonalization approach addresses open questions about uncountable classes and closure properties.

Abstract

We investigate language generation in the limit - a model by Kleinberg and Mullainathan [NeurIPS 2024] and extended by Li, Raman, and Tewari [COLT 2025]. While Kleinberg and Mullainathan proved generation is possible for all countable collections, Li et al. defined a hierarchy of generation notions (uniform, non-uniform, and generatable) and explored their feasibility for uncountable collections. Our first set of results resolve two open questions of Li et al. by proving finite unions of generatable or non-uniformly generatable classes need not be generatable. These follow from a stronger result: there is a non-uniformly generatable class and a uniformly generatable class whose union is non-generatable. This adds to the aspects along which language generation in the limit is different from traditional tasks in statistical learning theory like classification, which are closed under finite unions. In particular, it implies that given two generators for different collections, one cannot combine them to obtain a single "more powerful" generator, prohibiting this notion of boosting. Our construction also addresses a third open question of Li et al. on whether there are uncountable classes that are non-uniformly generatable and do not satisfy the eventually unbounded closure (EUC) condition introduced by Li, Raman, and Tewari. Our approach utilizes carefully constructed classes along with a novel diagonalization argument that could be of independent interest in the growing area of language generation.