Diminishing Returns in Expanding Generative Models and Godel-Tarski-Lob Limits

TL;DR

This paper develops a theoretical framework to analyze the limits of capability growth in expanding generative models, revealing that improvements diminish asymptotically and logical constraints persist.

Contribution

It introduces a task-space framework for understanding generative system capabilities and proves that marginal improvements must converge to zero as capacity increases.

Findings

Marginal improvement in task coverage diminishes with capacity growth.

Logical limitations like incompleteness and undefinability constrain reasoning systems.

Theoretical bounds on prediction improvements based on complexity-weighted hypothesis classes.

Abstract

Modern generative modelling systems are increasingly improved by expanding model capacity, training data, and computational resources. While empirical studies have documented such scaling behaviour across architectures including generative adversarial networks, variational autoencoders, transformer-based models, and diffusion models, the theoretical limits of capability growth in expanding generative systems remain poorly understood. In this paper we develop a general task-space framework for analysing expanding generative reasoning systems. Each system induces a subset of a global task space representing the tasks it can successfully solve, and system capability is measured by the probability mass of this solved-task set under a fixed task distribution. Within this framework we prove a structural result showing that, under mild assumptions, the marginal improvement in solved tasks must converge to zero as system capacity increases. Thus expanding generative systems may continue to gain capability, but the probability mass of newly solvable tasks necessarily diminishes asymptotically. We further provide a prediction-theoretic refinement based on complexity-weighted hypothesis classes inspired by algorithmic probability, yielding quantitative bounds on marginal improvement in prediction settings. Finally, we examine logical reasoning tasks and show that classical results from mathematical logic -- including Rosser incompleteness, Tarski's undefinability theorem, and L\"ob's theorem -- imply the persistence of unresolved logical tasks within sufficiently expressive reasoning systems. Together these results provide a mathematical perspective on the asymptotic behaviour of expanding generative systems, showing that long-run capability growth is constrained both by diminishing marginal improvements in task coverage and by fundamental logical limitations on internal reasoning.