Self-Improvement as Coherence Optimization: A Theoretical Account

TL;DR

This paper presents a theoretical framework explaining how language models can self-improve through coherence optimization, which involves finding compressible and predictable context-to-behavior mappings, supported by preliminary experiments.

Contribution

It introduces coherence optimization as a unifying theory for self-improvement methods and proves its optimality in semi-supervised learning scenarios derived from pretrained models.

Findings

Coherence optimization is equivalent to description-length regularization.

Self-improvement methods work by maximizing joint predictability and compressibility.

The theory predicts conditions under which feedback-free self-improvement succeeds or fails.

Abstract

Can language models improve their accuracy without external supervision? Methods such as debate, bootstrap, and internal coherence maximization achieve this surprising feat, even matching golden finetuning performance. Yet why they work remains theoretically unclear. We show that they are all special cases of coherence optimization: finding a context-to-behavior mapping that's most compressible and jointly predictable. We prove that coherence optimization is equivalent to description-length regularization, and that among all such regularization schemes, it is optimal for semi-supervised learning when the regularizer is derived from a pretrained model. Our theory, supported by preliminary experiments, explains why feedback-free self-improvement works and predicts when it should succeed or fail.