Prediction horizon shapes representations in predictive learning

TL;DR

This paper investigates how the prediction horizon influences the emergence of structured representations in predictive learning models, providing theoretical and empirical insights into when and why such representations form.

Contribution

It identifies the prediction horizon as a key factor shaping the structure of learned representations and explains its role in recovering latent task geometry.

Findings

Increasing the prediction horizon alters the learning problem's structure.

Implicit biases interact with structural changes to recover latent geometry.

Phenomena persist in nonlinear architectures and complex datasets.

Abstract

Predictive learning has emerged as a central paradigm for training models across diverse data domains and is increasingly viewed as a foundation for modern artificial intelligence. A common intuition for this success is that accurate prediction requires models to capture the underlying dynamics of the environment, leading to the emergence of structured world models. However, predictive learning does not universally yield such representations, and a mechanistic account of when and why it does remains incomplete. In this work, we identify the prediction horizon as a critical, but often implicit, component of predictive learning objectives. We show that increasing the prediction horizon fundamentally shapes the effective structure of the learning problem. In a minimal setting, we demonstrate both theoretically and empirically that the model's implicit biases interact with this structural change to recover the latent geometry of the task. We then extend these empirical results to nonlinear architectures and more complex datasets, where similar phenomena persist. These findings provide a principled explanation for the emergence of structured representations in predictive learning paradigms and clarify the conditions under which such representations should be expected.