Constrained latent state modeling: A unifying perspective on representation learning under competing constraints

TL;DR

This paper introduces constrained latent state modeling (CLSM) as a unifying framework that formalizes core properties and trade-offs in representation learning, aiming to improve interpretability and robustness.

Contribution

It formalizes a set of core properties for latent states and shows how existing methods fit into a common design space, addressing fragmentation and ambiguity in current approaches.

Findings

Existing approaches are interpretable as enforcing different subsets of constraints.

Persistent challenges like non-identifiability are due to underconstrained formulations.

CLSM offers a framework for explicit design choices and trade-off analysis.

Abstract

Learning latent representations from complex data is central to modern machine learning, spanning temporal, multimodal, and partially observed systems. In such settings, representations are better understood as latent states capturing underlying system dynamics, rather than as mere compressed summaries of observations. Yet current approaches remain fragmented, relying on distinct -- and often implicit -- assumptions about what these states should represent. We argue that this fragmentation reflects a more fundamental limitation: latent representations are typically learned from underconstrained objectives that fail to specify the properties that meaningful latent states should satisfy. As a result, multiple representations can satisfy the same objective, leading to ambiguity in their structure and interpretation. While many of the underlying principles have been explored in isolation, their interactions have not been explicitly formalized. In this work, we propose constrained latent state modeling (CLSM) as a unifying perspective. We identify a set of core properties -- predictive sufficiency, minimality, temporal coherence, observation compatibility, invariance to nuisance factors, and structural constraints -- and show that they are intrinsically coupled through fundamental trade-offs. Revisiting major modeling families through this lens, we show that existing approaches can be interpreted as enforcing different subsets of constraints, thereby occupying distinct regions of a common design space. This perspective reframes persistent challenges such as lack of identifiability as consequences of underconstrained formulations, rather than isolated technical limitations. More broadly, CLSM provides a principled framework to make design choices explicit, to analyze trade-offs, and to guide the development of more interpretable, robust, and task-aligned latent state models.