Self-supervised contrastive learning performs non-linear system identification
Rodrigo Gonz\'alez Laiz, Tobias Schmidt, Steffen Schneider
TL;DR
This paper demonstrates that self-supervised contrastive learning can be used for non-linear system identification by uncovering underlying dynamics in latent space, supported by theoretical guarantees and empirical validation.
Contribution
It introduces dynamics contrastive learning, a novel framework that identifies linear, switching linear, and non-linear dynamics from non-linear observations, with theoretical backing.
Findings
Successfully uncovers latent dynamics in complex systems.
Provides theoretical guarantees for the proposed method.
Validates the approach through empirical experiments.
Abstract
Self-supervised learning (SSL) approaches have brought tremendous success across many tasks and domains. It has been argued that these successes can be attributed to a link between SSL and identifiable representation learning: Temporal structure and auxiliary variables ensure that latent representations are related to the true underlying generative factors of the data. Here, we deepen this connection and show that SSL can perform system identification in latent space. We propose dynamics contrastive learning, a framework to uncover linear, switching linear and non-linear dynamics under a non-linear observation model, give theoretical guarantees and validate them empirically.
Peer Reviews
Decision·ICLR 2025 Poster
Applying contrastive learning to recover latent dynamics is itself a relatively new approach and the paper is well organised. The proofs use standard jacobian analysis tools and is easy to follow.
The paper needs refinement, with minor typos and inadequately captioned figures. While studying the identifiability of time-series contrastive learning might be novel, all the technical tools require carefully controlled assumptions and specific behavior of Jacobians under contrastive loss minimization, which typically do not hold in practice. Nonetheless, such assumptions are common in the literature on the identifiability of dynamical systems from observed time series.
1. The connection between contrastive learning and dynamic system identification is novel and could lead to simple encoder-only implementations favored in practice. 2. The synthetic experiments, especially the ablation studies, are extensive and investigate many aspects of the theoretical results.
1. The contributions (comparison with previous work), theoretical techniques, and novelty are very lightly discussed. I would appreciate a detailed discussion comparing this work's conditions with those in recent literature on temporal causal representation learning (e.g., [1] and its follow-up work). This would aid the contextualization of this work and make the contribution more transparent. 2. This work strikes me as theoretically oriented. There is a lack of discussion on the theoretical c
Automatic identification of latent variables or dynamics is of critical importance in modern machine / reinforcement learning. The method the authors propose follows a line of self-supervised methods in contrastive learning. In comparison to its closest relative in time-contrastive learning (Hyvarinen and Morioka, 2016), the proposed method is seemingly more well-fit for fitting non-linear time-series data by fitting a latent time-series, rather than predicting a categorical label as in the afor
In my opinion, the paper leaves quite a few critical questions unanswered, and in general suffers from a lack of polish. In its current state, I cannot recommend the paper for acceptance. The main weaknesses in my eyes are the following: The paper claims to perform latent nonlinear system identification. This is a key desideratum in various fields such as reinforcement learning and continuous control, and thus has a rich history and literature. However, the assumptions in this paper--and notabl
Code & Models
Videos
Taxonomy
TopicsNeural Networks and Applications