Toward Physics-guided Time Series Embedding
Jiaxi Hu, Bowen Zhang, Qingsong Wen, Fugee Tsung, Yuxuan Liang
TL;DR
This paper introduces Embedding Duality Theory, enabling physics-guided time series embedding that reduces parameters, accelerates computation, and improves performance across various tasks without hyper-parameter tuning.
Contribution
It proposes a novel Embedding Duality Theory that leverages physical priors for efficient, high-performance time series embedding, bypassing traditional nonlinear embedding layers.
Findings
10X reduction in parameters
3X increase in speed
Performance boosts up to 53% in zero-shot tasks
Abstract
In various scientific and engineering fields, the primary research areas have revolved around physics-based dynamical systems modeling and data-driven time series analysis. According to the embedding theory, dynamical systems and time series can be mutually transformed using observation functions and physical reconstruction techniques. Based on this, we propose Embedding Duality Theory, where the parameterized embedding layer essentially provides a linear estimation of the non-linear time series dynamics. This theory enables us to bypass the parameterized embedding layer and directly employ physical reconstruction techniques to acquire a data embedding representation. Utilizing physical priors results in a 10X reduction in parameters, a 3X increase in speed, and maximum performance boosts of 18% in expert, 22% in few-shot, and 53\% in zero-shot tasks without any hyper-parameter tuning.…
Peer Reviews
Decision·Submitted to ICLR 2025
This paper introduces the concept of "Embedding Duality Theory" in the context of time series analysis. The authors propose that a parameterized embedding layer in a deep learning model serves as a linear approximation of the underlying nonlinear dynamics in time series data. The paper suggests that, by using physics-based priors, the model can bypass traditional parameterized embeddings, resulting in reduced parameters, faster computation, and improved performance on various time series tasks.
See the summary
The strengths of this paper lie in the radical idea that parameterized embeddings are unnecessary, and the author make a good case for this through extensive experiments. The authors also provide intuition to why the physics-guided embeddings perform better. Thus, the reader can infer how to design such embeddings.
The pre-dominant weaknesses of this submission lies in its writing and presentation. While the sentences were well written, the writing was not done in a manner where the key ideas, jargon, and notation were introduced before using them. In the theory section, there were no transitions between lemmas, propositions, etc.. It was difficult to follow and connect the dots. In the experiments section, there were significant details of experiments that were not outlined and clear (even in the appendix
The summary in Figure 3 is clear, easy to understand, and provides a strong overview of the work. The methods are clearly inspired by dynamical systems theory. The experimental evaluation is reasonable, and appears to me consistent with norms in this area (particularly in comparison to Time-SSM and PatchTST), and addresses the research questions outlined in Section 5. The authors demonstrate that these alternative patching and embedding strategies can improve performance with smaller parameter
### Organization and Framing The paper’s organization and framing are somewhat confusing. The main contribution seems to be an empirical evaluation of patching/embedding strategies (time delay, principal components of time delay, finite differences, and signal summation) that improve performance in models with smaller parameter counts. These strategies draw some mild inspiration from dynamical systems theory as a form of feature engineering to incorporate prior knowledge about sequences. This i
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTime Series Analysis and Forecasting