Path Development Network with Finite-dimensional Lie Group Representation

TL;DR

This paper introduces a trainable path development layer based on finite-dimensional Lie group representations, reducing dimensionality and improving modeling of irregular time series in deep learning applications.

Contribution

It proposes a novel development layer leveraging Lie group representations with a manifold-optimized backpropagation, enhancing sequence modeling and outperforming signature features.

Findings

Outperforms signature features in accuracy and dimensionality reduction

Achieves state-of-the-art results when combined with LSTM models

Improves modeling of dynamics constrained to Lie groups

Abstract

Signature, lying at the heart of rough path theory, is a central tool for analysing controlled differential equations driven by irregular paths. Recently it has also found extensive applications in machine learning and data science as a mathematically principled, universal feature that boosts the performance of deep learning-based models in sequential data tasks. It, nevertheless, suffers from the curse of dimensionality when paths are high-dimensional. We propose a novel, trainable path development layer, which exploits representations of sequential data through finite-dimensional Lie groups, thus resulting in dimension reduction. Its backpropagation algorithm is designed via optimization on manifolds. Our proposed layer, analogous to recurrent neural networks (RNN), possesses an explicit, simple recurrent unit that alleviates the gradient issues. Our layer demonstrates its strength in irregular time series modelling. Empirical results on a range of datasets show that the development layer consistently and significantly outperforms signature features on accuracy and dimensionality. The compact hybrid model (stacking one-layer LSTM with the development layer) achieves state-of-the-art against various RNN and continuous time series models. Our layer also enhances the performance of modelling dynamics constrained to Lie groups. Code is available at https://github.com/PDevNet/DevNet.git.