Rough Transformers for Continuous and Efficient Time-Series Modelling

TL;DR

The paper introduces the Rough Transformer, a continuous-time model for time-series data that reduces computational costs while effectively capturing long-range dependencies, especially useful in medical applications.

Contribution

It proposes the Rough Transformer with multi-view signature attention, combining Neural ODEs and Transformers for efficient, long-range time-series modeling.

Findings

Outperforms vanilla attention models in accuracy.

Uses significantly less computational time and memory.

Effective on both synthetic and real-world data.

Abstract

Time-series data in real-world medical settings typically exhibit long-range dependencies and are observed at non-uniform intervals. In such contexts, traditional sequence-based recurrent models struggle. To overcome this, researchers replace recurrent architectures with Neural ODE-based models to model irregularly sampled data and use Transformer-based architectures to account for long-range dependencies. Despite the success of these two approaches, both incur very high computational costs for input sequences of moderate lengths and greater. To mitigate this, we introduce the Rough Transformer, a variation of the Transformer model which operates on continuous-time representations of input sequences and incurs significantly reduced computational costs, critical for addressing long-range dependencies common in medical contexts. In particular, we propose multi-view signature attention, which uses path signatures to augment vanilla attention and to capture both local and global dependencies in input data, while remaining robust to changes in the sequence length and sampling frequency. We find that Rough Transformers consistently outperform their vanilla attention counterparts while obtaining the benefits of Neural ODE-based models using a fraction of the computational time and memory resources on synthetic and real-world time-series tasks.