Parallelizing non-linear sequential models over the sequence length

TL;DR

This paper introduces a parallel algorithm that significantly accelerates GPU evaluation of non-linear sequential models like RNNs and Neural ODEs, enabling faster training without accuracy loss, thus broadening their applicability to long sequence tasks.

Contribution

The authors present a novel parallel evaluation method for sequential models that does not require special architecture modifications, achieving up to 1000x speedup and enabling practical training of long sequence models.

Findings

GPU evaluation speed increased by up to 3 orders of magnitude.

Training time reduced by more than 10 times without accuracy loss.

Demonstrated effectiveness of GRUs on long time series classification.

Abstract

Sequential models, such as Recurrent Neural Networks and Neural Ordinary Differential Equations, have long suffered from slow training due to their inherent sequential nature. For many years this bottleneck has persisted, as many thought sequential models could not be parallelized. We challenge this long-held belief with our parallel algorithm that accelerates GPU evaluation of sequential models by up to 3 orders of magnitude faster without compromising output accuracy. The algorithm does not need any special structure in the sequential models' architecture, making it applicable to a wide range of architectures. Using our method, training sequential models can be more than 10 times faster than the common sequential method without any meaningful difference in the training results. Leveraging this accelerated training, we discovered the efficacy of the Gated Recurrent Unit in a long time series classification problem with 17k time samples. By overcoming the training bottleneck, our work serves as the first step to unlock the potential of non-linear sequential models for long sequence problems.