Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences

TL;DR

The paper introduces Phased LSTM, an extension of LSTM with a time gate controlled by oscillation, enabling faster training and better handling of asynchronous, irregularly sampled sequence data.

Contribution

It presents a novel Phased LSTM model that improves training speed and efficiency for long, irregular, and event-based sequences by incorporating a time gate mechanism.

Findings

Faster convergence than regular LSTMs on long sequence tasks.

Effective integration of asynchronous sensor inputs.

Significant reduction in computational cost at runtime.

Abstract

Recurrent Neural Networks (RNNs) have become the state-of-the-art choice for extracting patterns from temporal sequences. However, current RNN models are ill-suited to process irregularly sampled data triggered by events generated in continuous time by sensors or other neurons. Such data can occur, for example, when the input comes from novel event-driven artificial sensors that generate sparse, asynchronous streams of events or from multiple conventional sensors with different update intervals. In this work, we introduce the Phased LSTM model, which extends the LSTM unit by adding a new time gate. This gate is controlled by a parametrized oscillation with a frequency range that produces updates of the memory cell only during a small percentage of the cycle. Even with the sparse updates imposed by the oscillation, the Phased LSTM network achieves faster convergence than regular LSTMs on tasks which require learning of long sequences. The model naturally integrates inputs from sensors of arbitrary sampling rates, thereby opening new areas of investigation for processing asynchronous sensory events that carry timing information. It also greatly improves the performance of LSTMs in standard RNN applications, and does so with an order-of-magnitude fewer computes at runtime.