Efficient Real Time Recurrent Learning through combined activity and parameter sparsity

TL;DR

This paper introduces a method to make Real-Time Recurrent Learning (RTRL) more practical by leveraging activity and parameter sparsity, enabling online training of larger recurrent networks without approximations.

Contribution

The authors demonstrate that combining activity and parameter sparsity significantly reduces RTRL's computational and memory costs, making it feasible for real-time applications.

Findings

Sparsity reduces RTRL computational complexity.

Combined activity and parameter sparsity enable practical RTRL.

No approximations are needed for efficiency gains.

Abstract

Backpropagation through time (BPTT) is the standard algorithm for training recurrent neural networks (RNNs), which requires separate simulation phases for the forward and backward passes for inference and learning, respectively. Moreover, BPTT requires storing the complete history of network states between phases, with memory consumption growing proportional to the input sequence length. This makes BPTT unsuited for online learning and presents a challenge for implementation on low-resource real-time systems. Real-Time Recurrent Learning (RTRL) allows online learning, and the growth of required memory is independent of sequence length. However, RTRL suffers from exceptionally high computational costs that grow proportional to the fourth power of the state size, making RTRL computationally intractable for all but the smallest of networks. In this work, we show that recurrent networks exhibiting high activity sparsity can reduce the computational cost of RTRL. Moreover, combining activity and parameter sparsity can lead to significant enough savings in computational and memory costs to make RTRL practical. Unlike previous work, this improvement in the efficiency of RTRL can be achieved without using any approximations for the learning process.