Learning Long-Range Dependencies with Temporal Predictive Coding

TL;DR

This paper introduces a novel learning method combining Temporal Predictive Coding with approximate RTRL, enabling efficient training of recurrent neural networks on long-range dependencies with performance comparable to backpropagation through time.

Contribution

It presents a new approach that extends Predictive Coding to recurrent networks, allowing for energy-efficient, parallelizable learning of long-range temporal dependencies.

Findings

Matches BPTT performance on synthetic benchmarks

Achieves competitive perplexity on large-scale machine translation

Demonstrates energy-efficient learning with local, parallelizable operations

Abstract

Predictive Coding (PC) is a biologically-inspired learning framework characterised by local, parallelisable operations, properties that enable energy-efficient implementation on neuromorphic hardware. Despite this, extending PC effectively to recurrent neural networks (RNNs) has been challenging, particularly for tasks involving long-range temporal dependencies. Backpropagation Through Time (BPTT) remains the dominant method for training RNNs, but its non-local computation, lack of spatial parallelism, and requirement to store extensive activation histories results in significant energy consumption. This work introduces a novel method combining Temporal Predictive Coding (tPC) with approximate Real-Time Recurrent Learning (RTRL), enabling effective spatio-temporal credit assignment. Results indicate that the proposed method can closely match the performance of BPTT on both synthetic benchmarks and real-world tasks. On a challenging machine translation task, with a 15-million parameter model, the proposed method achieves a test perplexity of 7.62 (vs. 7.49 for BPTT), marking one of the first applications of tPC to tasks of this scale. These findings demonstrate the potential of this method to learn complex temporal dependencies whilst retaining the local, parallelisable, and flexible properties of the original PC framework, paving the way for more energy-efficient learning systems.