Approximate Fixed-Points in Recurrent Neural Networks

TL;DR

This paper introduces a novel approach to reformulating recurrent neural networks as fixed-point problems, enabling parallel computation of approximate fixed-points for training and inference, which maintains competitive performance.

Contribution

It presents a fixed-point reformulation of RNNs allowing parallel approximation, addressing efficiency and inconsistency issues in traditional training and inference methods.

Findings

Approximate fixed-points can be computed in parallel.

The method achieves competitive performance on language modeling tasks.

It enables consistent training and inference for RNNs.

Abstract

Recurrent neural networks are widely used in speech and language processing. Due to dependency on the past, standard algorithms for training these models, such as back-propagation through time (BPTT), cannot be efficiently parallelised. Furthermore, applying these models to more complex structures than sequences requires inference time approximations, which introduce inconsistency between inference and training. This paper shows that recurrent neural networks can be reformulated as fixed-points of non-linear equation systems. These fixed-points can be computed using an iterative algorithm exactly and in as many iterations as the length of any given sequence. Each iteration of this algorithm adds one additional Markovian-like order of dependencies such that upon termination all dependencies modelled by the recurrent neural networks have been incorporated. Although exact fixed-points inherit the same parallelization and inconsistency issues, this paper shows that approximate fixed-points can be computed in parallel and used consistently in training and inference including tasks such as lattice rescoring. Experimental validation is performed in two tasks, Penn Tree Bank and WikiText-2, and shows that approximate fixed-points yield competitive prediction performance to recurrent neural networks trained using the BPTT algorithm.