Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN

TL;DR

This paper introduces Carry-lookahead RNN (CL-RNN), a novel architecture inspired by digital adder techniques, enabling parallel computation in RNNs and improving efficiency and flexibility in sequence modeling.

Contribution

We propose CL-RNN, a new RNN architecture that incorporates carry-lookahead modules to enable parallel processing, addressing the serial dependency issue in traditional RNNs.

Findings

CL-RNN outperforms traditional RNNs in sequence modeling tasks.

CL-RNN achieves higher computational efficiency due to parallelism.

The architecture offers a more flexible receptive field.

Abstract

The recurrent network architecture is a widely used model in sequence modeling, but its serial dependency hinders the computation parallelization, which makes the operation inefficient. The same problem was encountered in serial adder at the early stage of digital electronics. In this paper, we discuss the similarities between recurrent neural network (RNN) and serial adder. Inspired by carry-lookahead adder, we introduce carry-lookahead module to RNN, which makes it possible for RNN to run in parallel. Then, we design the method of parallel RNN computation, and finally Carry-lookahead RNN (CL-RNN) is proposed. CL-RNN takes advantages in parallelism and flexible receptive field. Through a comprehensive set of tests, we verify that CL-RNN can perform better than existing typical RNNs in sequence modeling tasks which are specially designed for RNNs.