An Optimized and Energy-Efficient Parallel Implementation of Non-Iteratively Trained Recurrent Neural Networks

TL;DR

This paper introduces extit{Opt}, a GPU-optimized parallel RNN training algorithm based on extreme learning machines, achieving significant speedups over traditional methods in time-series prediction tasks.

Contribution

The paper presents a novel GPU-based parallel RNN training method using ELM that significantly reduces training time compared to existing approaches.

Findings

Achieves up to 845x speedup over sequential training.

Requires up to 20x less training time than parallel BPTT.

Demonstrates effectiveness on ten time-series prediction applications.

Abstract

Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation through time (BPTT) which is prohibitively expensive, especially when the length of the time dependencies and the number of hidden neurons increase. To reduce the training time, extreme learning machines (ELMs) have been recently applied to RNN training, reaching a 99\% speedup on some applications. Due to its non-iterative nature, ELM training, when parallelized, has the potential to reach higher speedups than BPTT. In this work, we present \opt, an optimized parallel RNN training algorithm based on ELM that takes advantage of the GPU shared memory and of parallel QR factorization algorithms to efficiently reach optimal solutions. The theoretical analysis of the proposed algorithm is presented on six RNN architectures, including LSTM and GRU, and its performance is empirically tested on ten time-series prediction applications. \opt~is shown to reach up to 845 times speedup over its sequential counterpart and to require up to 20x less time to train than parallel BPTT.