Enhancing Transformer RNNs with Multiple Temporal Perspectives

TL;DR

This paper proposes a novel method of integrating multiple temporal perspectives into RNN architectures, specifically enhancing the RWKV model, to improve sequential data understanding with minimal additional parameters and computational overhead.

Contribution

It introduces a new approach for RNNs that maintains diverse temporal views, improving context interpretation without extensive retraining or large parameter increases.

Findings

Enhanced performance across multiple benchmarks.

Minimal parameter increase (as low as 0.04%).

Maintains linear inference complexity.

Abstract

We introduce the concept of multiple temporal perspectives, a novel approach applicable to Recurrent Neural Network (RNN) architectures for enhancing their understanding of sequential data. This method involves maintaining diverse temporal views of previously encountered text, significantly enriching the language models' capacity to interpret context. To show the efficacy of this approach, we incorporate it into the Receptance Weighted Key Value (RWKV) architecture, addressing its inherent challenge of retaining all historical information within a single hidden state. Notably, this improvement is achieved with a minimal increase in the number of parameters --even as little as $0.04\%$ of the original number of parameters. Further, the additional parameters necessary for the multiple temporal perspectives are fine-tuned with minimal computational overhead, avoiding the need for a full pre-training. The resulting model maintains linear computational complexity during prompt inference, ensuring consistent efficiency across various sequence lengths. The empirical results and ablation studies included in our research validate the effectiveness of our approach, showcasing improved performance across multiple benchmarks. The code, model weights and datasets are open-sourced at: https://github.com/RazvanDu/TemporalRNNs.