Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures

TL;DR

This paper explores hybrid RNN-Transformer architectures to improve language modeling in low-resource settings, achieving better performance with fewer parameters and reduced computational costs.

Contribution

It introduces a novel hybrid architecture that replaces some attention layers with feed-forward and quasi-recurrent layers, enhancing efficiency in low-data regimes.

Findings

Outperforms existing models with similar parameter counts

Achieves comparable performance to larger models

Reduces training costs and model size

Abstract

Transformer-based language models have recently been at the forefront of active research in text generation. However, these models' advances come at the price of prohibitive training costs, with parameter counts in the billions and compute requirements measured in petaflop/s-decades. In this paper, we investigate transformer-based architectures for improving model performance in a low-data regime by selectively replacing attention layers with feed-forward and quasi-recurrent neural network layers. We test these architectures on the standard Enwik8 and Wikitext-103 corpora. Our results show that our reduced architectures outperform existing models with a comparable number of parameters, and obtain comparable performance to larger models while significantly reducing the number of parameters.