SHAQ: Single Headed Attention with Quasi-Recurrence

TL;DR

SHAQ introduces a simplified, efficient neural network architecture combining single-headed attention with quasi-recurrence, achieving comparable accuracy to prior models but with significantly faster training times, making advanced NLP more accessible.

Contribution

The paper presents SHAQ, a novel architecture that merges single-headed attention with quasi-recurrence, offering a faster training alternative to complex transformer models.

Findings

SHAQ achieves similar accuracy to SHA-RNN.

SHAQ trains four times faster than SHA-RNN.

The architecture reduces computational complexity while maintaining performance.

Abstract

Natural Language Processing research has recently been dominated by large scale transformer models. Although they achieve state of the art on many important language tasks, transformers often require expensive compute resources, and days spanning to weeks to train. This is feasible for researchers at big tech companies and leading research universities, but not for scrappy start-up founders, students, and independent researchers. Stephen Merity's SHA-RNN, a compact, hybrid attention-RNN model, is designed for consumer-grade modeling as it requires significantly fewer parameters and less training time to reach near state of the art results. We analyze Merity's model here through an exploratory model analysis over several units of the architecture considering both training time and overall quality in our assessment. Ultimately, we combine these findings into a new architecture which we call SHAQ: Single Headed Attention Quasi-recurrent Neural Network. With our new architecture we achieved similar accuracy results as the SHA-RNN while accomplishing a 4x speed boost in training.