Supplementary Material: Implementation and Experiments for GAU-based Model

TL;DR

This paper analyzes the GAU-based Transformer variant FLASH, proposes a new GAU-based model, and demonstrates its superior speed and performance on the CLUE benchmark through pre-training on Chinese data.

Contribution

It introduces a novel GAU-based model, provides detailed implementation analysis, and achieves improved speed and accuracy on Chinese language tasks.

Findings

Model achieves 75.02 average score on CLUE benchmark.

Model is 45% faster than RoFormerV1.

Pre-trained on Chinese corpus with competitive results.

Abstract

In February this year Google proposed a new Transformer variant called FLASH, which has a faster speed, lower VRAM footprint and better performance. This is achieved by designing a performant layer named GAU (Gated Attention Unit), which combines the Attention layer and FFN. In this paper, some implementation details are re-analyzed both theoretically and practically. We then propose a novel GAU-based model and pre-train it on a Chinese corpus. Results of the CLUE benchmark show that our model achieves a dev average score of 75.02, 1% higher than RoFormerV1 and being 45% faster, which is also competitive with RoFormerV2.