Pre-Training a Graph Recurrent Network for Language Representation

TL;DR

This paper introduces a graph recurrent network for language representation that offers a more efficient, linear-complexity alternative to Transformers, demonstrating effectiveness in various language understanding tasks across English and Chinese.

Contribution

The paper proposes a novel graph recurrent network architecture optimized for self-supervised learning, improving efficiency and output diversity over attention-based models.

Findings

Achieves linear complexity, reducing inference time.

Generates more diverse outputs with less redundancy.

Effective in both English and Chinese language tasks.

Abstract

Transformer-based pre-trained models have gained much advance in recent years, becoming one of the most important backbones in natural language processing. Recent work shows that the attention mechanism inside Transformer may not be necessary, both convolutional neural networks and multi-layer perceptron based models have also been investigated as Transformer alternatives. In this paper, we consider a graph recurrent network for language model pre-training, which builds a graph structure for each sequence with local token-level communications, together with a sentence-level representation decoupled from other tokens. The original model performs well in domain-specific text classification under supervised training, however, its potential in learning transfer knowledge by self-supervised way has not been fully exploited. We fill this gap by optimizing the architecture and verifying its effectiveness in more general language understanding tasks, for both English and Chinese languages. As for model efficiency, instead of the quadratic complexity in Transformer-based models, our model has linear complexity and performs more efficiently during inference. Moreover, we find that our model can generate more diverse outputs with less contextualized feature redundancy than existing attention-based models.