Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning

TL;DR

This paper introduces T-TA, a deep bidirectional language model that computes contextual representations efficiently without repetitive inference, achieving significant speedups and competitive accuracy compared to BERT in unsupervised tasks.

Contribution

The paper presents T-TA, a novel Transformer-based autoencoder that enables fast, accurate, and non-repetitive contextual language representations for unsupervised learning.

Findings

T-TA is over six times faster than BERT in reranking tasks.

T-TA is twelve times faster than BERT in semantic similarity tasks.

T-TA achieves comparable or better accuracy than BERT on key tasks.

Abstract

Even though BERT achieves successful performance improvements in various supervised learning tasks, applying BERT for unsupervised tasks still holds a limitation that it requires repetitive inference for computing contextual language representations. To resolve the limitation, we propose a novel deep bidirectional language model called Transformer-based Text Autoencoder (T-TA). The T-TA computes contextual language representations without repetition and has benefits of the deep bidirectional architecture like BERT. In run-time experiments on CPU environments, the proposed T-TA performs over six times faster than the BERT-based model in the reranking task and twelve times faster in the semantic similarity task. Furthermore, the T-TA shows competitive or even better accuracies than those of BERT on the above tasks.