Pre-training Universal Language Representation

TL;DR

This paper introduces a universal language representation learning method that embeds various linguistic units into a single vector space, improving performance on analogy, downstream tasks, and question answering.

Contribution

It proposes the MiSAD training objective and demonstrates that a well-designed pre-training scheme can produce effective universal language representations.

Findings

Achieves highest accuracy on analogy tasks across language levels

Significantly improves downstream task performance on GLUE benchmark

Enhances question answering capabilities

Abstract

Despite the well-developed cut-edge representation learning for language, most language representation models usually focus on specific levels of linguistic units. This work introduces universal language representation learning, i.e., embeddings of different levels of linguistic units or text with quite diverse lengths in a uniform vector space. We propose the training objective MiSAD that utilizes meaningful n-grams extracted from large unlabeled corpus by a simple but effective algorithm for pre-trained language models. Then we empirically verify that well designed pre-training scheme may effectively yield universal language representation, which will bring great convenience when handling multiple layers of linguistic objects in a unified way. Especially, our model achieves the highest accuracy on analogy tasks in different language levels and significantly improves the performance on downstream tasks in the GLUE benchmark and a question answering dataset.