Learning Universal Representations from Word to Sentence

TL;DR

This paper proposes a method for learning universal linguistic representations across words, phrases, and sentences in a shared vector space, enabling more flexible language understanding and processing.

Contribution

It introduces a universal representation learning approach, constructs analogy datasets across linguistic levels, and demonstrates that pre-trained Transformer models can effectively produce such representations.

Findings

Pre-trained Transformer models can generate universal embeddings for multiple linguistic levels.

Fine-tuning ALBERT on NLI and PPDB datasets improves analogy task accuracy.

Universal representations enhance real-world NLP applications like FAQ understanding.

Abstract

Despite the well-developed cut-edge representation learning for language, most language representation models usually focus on specific level of linguistic unit, which cause great inconvenience when being confronted with handling multiple layers of linguistic objects in a unified way. Thus this work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space through a task-independent evaluation. We present our approach of constructing analogy datasets in terms of words, phrases and sentences and experiment with multiple representation models to examine geometric properties of the learned vector space. Then we empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation. Especially, our implementation of fine-tuning ALBERT on NLI and PPDB datasets achieves the highest accuracy on analogy tasks in different language levels. Further experiments on the insurance FAQ task show effectiveness of universal representation models in real-world applications.