A Survey on Word Meta-Embedding Learning

TL;DR

This survey reviews meta-embedding learning in NLP, categorizing methods based on their operation modes, training approaches, and domain adaptation, while discussing limitations and future research directions.

Contribution

It provides the first systematic classification and analysis of meta-embedding learning methods in NLP, filling a significant gap in existing literature.

Findings

Meta-embedding methods vary by static vs. contextualized embeddings

Unsupervised and task-specific fine-tuning approaches are identified

Limitations include scalability and domain adaptation challenges

Abstract

Meta-embedding (ME) learning is an emerging approach that attempts to learn more accurate word embeddings given existing (source) word embeddings as the sole input. Due to their ability to incorporate semantics from multiple source embeddings in a compact manner with superior performance, ME learning has gained popularity among practitioners in NLP. To the best of our knowledge, there exist no prior systematic survey on ME learning and this paper attempts to fill this need. We classify ME learning methods according to multiple factors such as whether they (a) operate on static or contextualised embeddings, (b) trained in an unsupervised manner or (c) fine-tuned for a particular task/domain. Moreover, we discuss the limitations of existing ME learning methods and highlight potential future research directions.