GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations

TL;DR

This paper introduces GLoMo, a framework for learning and transferring latent relational graphs from large-scale unlabeled data, enhancing performance across diverse NLP and vision tasks beyond traditional feature transfer methods.

Contribution

It proposes a novel method for unsupervised learning of relational graphs that are transferable across different data modalities and embedding types, extending transfer learning capabilities.

Findings

Improved performance on question answering, natural language inference, sentiment analysis, and image classification.

Relational graphs are transferable across different embeddings and even embedding-free data.

The learned graphs generalize well to various downstream tasks.

Abstract

Modern deep transfer learning approaches have mainly focused on learning generic feature vectors from one task that are transferable to other tasks, such as word embeddings in language and pretrained convolutional features in vision. However, these approaches usually transfer unary features and largely ignore more structured graphical representations. This work explores the possibility of learning generic latent relational graphs that capture dependencies between pairs of data units (e.g., words or pixels) from large-scale unlabeled data and transferring the graphs to downstream tasks. Our proposed transfer learning framework improves performance on various tasks including question answering, natural language inference, sentiment analysis, and image classification. We also show that the learned graphs are generic enough to be transferred to different embeddings on which the graphs have not been trained (including GloVe embeddings, ELMo embeddings, and task-specific RNN hidden unit), or embedding-free units such as image pixels.