RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations

TL;DR

RetroGAN introduces a GAN-based post-specialization system that enhances word embeddings, especially for out-of-knowledge and rare words, achieving state-of-the-art results in benchmarks and downstream tasks.

Contribution

It proposes a novel GAN-based approach for post-specialization of word vectors, enabling better generalization to unseen and rare words beyond traditional retrofitting methods.

Findings

Achieves state-of-the-art performance on CARD-660 benchmark.

Effectively generalizes to out-of-knowledge and rare words.

Improves downstream sentence simplification tasks.

Abstract

Retrofitting is a technique used to move word vectors closer together or further apart in their space to reflect their relationships in a Knowledge Base (KB). However, retrofitting only works on concepts that are present in that KB. RetroGAN uses a pair of Generative Adversarial Networks (GANs) to learn a one-to-one mapping between concepts and their retrofitted counterparts. It applies that mapping (post-specializes) to handle concepts that do not appear in the original KB in a manner similar to how some natural language systems handle out-of-vocabulary entries. We test our system on three word-similarity benchmarks and a downstream sentence simplification task and achieve the state of the art (CARD-660). Altogether, our results demonstrate our system's effectiveness for out-of-knowledge and rare word generalization.