Conformal retrofitting via Riemannian manifolds: distilling task-specific graphs into pretrained embeddings

TL;DR

This paper introduces a Riemannian manifold-based retrofitting method for pretrained embeddings, improving their ability to incorporate task-specific graph knowledge, especially for missing entities and complex graph structures.

Contribution

It proposes a conformality regularizer and a Riemannian feedforward layer to enhance retrofitting of pretrained embeddings with non-Euclidean geometry.

Findings

Outperforms Euclidean methods on WordNet link prediction

Prevents overfitting on missing entities

Enables better representation of complex graph structures

Abstract

Pretrained (language) embeddings are versatile, task-agnostic feature representations of entities, like words, that are central to many machine learning applications. These representations can be enriched through retrofitting, a class of methods that incorporate task-specific domain knowledge encoded as a graph over a subset of these entities. However, existing retrofitting algorithms face two limitations: they overfit the observed graph by failing to represent relationships with missing entities; and they underfit the observed graph by only learning embeddings in Euclidean manifolds, which cannot faithfully represent even simple tree-structured or cyclic graphs. We address these problems with two key contributions: (i) we propose a novel regularizer, a conformality regularizer, that preserves local geometry from the pretrained embeddings---enabling generalization to missing entities and (ii) a new Riemannian feedforward layer that learns to map pre-trained embeddings onto a non-Euclidean manifold that can better represent the entire graph. Through experiments on WordNet, we demonstrate that the conformality regularizer prevents even existing (Euclidean-only) methods from overfitting on link prediction for missing entities, and---together with the Riemannian feedforward layer---learns non-Euclidean embeddings that outperform them.