A Generative Model of Words and Relationships from Multiple Sources

TL;DR

This paper introduces a generative model that integrates multiple data sources and relationships to improve word embeddings, especially in specialized domains with limited data, by generalizing co-occurrence to include various relationships.

Contribution

The paper presents a novel generative model that incorporates diverse data sources and relationships into semantic embeddings, extending beyond traditional co-occurrence methods.

Findings

Outperforms recent models on link prediction tasks.

Effectively utilizes partially or fully unobserved training labels.

Learns from overlapping vocabularies across data sources.

Abstract

Neural language models are a powerful tool to embed words into semantic vector spaces. However, learning such models generally relies on the availability of abundant and diverse training examples. In highly specialised domains this requirement may not be met due to difficulties in obtaining a large corpus, or the limited range of expression in average use. Such domains may encode prior knowledge about entities in a knowledge base or ontology. We propose a generative model which integrates evidence from diverse data sources, enabling the sharing of semantic information. We achieve this by generalising the concept of co-occurrence from distributional semantics to include other relationships between entities or words, which we model as affine transformations on the embedding space. We demonstrate the effectiveness of this approach by outperforming recent models on a link prediction task and demonstrating its ability to profit from partially or fully unobserved data training labels. We further demonstrate the usefulness of learning from different data sources with overlapping vocabularies.