Latent Relation Language Models

TL;DR

Latent Relation Language Models (LRLMs) integrate knowledge graph relations into language modeling, enhancing performance and enabling entity span annotation, with empirical evidence showing improvements over baseline models.

Contribution

The paper introduces LRLMs, a novel approach that jointly models text and knowledge graph relations, improving language modeling and entity annotation capabilities.

Findings

Empirical improvements over baseline language models.

Successful annotation of entity spans using learned relations.

Ability to predict contextually appropriate relations.

Abstract

In this paper, we propose Latent Relation Language Models (LRLMs), a class of language models that parameterizes the joint distribution over the words in a document and the entities that occur therein via knowledge graph relations. This model has a number of attractive properties: it not only improves language modeling performance, but is also able to annotate the posterior probability of entity spans for a given text through relations. Experiments demonstrate empirical improvements over both a word-based baseline language model and a previous approach that incorporates knowledge graph information. Qualitative analysis further demonstrates the proposed model's ability to learn to predict appropriate relations in context.