RelBERT: Embedding Relations with Language Models

TL;DR

RelBERT is a fine-tuned masked language model that effectively captures relational similarities, outperforming larger models and enabling nuanced understanding of concept and entity relations with minimal training data.

Contribution

The paper introduces RelBERT, a novel relation embedding model derived from small language models, achieving state-of-the-art analogy performance with efficient training.

Findings

RelBERT surpasses larger models in analogy benchmarks.

It models relations beyond its training data, including named entities and morphological analogies.

RelBERT outperforms prompting strategies with significantly larger models.

Abstract

Many applications need access to background knowledge about how different concepts and entities are related. Although Knowledge Graphs (KG) and Large Language Models (LLM) can address this need to some extent, KGs are inevitably incomplete and their relational schema is often too coarse-grained, while LLMs are inefficient and difficult to control. As an alternative, we propose to extract relation embeddings from relatively small language models. In particular, we show that masked language models such as RoBERTa can be straightforwardly fine-tuned for this purpose, using only a small amount of training data. The resulting model, which we call RelBERT, captures relational similarity in a surprisingly fine-grained way, allowing us to set a new state-of-the-art in analogy benchmarks. Crucially, RelBERT is capable of modelling relations that go well beyond what the model has seen during training. For instance, we obtained strong results on relations between named entities with a model that was only trained on lexical relations between concepts, and we observed that RelBERT can recognise morphological analogies despite not being trained on such examples. Overall, we find that RelBERT significantly outperforms strategies based on prompting language models that are several orders of magnitude larger, including recent GPT-based models and open source models.