Extracting Multi-valued Relations from Language Models

TL;DR

This paper investigates how pre-trained language models can be used to extract multiple objects for a given subject-relation pair, addressing the limitation of existing single-object extraction methods.

Contribution

It formulates multi-valued relation extraction as a rank-then-select task and proposes new prompting techniques and selection thresholds to improve extraction accuracy.

Findings

Choosing objects with a likelihood above a learned threshold yields 49.5% F1 score.

Existing prompting techniques are evaluated and improved for multi-object extraction.

Highlighting the challenges and potential of using LMs for multi-valued relational knowledge extraction.

Abstract

The widespread usage of latent language representations via pre-trained language models (LMs) suggests that they are a promising source of structured knowledge. However, existing methods focus only on a single object per subject-relation pair, even though often multiple objects are correct. To overcome this limitation, we analyze these representations for their potential to yield materialized multi-object relational knowledge. We formulate the problem as a rank-then-select task. For ranking candidate objects, we evaluate existing prompting techniques and propose new ones incorporating domain knowledge. Among the selection methods, we find that choosing objects with a likelihood above a learned relation-specific threshold gives a 49.5% F1 score. Our results highlight the difficulty of employing LMs for the multi-valued slot-filling task and pave the way for further research on extracting relational knowledge from latent language representations.