A Study on Building Efficient Zero-Shot Relation Extraction Models

TL;DR

This paper evaluates the robustness of zero-shot relation extraction models in realistic scenarios, proposing new strategies and finding that current models lack robustness, with AlignRE performing best overall.

Contribution

It introduces a typology of models, strategies for single pass and rejection mechanisms, and benchmarks their robustness in realistic extraction scenarios.

Findings

No existing model is fully robust to realistic assumptions

AlignRE outperforms other models in the evaluated setting

Proposed strategies improve model robustness

Abstract

Zero-shot relation extraction aims to identify relations between entity mentions using textual descriptions of novel types (i.e., previously unseen) instead of labeled training examples. Previous works often rely on unrealistic assumptions: (1) pairs of mentions are often encoded directly in the input, which prevents offline pre-computation for large scale document database querying; (2) no rejection mechanism is introduced, biasing the evaluation when using these models in a retrieval scenario where some (and often most) inputs are irrelevant and must be ignored. In this work, we study the robustness of existing zero-shot relation extraction models when adapting them to a realistic extraction scenario. To this end, we introduce a typology of existing models, and propose several strategies to build single pass models and models with a rejection mechanism. We adapt several state-of-the-art tools, and compare them in this challenging setting, showing that no existing work is really robust to realistic assumptions, but overall AlignRE (Li et al., 2024) performs best along all criteria.