Relation Extraction or Pattern Matching? Unravelling the Generalisation Limits of Language Models for Biographical RE

TL;DR

This paper investigates the limits of language models in relation extraction, revealing that models often overfit to dataset artifacts and that data quality significantly impacts transferability, with no one-size-fits-all adaptation strategy.

Contribution

The study provides a comprehensive analysis of the generalisation challenges in RE models, highlighting the importance of data quality and benchmark structure in transfer performance.

Findings

Higher intra-dataset performance does not imply better transferability.

Data quality is more crucial than lexical similarity for robust transfer.

Zero-shot baselines can outperform cross-dataset models.

Abstract

Analysing the generalisation capabilities of relation extraction (RE) models is crucial for assessing whether they learn robust relational patterns or rely on spurious correlations. Our cross-dataset experiments find that RE models struggle with unseen data, even within similar domains. Notably, higher intra-dataset performance does not indicate better transferability, instead often signaling overfitting to dataset-specific artefacts. Our results also show that data quality, rather than lexical similarity, is key to robust transfer, and the choice of optimal adaptation strategy depends on the quality of data available: while fine-tuning yields the best cross-dataset performance with high-quality data, few-shot in-context learning (ICL) is more effective with noisier data. However, even in these cases, zero-shot baselines occasionally outperform all cross-dataset results. Structural issues in RE benchmarks, such as single-relation per sample constraints and non-standardised negative class definitions, further hinder model transferability.