Rethinking Relation Extraction: Beyond Shortcuts to Generalization with a Debiased Benchmark

TL;DR

This paper introduces a new debiased benchmark dataset, DREB, and a debiasing method, MixDebias, to improve relation extraction models' ability to generalize beyond shortcut biases.

Contribution

It presents DREB, a benchmark that reduces entity bias in relation extraction, and MixDebias, a novel debiasing technique that enhances model robustness.

Findings

MixDebias improves performance on DREB

DREB provides a more reliable evaluation of generalization

MixDebias maintains performance on original datasets

Abstract

Benchmarks are crucial for evaluating machine learning algorithm performance, facilitating comparison and identifying superior solutions. However, biases within datasets can lead models to learn shortcut patterns, resulting in inaccurate assessments and hindering real-world applicability. This paper addresses the issue of entity bias in relation extraction tasks, where models tend to rely on entity mentions rather than context. We propose a debiased relation extraction benchmark DREB that breaks the pseudo-correlation between entity mentions and relation types through entity replacement. DREB utilizes Bias Evaluator and PPL Evaluator to ensure low bias and high naturalness, providing a reliable and accurate assessment of model generalization in entity bias scenarios. To establish a new baseline on DREB, we introduce MixDebias, a debiasing method combining data-level and model training-level techniques. MixDebias effectively improves model performance on DREB while maintaining performance on the original dataset. Extensive experiments demonstrate the effectiveness and robustness of MixDebias compared to existing methods, highlighting its potential for improving the generalization ability of relation extraction models. We will release DREB and MixDebias publicly.