PRiSM: Enhancing Low-Resource Document-Level Relation Extraction with Relation-Aware Score Calibration

TL;DR

PRiSM is a calibration-based method that significantly improves low-resource document-level relation extraction by adjusting logits with relation semantic information, reducing calibration error and boosting F1 scores.

Contribution

This work introduces PRiSM, a novel calibration approach that enhances low-resource DocRE performance by learning to adapt logits based on relation semantics.

Findings

Performance improved by up to 26.38 F1 points.

Calibration error reduced by up to 36 times.

Effective with only about 3% of training data.

Abstract

Document-level relation extraction (DocRE) aims to extract relations of all entity pairs in a document. A key challenge in DocRE is the cost of annotating such data which requires intensive human effort. Thus, we investigate the case of DocRE in a low-resource setting, and we find that existing models trained on low data overestimate the NA ("no relation") label, causing limited performance. In this work, we approach the problem from a calibration perspective and propose PRiSM, which learns to adapt logits based on relation semantic information. We evaluate our method on three DocRE datasets and demonstrate that integrating existing models with PRiSM improves performance by as much as 26.38 F1 score, while the calibration error drops as much as 36 times when trained with about 3% of data. The code is publicly available at https://github.com/brightjade/PRiSM.