Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction

TL;DR

This paper introduces active testing, a new evaluation method that combines noisy test data with manual annotations to provide more accurate and unbiased performance assessments for distantly supervised relation extraction models.

Contribution

The paper proposes a novel active testing approach that mitigates bias in evaluation by integrating noisy datasets with limited manual annotations.

Findings

Active testing reduces bias in relation extraction evaluation.

The method achieves approximately unbiased performance estimates.

Experiments demonstrate improved evaluation accuracy on benchmark datasets.

Abstract

Distant supervision has been a widely used method for neural relation extraction for its convenience of automatically labeling datasets. However, existing works on distantly supervised relation extraction suffer from the low quality of test set, which leads to considerable biased performance evaluation. These biases not only result in unfair evaluations but also mislead the optimization of neural relation extraction. To mitigate this problem, we propose a novel evaluation method named active testing through utilizing both the noisy test set and a few manual annotations. Experiments on a widely used benchmark show that our proposed approach can yield approximately unbiased evaluations for distantly supervised relation extractors.