Making Efficient Use of a Domain Expert's Time in Relation Extraction

TL;DR

This paper explores an efficient hybrid approach combining distant supervision and active learning to improve relation extraction in text mining, reducing expert labeling effort while maintaining high accuracy.

Contribution

It introduces an active learning extension to a neural relation extraction model that refines noisy distant supervision labels with minimal expert input.

Findings

Active learning improves relation extraction accuracy.

Expert feedback reduces labeling noise.

Method achieves promising results on complex datasets.

Abstract

Scarcity of labeled data is one of the most frequent problems faced in machine learning. This is particularly true in relation extraction in text mining, where large corpora of texts exists in many application domains, while labeling of text data requires an expert to invest much time to read the documents. Overall, state-of-the art models, like the convolutional neural network used in this paper, achieve great results when trained on large enough amounts of labeled data. However, from a practical point of view the question arises whether this is the most efficient approach when one takes the manual effort of the expert into account. In this paper, we report on an alternative approach where we first construct a relation extraction model using distant supervision, and only later make use of a domain expert to refine the results. Distant supervision provides a mean of labeling data given known relations in a knowledge base, but it suffers from noisy labeling. We introduce an active learning based extension, that allows our neural network to incorporate expert feedback and report on first results on a complex data set.