Dialogue-Based Relation Extraction

TL;DR

This paper introduces DialogRE, a human-annotated dataset for dialogue-based relation extraction, emphasizing speaker information and proposing new evaluation metrics to improve RE performance in conversational contexts.

Contribution

The paper presents the first dialogue-based RE dataset, analyzes speaker importance, and develops a new metric for evaluating RE methods in dialogues.

Findings

Speaker-aware models improve RE performance.

DialogRE enables cross-sentence relation extraction analysis.

New evaluation metric better captures conversational RE challenges.

Abstract

We present the first human-annotated dialogue-based relation extraction (RE) dataset DialogRE, aiming to support the prediction of relation(s) between two arguments that appear in a dialogue. We further offer DialogRE as a platform for studying cross-sentence RE as most facts span multiple sentences. We argue that speaker-related information plays a critical role in the proposed task, based on an analysis of similarities and differences between dialogue-based and traditional RE tasks. Considering the timeliness of communication in a dialogue, we design a new metric to evaluate the performance of RE methods in a conversational setting and investigate the performance of several representative RE methods on DialogRE. Experimental results demonstrate that a speaker-aware extension on the best-performing model leads to gains in both the standard and conversational evaluation settings. DialogRE is available at https://dataset.org/dialogre/.