Extracting triples from dialogues for conversational social agents

TL;DR

This paper develops and evaluates models for extracting explicit symbolic triples from social conversations, addressing the unique challenges posed by dialogue phenomena like co-reference and ellipsis.

Contribution

It introduces datasets and five models specifically designed for triple extraction from social dialogue, highlighting the difficulty of capturing conversational knowledge.

Findings

Highest precision of 51.14% for complete triples on single utterances

Triple element precision reaches 69.32% on single utterances

Extraction from multi-turn conversations remains significantly more challenging

Abstract

Obtaining an explicit understanding of communication within a Hybrid Intelligence collaboration is essential to create controllable and transparent agents. In this paper, we describe a number of Natural Language Understanding models that extract explicit symbolic triples from social conversation. Triple extraction has mostly been developed and tested for Knowledge Base Completion using Wikipedia text and data for training and testing. However, social conversation is very different as a genre in which interlocutors exchange information in sequences of utterances that involve statements, questions, and answers. Phenomena such as co-reference, ellipsis, coordination, and implicit and explicit negation or confirmation are more prominent in conversation than in Wikipedia text. We therefore describe an attempt to fill this gap by releasing data sets for training and testing triple extraction from social conversation. We also created five triple extraction models and tested them in our evaluation data. The highest precision is 51.14 for complete triples and 69.32 for triple elements when tested on single utterances. However, scores for conversational triples that span multiple turns are much lower, showing that extracting knowledge from true conversational data is much more challenging.