Edina: Building an Open Domain Socialbot with Self-dialogues

TL;DR

Edina is a social chatbot developed for the Amazon Alexa Prize that uses innovative self-dialogues from Mechanical Turk data to train neural networks and improve response quality through a hybrid rule-based and machine learning architecture.

Contribution

The paper introduces a novel self-dialogue data collection technique and a hybrid architecture combining rule-based and neural network components for socialbots.

Findings

Self-dialogues are natural and efficient for data collection.

Confidence scores from soft rules correlate with reply quality.

Hybrid approach improves response coverage and quality.

Abstract

We present Edina, the University of Edinburgh's social bot for the Amazon Alexa Prize competition. Edina is a conversational agent whose responses utilize data harvested from Amazon Mechanical Turk (AMT) through an innovative new technique we call self-dialogues. These are conversations in which a single AMT Worker plays both participants in a dialogue. Such dialogues are surprisingly natural, efficient to collect and reflective of relevant and/or trending topics. These self-dialogues provide training data for a generative neural network as well as a basis for soft rules used by a matching score component. Each match of a soft rule against a user utterance is associated with a confidence score which we show is strongly indicative of reply quality, allowing this component to self-censor and be effectively integrated with other components. Edina's full architecture features a rule-based system backing off to a matching score, backing off to a generative neural network. Our hybrid data-driven methodology thus addresses both coverage limitations of a strictly rule-based approach and the lack of guarantees of a strictly machine-learning approach.