Dialogue Response Ranking Training with Large-Scale Human Feedback Data

TL;DR

This paper introduces DialogRPT, a large-scale feedback prediction model trained on 133 million human feedback pairs, which improves response ranking in open-domain dialogue systems by better aligning with human preferences.

Contribution

The paper presents a novel large-scale training dataset and a GPT-2 based ranking model that outperforms baselines in predicting engaging responses and correlates well with human preferences.

Findings

DialogRPT outperforms baselines in feedback prediction.

The model correlates better with human preferences.

Combining feedback and scoring models improves response ranking.

Abstract

Existing open-domain dialog models are generally trained to minimize the perplexity of target human responses. However, some human replies are more engaging than others, spawning more followup interactions. Current conversational models are increasingly capable of producing turns that are context-relevant, but in order to produce compelling agents, these models need to be able to predict and optimize for turns that are genuinely engaging. We leverage social media feedback data (number of replies and upvotes) to build a large-scale training dataset for feedback prediction. To alleviate possible distortion between the feedback and engagingness, we convert the ranking problem to a comparison of response pairs which involve few confounding factors. We trained DialogRPT, a set of GPT-2 based models on 133M pairs of human feedback data and the resulting ranker outperformed several baselines. Particularly, our ranker outperforms the conventional dialog perplexity baseline with a large margin on predicting Reddit feedback. We finally combine the feedback prediction models and a human-like scoring model to rank the machine-generated dialog responses. Crowd-sourced human evaluation shows that our ranking method correlates better with real human preferences than baseline models.