Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning

TL;DR

This paper investigates the reliability of human bandit feedback in sequence-to-sequence reinforcement learning, demonstrating that reliable feedback improves reward estimation and translation quality, with potential for large-scale applications.

Contribution

The study analyzes human feedback reliability and its impact on reward learning in sequence-to-sequence RL, highlighting the effectiveness of standardized cardinal feedback.

Findings

Cardinal feedback shows high reliability and ease of learning.

Reward estimator trained on cardinal feedback improves translation by over 1 BLEU.

RL can be effective with small, reliable human feedback datasets.

Abstract

We present a study on reinforcement learning (RL) from human bandit feedback for sequence-to-sequence learning, exemplified by the task of bandit neural machine translation (NMT). We investigate the reliability of human bandit feedback, and analyze the influence of reliability on the learnability of a reward estimator, and the effect of the quality of reward estimates on the overall RL task. Our analysis of cardinal (5-point ratings) and ordinal (pairwise preferences) feedback shows that their intra- and inter-annotator $\alpha$-agreement is comparable. Best reliability is obtained for standardized cardinal feedback, and cardinal feedback is also easiest to learn and generalize from. Finally, improvements of over 1 BLEU can be obtained by integrating a regression-based reward estimator trained on cardinal feedback for 800 translations into RL for NMT. This shows that RL is possible even from small amounts of fairly reliable human feedback, pointing to a great potential for applications at larger scale.