Reinforcing an Image Caption Generator Using Off-Line Human Feedback

TL;DR

This paper introduces a reinforcement learning approach that uses off-line human feedback to improve image captioning models, effectively leveraging limited human ratings to enhance caption quality and generalization.

Contribution

The paper presents a novel method that utilizes off-policy reinforcement learning with human ratings to improve image captioning models beyond traditional training data.

Findings

Model generalizes human ratings to unseen images

Improved caption quality judged by human evaluators

Effective use of limited human feedback data

Abstract

Human ratings are currently the most accurate way to assess the quality of an image captioning model, yet most often the only used outcome of an expensive human rating evaluation is a few overall statistics over the evaluation dataset. In this paper, we show that the signal from instance-level human caption ratings can be leveraged to improve captioning models, even when the amount of caption ratings is several orders of magnitude less than the caption training data. We employ a policy gradient method to maximize the human ratings as rewards in an off-policy reinforcement learning setting, where policy gradients are estimated by samples from a distribution that focuses on the captions in a caption ratings dataset. Our empirical evidence indicates that the proposed method learns to generalize the human raters' judgments to a previously unseen set of images, as judged by a different set of human judges, and additionally on a different, multi-dimensional side-by-side human evaluation procedure.