Transfer Reward Learning for Policy Gradient-Based Text Generation

TL;DR

This paper introduces a transfer learning approach for reward models in policy gradient-based text generation, improving semantic evaluation metrics in image captioning tasks.

Contribution

It proposes a transferable reward learner that enhances policy gradient models by using model-based rewards from sentence similarity tasks, outperforming n-gram overlap measures.

Findings

Improved semantic similarity scores on MSCOCO dataset.

Enhanced performance on Flickr-30k dataset.

Demonstrated general applicability of transfer learning in reward models.

Abstract

Task-specific scores are often used to optimize for and evaluate the performance of conditional text generation systems. However, such scores are non-differentiable and cannot be used in the standard supervised learning paradigm. Hence, policy gradient methods are used since the gradient can be computed without requiring a differentiable objective. However, we argue that current n-gram overlap based measures that are used as rewards can be improved by using model-based rewards transferred from tasks that directly compare the similarity of sentence pairs. These reward models either output a score of sentence-level syntactic and semantic similarity between entire predicted and target sentences as the expected return, or for intermediate phrases as segmented accumulative rewards. We demonstrate that using a \textit{Transferable Reward Learner} leads to improved results on semantical evaluation measures in policy-gradient models for image captioning tasks. Our InferSent actor-critic model improves over a BLEU trained actor-critic model on MSCOCO when evaluated on a Word Mover's Distance similarity measure by 6.97 points, also improving on a Sliding Window Cosine Similarity measure by 10.48 points. Similar performance improvements are also obtained on the smaller Flickr-30k dataset, demonstrating the general applicability of the proposed transfer learning method.