Off-Policy Self-Critical Training for Transformer in Visual Paragraph Generation

TL;DR

This paper introduces an off-policy reinforcement learning method using self-critical training and TRIS to improve Transformer-based visual paragraph generation, reducing variance and enhancing performance.

Contribution

It proposes a novel off-policy RL algorithm with TRIS and KL-control for Transformer models, enabling efficient training for visual paragraph generation.

Findings

Achieved state-of-the-art results on visual paragraph generation

Improved image captioning performance

Reduced variance in importance sampling

Abstract

Recently, several approaches have been proposed to solve language generation problems. Transformer is currently state-of-the-art seq-to-seq model in language generation. Reinforcement Learning (RL) is useful in solving exposure bias and the optimisation on non-differentiable metrics in seq-to-seq language learning. However, Transformer is hard to combine with RL as the costly computing resource is required for sampling. We tackle this problem by proposing an off-policy RL learning algorithm where a behaviour policy represented by GRUs performs the sampling. We reduce the high variance of importance sampling (IS) by applying the truncated relative importance sampling (TRIS) technique and Kullback-Leibler (KL)-control concept. TRIS is a simple yet effective technique, and there is a theoretical proof that KL-control helps to reduce the variance of IS. We formulate this off-policy RL based on self-critical sequence training. Specifically, we use a Transformer-based captioning model as the target policy and use an image-guided language auto-encoder as the behaviour policy to explore the environment. The proposed algorithm achieves state-of-the-art performance on the visual paragraph generation and improved results on image captioning.