Cold-Start Reinforcement Learning with Softmax Policy Gradient

TL;DR

This paper introduces a novel reinforcement learning method using a softmax value function that eliminates the need for warm-start training and variance reduction, improving efficiency in sequence generation tasks.

Contribution

It presents a new cold-start reinforcement learning approach combining policy-gradient benefits with maximum-likelihood simplicity, validated on summarization and captioning tasks.

Findings

Effective in automatic summarization

Improves image captioning performance

Reduces training overhead

Abstract

Policy-gradient approaches to reinforcement learning have two common and undesirable overhead procedures, namely warm-start training and sample variance reduction. In this paper, we describe a reinforcement learning method based on a softmax value function that requires neither of these procedures. Our method combines the advantages of policy-gradient methods with the efficiency and simplicity of maximum-likelihood approaches. We apply this new cold-start reinforcement learning method in training sequence generation models for structured output prediction problems. Empirical evidence validates this method on automatic summarization and image captioning tasks.