Dialog Action-Aware Transformer for Dialog Policy Learning

TL;DR

This paper introduces DaTrans, a dialog action-aware transformer that leverages pre-trained language models and a novel fine-tuning task to accelerate dialog policy learning in reinforcement learning settings, showing improved performance.

Contribution

It proposes DaTrans, a transformer encoder with a new masked last action task, combining pre-trained knowledge with RL for faster dialog policy learning.

Findings

Demonstrates improved learning speed in simulator evaluations.

Achieves higher reward optimization in human evaluations.

Effectively integrates pre-trained language models with RL for dialog policy.

Abstract

Recent works usually address Dialog policy learning DPL by training a reinforcement learning (RL) agent to determine the best dialog action. However, existing works on deep RL require a large volume of agent-user interactions to achieve acceptable performance. In this paper, we propose to make full use of the plain text knowledge from the pre-trained language model to accelerate the RL agent's learning speed. Specifically, we design a dialog action-aware transformer encoder (DaTrans), which integrates a new fine-tuning procedure named masked last action task to encourage DaTrans to be dialog-aware and distils action-specific features. Then, DaTrans is further optimized in an RL setting with ongoing interactions and evolves through exploration in the dialog action space toward maximizing long-term accumulated rewards. The effectiveness and efficiency of the proposed model are demonstrated with both simulator evaluation and human evaluation.