Cascaded LSTMs based Deep Reinforcement Learning for Goal-driven Dialogue

TL;DR

This paper introduces a cascaded LSTM-based deep reinforcement learning model that jointly learns dialogue understanding and management for goal-driven systems, outperforming traditional models in booking tasks.

Contribution

It presents a novel joint LSTM-based neural network architecture that integrates NLU and dialogue management without explicit state modeling, optimized via reinforcement learning.

Findings

Outperforms traditional MDP and single LSTM models in booking tasks

Learns meaningful dialogue state representations through visualization

Jointly optimizes dialogue understanding and management end-to-end

Abstract

This paper proposes a deep neural network model for joint modeling Natural Language Understanding (NLU) and Dialogue Management (DM) in goal-driven dialogue systems. There are three parts in this model. A Long Short-Term Memory (LSTM) at the bottom of the network encodes utterances in each dialogue turn into a turn embedding. Dialogue embeddings are learned by a LSTM at the middle of the network, and updated by the feeding of all turn embeddings. The top part is a forward Deep Neural Network which converts dialogue embeddings into the Q-values of different dialogue actions. The cascaded LSTMs based reinforcement learning network is jointly optimized by making use of the rewards received at each dialogue turn as the only supervision information. There is no explicit NLU and dialogue states in the network. Experimental results show that our model outperforms both traditional Markov Decision Process (MDP) model and single LSTM with Deep Q-Network on meeting room booking tasks. Visualization of dialogue embeddings illustrates that the model can learn the representation of dialogue states.