Post-processing Networks: Method for Optimizing Pipeline Task-oriented Dialogue Systems using Reinforcement Learning

TL;DR

This paper introduces post-processing networks (PPNs) that optimize pipeline task-oriented dialogue systems with diverse modules using reinforcement learning, enhancing overall dialogue performance without requiring modules to be differentiable.

Contribution

The paper presents a novel method for optimizing heterogeneous pipeline dialogue systems with non-neural modules via reinforcement learning applied to post-processing networks.

Findings

Improved dialogue performance on MultiWOZ dataset

Effective optimization of systems with arbitrary modules

Validated through dialogue simulation and human evaluation

Abstract

Many studies have proposed methods for optimizing the dialogue performance of an entire pipeline task-oriented dialogue system by jointly training modules in the system using reinforcement learning. However, these methods are limited in that they can only be applied to modules implemented using trainable neural-based methods. To solve this problem, we propose a method for optimizing a pipeline system composed of modules implemented with arbitrary methods for dialogue performance. With our method, neural-based components called post-processing networks (PPNs) are installed inside such a system to post-process the output of each module. All PPNs are updated to improve the overall dialogue performance of the system by using reinforcement learning, not necessitating each module to be differentiable. Through dialogue simulation and human evaluation on the MultiWOZ dataset, we show that our method can improve the dialogue performance of pipeline systems consisting of various modules.