A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts

TL;DR

This paper introduces a modular neural dialogue system using a mixture-of-experts approach, where specialized expert bots are coordinated by a chair bot to improve response generation in task-oriented dialogues.

Contribution

The paper proposes a novel Token-level Mixture-of-Experts model for end-to-end task-oriented dialogue systems, enabling adaptive expert selection and improved performance.

Findings

8.1% increase in inform rate

0.8% increase in success rate

Effective coordination of expert bots improves response quality

Abstract

End-to-end Task-oriented Dialogue Systems (TDSs) have attracted a lot of attention for their superiority (e.g., in terms of global optimization) over pipeline modularized TDSs. Previous studies on end-to-end TDSs use a single-module model to generate responses for complex dialogue contexts. However, no model consistently outperforms the others in all cases. We propose a neural Modular Task-oriented Dialogue System(MTDS) framework, in which a few expert bots are combined to generate the response for a given dialogue context. MTDS consists of a chair bot and several expert bots. Each expert bot is specialized for a particular situation, e.g., one domain, one type of action of a system, etc. The chair bot coordinates multiple expert bots and adaptively selects an expert bot to generate the appropriate response. We further propose a Token-level Mixture-of-Expert (TokenMoE) model to implement MTDS, where the expert bots predict multiple tokens at each timestamp and the chair bot determines the final generated token by fully taking into consideration the outputs of all expert bots. Both the chair bot and the expert bots are jointly trained in an end-to-end fashion. To verify the effectiveness of TokenMoE, we carry out extensive experiments on a benchmark dataset. Compared with the baseline using a single-module model, our TokenMoE improves the performance by 8.1% of inform rate and 0.8% of success rate.