BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems

TL;DR

BotSIM is an end-to-end simulation toolkit that enhances commercial task-oriented dialog systems by reducing manual testing, enabling holistic evaluation, and providing actionable troubleshooting insights.

Contribution

It introduces a novel, data-efficient framework combining dialog generation, simulation, and remediation for improved bot evaluation and troubleshooting.

Findings

Effective end-to-end evaluation demonstrated on commercial platforms

Reduces manual effort in test case creation

Provides actionable insights for bot improvement

Abstract

We present BotSIM, a data-efficient end-to-end Bot SIMulation toolkit for commercial text-based task-oriented dialog (TOD) systems. BotSIM consists of three major components: 1) a Generator that can infer semantic-level dialog acts and entities from bot definitions and generate user queries via model-based paraphrasing; 2) an agenda-based dialog user Simulator (ABUS) to simulate conversations with the dialog agents; 3) a Remediator to analyze the simulated conversations, visualize the bot health reports and provide actionable remediation suggestions for bot troubleshooting and improvement. We demonstrate BotSIM's effectiveness in end-to-end evaluation, remediation and multi-intent dialog generation via case studies on two commercial bot platforms. BotSIM's "generation-simulation-remediation" paradigm accelerates the end-to-end bot evaluation and iteration process by: 1) reducing manual test cases creation efforts; 2) enabling a holistic gauge of the bot in terms of NLU and end-to-end performance via extensive dialog simulation; 3) improving the bot troubleshooting process with actionable suggestions. A demo of our system can be found at https://tinyurl.com/mryu74cd and a demo video at https://youtu.be/qLi5iSoly30. We have open-sourced the toolkit at https://github.com/salesforce/botsim