BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems
Guangsen Wang, Shafiq Joty, Junnan Li, Steven Hoi

TL;DR
BotSIM is a comprehensive, open-source toolkit designed to facilitate large-scale evaluation, diagnosis, and improvement of commercial task-oriented dialog systems, aiming to accelerate development and reduce costs.
Contribution
It introduces a modular, layered architecture for bot simulation, integrating dialog generation, user simulation, and analytics in a unified environment.
Findings
Effective bot evaluation and remediation demonstrated with Einstein BotBuilder
Significantly reduces development and evaluation costs
Supports scalable, data-efficient end-to-end testing
Abstract
We introduce BotSIM, a modular, open-source Bot SIMulation environment with dialog generation, user simulation and conversation analytics capabilities. BotSIM aims to serve as a one-stop solution for large-scale data-efficient end-to-end evaluation, diagnosis and remediation of commercial task-oriented dialog (TOD) systems to significantly accelerate commercial bot development and evaluation, reduce cost and time-to-market. BotSIM adopts a layered design comprising the infrastructure layer, the adaptor layer and the application layer. The infrastructure layer hosts key models and components to support BotSIM's major functionalities via a streamlined "generation-simulation-remediation" pipeline. The adaptor layer is used to extend BotSIM to accommodate new bot platforms. The application layer provides a suite of command line tools and a Web App to significantly lower the entry barrier…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsService-Oriented Architecture and Web Services · Speech and dialogue systems · AI in Service Interactions
