RV4Chatbot: Are Chatbots Allowed to Dream of Electric Sheep?

TL;DR

This paper introduces RV4Chatbot, a runtime verification framework that monitors chatbot behaviour to ensure safety and adherence to expected interaction protocols, with implementations for Rasa and Dialogflow chatbots.

Contribution

It formalizes chatbot interaction protocols and provides two implementations for runtime verification in different chatbot frameworks.

Findings

Effective monitoring of chatbot behaviour demonstrated in factory automation scenarios.

RV4Chatbot successfully detects deviations from expected behaviour.

Framework adaptable to multiple chatbot platforms.

Abstract

Chatbots have become integral to various application domains, including those with safety-critical considerations. As a result, there is a pressing need for methods that ensure chatbots consistently adhere to expected, safe behaviours. In this paper, we introduce RV4Chatbot, a Runtime Verification framework designed to monitor deviations in chatbot behaviour. We formalise expected behaviours as interaction protocols between the user and the chatbot. We present the RV4Chatbot design and describe two implementations that instantiate it: RV4Rasa, for monitoring chatbots created with the Rasa framework, and RV4Dialogflow, for monitoring Dialogflow chatbots. Additionally, we detail experiments conducted in a factory automation scenario using both RV4Rasa and RV4Dialogflow.