The HCI Aspects of Public Deployment of Research Chatbots: A User Study, Design Recommendations, and Open Challenges

TL;DR

This paper investigates how interface design choices in public research chatbots influence user perception and interaction, providing empirical insights, design recommendations, and highlighting open challenges for HCI practitioners.

Contribution

It offers the first comprehensive user study on interaction paradigms and design approaches for public research chatbots, with practical recommendations and identification of open challenges.

Findings

Anthropomorphic representation affects user perception

AI explainability influences feedback rates

Designing diegetic and extradiegetic levels enhances user experience

Abstract

Publicly deploying research chatbots is a nuanced topic involving necessary risk-benefit analyses. While there have recently been frequent discussions on whether it is responsible to deploy such models, there has been far less focus on the interaction paradigms and design approaches that the resulting interfaces should adopt, in order to achieve their goals more effectively. We aim to pose, ground, and attempt to answer HCI questions involved in this scope, by reporting on a mixed-methods user study conducted on a recent research chatbot. We find that abstract anthropomorphic representation for the agent has a significant effect on user's perception, that offering AI explainability may have an impact on feedback rates, and that two (diegetic and extradiegetic) levels of the chat experience should be intentionally designed. We offer design recommendations and areas of further focus for the research community.