VIRATrustData: A Trust-Annotated Corpus of Human-Chatbot Conversations About COVID-19 Vaccines

TL;DR

This paper introduces VIRATrustData, a novel annotated dataset of human-chatbot conversations about COVID-19 vaccines, focusing on trust levels, and evaluates models for trust classification in these interactions.

Contribution

The paper presents the first annotated dataset of trust in COVID-19 vaccine chatbot conversations and compares models for trust level prediction.

Findings

Trust classification is challenging in human-chatbot dialogs.

Models can predict trust levels with varying accuracy.

VIRATrustData enables future research on trust in health chatbots.

Abstract

Public trust in medical information is crucial for successful application of public health policies such as vaccine uptake. This is especially true when the information is offered remotely, by chatbots, which have become increasingly popular in recent years. Here, we explore the challenging task of human-bot turn-level trust classification. We rely on a recently released data of observationally-collected (rather than crowdsourced) dialogs with VIRA chatbot, a COVID-19 Vaccine Information Resource Assistant. These dialogs are centered around questions and concerns about COVID-19 vaccines, where trust is particularly acute. We annotated $3k$ VIRA system-user conversational turns for Low Institutional Trust or Low Agent Trust vs. Neutral or High Trust. We release the labeled dataset, VIRATrustData, the first of its kind to the best of our knowledge. We demonstrate how this task is non-trivial and compare several models that predict the different levels of trust.