Can You Trust Your Trust Measure?

TL;DR

This paper evaluates the applicability of trust questionnaires in human-robot interaction, revealing limitations in their generalizability across different robot types and scenarios, and offers recommendations for improving trust measurement practices.

Contribution

It systematically examines how trust questionnaires perform across various robots and scenarios, highlighting their limitations and proposing guidelines for better trust assessment in HRI.

Findings

Participants rated questionnaire items as non-applicable depending on robot type and scenario.

Trust scores remained stable despite N/A ratings, indicating potential interpretative issues.

The study provides recommendations for developing and reporting trust measures in HRI.

Abstract

Trust in human-robot interactions (HRI) is measured in two main ways: through subjective questionnaires and through behavioral tasks. To optimize measurements of trust through questionnaires, the field of HRI faces two challenges: the development of standardized measures that apply to a variety of robots with different capabilities, and the exploration of social and relational dimensions of trust in robots (e.g., benevolence). In this paper we look at how different trust questionnaires fare given these challenges that pull in different directions (being general vs. being exploratory) by studying whether people think the items in these questionnaires are applicable to different kinds of robots and interactions. In Study 1 we show that after being presented with a robot (non-humanoid) and an interaction scenario (fire evacuation), participants rated multiple questionnaire items such as "This robot is principled" as "Non-applicable to robots in general" or "Non-applicable to this robot". In Study 2 we show that the frequency of these ratings change (indeed, even for items rated as N/A to robots in general) when a new scenario is presented (game playing with a humanoid robot). Finally, while overall trust scores remained robust to N/A ratings, our results revealed potential fallacies in the way these scores are commonly interpreted. We conclude with recommendations for the development, use and results-reporting of trust questionnaires for future studies, as well as theoretical implications for the field of HRI.