My Actions Speak Louder Than Your Words: When User Behavior Predicts Their Beliefs about Agents' Attributes

TL;DR

This paper reveals that users' experiences with agents influence their perceptions of the agents' attributes, often leading to biased judgments based on outcomes rather than relevant qualities, highlighting the need for improved modeling.

Contribution

It identifies a bias where user outcomes affect their ratings of agent attributes, and suggests augmenting models to account for and correct these perception biases.

Findings

Users with better outcomes rated agents more favorably.

Outcome-based biases influence perceptions of agent attributes.

Models should incorporate mechanisms to detect and correct perception biases.

Abstract

An implicit expectation of asking users to rate agents, such as an AI decision-aid, is that they will use only relevant information -- ask them about an agent's benevolence, and they should consider whether or not it was kind. Behavioral science, however, suggests that people sometimes use irrelevant information. We identify an instance of this phenomenon, where users who experience better outcomes in a human-agent interaction systematically rated the agent as having better abilities, being more benevolent, and exhibiting greater integrity in a post hoc assessment than users who experienced worse outcome -- which were the result of their own behavior -- with the same agent. Our analyses suggest the need for augmentation of models so that they account for such biased perceptions as well as mechanisms so that agents can detect and even actively work to correct this and similar biases of users.