Dice in the Black Box: User Experiences with an Inscrutable Algorithm

TL;DR

This study reveals how users tend to trust seemingly intelligent black box algorithms, even when such systems operate randomly, highlighting the need for safeguards to prevent misplaced trust.

Contribution

The paper demonstrates the susceptibility of users to trust inscrutable algorithms and offers insights into their trust-building processes and potential corrective measures.

Findings

Users often trust black box algorithms despite randomness.

Trust is influenced by perceived intelligence and framing.

Recommendations for reducing unwarranted trust in AI systems.

Abstract

We demonstrate that users may be prone to place an inordinate amount of trust in black box algorithms that are framed as intelligent. We deploy an algorithm that purportedly assesses the positivity and negativity of a users' writing emotional writing. In actuality, the algorithm responds in a random fashion. We qualitatively examine the paths to trust that users followed while testing the system. In light of the ease with which users may trust systems exhibiting "intelligent behavior" we recommend corrective approaches.