"A cold, technical decision-maker": Can AI provide explainability, negotiability, and humanity?

TL;DR

This study explores perceptions of AI as a decision-maker, highlighting its strengths in mechanical tasks and limitations in moral judgments, and introduces the concept of 'negotiability' in decision processes.

Contribution

It provides qualitative insights into how people perceive AI's explainability, humanity, and negotiability across different countries and decision-making contexts.

Findings

AI viewed as rigid and mechanical but lacking moral judgment

Participants value negotiability as flexibility beyond formal criteria

Concerns about AI's opacity and alignment with social values

Abstract

Algorithmic systems are increasingly deployed to make decisions in many areas of people's lives. The shift from human to algorithmic decision-making has been accompanied by concern about potentially opaque decisions that are not aligned with social values, as well as proposed remedies such as explainability. We present results of a qualitative study of algorithmic decision-making, comprised of five workshops conducted with a total of 60 participants in Finland, Germany, the United Kingdom, and the United States. We invited participants to reason about decision-making qualities such as explainability and accuracy in a variety of domains. Participants viewed AI as a decision-maker that follows rigid criteria and performs mechanical tasks well, but is largely incapable of subjective or morally complex judgments. We discuss participants' consideration of humanity in decision-making, and introduce the concept of 'negotiability,' the ability to go beyond formal criteria and work flexibly around the system.