Creative Uses of AI Systems and their Explanations: A Case Study from Insurance

TL;DR

This paper investigates how AI and explainability methods are creatively used in the insurance industry, revealing divergences from intended use and offering design recommendations to better manage unexpected user behaviors.

Contribution

It provides a qualitative case study highlighting creative uses and misalignments in AI explainability in insurance, with practical design insights.

Findings

AI is used creatively in daily workflows

Divergence exists between intended and actual AI use

Recommendations for designing better human-AI interactions

Abstract

Recent works have recognized the need for human-centered perspectives when designing and evaluating human-AI interactions and explainable AI methods. Yet, current approaches fall short at intercepting and managing unexpected user behavior resulting from the interaction with AI systems and explainability methods of different stake-holder groups. In this work, we explore the use of AI and explainability methods in the insurance domain. In an qualitative case study with participants with different roles and professional backgrounds, we show that AI and explainability methods are used in creative ways in daily workflows, resulting in a divergence between their intended and actual use. Finally, we discuss some recommendations for the design of human-AI interactions and explainable AI methods to manage the risks and harness the potential of unexpected user behavior.