Understanding the Impact of Physicians' Legal Considerations on XAI Systems

TL;DR

This paper explores how physicians' legal concerns influence the design of explainable AI systems in healthcare, emphasizing the need for contextual information to support legal risk mitigation.

Contribution

It provides insights from physician interviews on legal concerns and proposes design implications for XAI systems to address these issues.

Findings

Physicians anticipate risks and legal concerns with AI in patient care.

Legal risk mitigation strategies may change with new AI systems.

Designs should include contextual information to support legal risk management.

Abstract

Physicians are--and feel--ethically, professionally, and legally responsible for patient outcomes, buffering patients from harmful AI determinations from medical AI systems. Many have called for explainable AI (XAI) systems to help physicians incorporate medical AI recommendations into their workflows in a way that reduces the potential of harms to patients. While prior work has demonstrated how physicians' legal concerns impact their medical decision making, little work has explored how XAI systems should be designed in light of these concerns. In this study, we conducted interviews with 10 physicians to understand where and how they anticipate errors that may occur with a medical AI system and how these anticipated errors connect to their legal concerns. In our study, physicians anticipated risks associated with using an AI system for patient care, but voiced unknowns around how their legal risk mitigation strategies may change given a new technical system. Based on these findings, we describe the implications for designing XAI systems that can address physicians' legal concerns. Specifically, we identify the need to provide AI recommendations alongside contextual information that guides their risk mitigation strategies, including how non-legally related aspects of their systems, such as medical documentation and auditing requests, might be incorporated into a legal case.