What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use

TL;DR

This paper investigates what clinicians need from explainable machine learning models to trust and effectively use them in clinical settings, through surveys and analysis of clinician feedback.

Contribution

It provides a detailed characterization of explainability aspects that build trust, identifies key explanation classes, and proposes metrics for evaluating clinical explainability methods.

Findings

Clinicians value explanations that clarify model reasoning in context.

Certain explanation types are deemed most relevant for clinical trust.

Metrics for assessing explainability in healthcare are proposed.

Abstract

Translating machine learning (ML) models effectively to clinical practice requires establishing clinicians' trust. Explainability, or the ability of an ML model to justify its outcomes and assist clinicians in rationalizing the model prediction, has been generally understood to be critical to establishing trust. However, the field suffers from the lack of concrete definitions for usable explanations in different settings. To identify specific aspects of explainability that may catalyze building trust in ML models, we surveyed clinicians from two distinct acute care specialties (Intenstive Care Unit and Emergency Department). We use their feedback to characterize when explainability helps to improve clinicians' trust in ML models. We further identify the classes of explanations that clinicians identified as most relevant and crucial for effective translation to clinical practice. Finally, we discern concrete metrics for rigorous evaluation of clinical explainability methods. By integrating perceptions of explainability between clinicians and ML researchers we hope to facilitate the endorsement and broader adoption and sustained use of ML systems in healthcare.