Designing AI for Trust and Collaboration in Time-Constrained Medical Decisions: A Sociotechnical Lens

TL;DR

This paper explores how decision support tools for antidepressant treatment should be designed as sociotechnical systems that facilitate collaboration, transparency, and integration within clinical workflows to improve real-world adoption.

Contribution

It introduces a co-design approach to develop DSTs that align with clinical sociotechnical systems and highlights the importance of multi-user, explainable AI features for effective medical decision-making.

Findings

DSTs should support patient-provider collaboration

On-demand explanations improve trust and understanding

Designing for sociotechnical integration enhances clinical adoption

Abstract

Major depressive disorder is a debilitating disease affecting 264 million people worldwide. While many antidepressant medications are available, few clinical guidelines support choosing among them. Decision support tools (DSTs) embodying machine learning models may help improve the treatment selection process, but often fail in clinical practice due to poor system integration. We use an iterative, co-design process to investigate clinicians' perceptions of using DSTs in antidepressant treatment decisions. We identify ways in which DSTs need to engage with the healthcare sociotechnical system, including clinical processes, patient preferences, resource constraints, and domain knowledge. Our results suggest that clinical DSTs should be designed as multi-user systems that support patient-provider collaboration and offer on-demand explanations that address discrepancies between predictions and current standards of care. Through this work, we demonstrate how current trends in explainable AI may be inappropriate for clinical environments and consider paths towards designing these tools for real-world medical systems.