Multimodal AI for Serious Illness Communication: Opportunities and Best Practices

TL;DR

This paper explores using AI to improve communication during serious illness discussions by integrating multiple communication modes and addressing diverse patient needs.

Contribution

The paper introduces Clinical SmartReporter™, the first AI tool for patient-facing serious illness communication, combining multimodal and multilingual support.

Findings

Clinical SmartReporter™ can generate real-time transcribed reports using machine learning and natural language processing.

User-centered design insights from patient, caregiver, and clinician interviews informed the development of the tool.

The tool supports multilingual translation and speaker diarization in complex clinical settings.

Abstract

Patients and families face complex communication challenges in the setting of serious illness. These difficulties are compounded for individuals who are non-English-speaking, have low health literacy, have visual/auditory impairments, or possess unique learning styles. We hypothesize that augmenting standard verbal communication modalities with a multimodal communication approach (integrating written, visual, and other modalities) can improve medical information-sharing. In the first use of artificial intelligence (AI) for patient-facing serious illness communication, we developed Clinical SmartReporter™, a program capable of generating real-time transcribed reports of family meetings for patients, their caregivers, and other clinicians. Clinical SmartReporter™ uses machine learning, natural language processing, and speaker diarization modules to accurately integrate medical terminology, translate between multiple languages, and distinguish between a large number of different speakers. In this presentation, we will describe our model of multimodal communication, present results of our user-centered design process based on patient, caregiver, and clinician interviews, and demonstrate the use of Clinical SmartReporter™ in the setting of a simulated family meeting (professionally filmed video). We will outline technology-based opportunities for the advancement of communication science in palliative care research, quality improvement, and medical education. We will also facilitate a discussion of practical and ethical challenges, as well as the need to establish field-wide best practices for AI implementation in geriatrics and palliative care settings. These include data security considerations, partnership with institutional offices for technology licensing and privacy/security, and protection of the integrity of intimate patient-clinician relationships, emphasizing the need for clinician advocacy and cross-institutional collaboration.