# Multimodal AI for Serious Illness Communication: Opportunities and Best Practices

## TL;DR

This paper explores using AI to improve communication during serious illness discussions by integrating multiple communication modes and addressing diverse patient needs.

## Contribution

The paper introduces Clinical SmartReporter™, the first AI tool for patient-facing serious illness communication, combining multimodal and multilingual support.

## Key findings

- Clinical SmartReporter™ can generate real-time transcribed reports using machine learning and natural language processing.
- User-centered design insights from patient, caregiver, and clinician interviews informed the development of the tool.
- The tool supports multilingual translation and speaker diarization in complex clinical settings.

## Abstract

Patients and families face complex communication challenges in the setting of serious illness. These difficulties are compounded for individuals who are non-English-speaking, have low health literacy, have visual/auditory impairments, or possess unique learning styles. We hypothesize that augmenting standard verbal communication modalities with a multimodal communication approach (integrating written, visual, and other modalities) can improve medical information-sharing. In the first use of artificial intelligence (AI) for patient-facing serious illness communication, we developed Clinical SmartReporter™, a program capable of generating real-time transcribed reports of family meetings for patients, their caregivers, and other clinicians. Clinical SmartReporter™ uses machine learning, natural language processing, and speaker diarization modules to accurately integrate medical terminology, translate between multiple languages, and distinguish between a large number of different speakers. In this presentation, we will describe our model of multimodal communication, present results of our user-centered design process based on patient, caregiver, and clinician interviews, and demonstrate the use of Clinical SmartReporter™ in the setting of a simulated family meeting (professionally filmed video). We will outline technology-based opportunities for the advancement of communication science in palliative care research, quality improvement, and medical education. We will also facilitate a discussion of practical and ethical challenges, as well as the need to establish field-wide best practices for AI implementation in geriatrics and palliative care settings. These include data security considerations, partnership with institutional offices for technology licensing and privacy/security, and protection of the integrity of intimate patient-clinician relationships, emphasizing the need for clinician advocacy and cross-institutional collaboration.

---
Source: https://tomesphere.com/paper/PMC12761412