Evaluating ChatGPT in Portal Messaging for Dementia Care: Identifying Care Partners and Exploring Acceptance

TL;DR

ChatGPT can help identify care partners in dementia care messaging and is accepted if clinicians review and disclose its use.

Contribution

Demonstrates ChatGPT's effectiveness in identifying care partners and outlines conditions for its acceptance in dementia care.

Findings

ChatGPT accurately detected nonpatient authors of portal messages with an AUC of 0.92.

Participants accepted AI use if messages were reviewed by clinicians and AI involvement was transparent.

Care partners and persons with dementia showed positive acceptance of AI in portal messaging under specific conditions.

Abstract

ChatGPT, a popular large language model, is integrated in Epic’s patient portal and can create draft responses that clinicians may refine and use when answering patient messages. Application of GPT in messaging related to dementia care requires a judicious approach: persons with dementia and their care partners have distinct differences from the general population (e.g., higher care partner involvement).

We conducted a two-part study. We first used a de-identified set of patient portal messages sent from persons with dementia and their care partners to assess ChatGPT’s capacity to identify the sender of the message. We used a set of 1,973 messages where the sender (patient or nonpatient) was manually coded by two independent reviewers as the gold standard. We then conducted interviews with persons with dementia and their care partners to explore their perceptions of the use of artificial intelligence in patient portal messaging. We used team-based qualitative rapid analysis encapsulate major findings across the summaries of participants’ interviews.

We found that ChatGPT had acceptable performance in detecting a nonpatient author of a patient portal message (0.92 area under the receiver operating curve). In qualitative interviews (n = 13), we found that care partners and persons with dementia accepted the use of artificial intelligence to draft patient portal messages provided two key conditions were met: clinician review before sending and clear transparency regarding artificial intelligence’s involvement.

Use of ChatGPT in patient portal messaging is acceptable to care partners and persons with dementia, and may support care partner identification.