Words to Describe What I'm Feeling: Exploring the Potential of AI Agents for High Subjectivity Decisions in Advance Care Planning

TL;DR

This study explores how AI agents can serve as personal proxies in high-stakes, subjective decisions like Advance Care Planning, demonstrating high agreement with user preferences and highlighting future ethical considerations.

Contribution

Introduces an AI prototype for ACP decision support, analyzing user interactions and proposing a new role for AI as personal advocates in sensitive decision-making.

Findings

86.7% agreement between users and AI agent

Participants see AI as a potential personal advocate

Highlights need for ethical oversight and user expectation management

Abstract

Loss of decisional capacity, coupled with the increasing absence of reliable human proxies, raises urgent questions about how individuals' values can be represented in Advance Care Planning (ACP). To probe this fraught design space of high-risk, high-subjectivity decision support, we built an experience prototype (\acpagent{}) and asked 15 participants in 4 workshops to train it to be their personal ACP proxy. We analysed their coping strategies and feature requests and mapped the results onto axes of agent autonomy and human control. Our findings show a surprising 86.7\% agreement with \acpagent{}, arguing for a potential new role of AI in ACP where agents act as personal advocates for individuals, building mutual intelligibility over time. We propose that the key areas of future risk that must be addressed are the moderation of users' expectations and designing accountability and oversight over agent deployment and cutoffs.