Assessing the Human-Likeness of LLM-Driven Digital Twins in Simulating Health Care System Trust

TL;DR

This study evaluates how well LLM-driven digital twins can simulate human distrust in healthcare, revealing they capture broad trends but struggle with detailed subgroup differences, highlighting current limitations for policy use.

Contribution

It provides a systematic assessment of LLM-based digital twins' ability to simulate complex human attitudes like distrust in healthcare systems.

Findings

Digital twins produce more centralized responses with lower variance.

They replicate major demographic patterns such as age and gender.

They show limited sensitivity to minor differences like education levels.

Abstract

Serving as an emerging and powerful tool, Large Language Model (LLM)-driven Human Digital Twins are showing great potential in healthcare system research. However, its actual simulation ability for complex human psychological traits, such as distrust in the healthcare system, remains unclear. This research gap particularly impacts health professionals' trust and usage of LLM-based Artificial Intelligence (AI) systems in assisting their routine work. In this study, based on the Twin-2K-500 dataset, we systematically evaluated the simulation results of the LLM-driven human digital twin using the Health Care System Distrust Scale (HCSDS) with an established human-subject sample, analyzing item-level distributions, summary statistics, and demographic subgroup patterns. Results showed that the simulated responses by the digital twin were significantly more centralized with lower variance and had fewer selections of extreme options (all p<0.001). While the digital twin broadly reproduces human results in major demographic patterns, such as age and gender, it exhibits relatively low sensitivity in capturing minor differences in education levels. The LLM-based digital twin simulation has the potential to simulate population trends, but it also presents challenges in making detailed, specific distinctions in subgroups of human beings. This study suggests that the current LLM-driven Digital Twins have limitations in modeling complex human attitudes, which require careful calibration and validation before applying them in inferential analyses or policy simulations in health systems engineering. Future studies are necessary to examine the emotional reasoning mechanism of LLMs before their use, particularly for studies that involve simulations sensitive to social topics, such as human-automation trust.