Communication Styles and Reader Preferences of LLM and Human Experts in Explaining Health Information

TL;DR

This study compares the communication styles of LLMs and human experts in health misinformation explanations, revealing that despite lower traditional quality scores, LLMs are preferred by readers for clarity and engagement.

Contribution

It provides a detailed analysis of style differences between LLMs and humans in health communication and evaluates reader perceptions, highlighting LLMs' potential for effective health information delivery.

Findings

LLMs scored lower on persuasive strategies and social value alignment.

Readers preferred LLM explanations for clarity and completeness.

LLMs' structured presentation enhances reader engagement despite lower traditional scores.

Abstract

With the wide adoption of large language models (LLMs) in information assistance, it is essential to examine their alignment with human communication styles and values. We situate this study within the context of fact-checking health information, given the critical challenge of rectifying conceptions and building trust. Recent studies have explored the potential of LLM for health communication, but style differences between LLMs and human experts and associated reader perceptions remain under-explored. In this light, our study evaluates the communication styles of LLMs, focusing on how their explanations differ from those of humans in three core components of health communication: information, sender, and receiver. We compiled a dataset of 1498 health misinformation explanations from authoritative fact-checking organizations and generated LLM responses to inaccurate health information. Drawing from health communication theory, we evaluate communication styles across three key dimensions of information linguistic features, sender persuasive strategies, and receiver value alignments. We further assessed human perceptions through a blinded evaluation with 99 participants. Our findings reveal that LLM-generated articles showed significantly lower scores in persuasive strategies, certainty expressions, and alignment with social values and moral foundations. However, human evaluation demonstrated a strong preference for LLM content, with over 60% responses favoring LLM articles for clarity, completeness, and persuasiveness. Our results suggest that LLMs' structured approach to presenting information may be more effective at engaging readers despite scoring lower on traditional measures of quality in fact-checking and health communication.