The Promise of Large Language Models in Digital Health: Evidence from Sentiment Analysis in Online Health Communities

TL;DR

This paper demonstrates that large language models, enhanced with expert-encoded prompts, can perform high-quality sentiment analysis on online health community posts, addressing data scarcity and privacy issues in digital health analytics.

Contribution

It introduces a structured codebook for expert knowledge encoding in LLM prompts, enabling scalable, expert-level sentiment analysis without extensive model retraining.

Findings

LLMs outperform traditional models in sentiment analysis accuracy

Expert-level agreement achieved by LLMs is comparable to inter-expert agreement

In-context learning with structured prompts enhances LLM performance in health data analysis

Abstract

Digital health analytics face critical challenges nowadays. The sophisticated analysis of patient-generated health content, which contains complex emotional and medical contexts, requires scarce domain expertise, while traditional ML approaches are constrained by data shortage and privacy limitations in healthcare settings. Online Health Communities (OHCs) exemplify these challenges with mixed-sentiment posts, clinical terminology, and implicit emotional expressions that demand specialised knowledge for accurate Sentiment Analysis (SA). To address these challenges, this study explores how Large Language Models (LLMs) can integrate expert knowledge through in-context learning for SA, providing a scalable solution for sophisticated health data analysis. Specifically, we develop a structured codebook that systematically encodes expert interpretation guidelines, enabling LLMs to apply domain-specific knowledge through targeted prompting rather than extensive training. Six GPT models validated alongside DeepSeek and LLaMA 3.1 are compared with pre-trained language models (BioBERT variants) and lexicon-based methods, using 400 expert-annotated posts from two OHCs. LLMs achieve superior performance while demonstrating expert-level agreement. This high agreement, with no statistically significant difference from inter-expert agreement levels, suggests knowledge integration beyond surface-level pattern recognition. The consistent performance across diverse LLM models, supported by in-context learning, offers a promising solution for digital health analytics. This approach addresses the critical challenge of expert knowledge shortage in digital health research, enabling real-time, expert-quality analysis for patient monitoring, intervention assessment, and evidence-based health strategies.