It's Difficult to be Neutral -- Human and LLM-based Sentiment Annotation of Patient Comments

TL;DR

This study compares human and large language model-based sentiment annotation of Norwegian patient comments, highlighting the strengths and limitations of LLMs as alternatives to human annotators in healthcare sentiment analysis.

Contribution

It provides an extensive evaluation of LLMs for sentiment annotation in healthcare data, demonstrating their potential and current limitations compared to human annotators.

Findings

LLMs perform well above baseline in binary sentiment detection

LLMs cannot yet match human annotators on full datasets

Zero-shot LLM performance is promising but not sufficient for full accuracy

Abstract

Sentiment analysis is an important tool for aggregating patient voices, in order to provide targeted improvements in healthcare services. A prerequisite for this is the availability of in-domain data annotated for sentiment. This article documents an effort to add sentiment annotations to free-text comments in patient surveys collected by the Norwegian Institute of Public Health (NIPH). However, annotation can be a time-consuming and resource-intensive process, particularly when it requires domain expertise. We therefore also evaluate a possible alternative to human annotation, using large language models (LLMs) as annotators. We perform an extensive evaluation of the approach for two openly available pretrained LLMs for Norwegian, experimenting with different configurations of prompts and in-context learning, comparing their performance to human annotators. We find that even for zero-shot runs, models perform well above the baseline for binary sentiment, but still cannot compete with human annotators on the full dataset.