Labels Generated by Large Language Models Help Measure People's Empathy in Vitro

TL;DR

This paper investigates how large language model-generated labels can enhance supervised training for empathy prediction tasks, leading to improved accuracy and more equitable models.

Contribution

It introduces methods for using LLM-generated labels to correct noise and augment data in empathy computing, achieving state-of-the-art results.

Findings

LLM-generated labels improve empathy prediction accuracy.

Replacing noisy labels with LLM labels yields significant gains.

The RoBERTa model trained with these labels achieves a Pearson correlation of 0.648.

Abstract

Large language models (LLMs) have revolutionised many fields, with LLM-as-a-service (LLMSaaS) offering accessible, general-purpose solutions without costly task-specific training. In contrast to the widely studied prompt engineering for directly solving tasks (in vivo), this paper explores LLMs' potential for in-vitro applications: using LLM-generated labels to improve supervised training of mainstream models. We examine two strategies - (1) noisy label correction and (2) training data augmentation - in empathy computing, an emerging task to predict psychology-based questionnaire outcomes from inputs like textual narratives. Crowdsourced datasets in this domain often suffer from noisy labels that misrepresent underlying empathy. We show that replacing or supplementing these crowdsourced labels with LLM-generated labels, developed using psychology-based scale-aware prompts, achieves statistically significant accuracy improvements. Notably, the RoBERTa pre-trained language model (PLM) trained with noise-reduced labels yields a state-of-the-art Pearson correlation coefficient of 0.648 on the public NewsEmp benchmarks. This paper further analyses evaluation metric selection and demographic biases to help guide the future development of more equitable empathy computing models. Code and LLM-generated labels are available at https://github.com/hasan-rakibul/LLMPathy.