Automated Data Enrichment using Confidence-Aware Fine-Grained Debate among Open-Source LLMs for Mental Health and Online Safety

TL;DR

This paper introduces a Confidence-Aware Fine-Grained Debate framework that enhances automated multi-label data annotation for mental health and online safety, leveraging collaborative debate among LLMs to improve downstream NLP tasks.

Contribution

The paper presents a novel debate-based annotation framework and new datasets, improving multi-label data enrichment and downstream task performance in sensitive NLP applications.

Findings

CFD outperforms baseline annotation methods in robustness.

Enriched features via debate transcripts improve task accuracy by 9.9%.

New datasets support mental health and online safety research.

Abstract

Real-world indicators play an important role in many natural language processing (NLP) applications, such as life-event for mental health analysis and risky behaviour for online safety, yet labelling such information in training datasets is often costly and/or difficult due to their dynamic nature. Large language models (LLMs) show promising potential for automated annotation, yet multi-label prediction remains challenging. In this work, we propose a Confidence-Aware Fine-Grained Debate (CFD) framework that simulates collaborative annotation using fine-grained information to better support automated multi-label enrichment. We introduce two new expert-annotated resources: A mental health Reddit well-being dataset and an online safety Facebook sharenting risk dataset. Experiments show that CFD achieves the most robust enrichment performance compared to a range of baseline approaches. We further evaluate various training-free enrichment incorporation strategies and demonstrate that LLM-enriched indicators consistently improves our downstream tasks. Enriched features incorporated via debate transcripts yield the largest gains, outperforming the non-enriched baseline by 9.9\% on the online safety task.