Prompting Large Language Models to Detect Dementia Family Caregivers

TL;DR

This paper presents a system using large language models with prompting techniques to accurately identify tweets from caregivers of dementia patients, achieving high classification performance.

Contribution

It introduces a prompting-based approach with LLMs for detecting caregiver tweets, demonstrating effectiveness in a shared task setting.

Findings

Zero-shot prompting on fine-tuned models achieved best results.

System achieved macro F1-score of 0.95 on validation and test sets.

Prompting methods significantly impact detection accuracy.

Abstract

Social media, such as Twitter, provides opportunities for caregivers of dementia patients to share their experiences and seek support for a variety of reasons. Availability of this information online also paves the way for the development of internet-based interventions in their support. However, for this purpose, tweets written by caregivers of dementia patients must first be identified. This paper demonstrates our system for the SMM4H 2025 shared task 3, which focuses on detecting tweets posted by individuals who have a family member with dementia. The task is outlined as a binary classification problem, differentiating between tweets that mention dementia in the context of a family member and those that do not. Our solution to this problem explores large language models (LLMs) with various prompting methods. Our results show that a simple zero-shot prompt on a fine-tuned model yielded the best results. Our final system achieved a macro F1-score of 0.95 on the validation set and the test set. Our full code is available on GitHub.