BERT based Transformers lead the way in Extraction of Health Information from Social Media

TL;DR

This paper presents BERT-based transformer models fine-tuned for extracting health-related information from social media, achieving top performance in shared tasks on adverse drug effects and COVID-19 symptoms.

Contribution

The study demonstrates the effectiveness of domain-specific BERT models like RoBERTa and BERTweet in health information extraction from social media, with state-of-the-art results.

Findings

First in subtask-1(a) with 61% F1-score

Improved F1 by 8% over average submissions in subtask-1(b)

94% F1-score on COVID-19 symptom classification

Abstract

This paper describes our submissions for the Social Media Mining for Health (SMM4H)2021 shared tasks. We participated in 2 tasks:(1) Classification, extraction and normalization of adverse drug effect (ADE) mentions in English tweets (Task-1) and (2) Classification of COVID-19 tweets containing symptoms(Task-6). Our approach for the first task uses the language representation model RoBERTa with a binary classification head. For the second task, we use BERTweet, based on RoBERTa. Fine-tuning is performed on the pre-trained models for both tasks. The models are placed on top of a custom domain-specific processing pipeline. Our system ranked first among all the submissions for subtask-1(a) with an F1-score of 61%. For subtask-1(b), our system obtained an F1-score of 50% with improvements up to +8% F1 over the score averaged across all submissions. The BERTweet model achieved an F1 score of 94% on SMM4H 2021 Task-6.