View Distillation with Unlabeled Data for Extracting Adverse Drug Effects from User-Generated Data

TL;DR

This paper introduces a novel view distillation approach using unlabeled social media data and multi-layer transformers to improve the extraction of adverse drug reactions, significantly outperforming existing models.

Contribution

It proposes a new multi-view distillation method leveraging unlabeled data and transformer embeddings for ADR detection, enhancing performance over pretrained models.

Findings

Model outperforms domain-specific pretrained transformers on ADR data

Utilizes unlabeled social media data effectively for ADR extraction

Significant improvement in ADR detection accuracy

Abstract

We present an algorithm based on multi-layer transformers for identifying Adverse Drug Reactions (ADR) in social media data. Our model relies on the properties of the problem and the characteristics of contextual word embeddings to extract two views from documents. Then a classifier is trained on each view to label a set of unlabeled documents to be used as an initializer for a new classifier in the other view. Finally, the initialized classifier in each view is further trained using the initial training examples. We evaluated our model in the largest publicly available ADR dataset. The experiments testify that our model significantly outperforms the transformer-based models pretrained on domain-specific data.