MultiADE: A Multi-domain Benchmark for Adverse Drug Event Extraction

TL;DR

This paper introduces MultiADE, a comprehensive multi-domain benchmark for adverse drug event extraction, highlighting the challenges of domain generalization and the need for improved models across diverse text sources.

Contribution

It presents a new multi-domain benchmark dataset, CADECv2, and evaluates the generalization capabilities of ADE extraction models across various text types.

Findings

Model generalization remains limited across domains.

Intermediate transfer learning shows promise but needs better domain adaptation methods.

CADECv2 dataset and benchmark scripts are publicly available.

Abstract

Active adverse event surveillance monitors Adverse Drug Events (ADE) from different data sources, such as electronic health records, medical literature, social media and search engine logs. Over the years, many datasets have been created, and shared tasks have been organised to facilitate active adverse event surveillance. However, most - if not all - datasets or shared tasks focus on extracting ADEs from a particular type of text. Domain generalisation - the ability of a machine learning model to perform well on new, unseen domains (text types) - is under-explored. Given the rapid advancements in natural language processing, one unanswered question is how far we are from having a single ADE extraction model that is effective on various types of text, such as scientific literature and social media posts. We contribute to answering this question by building a multi-domain benchmark for adverse drug event extraction, which we named MultiADE. The new benchmark comprises several existing datasets sampled from different text types and our newly created dataset - CADECv2, which is an extension of CADEC, covering online posts regarding more diverse drugs than CADEC. Our new dataset is carefully annotated by human annotators following detailed annotation guidelines. Our benchmark results show that the generalisation of the trained models is far from perfect, making it infeasible to be deployed to process different types of text. In addition, although intermediate transfer learning is a promising approach to utilising existing resources, further investigation is needed on methods of domain adaptation, particularly cost-effective methods to select useful training instances. The newly created CADECv2 and the scripts for building the benchmark are publicly available at CSIRO's Data Portal.