Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study

TL;DR

This study evaluates ChatGPT's effectiveness in extracting pharmacovigilance events from medical texts, highlighting its moderate performance, challenges with data augmentation, and potential strategies for improvement.

Contribution

The paper provides an empirical assessment of ChatGPT's capabilities in pharmacovigilance event extraction and explores data augmentation techniques with filtering strategies.

Findings

ChatGPT shows reasonable performance with proper prompts.

Synthesized data may decrease fine-tuning performance due to noise.

Filtering strategies can stabilize performance but do not guarantee improvement.

Abstract

With the advent of large language models (LLMs), there has been growing interest in exploring their potential for medical applications. This research aims to investigate the ability of LLMs, specifically ChatGPT, in the context of pharmacovigilance event extraction, of which the main goal is to identify and extract adverse events or potential therapeutic events from textual medical sources. We conduct extensive experiments to assess the performance of ChatGPT in the pharmacovigilance event extraction task, employing various prompts and demonstration selection strategies. The findings demonstrate that while ChatGPT demonstrates reasonable performance with appropriate demonstration selection strategies, it still falls short compared to fully fine-tuned small models. Additionally, we explore the potential of leveraging ChatGPT for data augmentation. However, our investigation reveals that the inclusion of synthesized data into fine-tuning may lead to a decrease in performance, possibly attributed to noise in the ChatGPT-generated labels. To mitigate this, we explore different filtering strategies and find that, with the proper approach, more stable performance can be achieved, although constant improvement remains elusive.