Validation and Transparency in AI systems for pharmacovigilance: a case study applied to the medical literature monitoring of adverse events

TL;DR

This paper presents a validated AI system for medical literature monitoring of adverse events, emphasizing transparency and regulatory compliance to enhance trustworthiness and reduce screening effort in pharmacovigilance.

Contribution

It demonstrates how to operationalize existing guidance for validated AI in pharmacovigilance, integrating transparency measures and providing experimental evidence of efficiency and high recall.

Findings

System filters 55% of irrelevant articles while maintaining 0.99 recall

Uses public disclosures as risk control measures

Effective tuning of recall to match risk profiles

Abstract

Recent advances in artificial intelligence applied to biomedical text are opening exciting opportunities for improving pharmacovigilance activities currently burdened by the ever growing volumes of real world data. To fully realize these opportunities, existing regulatory guidance and industry best practices should be taken into consideration in order to increase the overall trustworthiness of the system and enable broader adoption. In this paper we present a case study on how to operationalize existing guidance for validated AI systems in pharmacovigilance focusing on the specific task of medical literature monitoring (MLM) of adverse events from the scientific literature. We describe an AI system designed with the goal of reducing effort in MLM activities built in close collaboration with subject matter experts and considering guidance for validated systems in pharmacovigilance and AI transparency. In particular we make use of public disclosures as a useful risk control measure to mitigate system misuse and earn user trust. In addition we present experimental results showing the system can significantly remove screening effort while maintaining high levels of recall (filtering 55% of irrelevant articles on average, for a target recall of 0.99 on suspected adverse articles) and provide a robust method for tuning the desired recall to suit a particular risk profile.