Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI models

TL;DR

This paper proposes a method to identify potentially harmful medical product recommendations from generative AI models, addressing safety concerns in medical applications by demonstrating its effectiveness on a recent multimodal large language model.

Contribution

It introduces a novel approach to detect unsafe medical suggestions in generative AI, enhancing safety and regulatory compliance in medical AI deployment.

Findings

Successfully identified potentially harmful recommendations

Demonstrated effectiveness on a multimodal large language model

Provides a framework for safer medical AI use

Abstract

Generative AI (GenAI) models have demonstrated remarkable capabilities in a wide variety of medical tasks. However, as these models are trained using generalist datasets with very limited human oversight, they can learn uses of medical products that have not been adequately evaluated for safety and efficacy, nor approved by regulatory agencies. Given the scale at which GenAI may reach users, unvetted recommendations pose a public health risk. In this work, we propose an approach to identify potentially harmful product recommendations, and demonstrate it using a recent multimodal large language model.