Medical Foundation Models are Susceptible to Targeted Misinformation Attacks

TL;DR

This paper reveals that large language models used in medicine are vulnerable to targeted weight manipulations, allowing malicious injection of false biomedical facts without affecting overall performance, raising security concerns.

Contribution

It demonstrates a novel vulnerability in medical LLMs where minimal weight changes can inject false information, highlighting the need for improved safeguards.

Findings

Targeted weight manipulation can inject false biomedical facts.

Model performance on other tasks remains unaffected.

Raises security concerns for medical AI deployment.

Abstract

Large language models (LLMs) have broad medical knowledge and can reason about medical information across many domains, holding promising potential for diverse medical applications in the near future. In this study, we demonstrate a concerning vulnerability of LLMs in medicine. Through targeted manipulation of just 1.1% of the model's weights, we can deliberately inject an incorrect biomedical fact. The erroneous information is then propagated in the model's output, whilst its performance on other biomedical tasks remains intact. We validate our findings in a set of 1,038 incorrect biomedical facts. This peculiar susceptibility raises serious security and trustworthiness concerns for the application of LLMs in healthcare settings. It accentuates the need for robust protective measures, thorough verification mechanisms, and stringent management of access to these models, ensuring their reliable and safe use in medical practice.