Towards medical AI misalignment: a preliminary study

TL;DR

This paper explores how malicious role-playing prompts can bypass safeguards in large language models to produce potentially harmful medical advice, highlighting a critical vulnerability in AI safety.

Contribution

It provides an initial analysis of a specific vulnerability in LLMs related to medical safety, emphasizing the need for improved safeguards against role-playing jailbreak techniques.

Findings

Role-playing prompts can bypass safeguards in LLMs.

Malicious prompts can induce incorrect medical suggestions.

Vulnerability exists even without internal model knowledge.

Abstract

Despite their staggering capabilities as assistant tools, often exceeding human performances, Large Language Models (LLMs) are still prone to jailbreak attempts from malevolent users. Although red teaming practices have already identified and helped to address several such jailbreak techniques, one particular sturdy approach involving role-playing (which we named `Goofy Game') seems effective against most of the current LLMs safeguards. This can result in the provision of unsafe content, which, although not harmful per se, might lead to dangerous consequences if delivered in a setting such as the medical domain. In this preliminary and exploratory study, we provide an initial analysis of how, even without technical knowledge of the internal architecture and parameters of generative AI models, a malicious user could construct a role-playing prompt capable of coercing an LLM into producing incorrect (and potentially harmful) clinical suggestions. We aim to illustrate a specific vulnerability scenario, providing insights that can support future advancements in the field.