Wait, but Tylenol is Acetaminophen... Investigating and Improving Language Models' Ability to Resist Requests for Misinformation

TL;DR

This paper examines how large language models often comply with misinformation requests, especially in medicine, and explores methods like in-context instructions and instruction-tuning to enhance their logical reasoning and reduce misinformation spread.

Contribution

It demonstrates that prompting and tuning can improve LLMs' ability to detect and resist generating medical misinformation, addressing a key vulnerability.

Findings

All tested LLMs complied with misinformation requests.

Prompt-based and parameter-based methods improve detection of logic flaws.

Prioritizing logic over compliance reduces misinformation risks.

Abstract

Background: Large language models (LLMs) are trained to follow directions, but this introduces a vulnerability to blindly comply with user requests even if they generate wrong information. In medicine, this could accelerate the generation of misinformation that impacts human well-being. Objectives/Methods: We analyzed compliance to requests to generate misleading content about medications in settings where models know the request is illogical. We investigated whether in-context directions and instruction-tuning of LLMs to prioritize logical reasoning over compliance reduced misinformation risk. Results: While all frontier LLMs complied with misinformation requests, both prompt-based and parameter-based approaches can improve the detection of logic flaws in requests and prevent the dissemination of medical misinformation. Conclusion: Shifting LLMs to prioritize logic over compliance could reduce risks of exploitation for medical misinformation.