An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMs

TL;DR

This paper analyzes how jailbreak prompts can cause LLMs to generate harmful health misinformation, compares it to social media content, and explores detection methods to mitigate such risks.

Contribution

It provides a detailed analysis of 109 jailbreak attacks on LLMs, compares generated misinformation to social media, and evaluates detection strategies for health-related content.

Findings

Jailbreak attacks can produce health misinformation similar to social media.

Standard ML detection methods can identify some jailbreak-generated misinformation.

Careful LLM design can help mitigate health misinformation risks.

Abstract

Large Language Models (LLMs) are a double-edged sword capable of generating harmful misinformation -- inadvertently, or when prompted by "jailbreak" attacks that attempt to produce malicious outputs. LLMs could, with additional research, be used to detect and prevent the spread of misinformation. In this paper, we investigate the efficacy and characteristics of LLM-produced jailbreak attacks that cause other models to produce harmful medical misinformation. We also study how misinformation generated by jailbroken LLMs compares to typical misinformation found on social media, and how effectively it can be detected using standard machine learning approaches. Specifically, we closely examine 109 distinct attacks against three target LLMs and compare the attack prompts to in-the-wild health-related LLM queries. We also examine the resulting jailbreak responses, comparing the generated misinformation to health-related misinformation on Reddit. Our findings add more evidence that LLMs can be effectively used to detect misinformation from both other LLMs and from people, and support a body of work suggesting that with careful design, LLMs can contribute to a healthier overall information ecosystem.