Red Teaming Large Language Models for Healthcare

TL;DR

This paper discusses a workshop where experts tested large language models in healthcare for vulnerabilities that could lead to clinical harm, revealing critical weaknesses and assessing their prevalence across models.

Contribution

It introduces a collaborative red-teaming process combining clinical and technical expertise to identify and categorize vulnerabilities in healthcare LLMs.

Findings

Identified multiple vulnerabilities that could cause clinical harm

Categorized types of vulnerabilities found in LLMs

Assessed vulnerability prevalence across different LLMs

Abstract

We present the design process and findings of the pre-conference workshop at the Machine Learning for Healthcare Conference (2024) entitled Red Teaming Large Language Models for Healthcare, which took place on August 15, 2024. Conference participants, comprising a mix of computational and clinical expertise, attempted to discover vulnerabilities -- realistic clinical prompts for which a large language model (LLM) outputs a response that could cause clinical harm. Red-teaming with clinicians enables the identification of LLM vulnerabilities that may not be recognised by LLM developers lacking clinical expertise. We report the vulnerabilities found, categorise them, and present the results of a replication study assessing the vulnerabilities across all LLMs provided.