Benchmarking the Safety of Large Language Models for Robotic Health Attendant Control

TL;DR

This study evaluates the safety of 72 large language models in robotic health attendant control using a new dataset of harmful instructions, revealing significant safety concerns and the limited effectiveness of current mitigation strategies.

Contribution

It introduces a comprehensive dataset for safety evaluation and provides empirical insights into the safety performance of various LLMs in medical robotic contexts.

Findings

Over half of the models exceeded 50% violation rate.

Proprietary models were significantly safer than open-weight models.

Prompt-based defenses only modestly reduced violation rates.

Abstract

Large language models (LLMs) are increasingly considered for deployment as the control component of robotic health attendants, yet their safety in this context remains poorly characterized. We introduce a dataset of 270 harmful instructions spanning nine prohibited behavior categories grounded in the American Medical Association Principles of Medical Ethics, and use it to evaluate 72 LLMs in a simulation environment based on the Robotic Health Attendant framework. The mean violation rate across all models was 54.4\%, with more than half exceeding 50\%, and violation rates varied substantially across behavior categories, with superficially plausible instructions such as device manipulation and emergency delay proving harder to refuse than overtly destructive ones. Model size and release date were the primary determinants of safety performance among open-weight models, and proprietary models were substantially safer than open-weight counterparts (median 23.7\% versus 72.8\%). Medical domain fine-tuning conferred no significant overall safety benefit, and a prompt-based defense strategy produced only a modest reduction in violation rates among the least safe models, leaving absolute violation rates at levels that would preclude safe clinical deployment. These findings demonstrate that safety evaluation must be treated as a first-class criterion in the development and deployment of LLMs for robotic health attendants.