PsychEthicsBench: Evaluating Large Language Models Against Australian Mental Health Ethics

TL;DR

This paper introduces PsychEthicsBench, a new benchmark based on Australian mental health ethics, to evaluate large language models' ethical responses beyond simple refusal metrics, revealing insights into their safety and clinical appropriateness.

Contribution

It presents the first principle-grounded benchmark for assessing LLMs' ethical knowledge in mental health, moving beyond refusal-based safety signals and highlighting the impact of domain-specific fine-tuning.

Findings

Refusal rates are poor indicators of ethical behavior.

Domain-specific fine-tuning can reduce ethical robustness.

Specialized models may underperform base models in ethical alignment.

Abstract

The increasing integration of large language models (LLMs) into mental health applications necessitates robust frameworks for evaluating professional safety alignment. Current evaluative approaches primarily rely on refusal-based safety signals, which offer limited insight into the nuanced behaviors required in clinical practice. In mental health, clinically inadequate refusals can be perceived as unempathetic and discourage help-seeking. To address this gap, we move beyond refusal-centric metrics and introduce \texttt{PsychEthicsBench}, the first principle-grounded benchmark based on Australian psychology and psychiatry guidelines, designed to evaluate LLMs' ethical knowledge and behavioral responses through multiple-choice and open-ended tasks with fine-grained ethicality annotations. Empirical results across 14 models reveal that refusal rates are poor indicators of ethical behavior, revealing a significant divergence between safety triggers and clinical appropriateness. Notably, we find that domain-specific fine-tuning can degrade ethical robustness, as several specialized models underperform their base backbones in ethical alignment. PsychEthicsBench provides a foundation for systematic, jurisdiction-aware evaluation of LLMs in mental health, encouraging more responsible development in this domain.