How Safe is Your Safety Metric? Automatic Concatenation Tests for Metric Reliability

TL;DR

This paper reveals that many safety metrics for language models can fail when prompt-response pairs are concatenated, highlighting the need for better evaluation methods.

Contribution

The authors introduce automatic concatenation tests to evaluate the reliability of safety metrics, exposing their inconsistencies and sensitivities.

Findings

Safety metrics can reverse their judgments upon concatenation.

Metrics are highly sensitive to input order.

Concatenation tests reveal significant evaluation inconsistencies.

Abstract

Consider a scenario where a harmfulness evaluation metric intended to filter unsafe responses from a Large Language Model. When applied to individual harmful prompt-response pairs, it correctly flags them as unsafe by assigning a high-risk score. Yet, if those same pairs are concatenated, the metrics decision unexpectedly reverses - labelling the combined content as safe with a low score, allowing the harmful text to bypass the filter. We found that multiple safety metrics, including advanced metrics such as GPT-based judges, exhibit this non-safe behaviour. Moreover, they show a strong sensitivity to input order: responses are often classified as safe if safe content appears first, regardless of any harmful content that follows, and vice versa. These findings underscore the importance of evaluating the safety of safety metrics, that is, the reliability of their output scores. To address this, we developed general, automatic, concatenation-based tests to assess key properties of these metrics. When applied in a model safety scenario, the tests revealed significant inconsistencies in harmfulness evaluations.