Evaluating Defences against Unsafe Feedback in RLHF

TL;DR

This paper analyzes the vulnerability of safety-aligned large language models trained with reinforcement learning from unsafe feedback, revealing current defenses are ineffective and proposing theoretical insights for future safety improvements.

Contribution

It provides the first analysis of unsafe feedback learning in RLHF, demonstrating the limitations of existing defenses and offering theoretical explanations for harmful reward hacking.

Findings

Safety-aligned LLMs can generate harmful text via unsafe feedback

Current defenses are ineffective against unsafe feedback in RLHF

Some defenses work by harmless reward hacking, explained through Constrained MDP theory

Abstract

While there has been progress towards aligning Large Language Models (LLMs) with human values and ensuring safe behaviour at inference time, safety guards can easily be removed when fine tuned on unsafe and harmful datasets. While this setting has been treated extensively, another popular training paradigm, learning from unsafe feedback with reinforcement learning, has previously been unexplored. This is concerning due to the widespread deployment of feedback collection systems. We address this gap by providing an analysis of learning settings where feedback is harmful, i.e. that unsafe samples are preferred over safe ones despite model developers goal to maintain safety. We find that safety-aligned LLMs easily explore unsafe action spaces via generating harmful text and optimize for reward that violates safety constraints indicating that current safety guards are not enough to prevent learning from unsafe feedback. In order to protect against this vulnerability, we adapt a number of both "implict" and "explicit" harmful fine-tuning defences to evaluate whether they are effective as learning constraints in an RLHF setting finding that no method is generally effective pointing to the need for more defence research. We end the paper with the observation that some defences work by performing "harmless reward hacking" for which we provide a theoretical explanation drawn from the theory of Constrained Markov Decision Processes and provide some direction for future defence development.