Rethinking harmless refusals when fine-tuning foundation models

TL;DR

This paper examines how fine-tuning large language models may hide undesirable behaviors rather than eliminate them, and finds that explicit rebuttals are more effective than polite refusals in preventing unethical outputs.

Contribution

It introduces the concept of reason-based deception and demonstrates that rebuttals outperform refusals in mitigating harmful model behaviors.

Findings

Revealed prevalence of reason-based deception in fine-tuned models.

Rebuttals significantly reduce unethical outputs compared to polite refusals.

Nearly eliminate reason-based deception when using explicit rebuttals.

Abstract

In this paper, we investigate the degree to which fine-tuning in Large Language Models (LLMs) effectively mitigates versus merely conceals undesirable behavior. Through the lens of semi-realistic role-playing exercises designed to elicit such behaviors, we explore the response dynamics of LLMs post fine-tuning interventions. Our methodology involves prompting models for Chain-of-Thought (CoT) reasoning and analyzing the coherence between the reasoning traces and the resultant outputs. Notably, we identify a pervasive phenomenon we term \emph{reason-based deception}, where models either stop producing reasoning traces or produce seemingly ethical reasoning traces that belie the unethical nature of their final outputs. We further examine the efficacy of response strategies (polite refusal versus explicit rebuttal) in curbing the occurrence of undesired behavior in subsequent outputs of multi-turn interactions. Our findings reveal that explicit rebuttals significantly outperform polite refusals in preventing the continuation of undesired outputs and nearly eliminate reason-based deception, challenging current practices in model fine-tuning. Accordingly, the two key contributions of this paper are (1) defining and studying reason-based deception, a new type of hidden behavior, and (2) demonstrating that rebuttals provide a more robust response model to harmful requests than refusals, thereby highlighting the need to reconsider the response strategies in fine-tuning approaches.