Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMs

TL;DR

This paper assesses safety risks in role-play fine-tuning of LLMs, revealing safety performance declines and proposing SaRFT to balance role capabilities with safety, validated across multiple models.

Contribution

It provides the first comprehensive evaluation of role-play fine-tuning safety risks and introduces SaRFT, a novel method to mitigate these risks while maintaining role performance.

Findings

Role-play fine-tuning reduces safety performance.

Safety risks vary with character traits.

SaRFT outperforms existing baselines in safety and role ability.

Abstract

Role-playing enables large language models (LLMs) to engage users in immersive and personalized interactions, but it also introduces significant safety risks. Existing role-play fine-tuning techniques improve role adaptability but may degrade safety performance, particularly for villainous characters. In this work, we conduct the first comprehensive assessment of role-play fine-tuning risks by training 95 role-specific LLMs using RoleBench. Our experiments reveal that role-play fine-tuning leads to a noticeable decline in safety performance, with safety risks varying based on character traits. To tackle this challenge, we propose Safety-Aware Role-Play Fine-Tuning (SaRFT), a novel method designed to balance role-playing capabilities and safety. Extensive experiments on LLaMA-3-8B-Instruct, Gemma-2-9B-it, and Qwen2.5-7B-Instruct demonstrate that SaRFT consistently outperforms state-of-the-art baselines under both LoRA and full-parameter fine-tuning settings. Our findings highlight the necessity of role-adaptive safety measures and provide insights into mitigating role-specific safety risks in role-playing LLMs.