SAFER: Probing Safety in Reward Models with Sparse Autoencoder

TL;DR

SAFER introduces a novel method using sparse autoencoders to interpret, audit, and improve safety alignment in reward models for large language models, enabling targeted safety modifications without affecting overall performance.

Contribution

The paper presents SAFER, a framework that uses sparse autoencoders to uncover interpretable safety-related features in reward models, facilitating targeted safety improvements.

Findings

SAFER can precisely degrade safety alignment with minimal data.

SAFER can enhance safety alignment without harming general performance.

The approach enables interpretability and targeted safety interventions in reward models.

Abstract

Reinforcement learning from human feedback (RLHF) is a key paradigm for aligning large language models (LLMs) with human values, yet the reward models at its core remain largely opaque. In this work, we present Sparse Autoencoder For Enhanced Reward model (\textbf{SAFER}), a novel framework for interpreting and improving reward models through mechanistic analysis. Leveraging Sparse Autoencoders (SAEs), we uncover human-interpretable features in reward model activations, enabling insight into safety-relevant decision-making. We apply SAFER to safety-oriented preference datasets and quantify the salience of individual features by activation differences between chosen and rejected responses. Using these feature-level signals, we design targeted data poisoning and denoising strategies. Experiments show that SAFER can precisely degrade or enhance safety alignment with minimal data modification, without sacrificing general chat performance. Our approach contributes to interpreting, auditing and refining reward models in high-stakes LLM alignment tasks. Our codes are available at https://github.com/xzy-101/SAFER-code. \textit{This paper discusses topics related to reward model safety and may include discussions or examples that highlight potential risks or unsafe outcomes.}