SparseRM: A Lightweight Preference Modeling with Sparse Autoencoder

TL;DR

SparseRM introduces a lightweight, interpretable reward model using sparse autoencoders to efficiently capture preference features in LLM representations, reducing training costs while maintaining high performance.

Contribution

The paper presents SparseRM, a novel approach that leverages sparse autoencoders to build efficient, interpretable reward models with minimal parameters for preference modeling in LLMs.

Findings

SparseRM outperforms most mainstream reward models.

Uses less than 1% of trainable parameters.

Easily integrates into downstream alignment pipelines.

Abstract

Reward models (RMs) are a core component in the post-training of large language models (LLMs), serving as proxies for human preference evaluation and guiding model alignment. However, training reliable RMs under limited resources remains challenging due to the reliance on large-scale preference annotations and the high cost of fine-tuning LLMs. To address this, we propose SparseRM, which leverages Sparse Autoencoder (SAE) to extract preference-relevant information encoded in model representations, enabling the construction of a lightweight and interpretable reward model. SparseRM first employs SAE to decompose LLM representations into interpretable directions that capture preference-relevant features. The representations are then projected onto these directions to compute alignment scores, which quantify the strength of each preference feature in the representations. A simple reward head aggregates these scores to predict preference scores. Experiments on three preference modeling tasks show that SparseRM achieves superior performance over most mainstream RMs while using less than 1% of trainable parameters. Moreover, it integrates seamlessly into downstream alignment pipelines, highlighting its potential for efficient alignment.