Tiny Reward Models

TL;DR

TinyRM introduces small, efficient bidirectional language models that match large models in reward and safety tasks, offering a resource-efficient alternative for reinforcement learning from human feedback.

Contribution

The paper presents TinyRM, a family of compact bidirectional models that achieve competitive performance on reward modeling tasks using novel tuning strategies and lightweight finetuning.

Findings

TinyRM models perform comparably to much larger models on reasoning tasks.

Domain-specific tuning enhances small model performance.

Lightweight finetuning methods are particularly effective for reasoning.

Abstract

Large decoder-based language models have become the dominant architecture for reward modeling in reinforcement learning from human feedback (RLHF). However, as reward models are increasingly deployed in test-time strategies, their inference costs become a growing concern. We present TinyRM, a family of small, bidirectional masked language models (MLMs) with as few as 400 million parameters, that rival the capabilities of models over 175 times larger on reasoning and safety preference modeling tasks. TinyRM combines FLAN-style prompting, Directional Low-Rank Adaptation (DoRA), and layer freezing to achieve strong performance on RewardBench, despite using significantly fewer resources. Our experiments suggest that small models benefit from domain-specific tuning strategies, particularly in reasoning, where lightweight finetuning methods are especially effective. While challenges remain in building generalist models and conversational preference modeling, our preliminary results highlight the promise of lightweight bidirectional architectures as efficient, scalable alternatives for preference modeling.