Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability

TL;DR

This paper introduces RLFR, a reinforcement learning framework that uses interpretability-derived features as scalable rewards to reduce hallucinations in language models while maintaining their performance.

Contribution

It presents a novel RL pipeline that leverages feature-based rewards for open-ended tasks, enabling scalable supervision and improved factual accuracy.

Findings

58% reduction in hallucinations compared to original model

Maintains performance on standard benchmarks

Operates efficiently with scalable test-time compute

Abstract

Language models trained on large-scale datasets have been shown to learn features that encode abstract concepts such as factuality or intent. Such features are traditionally used for test-time monitoring or steering. We present an alternative affordance: features as scalable supervision for open-ended tasks. We consider the case of hallucination-reduction as a desirable, yet open-ended behavior and design a reinforcement learning (RL) pipeline, titled RLFR (Reinforcement Learning from Feature Rewards), that uses features as reward functions. Grounded in a novel probing framework that identifies candidate hallucinated claims, our pipeline teaches a model to intervene and correct its completions when it is uncertain of their factuality. Furthermore, the pipeline enables scalable test-time compute, guided once more by our reward features. This end-to-end process operationalized on Gemma-3-12B-IT results in a policy that is 58% less likely to hallucinate compared to the original model (when run in tandem with our probing harness), while preserving performance on standard benchmarks. Taken together, by grounding supervision in the language of features, this paper introduces a novel paradigm in the use of interpretability for learning open-ended tasks.