ClaHF: A Human Feedback-inspired Reinforcement Learning Framework for Improving Classification Tasks

TL;DR

ClaHF introduces a reinforcement learning framework inspired by human feedback to enhance text classification models by integrating preference modeling and reward optimization without extra human labels.

Contribution

It presents a novel method that converts label supervision into preference signals, improving classification accuracy and calibration across multiple tasks and language models.

Findings

Consistently improves classification performance across eight tasks.

Enhances confidence calibration in diverse language models.

Operates without requiring additional human annotations.

Abstract

Text classification models are typically trained via supervised fine-tuning (SFT). However, SFT essentially performs behavior cloning from instance-wise labels and thus fails to adequately capture relative preference relations among samples, which limits the model's ability to shape decision boundaries and calibrate predictive confidence. In this paper, we propose ClaHF, a human feedback-inspired reinforcement learning (RL) framework for text classification that integrates preference modeling and RL optimization into the classification pipeline without requiring additional human annotations. Unlike prior work that relies solely on instance-wise supervision, ClaHF constructs multiple candidate predictions together with their relative ranking relations, and jointly models the Top-1 preference and the ordering among non-optimal candidates within a reward model (RM). This design converts conventional label supervision into preference signals that are directly applicable to policy optimization. We conduct systematic evaluations on eight classification tasks spanning three categories of scenarios. Results demonstrate that ClaHF consistently improves both classification performance and confidence calibration across diverse language models (LMs). The data and code are available at https://anonymous.4open.science/r/ClaHF.