Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models

TL;DR

This paper introduces RLSC, a reinforcement learning method that uses a language model's own confidence as a reward signal, enabling effective fine-tuning with minimal labeled data for improved mathematical reasoning accuracy.

Contribution

The paper presents RLSC, a novel self-confidence based RL approach that eliminates the need for external rewards or labels in fine-tuning large language models for reasoning tasks.

Findings

RLSC improves accuracy on multiple math benchmarks significantly.

It requires only 16 samples per question and few training steps.

The method is simple, scalable, and does not rely on external reward models.

Abstract

Large language models (LLMs) excel at reasoning, yet post-training remains critical for aligning their behavior with task goals. Existing reinforcement learning (RL) methods often depend on costly human annotations or external reward models. We propose Reinforcement Learning via Self-Confidence (RLSC), which uses the model's own confidence as reward signals-eliminating the need for labels, preference models, or reward engineering. Applied to Qwen2.5-Math-7B with only 16 samples per question and 10 or 20 training steps, RLSC improves accuracy by +13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23. RLSC provides a simple, scalable post-training method for inference models, requiring only a small number of samples and unlabelled supervision.