Reasoning with Reinforced Functional Token Tuning

TL;DR

This paper introduces Reinforced Functional Token Tuning (RFTT), a novel framework that enhances large language models' reasoning abilities by embedding learnable functional tokens and using reinforcement learning to improve reasoning pathways, achieving state-of-the-art results on math benchmarks.

Contribution

The paper presents a new reinforcement learning-based fine-tuning method that incorporates functional tokens directly into LLMs for improved reasoning capabilities, surpassing previous prompt-driven approaches.

Findings

Significant performance improvements on math benchmarks.

RFTT outperforms prior models on the MATH dataset.

Performance improves with more inference rollouts.

Abstract

In this work, we propose Reinforced Functional Token Tuning (RFTT), a novel reinforced fine-tuning framework that empowers Large Language Models (LLMs) with self-play learn-to-reason capabilities. Unlike prior prompt-driven reasoning efforts, RFTT embeds a rich set of learnable functional tokens (e.g., <analyze>, <verify>, <refine>) directly into the model vocabulary, enabling chain-of-thought construction with diverse human-like reasoning behaviors. Specifically, RFTT comprises two phases: (1) supervised fine-tuning performs prompt-driven tree search to obtain self-generated training data annotated with functional tokens, which warms up the model to learn these tokens for reasoning; and (2) online reinforcement learning further allows the model to explore different reasoning pathways through functional token sampling without relying on prompts, thereby facilitating effective self-improvement for functional reasoning. Extensive experiments demonstrate the superiority of the proposed RFTT on mathematical benchmarks, significantly boosting Qwen-2.5-7B-Instruct (70.6% to 79.8%) and LLaMA-3.1-8B-Instruct (32.2% to 60.2%) on the MATH dataset. Moreover, the performance of RFTT consistently improves with more search rollouts at inference time. Our code is available at https://github.com/sastpg/RFTT.