Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning

TL;DR

This paper introduces a novel method for disentangling reasoning tokens from boilerplate tokens in language model fine-tuning, leading to improved performance by emphasizing reasoning tokens during training.

Contribution

The paper proposes SHAD, a shuffle-aware discriminator, and RFT, a fine-tuning method that adaptively emphasizes reasoning tokens, addressing token role differentiation in LLM training.

Findings

RFT outperforms standard supervised fine-tuning.

SHAD effectively classifies tokens based on predictability.

Enhanced reasoning capabilities in LLMs.

Abstract

When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format) - differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT).