Bias-Restrained Prefix Representation Finetuning for Mathematical Reasoning

TL;DR

This paper introduces BREP ReFT, a novel fine-tuning method that improves mathematical reasoning in language models by optimizing early inference stages and constraining interventions, outperforming existing PEFT and ReFT methods.

Contribution

The paper proposes BREP ReFT, a new representation fine-tuning approach that enhances mathematical reasoning by focusing on early inference optimization and intervention constraints.

Findings

BREP ReFT outperforms standard ReFT and PEFT on mathematical reasoning tasks.

It demonstrates superior effectiveness, efficiency, and generalization across various models.

Extensive experiments validate the robustness of the proposed method.

Abstract

Parameter-Efficient finetuning (PEFT) enhances model performance on downstream tasks by updating a minimal subset of parameters. Representation finetuning (ReFT) methods further improve efficiency by freezing model weights and optimizing internal representations with fewer parameters than PEFT, outperforming PEFT on several tasks. However, ReFT exhibits a significant performance decline on mathematical reasoning tasks. To address this problem, the paper demonstrates that ReFT's poor performance on mathematical tasks primarily stems from its struggle to generate effective reasoning prefixes during the early inference phase. Moreover, ReFT disturbs the numerical encoding and the error accumulats during the CoT stage. Based on these observations, this paper proposes Bias-REstrained Prefix Representation FineTuning (BREP ReFT), which enhances ReFT's mathematical reasoning capability by truncating training data to optimize the generation of initial reasoning prefixes, intervening on the early inference stage to prevent error accumulation, and constraining the intervention vectors' magnitude to avoid disturbing numerical encoding. Extensive experiments across diverse model architectures demonstrate BREP's superior effectiveness, efficiency, and robust generalization capability, outperforming both standard ReFT and weight-based PEFT methods on the task of mathematical reasoning. The source code is available at https://github.com/LiangThree/BREP.