FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning

TL;DR

This paper introduces FAPO, a method that improves reasoning in language models by penalizing flawed reasoning patterns during reinforcement learning, leading to more reliable and stable performance without extra token costs.

Contribution

FAPO is a novel, parameter-free reward penalty method that detects and mitigates flawed reasoning in RL, enhancing model reliability and training stability.

Findings

FAPO improves correctness and reliability across multiple domains.

It stabilizes training without increasing token usage.

FAPO effectively detects flawed reasoning with GenRM.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has emerged as a promising paradigm for enhancing the reasoning capabilities of large language models (LLMs). In this context, models explore reasoning trajectories and exploit rollouts with correct answers as positive signals for policy optimization. However, these rollouts might involve flawed patterns such as answer-guessing and jump-in-reasoning. Such flawed-positive rollouts are rewarded identically to fully correct ones, causing policy models to internalize these unreliable reasoning patterns. In this work, we first conduct a systematic study of flawed-positive rollouts in RL and find that they enable rapid capability gains during the early optimization stage, while constraining reasoning capability later by reinforcing unreliable patterns. Building on these insights, we propose Flawed-Aware Policy Optimization (FAPO), which presents a parameter-free reward penalty for flawed-positive rollouts, enabling the policy to leverage them as useful shortcuts in the warm-up stage, securing stable early gains, while gradually shifting optimization toward reliable reasoning in the later refinement stage. To accurately and comprehensively detect flawed-positive rollouts, we introduce a generative reward model (GenRM) with a process-level reward that precisely localizes reasoning errors. Experiments show that FAPO is effective in broad domains, improving outcome correctness, process reliability, and training stability without increasing the token budget.