Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

TL;DR

This paper reveals that current Process Reward Models are vulnerable to adversarial attacks, often exploiting superficial cues rather than genuine reasoning, which undermines their reliability in guiding language model training.

Contribution

The authors introduce a three-tiered diagnostic framework and release tools to evaluate and improve the robustness of Process Reward Models against adversarial manipulation.

Findings

PRMs show high invariance to style changes but fail on logical reasoning detection.

Gradient-based attacks can inflate rewards on invalid trajectories.

Reward hacking allows policies to achieve high rewards with low true accuracy.

Abstract

Process Reward Models (PRMs) are rapidly becoming the backbone of LLM reasoning pipelines, yet we demonstrate that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure. To address this, we introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities. Static perturbation analysis uncovers a fluency-logic dissociation: high invariance to surface-level style changes reward changes $<$0.1, yet inconsistent detection of logically-corrupted reasoning, with different models failing on different attack types. Adversarial optimization demonstrates that gradient-based attacks inflate rewards on invalid trajectories, with reward landscapes exhibiting wide, exploitable peaks. RL-induced reward hacking exposes the critical failure mode: policies trained on AIME problems achieve near-perfect PRM rewards ($>$0.9), while ground-truth accuracy remains low (below 4%), with 43% of reward gains attributable to stylistic shortcuts. These findings reveal that current PRMs function as fluency detectors rather than reasoning verifiers, creating systematic blind spots that undermine their use as training signals. We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment. The code and dataset are available at https://github.com/SqueezeAILab/reward-under-attack.