Forward versus Backward: Comparing Reasoning Objectives in Direct Preference Optimization

TL;DR

This paper compares forward and backward reasoning objectives in Direct Preference Optimization for large language models, revealing a trade-off where forward training improves accuracy and backward training enhances verification reliability.

Contribution

It introduces and empirically evaluates the combined effects of forward and backward reasoning objectives in DPO, highlighting their complementary roles in improving reasoning and verification.

Findings

Forward training increases accuracy from 83.1% to 86.6%.

Backward training reduces false positive verification from 13.4% to 4.3%.

Both methods decrease model acknowledgment rate, indicating increased confidence.

Abstract

Large language models exhibit impressive reasoning capabilities yet frequently generate plausible but incorrect solutions, a phenomenon commonly termed hallucination. This paper investigates the effect of training objective composition on reasoning reliability through Direct Preference Optimization. Two complementary training signals are examined: forward chain-of-thought generation, which trains the model to produce correct reasoning traces, and backward verification, which trains the model to verify and acknowledge errors in candidate solutions. Experiments on GSM8K reveal a fundamental trade-off between these objectives. Forward-only DPO training achieves the highest accuracy improvement, increasing from 83.1% to 86.6% (+3.5 percentage points), while backward-only training yields minimal accuracy gains but substantially reduces the false positive rate from 13.4% to 4.3%. Notably, both training variants reduce acknowledgement rate compared to the baseline, suggesting that preference optimization increases model confidence in its outputs. These findings indicate that forward and backward reasoning objectives provide distinct and complementary learning signals: forward training improves problem-solving capability, while backward training improves verification calibration. The complete training and evaluation pipeline, implemented efficiently through Low-Rank Adaptation, is released to facilitate further research.