Rationale-Aware Answer Verification by Pairwise Self-Evaluation

TL;DR

This paper introduces REPS, a method to improve answer verification by selecting valid rationales through pairwise self-evaluation, leading to more reliable verifiers in reasoning tasks.

Contribution

It proposes REPS, a novel rationale selection technique that enhances verifier training by focusing on rationale validity without extra human labeling.

Findings

Verifiers trained on REPS-selected rationales outperform baseline methods.

Only 19% of LLM solutions with correct answers have valid rationales in StrategyQA.

REPS improves verifier accuracy across multiple reasoning benchmarks.

Abstract

Answer verification identifies correct solutions among candidates generated by large language models (LLMs). Current approaches typically train verifier models by labeling solutions as correct or incorrect based solely on whether the final answer matches the gold answer. However, this approach neglects any flawed rationale in the solution yielding the correct answer, undermining the verifier's ability to distinguish between sound and flawed rationales. We empirically show that in StrategyQA, only 19% of LLM-generated solutions with correct answers have valid rationales, thus leading to an unreliable verifier. Furthermore, we demonstrate that training a verifier on valid rationales significantly improves its ability to distinguish valid and flawed rationale. To make a better verifier without extra human supervision, we introduce REPS (Rationale Enhancement through Pairwise Selection), a method for selecting valid rationales from candidates by iteratively applying pairwise self-evaluation using the same LLM that generates the solutions. Verifiers trained on solutions selected by REPS outperform those trained using conventional training methods on three reasoning benchmarks (ARC-Challenge, DROP, and StrategyQA). Our results suggest that training reliable verifiers requires ensuring the validity of rationales in addition to the correctness of the final answers, which would be critical for models assisting humans in solving complex reasoning tasks.