PORT: Preference Optimization on Reasoning Traces

TL;DR

This paper introduces preference optimization on reasoning traces to enhance mathematical and symbolic reasoning in language models, achieving significant accuracy improvements without extra annotations.

Contribution

It proposes novel schemes for generating rejected answers and demonstrates improved reasoning performance on multiple benchmarks, highlighting the importance of high-quality reasoning trace datasets.

Findings

Up to 8.47% and 18.73% accuracy improvements on GSM8K and AQuA-RAT.

Enhanced transfer of reasoning abilities to non-mathematical tasks.

Preference optimization on reasoning traces boosts model reasoning performance.

Abstract

Preference optimization methods have been successfully applied to improve not only the alignment of large language models (LLMs) with human values, but also specific natural language tasks such as summarization and stylistic continuations. This paper proposes using preference optimization methods on Chain-of-Thought steps in order to improve the mathematical reasoning performances of language models. While the chosen answers are obtained from datasets that include reasoning traces, we propose two complementary schemes for generating rejected answers: weak LLM prompting, and digit corruption. Our approach leads to increased accuracy on the GSM8K and AQuA-RAT mathematical reasoning benchmarks for Falcon2-11B and Mistral-7B. Additionally, the improved abilities transfer to non-mathematical tasks, including the ARC benchmark and symbolic reasoning challenges. For example, our method can lead to up to relative 8.47% and 18.73% increases in accuracy on the GSM8K and AQuA benchmarks respectively, without any extra annotations. This work suggests that the path towards better language reasoning abilities goes through spending resources on creating high-quality datasets of reasoning traces.