Demystifying Multilingual Chain-of-Thought in Process Reward Modeling

TL;DR

This paper extends process reward modeling to multilingual settings, training models on seven languages and demonstrating improved reasoning accuracy and error reduction across 11 languages, advancing multilingual complex reasoning capabilities.

Contribution

It introduces multilingual process reward models trained on translated datasets, addressing the challenge of multilingual multi-step reasoning in LLMs.

Findings

Multilingual PRMs improve average accuracy across 11 languages.

Multilingual PRMs reduce early-stage reasoning errors.

Performance is sensitive to training languages and data volume.

Abstract

Large language models (LLMs) are designed to perform a wide range of tasks. To improve their ability to solve complex problems requiring multi-step reasoning, recent research leverages process reward modeling to provide fine-grained feedback at each step of the reasoning process for reinforcement learning (RL), but it predominantly focuses on English. In this paper, we tackle the critical challenge of extending process reward models (PRMs) to multilingual settings. To achieve this, we train multilingual PRMs on a dataset spanning seven languages, which is translated from English. Through comprehensive evaluations on two widely used reasoning benchmarks across 11 languages, we demonstrate that multilingual PRMs not only improve average accuracy but also reduce early-stage reasoning errors. Furthermore, our results highlight the sensitivity of multilingual PRMs to both the number of training languages and the volume of English data, while also uncovering the benefits arising from more candidate responses and trainable parameters. This work opens promising avenues for robust multilingual applications in complex, multi-step reasoning tasks. In addition, we release the code to foster research along this line.