Reveal-Bangla: A Dataset for Cross-Lingual Multi-Step Reasoning Evaluation

TL;DR

This paper introduces Reveal-Bangla, a new dataset for evaluating multi-step reasoning in Bangla, enabling cross-lingual assessment of language models' reasoning capabilities and highlighting challenges in non-English contexts.

Contribution

It presents a manually translated Bangla dataset for multi-step reasoning, facilitating cross-lingual evaluation of multilingual models' reasoning abilities.

Findings

Reasoning context improves performance on complex non-binary questions.

Models struggle to effectively utilize Bangla reasoning steps.

Cross-lingual evaluation reveals language-specific reasoning challenges.

Abstract

Language models have demonstrated remarkable performance on complex multi-step reasoning tasks. However, their evaluation has been predominantly confined to high-resource languages such as English. In this paper, we introduce a manually translated Bangla multi-step reasoning dataset derived from the English Reveal dataset, featuring both binary and non-binary question types. We conduct a controlled evaluation of English-centric and Bangla-centric multilingual small language models on the original dataset and our translated version to compare their ability to exploit relevant reasoning steps to produce correct answers. Our results show that, in comparable settings, reasoning context is beneficial for more challenging non-binary questions, but models struggle to employ relevant Bangla reasoning steps effectively. We conclude by exploring how reasoning steps contribute to models' predictions, highlighting different trends across models and languages.