Large Reasoning Models Struggle to Transfer Parametric Knowledge Across Scripts

TL;DR

This paper investigates why large reasoning language models struggle to transfer knowledge across different scripts, identifying script mismatch as the main barrier and proposing methods to improve cross-script reasoning and knowledge transfer.

Contribution

The study reveals script mismatch as the key factor in transfer failure and introduces a synthetic training pipeline to enhance models' reasoning about transliteration ambiguities.

Findings

Script match predicts transfer success more than language family.

Providing key entities in source language improves cross-script reasoning.

Enhanced reasoning training reduces transfer gap.

Abstract

In this work, we analyze shortcomings in cross-lingual knowledge transfer in large, modern reasoning LLMs. We demonstrate that the perceived gap in knowledge transfer is primarily a script barrier. First, we conduct an observational data analysis on the performance of thinking models on two datasets with local knowledge from around the world, ECLeKTic and MultiLoKo. Our regression analysis shows that script match - not language or family - is the primary predictor of knowledge transfer failure once model capability and question difficulty are accounted for. We further this finding by providing the LLMs with the key entities of the questions in their source language and find that this disproportionately improves cross-script questions. We then posit that these LLMs could be reasoning better at test-time. To evaluate this, we develop a synthetic generation pipeline to design SFT samples to encourage the model to better reason about transliteration ambiguities when trying to fetch parametric knowledge at inference-time. We show that teaching two models to reason better reduces the cross-script transfer gap. As a result, we conclude that there is potential to improve cross-lingual parametric knowledge transfer during post-training.