Testing Cross-Lingual Text Comprehension In LLMs Using Next Sentence Prediction

TL;DR

This study evaluates large language models' cross-lingual comprehension using a Next Sentence Prediction benchmark across English, Swahili, and Hausa, revealing resource-dependent performance and nuanced effects of Chain-of-Thought prompting.

Contribution

It introduces a large-scale cross-lingual NSP benchmark and analyzes how different models and prompting techniques perform across resource-scarce languages.

Findings

Models excel in English but struggle in low-resource languages.

Chain-of-Thought prompting improves performance for weaker models like LLaMA 3.

For stronger models, Chain-of-Thought can sometimes decrease accuracy.

Abstract

While large language models are trained on massive datasets, this data is heavily skewed towards English. Does their impressive performance reflect genuine ability or just this data advantage? To find out, we tested them in a setting where they could not rely on data abundance: low-resource languages. Building on prior work Agarwal et al. (2025) that used Next Sentence Prediction (NSP) as a test, we created a large-scale benchmark with 10,000 questions each for English (a high-resource language), Swahili (medium-resource), and Hausa (low-resource). We then tested several top models, including GPT-4 Turbo, Gemini 1.5 Flash, and LLaMA 3 70B, to see how their performance holds up. The results painted a clear picture of how levels of language resources impact outcomes. While all models excelled in English, their accuracy dropped in Swahili and fell sharply in Hausa, with LLaMA 3 struggling the most. The story became even more interesting when we introduced Chain-of-Thought (CoT) prompting. For the struggling LLaMA 3, CoT acted as a helpful guide, significantly boosting its accuracy. However, for the more capable GPT-4 and Gemini, the same technique often backfired, leading to a kind of "overthinking" that hurt their results in the cross-lingual context. This reveals that Chain-of-Thought is not a universal solution; its effectiveness depends heavily on the model's baseline capability and the specific context of the task. Our framework pinpoints LLM weaknesses, highlights when CoT helps or hinders cross-lingual NSP performance, and factors influencing their decisions.