What Really Counts? Examining Step and Token Level Attribution in Multilingual CoT Reasoning

TL;DR

This paper examines how attribution methods reveal the reasoning process of multilingual LLMs, uncovering biases and limitations in interpretability and robustness across languages.

Contribution

It introduces a comprehensive analysis of step and token attribution in multilingual CoT reasoning, highlighting challenges in faithfulness and interpretability across languages.

Findings

Attribution scores focus excessively on the final reasoning step.

Structured CoT improves accuracy mainly in high-resource Latin-script languages.

Perturbations like negation reduce model accuracy and attribution coherence.

Abstract

This study investigates the attribution patterns underlying Chain-of-Thought (CoT) reasoning in multilingual LLMs. While prior works demonstrate the role of CoT prompting in improving task performance, there are concerns regarding the faithfulness and interpretability of the generated reasoning chains. To assess these properties across languages, we applied two complementary attribution methods--ContextCite for step-level attribution and Inseq for token-level attribution--to the Qwen2.5 1.5B-Instruct model using the MGSM benchmark. Our experimental results highlight key findings such as: (1) attribution scores excessively emphasize the final reasoning step, particularly in incorrect generations; (2) structured CoT prompting significantly improves accuracy primarily for high-resource Latin-script languages; and (3) controlled perturbations via negation and distractor sentences reduce model accuracy and attribution coherence. These findings highlight the limitations of CoT prompting, particularly in terms of multilingual robustness and interpretive transparency.