MALT: Mechanistic Ablation of Lossy Translation in LLMs for a Low-Resource Language: Urdu

TL;DR

This paper investigates how LLMs process low-resource languages like Urdu, identifies lossy translation features in their internal responses, and proposes a mechanistic ablation method to improve performance and cultural preservation.

Contribution

The study introduces a novel mechanistic ablation approach that removes lossy translation features in LLMs, enhancing low-resource language processing.

Findings

Removing translation features improves LLM performance on Urdu.

Using a separate translation model preserves cultural nuances.

Internal English responses are coherent despite translation issues.

Abstract

LLMs are predominantly trained on English data, which leads to a significant drop in performance on low-resource languages. Understanding how LLMs handle these languages is crucial for improving their effectiveness. This study focuses on Urdu as a use case for exploring the challenges faced by LLMs in processing low-resource languages. LLMs primarily reason in English when prompted in another language, with the final layers acting as translators to convert the English response into the target language. This study finds that even for low-resource languages, the internal latent response of LLMs in English is quite coherent; however, the translation features are lossy and result in poor translations, leading to reduced performance. By mechanistically removing these translation features and using a separate translation model to translate the internal latent response of LLM, the performance of LLMs improves significantly while also preserving the cultural nuances of the input in low-resource languages.