Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax

TL;DR

This paper introduces a semantic-space alignment method using reinforcement learning with semantic rewards, enabling low-resource language expansion in large language models while reducing the alignment tax and preserving general capabilities.

Contribution

The authors propose a novel semantic RL approach with Group Relative Policy Optimization that improves low-resource language tasks without sacrificing pretrained knowledge.

Findings

Achieves low-resource language capabilities with less alignment tax.

Preserves general competence better than supervised fine-tuning.

Yields higher semantic quality and robustness in low-resource settings.

Abstract

Extending large language models (LLMs) to low-resource languages often incurs an "alignment tax": improvements in the target language come at the cost of catastrophic forgetting in general capabilities. We argue that this trade-off arises from the rigidity of supervised fine-tuning (SFT), which enforces token-level surface imitation on narrow and biased data distributions. To address this limitation, we propose a semantic-space alignment paradigm powered by Group Relative Policy Optimization (GRPO), where the model is optimized using embedding-level semantic rewards rather than likelihood maximization. This objective encourages meaning preservation through flexible realizations, enabling controlled updates that reduce destructive interference with pretrained knowledge. We evaluate our approach on Tibetan-Chinese machine translation and Tibetan headline generation. Experiments show that our method acquires low-resource capabilities while markedly mitigating alignment tax, preserving general competence more effectively than SFT. Despite producing less rigid surface overlap, semantic RL yields higher semantic quality and preference in open-ended generation, and few-shot transfer results indicate that it learns more transferable and robust representations under limited supervision. Overall, our study demonstrates that reinforcement learning with semantic rewards provides a safer and more reliable pathway for inclusive low-resource language expansion.