Smoothie-Qwen: Post-Hoc Smoothing to Reduce Language Bias in Multilingual LLMs

TL;DR

Smoothie-Qwen is a post-hoc technique that reduces language bias in multilingual LLMs by adjusting output probabilities, significantly decreasing unintended language outputs while maintaining task performance.

Contribution

It introduces a lightweight, post-hoc method to mitigate language bias in multilingual LLMs without retraining, improving language controllability.

Findings

Reduces unintended Chinese output by over 95%.

Preserves task accuracy on multilingual benchmarks.

Provides a practical solution for language bias mitigation.

Abstract

Multilingual large language models (LLMs) often exhibit language confusion, a tendency to generate responses in a dominant language irrespective of the prompt's language. To address this, we propose Smoothie-Qwen, a lightweight, post-hoc method that mitigates language bias without retraining. This technique selectively adjusts token-level output probabilities to effectively suppress undesired language generation. Applied to the Qwen model, our method reduces unintended Chinese output by over 95% while preserving task accuracy on multilingual benchmarks. This work provides a practical and efficient solution for enhancing the language controllability of LLMs, making them more reliable for global applications.