Teaching LLMs to Abstain across Languages via Multilingual Feedback

TL;DR

This paper introduces a multilingual feedback method to improve LLM abstention in low-resource languages, reducing knowledge gaps and hallucinations across diverse languages and cultures.

Contribution

It proposes a novel multilingual feedback strategy enabling LLMs to better identify knowledge gaps and abstain appropriately in under-resourced languages.

Findings

Up to 9.2% improvement in low-resource language performance.

Multilingual feedback enhances LLM calibration and reasoning.

Cultural factors influence language selection and abstention behavior.

Abstract

Multilingual LLMs often have knowledge disparities across languages, with larger gaps in under-resourced languages. Teaching LLMs to abstain in the face of knowledge gaps is thus a promising strategy to mitigate hallucinations in multilingual settings. However, previous studies on LLM abstention primarily focus on English; we find that directly applying existing solutions beyond English results in up to 20.5% performance gaps between high and low-resource languages, potentially due to LLMs' drop in calibration and reasoning beyond a few resource-rich languages. To this end, we propose strategies to enhance LLM abstention by learning from multilingual feedback, where LLMs self-reflect on proposed answers in one language by generating multiple feedback items in related languages: we show that this helps identifying the knowledge gaps across diverse languages, cultures, and communities. Extensive experiments demonstrate that our multilingual feedback approach outperforms various strong baselines, achieving up to 9.2% improvement for low-resource languages across three black-box and open models on three datasets, featuring open-book, closed-book, and commonsense QA. Further analysis reveals that multilingual feedback is both an effective and a more equitable abstain strategy to serve diverse language speakers, and cultural factors have great impact on language selection and LLM abstention behavior, highlighting future directions for multilingual and multi-cultural reliable language modeling.