Humans overrely on overconfident language models, across languages

TL;DR

This study reveals that large language models exhibit overconfidence across multiple languages, leading to high risks of user overreliance, and emphasizes the need for culturally aware safety evaluations.

Contribution

It provides a comprehensive analysis of multilingual epistemic marker usage and human reliance behaviors, highlighting calibration challenges in LLMs across five languages.

Findings

LLMs are overconfident across languages, often generating unwarranted certainty.

Human reliance on model outputs varies significantly across languages.

Models generate more uncertainty markers in Japanese and more certainty markers in German and Mandarin.

Abstract

As large language models (LLMs) are deployed globally, it is crucial that their responses are calibrated across languages to accurately convey uncertainty and limitations. Prior work shows that LLMs are linguistically overconfident in English, leading users to overrely on confident generations. However, the usage and interpretation of epistemic markers (e.g., 'I think it's') differs sharply across languages. Here, we study the risks of multilingual linguistic (mis)calibration, overconfidence, and overreliance across five languages to evaluate LLM safety in a global context. Our work finds that overreliance risks are high across languages. We first analyze the distribution of LLM-generated epistemic markers and observe that LLMs are overconfident across languages, frequently generating strengtheners even as part of incorrect responses. Model generations are, however, sensitive to documented cross-linguistic variation in usage: for example, models generate the most markers of uncertainty in Japanese and the most markers of certainty in German and Mandarin. Next, we measure human reliance rates across languages, finding that reliance behaviors differ cross-linguistically: for example, participants are significantly more likely to discount expressions of uncertainty in Japanese than in English (i.e., ignore their 'hedging' function and rely on generations that contain them). Taken together, these results indicate a high risk of reliance on overconfident model generations across languages. Our findings highlight the challenges of multilingual linguistic calibration and stress the importance of culturally and linguistically contextualized model safety evaluations.