Same Claim, Different Judgment: Benchmarking Scenario-Induced Bias in Multilingual Financial Misinformation Detection

TL;DR

This paper introduces MFMDScen, a benchmark for evaluating behavioral biases of multilingual LLMs in complex financial misinformation detection scenarios, revealing persistent biases across models.

Contribution

It presents a comprehensive, multilingual benchmark with diverse economic scenarios to systematically assess biases in financial misinformation detection by LLMs.

Findings

Behavioral biases are prevalent across both commercial and open-source LLMs.

The benchmark covers complex, multilingual financial scenarios involving ethnicity, region, and personality.

Persistent biases highlight the need for bias mitigation in financial LLM applications.

Abstract

Large language models (LLMs) have been widely applied across various domains of finance. Since their training data are largely derived from human-authored corpora, LLMs may inherit a range of human biases. Behavioral biases can lead to instability and uncertainty in decision-making, particularly when processing financial information. However, existing research on LLM bias has mainly focused on direct questioning or simplified, general-purpose settings, with limited consideration of the complex real-world financial environments and high-risk, context-sensitive, multilingual financial misinformation detection tasks MFMD. In this work, we propose MFMDScen, a comprehensive benchmark for evaluating behavioral biases of LLMs in MFMD across diverse economic scenarios. In collaboration with financial experts, we construct three types of complex financial scenarios: (i) role- and personality-based, (ii) role- and region-based, and (iii) role-based scenarios incorporating ethnicity and religious beliefs. We further develop a multilingual financial misinformation dataset covering English, Chinese, Greek, and Bengali. By integrating these scenarios with misinformation claims, MFMDScen enables a systematic evaluation of 22 mainstream LLMs. Our findings reveal that pronounced behavioral biases persist across both commercial and open-source models. This project is available at https://github.com/lzw108/FMD.