Fin-Bias: Comprehensive Evaluation for LLM Decision-Making under human bias in Finance Domain

TL;DR

This paper introduces Fin-Bias, a comprehensive benchmark for evaluating how large language models make financial decisions under human bias, revealing their tendency to herd bias and proposing methods to improve independent reasoning.

Contribution

The paper presents a new benchmark with a large dataset of analyst reports, and demonstrates how LLMs tend to herd bias, offering a method to detect and mitigate this effect.

Findings

LLMs tend to herd explicit human bias in financial contexts.

A method to detect potential human opinions can encourage independent LLM reasoning.

Some LLMs outperform humans in predicting future stock returns.

Abstract

Large language models (LLMs) are increasingly deployed in financial contexts, raising critical concerns about reliability, alignment, and susceptibility to adversarial manipulation. While prior finance-related benchmarks assess LLMs' capabilities in stock trading, they are often restricted to small sample and fail to demonstrate LLM susceptibility to context with potential human bias. We introduce Fin-Bias (financial herding under long and uncertain financial context), a benchmark for evaluating LLM investment decision-making when faced with uncertainty and possible human-biased opinions. Fin-Bias includes 8868 long firm-specific analyst reports, including firm aspects summarized and analyzed by sophisticated analysts with investment ratings (Bullish/Neutral/Bearish) spanning from various industries. We present large language models with firm analyst reports with/without analyst investment ratings and even with 'fake' rating, to get investment ratings generated by LLMs. Our results reveal that LLMs tend to herd the explicit bias in context. We also develop a method to detect potential human opinions, which can encourage LLMs to think independently, some models even exceed human performance in predicting future stock return.