FIBER: A Multilingual Evaluation Resource for Factual Inference Bias

TL;DR

FIBER is a multilingual benchmark designed to evaluate factual inference bias in large language models across different languages and entity settings, revealing language-dependent biases and performance variations.

Contribution

This paper introduces FIBER, a novel multilingual benchmark for assessing factual knowledge and inference biases in language models across multiple languages and entity types.

Findings

Language of prompt influences model output bias, especially for country-associated entities.

Models perform worse on multi-entity questions than single-entity ones.

Larger models outperform smaller ones and English prompts yield higher accuracy.

Abstract

Large language models are widely used across domains, yet there are concerns about their factual reliability and biases. Factual knowledge probing offers a systematic means to evaluate these aspects. Most existing benchmarks focus on single-entity facts and monolingual data. We therefore present FIBER, a multilingual benchmark for evaluating factual knowledge in single- and multi-entity settings. The dataset includes sentence completion, question-answering, and object-count prediction tasks in English, Italian, and Turkish. Using FIBER, we examine whether the prompt language induces inference bias in entity selection and how large language models perform on multi-entity versus single-entity questions. The results indicate that the language of the prompt can influence the model's generated output, particularly for entities associated with the country corresponding to that language. However, this effect varies across different topics such that 31% of the topics exhibit factual inference bias score greater than 0.5. Moreover, the level of bias differs across languages such that Turkish prompts show higher bias compared to Italian in 83% of the topics, suggesting a language-dependent pattern. Our findings also show that models face greater difficulty when handling multi-entity questions than the single-entity questions. Model performance differs across both languages and model sizes. The highest mean average precision is achieved in English, while Turkish and Italian lead to noticeably lower scores. Larger models, including Llama-3.1-8B and Qwen-2.5-7B, show consistently better performance than smaller 3B-4B models.