Format as a Prior: Quantifying and Analyzing Bias in LLMs for Heterogeneous Data

TL;DR

This paper investigates the systematic biases of large language models towards different data formats, analyzing their causes, internal mechanisms, and proposing interventions to improve fairness and robustness in heterogeneous data processing.

Contribution

It provides the first comprehensive empirical analysis of format bias in LLMs, identifying key factors and internal attention mechanisms influencing bias, and suggests practical strategies for mitigation.

Findings

Format bias is consistent across model types.

Bias is driven by information richness, structure quality, and representation.

Attention imbalance correlates with format bias.

Abstract

Large Language Models (LLMs) are increasingly employed in applications that require processing information from heterogeneous formats, including texts, tables, infoboxes, and knowledge graphs. However, systematic biases toward particular formats may undermine LLMs' ability to integrate heterogeneous data impartially, potentially resulting in reasoning errors and increased risks in downstream tasks. Yet it remains unclear whether such biases are systematic, which data-level factors drive them, and what internal mechanisms underlie their emergence. In this paper, we present the first comprehensive study of format bias in LLMs through a three-stage empirical analysis. The first stage explores the presence and direction of bias across a diverse range of LLMs. The second stage examines how key data-level factors influence these biases. The third stage analyzes how format bias emerges within LLMs' attention patterns and evaluates a lightweight intervention to test its effectiveness. Our results show that format bias is consistent across model families, driven by information richness, structure quality, and representation type, and is closely associated with attention imbalance within the LLMs. Based on these investigations, we identify three future research directions to reduce format bias: enhancing data pre-processing through format repair and normalization, introducing inference-time interventions such as attention re-weighting, and developing format-balanced training corpora. These directions will support the design of more robust and fair heterogeneous data processing systems.