Assessing Y-Axis Influence: Bias in Multimodal Language Models on Chart-to-Table Translation

TL;DR

This paper investigates biases related to y-axis features in chart-to-table translation by multimodal language models, revealing significant biases and proposing a framework to analyze and mitigate them.

Contribution

It introduces FairChart2Table, a framework for analyzing y-axis bias in chart-to-table translation models, and uncovers key factors influencing model performance.

Findings

Y-axis biases relate to digit length, tick count, value range, and format.

Number of legends/entities affects MLM performance.

Prompting with y-axis info improves some MLMs' accuracy.

Abstract

Chart-to-table translation converts chart images into structured tabular data. Accurate translation is crucial for Multimodal Language Model (MLM) to answer complex queries. We observe imbalances in the number of images across different aspects of the y-axis information in public chart datasets. Such imbalances can introduce unintended biases, causing uneven MLM performance. Previous works have not systematically examined these biases. To address this gap, we propose a new framework, FairChart2Table, for analyzing y-axis-related bias on five state-of-the-art models. Key Findings: (1) There are significant y-axis biases related to the digit length of the major tick values, the number of major ticks, the range of values, and the tick value format (e.g., abbreviation or scientific format). (2) The number of legends/entities in chart images impacts MLM performance. (3) Prompting MLM with y-axis information can significantly enhance the performance for some MLMs.