SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understanding

TL;DR

This paper introduces SimVec, a simplified vector format for charts, and a new dataset SimVecVis to improve multimodal large language models' understanding of visualizations through fine-tuning and data-centric QA tasks.

Contribution

The paper presents SimVec, a novel chart encoding format, and creates the SimVecVis dataset to enhance MLLMs' visualization understanding capabilities.

Findings

MLLMs show improved performance on visualization tasks after fine-tuning with SimVecVis.

SimVec encoding enables better decoding of chart information by MLLMs.

Fine-tuned MLLMs achieve higher accuracy in data-centric QA with visualization understanding.

Abstract

Current multimodal large language models (MLLMs), while effective in natural image understanding, struggle with visualization understanding due to their inability to decode the data-to-visual mapping and extract structured information. To address these challenges, we propose SimVec, a novel simplified vector format that encodes chart elements such as mark type, position, and size. The effectiveness of SimVec is demonstrated by using MLLMs to reconstruct chart information from SimVec formats. Then, we build a new visualization dataset, SimVecVis, to enhance the performance of MLLMs in visualization understanding, which consists of three key dimensions: bitmap images of charts, their SimVec representations, and corresponding data-centric question-answering (QA) pairs with explanatory chain-of-thought (CoT) descriptions. We finetune state-of-the-art MLLMs (e.g., MiniCPM and Qwen-VL), using SimVecVis with different dataset dimensions. The experimental results show that it leads to substantial performance improvements of MLLMs with good spatial perception capabilities (e.g., MiniCPM) in data-centric QA tasks. Our dataset and source code are available at: https://github.com/VIDA-Lab/SimVecVis.