JSynFlow: Japanese Synthesised Flowchart Visual Question Answering Dataset built with Large Language Models

TL;DR

JSynFlow is a newly created Japanese flowchart visual question answering dataset generated with large language models, designed to enhance the ability of vision and language models to interpret complex flowcharts.

Contribution

The paper introduces JSynFlow, a large-scale synthetic dataset for Japanese flowchart VQA, created using LLMs, and demonstrates its effectiveness in improving VLM performance.

Findings

Fine-tuning with JSynFlow improves VLM accuracy on flowchart QA tasks.

The dataset enables better flowchart understanding in Japanese VLMs.

JSynFlow is publicly available for research use.

Abstract

Vision and language models (VLMs) are expected to analyse complex documents, such as those containing flowcharts, through a question-answering (QA) interface. The ability to recognise and interpret these flowcharts is in high demand, as they provide valuable insights unavailable in text-only explanations. However, developing VLMs with precise flowchart understanding requires large-scale datasets of flowchart images and corresponding text, the creation of which is highly time-consuming. To address this challenge, we introduce JSynFlow, a synthesised visual QA dataset for Japanese flowcharts, generated using large language models (LLMs). Our dataset comprises task descriptions for various business occupations, the corresponding flowchart images rendered from domain-specific language (DSL) code, and related QA pairs. This paper details the dataset's synthesis procedure and demonstrates that fine-tuning with JSynFlow significantly improves VLM performance on flowchart-based QA tasks. Our dataset is publicly available at https://huggingface.co/datasets/jri-advtechlab/jsynflow.