FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding

TL;DR

This paper introduces the FlowLearn dataset to evaluate large vision-language models on flowchart understanding, revealing current limitations and setting the stage for future improvements in scientific visual comprehension.

Contribution

The paper presents a new dataset, FlowLearn, with detailed annotations for flowchart understanding and evaluates state-of-the-art LVLMs, highlighting their strengths and weaknesses.

Findings

GPT-4V achieved 58% accuracy in node counting on simulated flowcharts.

Claude achieved 83% accuracy in OCR tasks.

No single model excels across all flowchart understanding tasks.

Abstract

Flowcharts are graphical tools for representing complex concepts in concise visual representations. This paper introduces the FlowLearn dataset, a resource tailored to enhance the understanding of flowcharts. FlowLearn contains complex scientific flowcharts and simulated flowcharts. The scientific subset contains 3,858 flowcharts sourced from scientific literature and the simulated subset contains 10,000 flowcharts created using a customizable script. The dataset is enriched with annotations for visual components, OCR, Mermaid code representation, and VQA question-answer pairs. Despite the proven capabilities of Large Vision-Language Models (LVLMs) in various visual understanding tasks, their effectiveness in decoding flowcharts - a crucial element of scientific communication - has yet to be thoroughly investigated. The FlowLearn test set is crafted to assess the performance of LVLMs in flowchart comprehension. Our study thoroughly evaluates state-of-the-art LVLMs, identifying existing limitations and establishing a foundation for future enhancements in this relatively underexplored domain. For instance, in tasks involving simulated flowcharts, GPT-4V achieved the highest accuracy (58%) in counting the number of nodes, while Claude recorded the highest accuracy (83%) in OCR tasks. Notably, no single model excels in all tasks within the FlowLearn framework, highlighting significant opportunities for further development.