Towards Making Flowchart Images Machine Interpretable

TL;DR

This paper introduces FloCo-T5, a transformer-based model that converts flowchart images into executable Python code, leveraging a new dataset and pre-training techniques to improve code generation from flowcharts.

Contribution

The paper presents a novel transformer framework, FloCo-T5, and a large dataset, FloCo, for translating flowchart images into Python code, advancing machine interpretability of flowcharts.

Findings

FloCo-T5 outperforms baseline models on code generation metrics.

The dataset FloCo contains 11,884 flowchart images with corresponding Python code.

Pre-training with logic-preserving augmented code improves model performance.

Abstract

Computer programming textbooks and software documentations often contain flowcharts to illustrate the flow of an algorithm or procedure. Modern OCR engines often tag these flowcharts as graphics and ignore them in further processing. In this paper, we work towards making flowchart images machine-interpretable by converting them to executable Python codes. To this end, inspired by the recent success in natural language to code generation literature, we present a novel transformer-based framework, namely FloCo-T5. Our model is well-suited for this task,as it can effectively learn semantics, structure, and patterns of programming languages, which it leverages to generate syntactically correct code. We also used a task-specific pre-training objective to pre-train FloCo-T5 using a large number of logic-preserving augmented code samples. Further, to perform a rigorous study of this problem, we introduce theFloCo dataset that contains 11,884 flowchart images and their corresponding Python codes. Our experiments show promising results, and FloCo-T5 clearly outperforms related competitive baselines on code generation metrics. We make our dataset and implementation publicly available.