Flow2Code: Evaluating Large Language Models for Flowchart-based Code Generation Capability

TL;DR

Flow2Code introduces a new benchmark for evaluating large language models' ability to generate code from flowcharts across multiple programming languages, highlighting current limitations and the impact of fine-tuning.

Contribution

This work presents the first comprehensive benchmark dataset and evaluation for flowchart-based code generation using large language models, covering 15 languages and multiple flowchart types.

Findings

Current LLMs struggle with perfect flowchart-to-code translation.

Supervised fine-tuning significantly improves model performance.

The dataset and code are publicly available for further research.

Abstract

While large language models (LLMs) show promise in code generation, existing benchmarks neglect the flowchart-based code generation. To promote further research on flowchart-based code generation, this work presents Flow2Code, a novel benchmark for flowchart-based code generation evaluation. The evaluation dataset spans 15 programming languages and includes 5,622 code segments paired with 16,866 flowcharts of three types: code, UML, and pseudocode. Extensive experiments with 13 multimodal LLMs reveal that current LLMs can not generate code based on flowcharts perfectly. Besides, experiment results show that the supervised fine-tuning technique contributes greatly to the models' performance. We publicly release our code and datasets at https://github.com/hml-github/Flow2Code.