Program Synthesis Benchmark for Visual Programming in XLogoOnline Environment

TL;DR

This paper introduces a new program synthesis benchmark in the XLogoOnline environment, evaluates current models' performance, and proposes fine-tuning methods to significantly improve results on complex visual programming tasks.

Contribution

It presents a novel benchmark combining multiple skills, evaluates state-of-the-art models, and develops a fine-tuning pipeline with curriculum learning to enhance model performance.

Findings

GPT-4V achieves 20% success rate

Llama3-70B achieves 2.35% success rate

Fine-tuning with synthetic data and curriculum improves performance

Abstract

Large language and multimodal models have shown remarkable success on various benchmarks focused on specific skills such as general-purpose programming, math word problem-solving, and visual question answering. However, it is unclear how well these models perform on tasks that require a combination of these skills. In this paper, we curate a novel program synthesis benchmark based on the real-world tasks in the XLogoOnline visual programming environment. Each task requires a combination of different skills such as spatial planning, basic programming, and logical reasoning. Our evaluation shows that current state-of-the-art models like GPT-4V and Llama3-70B struggle to solve these tasks, achieving only 20% and 2.35% success rates, respectively. Next, we develop a fine-tuning pipeline to boost the performance of models by leveraging a large-scale synthetic training dataset with over 80,000 tasks. Moreover, we showcase how emulator-driven feedback can be used to design a curriculum over training data distribution, through which a fine-tuned Llama3-8B drastically outperforms GPT-4V and Llama3-70B models. Finally, we provide an in-depth failure analysis to understand the limitations of different models. We will publicly release the benchmark for future research on program synthesis in visual programming.