GP and LLMs for Program Synthesis: No Clear Winners

TL;DR

This study compares genetic programming and large language models in program synthesis, revealing no clear overall winner but highlighting the importance of different approaches and prompting strategies for success.

Contribution

It systematically evaluates PushGP and GPT-4o across various prompting methods and task sizes, providing insights into their complementary strengths and differences in program synthesis.

Findings

PushGP and GPT-4o with combined data-text prompts solved most tasks

Success rates decrease for PushGP with less data, unlike other methods

Significant differences in program similarity depending on prompting style

Abstract

Genetic programming (GP) and large language models (LLMs) differ in how program specifications are provided: GP uses input-output examples, and LLMs use text descriptions. In this work, we compared the ability of PushGP and GPT-4o to synthesize computer programs for tasks from the PSB2 benchmark suite. We used three prompt variants with GPT-4o: input-output examples (data-only), textual description of the task (text-only), and a combination of both textual descriptions and input-output examples (data-text). Additionally, we varied the number of input-output examples available for building programs. For each synthesizer and task combination, we compared success rates across all program synthesizers, as well as the similarity between successful GPT-4o synthesized programs. We found that the combination of PushGP and GPT-4o with data-text prompting led to the greatest number of tasks solved (23 of the 25 tasks), even though several tasks were solved exclusively by only one of the two synthesizers. We also observed that PushGP and GPT-4o with data-only prompting solved fewer tasks with the decrease in the training set size, while the remaining synthesizers saw no decrease. We also detected significant differences in similarity between the successful programs synthesized for GPT-4o with text-only and data-only prompting. With there being no dominant program synthesizer, this work highlights the importance of different optimization techniques used by PushGP and LLMs to synthesize programs.