QualityFlow: An Agentic Workflow for Program Synthesis Controlled by LLM Quality Checks

TL;DR

QualityFlow is a novel agentic workflow leveraging large language models for program synthesis, incorporating explicit quality checks and dynamic control to improve correctness and achieve state-of-the-art results on multiple benchmarks.

Contribution

Introduction of QualityFlow, an agentic LLM-based workflow with explicit quality checks for improved program synthesis accuracy.

Findings

Achieves state-of-the-art results on MBPP and HumanEval benchmarks.

Effectively filters incorrect programs and mitigates faulty tests.

Enhances workflow control and correctness in program synthesis.

Abstract

We introduce QualityFlow, a dynamic agentic workflow for program synthesis. Given the English description of a programming problem and a set of unit tests, the model's goal is to synthesize the correct program that solves the problem and passes the tests. QualityFlow includes large language model (LLM) agents resembling a software development team, including code generation, testing, and self-debugging. We propose the LLM Quality Checker, which explicitly "imagines" whether the synthesized programs' execution would conform to the unit tests. The Quality Checks dynamically control the workflow, including actions to submit the final answer, clarify the problem statement, and revert previous workflow steps. Our experiments show that the Quality Checker can precisely accept any correct program, mitigate faulty synthesized tests, and prevent potential workflow deviation. QualityFlow establishes the state-of-the-art results on four program synthesis benchmarks: MBPP, HumanEval, and stricter evaluations from MBPP-EvalPlus and HumanEval-EvalPlus.