Multi-modal Program Inference: a Marriage of Pre-trainedLanguage Models and Component-based Synthesis

TL;DR

This paper introduces a multi-modal program synthesis approach combining pre-trained language models with component-based synthesis, effectively generating precise code from ambiguous natural language and examples across different programming domains.

Contribution

It presents a novel hybrid method that leverages PTMs for natural language understanding and CBS for precise code generation, applicable across multiple programming domains.

Findings

Outperforms state-of-the-art specialized systems in experiments.

Demonstrates effectiveness in regular expressions and CSS selectors domains.

Shows generality and domain-agnostic capabilities.

Abstract

Multi-modal program synthesis refers to the task of synthesizing programs (code) from their specification given in different forms, such as a combination of natural language and examples. Examples provide a precise but incomplete specification, and natural language provides an ambiguous but more "complete" task description. Machine-learned pre-trained models (PTMs) are adept at handling ambiguous natural language, but struggle with generating syntactically and semantically precise code. Program synthesis techniques can generate correct code, often even from incomplete but precise specifications, such as examples, but they are unable to work with the ambiguity of natural languages. We present an approach that combines PTMs with component-based synthesis (CBS): PTMs are used to generate candidates programs from the natural language description of the task, which are then used to guide the CBS procedure to find the program that matches the precise examples-based specification. We use our combination approach to instantiate multi-modal synthesis systems for two programming domains: the domain of regular expressions and the domain of CSS selectors. Our evaluation demonstrates the effectiveness of our domain-agnostic approach in comparison to a state-of-the-art specialized system, and the generality of our approach in providing multi-modal program synthesis from natural language and examples in different programming domains.