Grammar Prompting for Domain-Specific Language Generation with Large Language Models

TL;DR

This paper introduces grammar prompting, a method that leverages domain-specific BNF grammars to improve large language models' ability to generate structured, domain-specific language outputs more accurately and reliably.

Contribution

The paper presents a novel grammar prompting technique that incorporates external grammar constraints into LLMs for better domain-specific language generation.

Findings

Enables LLMs to generate complex domain-specific languages more accurately.

Improves performance on semantic parsing, planning, and molecule generation tasks.

Demonstrates competitive results across diverse DSL tasks.

Abstract

Large language models (LLMs) can learn to perform a wide range of natural language tasks from just a handful of in-context examples. However, for generating strings from highly structured languages (e.g., semantic parsing to complex domain-specific languages), it is challenging for the LLM to generalize from just a few exemplars. We propose \emph{grammar prompting}, a simple approach to enable LLMs to use external knowledge and domain-specific constraints, expressed through a grammar in Backus--Naur Form (BNF), during in-context learning. Grammar prompting augments each demonstration example with a specialized grammar that is minimally sufficient for generating the particular output example, where the specialized grammar is a subset of the full DSL grammar. For inference, the LLM first predicts a BNF grammar given a test input, and then generates the output according to the rules of the grammar. Experiments demonstrate that grammar prompting can enable LLMs to perform competitively on a diverse set of DSL generation tasks, including semantic parsing (SMCalFlow, Overnight, GeoQuery), PDDL planning, and SMILES-based molecule generation.