Guiding Large Language Models to Generate Computer-Parsable Content

TL;DR

This paper introduces a coroutine-based method to guide Large Language Models in generating structured, computer-parsable content adhering to specific grammar constraints, improving accuracy and stability without fine-tuning.

Contribution

It presents YieldLang, a novel framework for constrained content generation using CFG-guided decoding, enhancing LLM output quality for formal language tasks.

Findings

Error rates exceed 95% for long DSLs in GPT-2 and Gemma.

Our approach improves accuracy by up to 11.6 times over benchmarks.

LLMs require only 16.5% of samples to generate effective JSON content.

Abstract

We propose a method to guide Large Language Models (LLMs) in generating structured content adhering to specific conventions without fine-tuning. By utilizing coroutine-based content generation constraints through a pre-agreed context-free grammar (CFG), LLMs are directed during decoding to produce formal language compliant outputs. This enhances stability and consistency in generating target data structures, types, or instructions, reducing application development complexities. Experimentally, error rates of GPT-2 and Gemma exceed 95% for DSLs longer than 36 and 282 tokens, respectively. We introduce YieldLang, a coroutine-based DSL generation framework, and evaluate it with LLMs on various tasks including JSON and Mermaid flowchart generation. Compared to benchmarks, our approach improves accuracy by 1.09 to 11.6 times, with LLMs requiring only about 16.5% of the samples to generate JSON effectively. This enhances usability of LLM-generated content for computer programs.