Lyra: A Benchmark for Turducken-Style Code Generation
Qingyuan Liang, Zeyu Sun, Qihao Zhu, Wenjie Zhang, Lian Yu, Yingfei, Xiong, Lu Zhang

TL;DR
Lyra introduces a new benchmark dataset for turducken-style code generation, focusing on generating imperative programs with embedded declarative languages from natural language comments, highlighting challenges and potential for real-world applications.
Contribution
This paper defines the first turducken-style code generation task and provides a dataset with annotated programs in Python and embedded SQL, along with baseline evaluations.
Findings
GPT-style models outperform others in accuracy.
AST exact matching accuracy reaches 24-25.5%.
Lyra presents a new challenge for code generation models.
Abstract
Recently, neural techniques have been used to generate source code automatically. While promising for declarative languages, these approaches achieve much poorer performance on datasets for imperative languages. Since a declarative language is typically embedded in an imperative language (i.e., the turducken-style programming) in real-world software development, the promising results on declarative languages can hardly lead to significant reduction of manual software development efforts. In this paper, we define a new code generation task: given a natural language comment, this task aims to generate a program in a base imperative language with an embedded declarative language. To our knowledge, this is the first turducken-style code generation task. For this task, we present Lyra: a dataset in Python with embedded SQL. This dataset contains 2,000 carefully annotated database…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Testing and Debugging Techniques · Software Engineering Research · Topic Modeling
MethodsAttention Is All You Need · Linear Layer · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Dense Connections · Label Smoothing · Dropout · Adam · Byte Pair Encoding · Softmax
