Anka: A Domain-Specific Language for Reliable LLM Code Generation

TL;DR

Anka is a new domain-specific language designed for data transformation tasks that significantly improves the accuracy of large language models in generating complex, multi-step code compared to general-purpose languages like Python.

Contribution

The paper introduces Anka, a DSL that reduces ambiguity in code generation, enabling LLMs to learn and generate reliable code without prior training on the language.

Findings

Anka achieves 99.9% parse success and 95.8% task accuracy without prior training.

Anka outperforms Python by 40 percentage points on multi-step pipeline tasks.

Cross-model validation confirms Anka's effectiveness across different LLMs.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in code generation, yet they exhibit systematic errors on complex, multi-step programming tasks. We hypothesize that these errors stem from the flexibility of general-purpose languages, which permits multiple valid approaches and requires implicit state management. To test this hypothesis, we introduce Anka, a domain-specific language (DSL) for data transformation pipelines designed with explicit, constrained syntax that reduces ambiguity in code generation. Despite having zero prior training exposure to Anka, Claude 3.5 Haiku achieves 99.9% parse success and 95.8% overall task accuracy across 100 benchmark problems. Critically, Anka demonstrates a 40 percentage point accuracy advantage over Python on multi-step pipeline tasks (100% vs. 60%), where Python's flexible syntax leads to frequent errors in operation sequencing and variable management. Cross-model validation with GPT-4o-mini confirms this advantage (+26.7 percentage points on multi-step tasks). Our results demonstrate that: (1) LLMs can learn novel DSLs entirely from in-context prompts, achieving near-native accuracy; (2) constrained syntax significantly reduces errors on complex tasks; and (3) domain-specific languages purposefully designed for LLM generation can outperform general-purpose languages on which the LLM has extensive training. We release the complete language implementation, benchmark suite, and evaluation framework to facilitate further research.