CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language

TL;DR

CangjieBench introduces a contamination-free benchmark for evaluating large language models on a low-resource, general-purpose programming language, revealing insights into model performance and transfer phenomena.

Contribution

The paper presents CangjieBench, a novel benchmark for low-resource general-purpose language Cangjie, including a comprehensive evaluation of LLMs across multiple generation settings.

Findings

Syntax-Constrained Generation balances accuracy and cost

Agent models achieve highest accuracy but with high token use

Code-to-Code translation often underperforms Text-to-Code

Abstract

Large Language Models excel in high-resource programming languages but struggle with low-resource ones. Existing research related to low-resource programming languages primarily focuses on Domain-Specific Languages (DSLs), leaving general-purpose languages that suffer from data scarcity underexplored. To address this gap, we introduce CangjieBench, a contamination-free benchmark for Cangjie, a representative low-resource general-purpose language. The benchmark comprises 248 high-quality samples manually translated from HumanEval and ClassEval, covering both Text-to-Code and Code-to-Code tasks. We conduct a systematic evaluation of diverse LLMs under four settings: Direct Generation, Syntax-Constrained Generation, Retrieval-Augmented Generation (RAG), and Agent. Experiments reveal that Direct Generation performs poorly, whereas Syntax-Constrained Generation offers the best trade-off between accuracy and computational cost. Agent achieve state-of-the-art accuracy but incur high token consumption. Furthermore, we observe that Code-to-Code translation often underperforms Text-to-Code generation, suggesting a negative transfer phenomenon where models overfit to the source language patterns. We hope that our work will offer valuable insights into LLM generalization to unseen and low-resource programming languages. Our code and data are available at https://github.com/cjhCoder7/CangjieBench.