ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS

TL;DR

ArkEval introduces the first comprehensive benchmark for automated code repair in ArkTS, leveraging large language models and a novel test generation approach to evaluate and advance repair capabilities in this emerging mobile ecosystem.

Contribution

This paper presents ArkEval, a new benchmark and evaluation framework specifically for ArkTS automated repair, including issue mining, test generation, and LLM evaluation.

Findings

LLMs show promising but limited repair capabilities for ArkTS.

The benchmark reveals current LLM limitations in low-resource languages.

ArkEval facilitates future research in automated repair for ArkTS.

Abstract

Large language models have transformed code generation, enabling unprecedented automation in software development. As mobile ecosystems evolve, HarmonyOS has emerged as a critical platform requiring robust development tools. Software development for the HarmonyOS ecosystem relies heavily on ArkTS, a statically typed extension of TypeScript. Despite its growing importance, the ecosystem lacks robust tools for automated code repair, primarily due to the absence of a high-quality benchmark for evaluation. To address this gap, we present ArkEval, a unified framework for ArkTS automated repair workflow evaluation and benchmark construction. It provides the first comprehensive benchmark specifically designed for ArkTS automated program repair. We constructed this benchmark by mining issues from a large-scale official Huawei repository containing over 400 independent ArkTS applications. Through a rigorous multi-stage filtering process, we curated 502 reproducible issues. To ensure testability, we employed a novel LLM-based test generation and voting mechanism involving Claude and other models. Furthermore, we standardized problem statements to facilitate fair evaluation. Finally, we evaluated four state-of-the-art Large Language Models (LLMs) on our benchmark using a retrieval-augmented repair workflow. Our results highlight the current capabilities and limitations of LLMs in repairing ArkTS code, paving the way for future research in this low-resource language domain.