How Natural Language Proficiency Shapes GenAI Code for Software Engineering Tasks

TL;DR

This study explores how English language proficiency in prompts influences the quality and correctness of code generated by large language models in software engineering, revealing that higher proficiency leads to more accurate outputs.

Contribution

It systematically analyzes the impact of natural language proficiency on code generation quality, highlighting its significance beyond prompt structure in LLM-based software engineering.

Findings

Higher language proficiency prompts improve code correctness across models

LLMs tend to default to an intermediate (B2) English level

Proficiency impacts code quality more than prompt structure alone

Abstract

With the widespread adoption of Foundation Model (FM)-powered tools in software engineering, the natural language prompt has become a critical interface between developers and Large Language Models (LLMs). While much research has focused on prompt structure, the natural language proficiency is an underexplored factor that can influence the quality of generated code. This paper investigates whether the English language proficiency itself independent of the prompting technique affects the proficiency and correctness of code generated by LLMs. Using the HumanEval dataset, we systematically varied the English proficiency of prompts from basic to advanced for 164 programming tasks and measured the resulting code proficiency and correctness. Our findings show that LLMs default to an intermediate (B2) natural language level. While the effect on the resulting code proficiency was model-dependent, we found that higher-proficiency prompts consistently yielded more correct code across all models. These results demonstrate that natural language proficiency is a key lever for controlling code generation, helping developers tailor AI output and improve the reliability of solutions.