Strategic Optimization and Challenges of Large Language Models in Object-Oriented Programming

TL;DR

This paper investigates how different levels of contextual information in prompts affect large language models' performance in object-oriented code generation, emphasizing cost-effectiveness and model selection based on task coupling.

Contribution

It introduces the concept of prompt-token cost-effectiveness and analyzes how prompt content and model choice impact code generation quality and efficiency.

Findings

Method invocation details improve cost-effectiveness.

Larger models do not always outperform smaller ones.

Task coupling influences model performance and suitability.

Abstract

In the area of code generation research, the emphasis has transitioned from crafting individual functions to developing class-level method code that integrates contextual information. This shift has brought several benchmarks such as ClassEval and CoderEval, which consider class-level contexts. Nevertheless, the influence of specific contextual factors at the method level remains less explored. This research focused on method-level code generation within the Object-Oriented Programming (OOP) framework. Based on CoderEval, we devised experiments that varied the extent of contextual information in the prompts, ranging from method-specific to project-wide details. We introduced the innovative metric of "Prompt-Token Cost-Effectiveness" to evaluate the economic viability of incorporating additional contextual layers. Our findings indicate that prompts enriched with method invocation details yield the highest cost-effectiveness. Additionally, our study revealed disparities among Large Language Models (LLMs) regarding error type distributions and the level of assistance they provide to developers. Notably, larger LLMs do not invariably perform better. We also observed that tasks with higher degrees of coupling present more substantial challenges, suggesting that the choice of LLM should be tailored to the task's coupling degree. For example, GPT-4 exhibited improved performance in low-coupling scenarios, whereas GPT-3.5 seemed better suited for tasks with high coupling. By meticulously curating prompt content and selecting the appropriate LLM, developers can optimize code quality while maximizing cost-efficiency during the development process.