Towards Advancing Code Generation with Large Language Models: A Research Roadmap

TL;DR

This paper presents a comprehensive research roadmap for advancing code generation using large language models, analyzing current challenges, proposing a structured framework, and offering actionable recommendations to improve system reliability and usability.

Contribution

It introduces a six-layer framework for code generation processes and provides a systematic analysis of challenges and solutions for LLM-based code generation.

Findings

Identified key challenges in LLM-based code generation.

Proposed a structured six-layer framework for process categorization.

Offered practical guidelines to enhance system robustness and usability.

Abstract

Recently, we have witnessed the rapid development of large language models, which have demonstrated excellent capabilities in the downstream task of code generation. However, despite their potential, LLM-based code generation still faces numerous technical and evaluation challenges, particularly when embedded in real-world development. In this paper, we present our vision for current research directions, and provide an in-depth analysis of existing studies on this task. We propose a six-layer vision framework that categorizes code generation process into distinct phases, namely Input Phase, Orchestration Phase, Development Phase, and Validation Phase. Additionally, we outline our vision workflow, which reflects on the currently prevalent frameworks. We systematically analyse the challenges faced by large language models, including those LLM-based agent frameworks, in code generation tasks. With these, we offer various perspectives and actionable recommendations in this area. Our aim is to provide guidelines for improving the reliability, robustness and usability of LLM-based code generation systems. Ultimately, this work seeks to address persistent challenges and to provide practical suggestions for a more pragmatic LLM-based solution for future code generation endeavors.