Renaissance of Literate Programming in the Era of LLMs: Enhancing LLM-Based Code Generation in Large-Scale Projects

TL;DR

This paper explores how integrating Literate Programming principles with Large Language Models can improve code generation and comprehension in large-scale software projects, addressing current challenges in managing complex dependencies.

Contribution

It introduces Interoperable Literate Programming (ILP) and a prompt engineering method to better utilize LLMs for large-scale project development and code generation.

Findings

ILP enhances LLM performance in large-scale projects

Proposed prompt engineering improves LLM involvement in code tasks

LLMs show improved code generation on RepoBench benchmark

Abstract

Large Language Models (LLMs) have helped programmers increase efficiency through code generation, comprehension, and repair. However, their application to large-scale projects remains challenging due to complex interdependencies and the extensive size of modern codebases. Although Knuth's concept of Literate Programming (LP) combines code and natural language to convey logic and intent, its potential for enhancing relationships in large projects has not been fully explored. In this study, we introduce the idea of Interoperable LP (ILP), which leverages literate programming principles to enhance the development of both small-scale documents and large-scale projects with LLMs. We investigate how LLMs perform under ILP-style instructions for both document-oriented tasks and entire projects. Recognizing that many researchers rely on well-structured templates to guide LLMs, we propose a concise prompt engineering method to write LP documents so LLMs can better be involved in code generation. We also examine the capacity of various LLMs to generate Scheme and Python code on the RepoBench benchmark, illustrating the advantages of our approach. Our findings indicate that ILP with LLMs can enhance LLM-based code generation in large-scale project development.