Exploring Direct Instruction and Summary-Mediated Prompting in LLM-Assisted Code Modification

TL;DR

This study compares two prompting strategies—direct instruction and summary-mediated—for using large language models to assist in code modification, revealing their respective advantages and influencing factors in developer workflows.

Contribution

It introduces and empirically evaluates two novel prompting strategies for LLM-assisted code modification, highlighting their trade-offs and developer preferences.

Findings

Direct instruction prompting is more flexible and easier to specify.

Summary-mediated prompting enhances comprehension and control.

Developer choice depends on task goals and context.

Abstract

This paper presents a study of using large language models (LLMs) in modifying existing code. While LLMs for generating code have been widely studied, their role in code modification remains less understood. Although "prompting" serves as the primary interface for developers to communicate intents to LLMs, constructing effective prompts for code modification introduces challenges different from generation. Prior work suggests that natural language summaries may help scaffold this process, yet such approaches have been validated primarily in narrow domains like SQL rewriting. This study investigates two prompting strategies for LLM-assisted code modification: Direct Instruction Prompting, where developers describe changes explicitly in free-form language, and Summary-Mediated Prompting, where changes are made by editing the generated summaries of the code. We conducted an exploratory study with 15 developers who completed modification tasks using both techniques across multiple scenarios. Our findings suggest that developers followed an iterative workflow: understanding the code, localizing the edit, and validating outputs through execution or semantic reasoning. Each prompting strategy presented trade-offs: direct instruction prompting was more flexible and easier to specify, while summary-mediated prompting supported comprehension, prompt scaffolding, and control. Developers' choice of strategy was shaped by task goals and context, including urgency, maintainability, learning intent, and code familiarity. These findings highlight the need for more usable prompt interactions, including adjustable summary granularity, reliable summary-code traceability, and consistency in generated summaries.