Reporting LLM Prompting in Automated Software Engineering: A Guideline Based on Current Practices and Expectations

TL;DR

This paper analyzes current reporting practices of LLM prompting in software engineering research, highlighting gaps and proposing a structured guideline to enhance transparency, reproducibility, and alignment with reviewer expectations.

Contribution

It provides the first systematic analysis of prompt reporting in SE papers and introduces a comprehensive guideline to standardize and improve prompt documentation.

Findings

Significant gaps between current practices and reviewer expectations.

Common omissions include prompt versioning and justification.

The guideline aims to improve transparency and reproducibility.

Abstract

Large Language Models, particularly decoder-only generative models such as GPT, are increasingly used to automate Software Engineering tasks. These models are primarily guided through natural language prompts, making prompt engineering a critical factor in system performance and behavior. Despite their growing role in SE research, prompt-related decisions are rarely documented in a systematic or transparent manner, hindering reproducibility and comparability across studies. To address this gap, we conducted a two-phase empirical study. First, we analyzed nearly 300 papers published at the top-3 SE conferences since 2022 to assess how prompt design, testing, and optimization are currently reported. Second, we surveyed 105 program committee members from these conferences to capture their expectations for prompt reporting in LLM-driven research. Based on the findings, we derived a structured guideline that distinguishes essential, desirable, and exceptional reporting elements. Our results reveal significant misalignment between current practices and reviewer expectations, particularly regarding version disclosure, prompt justification, and threats to validity. We present our guideline as a step toward improving transparency, reproducibility, and methodological rigor in LLM-based SE research.