Guidelines for Empirical Studies in Software Engineering involving Large Language Models

TL;DR

This paper provides a taxonomy of study types and comprehensive guidelines for conducting and reporting empirical research involving Large Language Models in software engineering to enhance reproducibility.

Contribution

It introduces a structured taxonomy of study types and presents detailed, context-aware guidelines for designing, reporting, and validating LLM-based SE studies.

Findings

Established a taxonomy of seven study types involving LLMs in SE.

Developed eight guidelines for transparent and reproducible LLM studies.

Created an online living resource for the community to adopt and evolve.

Abstract

Large Language Models (LLMs) are widely used in software engineering (SE) research and practice, yet their non-determinism, opaque training data, and rapidly evolving models threaten the reproducibility and replicability of empirical studies. We address this challenge through a collaborative effort of 22 researchers, presenting a taxonomy of seven study types that organizes how LLMs are used in SE research, together with eight guidelines for designing and reporting such studies. Each guideline distinguishes requirements (must) from recommended practices (should) and is contextualized by the study types it applies to. Our guidelines recommend that researchers: (1) declare LLM usage and role; (2) report model versions, configurations, and customizations; (3) document the tool architecture beyond the model; (4) disclose prompts, their development, and interaction logs; (5) validate LLM outputs with humans; (6) include an open LLM as a baseline; (7) use suitable baselines, benchmarks, and metrics; and (8) articulate limitations and mitigations. We complement the guidelines with an applicability matrix mapping guidelines to study types and a reporting checklist for authors and reviewers. We maintain the study types and guidelines online as a living resource for the community to use and shape (llm-guidelines$.$org).