Supporting Artifact Evaluation with LLMs: A Study with Published Security Research Papers

TL;DR

This paper explores how Large Language Models can enhance Artifact Evaluation in cybersecurity research by automating reproducibility assessments, environment setup, and pitfall detection, thereby improving efficiency and artifact quality.

Contribution

It introduces a novel LLM-based toolkit that automates key AE tasks, achieving over 72% accuracy in reproducibility ratings and high accuracy in pitfall detection.

Findings

Reproducibility rating accuracy exceeds 72%.

Automated environment setup for 28% of artifacts.

High accuracy (F1 > 92%) in detecting common pitfalls.

Abstract

Artifact Evaluation (AE) is essential for ensuring the transparency and reliability of research, closing the gap between exploratory work and real-world deployment is particularly important in cybersecurity, particularly in IoT and CPSs, where large-scale, heterogeneous, and privacy-sensitive data meet safety-critical actuation. Yet, manual reproducibility checks are time-consuming and do not scale with growing submission volumes. In this work, we demonstrate that Large Language Models (LLMs) can provide powerful support for AE tasks: (i) text-based reproducibility rating, (ii) autonomous sandboxed execution environment preparation, and (iii) assessment of methodological pitfalls. Our reproducibility-assessment toolkit yields an accuracy of over 72% and autonomously sets up execution environments for 28% of runnable cybersecurity artifacts. Our automated pitfall assessment detects seven prevalent pitfalls with high accuracy ($F_1$ > 92%). Hence, the toolkit significantly reduces reviewer effort and, when integrated into established AE processes, could incentivize authors to submit higher-quality and more reproducible artifacts. IoT, CPS, and cybersecurity conferences and workshops may integrate the toolkit into their peer-review processes to support reviewers' decisions on awarding artifact badges, improving the overall sustainability of the process.