On the Security of Research Artifacts

TL;DR

This paper highlights security risks in research artifacts, analyzing 509 artifacts from top security venues, and introduces SAFE, a framework for automated security risk assessment in artifact evaluation.

Contribution

It presents a taxonomy for security assessment, performs static analysis to identify real risks, and develops SAFE, an autonomous framework with high accuracy for security risk detection.

Findings

41.60% of findings may pose security concerns in practice

SAFE achieves 84.80% accuracy in risk classification

Source code for SAFE is publicly available at the provided GitHub link

Abstract

Research artifacts are widely shared to support reproducibility, and artifact evaluation (AE) has become common at many leading conferences. However, AE mainly checks whether artifacts work as claimed and can be reproduced. It largely overlooks potential security risks. Since these artifacts are publicly released and reused, they may unintentionally create opportunities for misuse and raise concerns about safe and responsible sharing. We study 509 research artifacts from top-tier security venues and find that many contain insecure code patterns that may introduce potential attack vectors. We propose a taxonomy for context-aware security assessment to enable structured analysis of such risks. We perform static analysis and examine the resulting findings, filtering false positives and identifying real security risks. Our analysis shows that 41.60% of the prevalent findings may pose security concerns under practical usage. To support scalable analysis, we introduce SAFE (Security-Aware Framework for Artifact Evaluation), a first step toward an autonomous framework that analyzes tool-reported findings by considering code semantics, execution context, and practical exploitability. SAFE achieves 84.80% accuracy and 84.63% F1-score in distinguishing security and non-security risks. Overall, our results show that security is also important in AE for promoting safe and responsible research sharing. The source code is available at: https://github.com/nanda-rani/SAFE