Chasing Shadows: Pitfalls in LLM Security Research

TL;DR

This paper identifies nine common pitfalls in LLM security research that threaten validity, assesses their prevalence in recent papers, and provides guidelines to improve research rigor and reproducibility.

Contribution

It introduces a comprehensive list of pitfalls specific to LLM security research and evaluates their impact across recent publications, offering practical recommendations.

Findings

Every paper contained at least one pitfall

Most pitfalls are unrecognized in current research

Empirical case studies show pitfalls can mislead evaluation

Abstract

Large language models (LLMs) are increasingly prevalent in security research. Their unique characteristics, however, introduce challenges that undermine established paradigms of reproducibility, rigor, and evaluation. Prior work has identified common pitfalls in traditional machine learning research, but these studies predate the advent of LLMs. In this paper, we identify nine common pitfalls that have become (more) relevant with the emergence of LLMs and that can compromise the validity of research involving them. These pitfalls span the entire computation process, from data collection, pre-training, and fine-tuning to prompting and evaluation. We assess the prevalence of these pitfalls across all 72 peer-reviewed papers published at leading Security and Software Engineering venues between 2023 and 2024. We find that every paper contains at least one pitfall, and each pitfall appears in multiple papers. Yet only 15.7% of the present pitfalls were explicitly discussed, suggesting that the majority remain unrecognized. To understand their practical impact, we conduct four empirical case studies showing how individual pitfalls can mislead evaluation, inflate performance, or impair reproducibility. Based on our findings, we offer actionable guidelines to support the community in future work.