From Reviewers' Lens: Understanding Bug Bounty Report Invalid Reasons with LLMs

TL;DR

This paper investigates the use of large language models to identify invalid bug bounty reports, proposing a retrieval-augmented approach to improve detection accuracy and understand reviewer decision factors.

Contribution

It introduces a taxonomy of rejection reasons and a RAG framework to enhance invalid report classification and interpretability in bug bounty platforms.

Findings

LLMs like GPT-5 and RoBERTa achieve high accuracy but often over-accept invalid reports.

The RAG framework improves classification consistency and reduces bias.

Reviewer reputation influences decision outcomes in borderline cases.

Abstract

Bug bounty platforms (e.g., HackerOne, BugCrowd) leverage crowd-sourced vulnerability discovery to improve continuous coverage, reduce the cost of discovery, and serve as an integral complement to internal red teams. With the rise of AI-generated bug reports, little work exists to help bug hunters understand why these reports are labeled as invalid. To improve report quality and reduce reviewers' burden, it is critical to predict invalid reports and interpret invalid reasons. In this work, we conduct an empirical study with the purpose of helping bug hunters understand the validity of reports. We collect a dataset of 9,942 disclosed bug bounty reports, including 1,400 invalid reports, and evaluate whether state-of-the-art large language models can identify invalid reports. While models such as GPT-5, DeepSeek, and a fine-tuned RoBERTa achieve strong overall accuracy, they consistently struggle to detect invalid cases, showing a tendency to over-accept reports. To improve invalidity detection, we build a taxonomy of rejection reasons for Information Disclosure vulnerabilities and incorporate it into a retrieval-augmented generation (RAG) framework. This approach substantially improves classification consistency and reduces bias. We also examine whether reviewer decisions may be influenced by factors beyond the content of the report. Our analysis shows that reporters with higher reputations tend to receive more favorable outcomes in borderline cases, suggesting that perceived expertise can influence review judgments. Overall, our findings highlight the challenges of invalid report identification and show that combining LLMs with structured reviewer knowledge can support more transparent and consistent vulnerability report review.