On the Difficulty of Selecting Few-Shot Examples for Effective LLM-based Vulnerability Detection

TL;DR

This paper investigates how the selection of few-shot examples influences the effectiveness of large language models in detecting code vulnerabilities, revealing that selection strategies improve results for some languages but not others.

Contribution

It introduces and evaluates two criteria for selecting few-shot examples in vulnerability detection, highlighting their impact across multiple programming languages.

Findings

Selection improves Python and JavaScript vulnerability detection.

Limited impact of selection on C and C++ programs.

More advanced methods may be needed for C/C++.

Abstract

Large language models (LLMs) have demonstrated impressive capabilities across a wide range of coding tasks, including summarization, translation, completion, and code generation. Despite these advances, detecting code vulnerabilities remains a challenging problem for LLMs. In-context learning (ICL) has emerged as an effective mechanism for improving model performance by providing a small number of labeled examples within the prompt. Prior work has shown, however, that the effectiveness of ICL depends critically on how these few-shot examples are selected. In this paper, we study two intuitive criteria for selecting few-shot examples for ICL in the context of code vulnerability detection. The first criterion leverages model behavior by prioritizing samples on which the LLM consistently makes mistakes, motivated by the intuition that such samples can expose and correct systematic model weaknesses. The second criterion selects examples based on semantic similarity to the query program, using k-nearest-neighbor retrieval to identify relevant contexts. We conduct extensive evaluations using open-source LLMs and datasets spanning multiple programming languages. Our results show that for Python and JavaScript, careful selection of few-shot examples can lead to measurable performance improvements in vulnerability detection. In contrast, for C and C++ programs, few-shot example selection has limited impact, suggesting that more powerful but also more expensive approaches, such as re-training or fine-tuning, may be required to substantially improve model performance.