Probing Model Signal-Awareness via Prediction-Preserving Input Minimization

TL;DR

This paper introduces a method called prediction-preserving input minimization (P2IM) to evaluate how well AI models for source code vulnerability detection rely on true vulnerability signals, revealing potential over-reliance on dataset noise.

Contribution

The paper proposes P2IM and a new metric, Signal-aware Recall (SAR), to systematically assess model signal-awareness in source code understanding tasks.

Findings

Models show a significant drop in Signal-aware Recall when minimal code snippets are used.

P2IM uncovers reliance on dataset noise rather than true vulnerability signals.

SAR provides a data-driven measure of a model's signal-awareness.

Abstract

This work explores the signal awareness of AI models for source code understanding. Using a software vulnerability detection use case, we evaluate the models' ability to capture the correct vulnerability signals to produce their predictions. Our prediction-preserving input minimization (P2IM) approach systematically reduces the original source code to a minimal snippet which a model needs to maintain its prediction. The model's reliance on incorrect signals is then uncovered when the vulnerability in the original code is missing in the minimal snippet, both of which the model however predicts as being vulnerable. We measure the signal awareness of models using a new metric we propose- Signal-aware Recall (SAR). We apply P2IM on three different neural network architectures across multiple datasets. The results show a sharp drop in the model's Recall from the high 90s to sub-60s with the new metric, highlighting that the models are presumably picking up a lot of noise or dataset nuances while learning their vulnerability detection logic. Although the drop in model performance may be perceived as an adversarial attack, but this isn't P2IM's objective. The idea is rather to uncover the signal-awareness of a black-box model in a data-driven manner via controlled queries. SAR's purpose is to measure the impact of task-agnostic model training, and not to suggest a shortcoming in the Recall metric. The expectation, in fact, is for SAR to match Recall in the ideal scenario where the model truly captures task-specific signals.