Syntax-Guided Program Reduction for Understanding Neural Code Intelligence Models

TL;DR

This paper introduces a syntax-guided program reduction method that enhances the interpretability of neural code models by considering programming language grammar, resulting in faster reductions and more concise key features for understanding and adversarial testing.

Contribution

It presents a novel syntax-aware reduction technique that improves transparency and efficiency in analyzing neural code intelligence models.

Findings

Faster reduction process compared to syntax-unaware methods

Produces smaller sets of key tokens in reduced programs

Key tokens can generate adversarial examples for up to 65% of inputs

Abstract

Neural code intelligence (CI) models are opaque black-boxes and offer little insight on the features they use in making predictions. This opacity may lead to distrust in their prediction and hamper their wider adoption in safety-critical applications. Recently, input program reduction techniques have been proposed to identify key features in the input programs to improve the transparency of CI models. However, this approach is syntax-unaware and does not consider the grammar of the programming language. In this paper, we apply a syntax-guided program reduction technique that considers the grammar of the input programs during reduction. Our experiments on multiple models across different types of input programs show that the syntax-guided program reduction technique is faster and provides smaller sets of key tokens in reduced programs. We also show that the key tokens could be used in generating adversarial examples for up to 65% of the input programs.