Exploring the Naturalness of Buggy Code with Recurrent Neural Networks

TL;DR

This paper investigates the use of LSTM recurrent neural networks to model source code and detect buggy lines by measuring entropy, showing slight improvements over traditional n-gram models.

Contribution

It introduces a novel application of LSTM language models for bug detection in source code, outperforming n-gram models in classification accuracy.

Findings

LSTM models slightly outperform n-gram models in bug detection.

Entropy-based classification effectively identifies buggy lines.

LSTM captures longer-range dependencies in code.

Abstract

Statistical language models are powerful tools which have been used for many tasks within natural language processing. Recently, they have been used for other sequential data such as source code.(Ray et al., 2015) showed that it is possible train an n-gram source code language mode, and use it to predict buggy lines in code by determining "unnatural" lines via entropy with respect to the language model. In this work, we propose using a more advanced language modeling technique, Long Short-term Memory recurrent neural networks, to model source code and classify buggy lines based on entropy. We show that our method slightly outperforms an n-gram model in the buggy line classification task using AUC.