Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit

TL;DR

This paper provides a comprehensive survey, benchmark, and toolkit for deep learning-based code intelligence, facilitating research and development in understanding and improving code analysis tools.

Contribution

It offers a detailed literature review, benchmarks state-of-the-art models, and releases an open-source toolkit and dataset for rapid prototyping and evaluation.

Findings

Benchmark results of neural models for code tasks

Open-source toolkit for code intelligence research

Public dataset for model evaluation

Abstract

Code intelligence leverages machine learning techniques to extract knowledge from extensive code corpora, with the aim of developing intelligent tools to improve the quality and productivity of computer programming. Currently, there is already a thriving research community focusing on code intelligence, with efforts ranging from software engineering, machine learning, data mining, natural language processing, and programming languages. In this paper, we conduct a comprehensive literature review on deep learning for code intelligence, from the aspects of code representation learning, deep learning techniques, and application tasks. We also benchmark several state-of-the-art neural models for code intelligence, and provide an open-source toolkit tailored for the rapid prototyping of deep-learning-based code intelligence models. In particular, we inspect the existing code intelligence models under the basis of code representation learning, and provide a comprehensive overview to enhance comprehension of the present state of code intelligence. Furthermore, we publicly release the source code and data resources to provide the community with a ready-to-use benchmark, which can facilitate the evaluation and comparison of existing and future code intelligence models (https://xcodemind.github.io). At last, we also point out several challenging and promising directions for future research.