Modeling Vocabulary for Big Code Machine Learning

TL;DR

This paper examines key decisions in modeling source code vocabulary for machine learning, demonstrating how these choices impact model training and performance on a large-scale corpus of open-source projects.

Contribution

It identifies critical vocabulary modeling decisions and evaluates their effects, enabling efficient training of neural language models on extensive code datasets.

Findings

Certain vocabulary choices are decisive for model training success.

Optimized vocabulary modeling enables rapid training on large code corpora.

Effective vocabulary decisions improve neural language model performance.

Abstract

When building machine learning models that operate on source code, several decisions have to be made to model source-code vocabulary. These decisions can have a large impact: some can lead to not being able to train models at all, others significantly affect performance, particularly for Neural Language Models. Yet, these decisions are not often fully described. This paper lists important modeling choices for source code vocabulary, and explores their impact on the resulting vocabulary on a large-scale corpus of 14,436 projects. We show that a subset of decisions have decisive characteristics, allowing to train accurate Neural Language Models quickly on a large corpus of 10,106 projects.