JavaBERT: Training a transformer-based model for the Java programming language

TL;DR

This paper introduces JavaBERT, a transformer-based model trained on Java code, demonstrating high accuracy in masked language modeling and highlighting its potential for improving software engineering tools.

Contribution

The paper presents a novel pipeline for training transformer models on software code, specifically Java, and shows promising results for code processing tasks.

Findings

JavaBERT achieves high accuracy on masked language modeling tasks

The model demonstrates potential for enhancing software engineering tools

A new data retrieval pipeline for training code-based models

Abstract

Code quality is and will be a crucial factor while developing new software code, requiring appropriate tools to ensure functional and reliable code. Machine learning techniques are still rarely used for software engineering tools, missing out the potential benefits of its application. Natural language processing has shown the potential to process text data regarding a variety of tasks. We argue, that such models can also show similar benefits for software code processing. In this paper, we investigate how models used for natural language processing can be trained upon software code. We introduce a data retrieval pipeline for software code and train a model upon Java software code. The resulting model, JavaBERT, shows a high accuracy on the masked language modeling task showing its potential for software engineering tools.