Code quality assessment using transformers

TL;DR

This paper explores using transformer-based models, specifically CodeBERT, to automatically evaluate subjective code quality attributes like readability and maintainability in Java code, which are difficult for traditional automated methods.

Contribution

It demonstrates that transformer models with task-specific pre-training can effectively predict code quality, outperforming other techniques on a new dataset.

Findings

Transformers can predict code quality to some extent.

Task-adapted pre-training improves model performance.

Saliency maps help interpret model predictions.

Abstract

Automatically evaluate the correctness of programming assignments is rather straightforward using unit and integration tests. However, programming tasks can be solved in multiple ways, many of which, although correct, are inelegant. For instance, excessive branching, poor naming or repetitiveness make the code hard to understand and maintain. These subjective qualities of code are hard to automatically assess using current techniques. In this work we investigate the use of CodeBERT to automatically assign quality score to Java code. We experiment with different models and training paradigms. We explore the accuracy of the models on a novel dataset for code quality assessment. Finally, we assess the quality of the predictions using saliency maps. We find that code quality to some extent is predictable and that transformer based models using task adapted pre-training can solve the task more efficiently than other techniques.