On the Impact of Language Selection for Training and Evaluating Programming Language Models

TL;DR

This paper investigates how the choice of programming languages affects the training and evaluation of language models, revealing language similarities and differences that impact model performance.

Contribution

It introduces a method to measure programming language similarity using CodeBERT, guiding better language selection for training multilingual models.

Findings

Languages like C++, Python, Java are similar in representation.

Mathematica and R show significant dissimilarity in token representations.

Diverse language selection can improve model robustness and performance.

Abstract

The recent advancements in Transformer-based Language Models have demonstrated significant potential in enhancing the multilingual capabilities of these models. The remarkable progress made in this domain not only applies to natural language tasks but also extends to the domain of programming languages. Despite the ability of these models to learn from multiple languages, evaluations typically focus on particular combinations of the same languages. In this study, we evaluate the similarity of programming languages by analyzing their representations using a CodeBERT-based model. Our experiments reveal that token representation in languages such as C++, Python, and Java exhibit proximity to one another, whereas the same tokens in languages such as Mathematica and R display significant dissimilarity. Our findings suggest that this phenomenon can potentially result in performance challenges when dealing with diverse languages. Thus, we recommend using our similarity measure to select a diverse set of programming languages when training and evaluating future models.