Which Features are Learned by CodeBert: An Empirical Study of the BERT-based Source Code Representation Learning

TL;DR

This paper empirically investigates what features CodeBERT learns for source code representation, revealing its reliance on variable and function names rather than understanding code logic.

Contribution

It provides an empirical analysis showing that CodeBERT primarily captures identifier information, highlighting limitations in understanding source code logic.

Findings

CodeBERT heavily relies on variable and function names.

Current models do not effectively understand source code logic.

Insights suggest directions for improving code representation learning.

Abstract

The Bidirectional Encoder Representations from Transformers (BERT) were proposed in the natural language process (NLP) and shows promising results. Recently researchers applied the BERT to source-code representation learning and reported some good news on several downstream tasks. However, in this paper, we illustrated that current methods cannot effectively understand the logic of source codes. The representation of source code heavily relies on the programmer-defined variable and function names. We design and implement a set of experiments to demonstrate our conjecture and provide some insights for future works.