Seamlessly Integrating Tree-Based Positional Embeddings into Transformer Models for Source Code Representation

TL;DR

This paper introduces a tree-based positional embedding method that explicitly encodes hierarchical AST structures into transformer models, significantly improving source code understanding tasks like MLM and clone detection.

Contribution

It presents a novel hierarchical embedding approach integrated into transformers, enhancing code representations by capturing intrinsic AST structures.

Findings

Outperforms baseline in loss and accuracy

Achieves higher F1, precision, and recall in clone detection

Demonstrates the effectiveness of hierarchical embeddings in code models

Abstract

Transformer-based models have demonstrated significant success in various source code representation tasks. Nonetheless, traditional positional embeddings employed by these models inadequately capture the hierarchical structure intrinsic to source code, typically represented as Abstract Syntax Trees (ASTs). To address this, we propose a novel tree-based positional embedding approach that explicitly encodes hierarchical relationships derived from ASTs, including node depth and sibling indices. These hierarchical embeddings are integrated into the transformer architecture, specifically enhancing the CodeBERTa model. We thoroughly evaluate our proposed model through masked language modeling (MLM) pretraining and clone detection fine-tuning tasks. Experimental results indicate that our Tree-Enhanced CodeBERTa consistently surpasses the baseline model in terms of loss, accuracy, F1 score, precision, and recall, emphasizing the importance of incorporating explicit structural information into transformer-based representations of source code.