Understanding Code Semantics: An Evaluation of Transformer Models in Summarization

TL;DR

This study evaluates whether transformer models genuinely understand code semantics in summarization tasks by testing their robustness against semantic alterations and adversarial code snippets across multiple programming languages.

Contribution

The paper introduces a comprehensive empirical evaluation of transformer models' understanding of code semantics, including adversarial code modifications, across Python, Javascript, and Java.

Findings

Models rely heavily on textual cues rather than true semantic understanding.

Adversarial code snippets significantly reduce model performance.

Transformer models show limited robustness to semantic alterations.

Abstract

This paper delves into the intricacies of code summarization using advanced transformer-based language models. Through empirical studies, we evaluate the efficacy of code summarization by altering function and variable names to explore whether models truly understand code semantics or merely rely on textual cues. We have also introduced adversaries like dead code and commented code across three programming languages (Python, Javascript, and Java) to further scrutinize the model's understanding. Ultimately, our research aims to offer valuable insights into the inner workings of transformer-based LMs, enhancing their ability to understand code and contributing to more efficient software development practices and maintenance workflows.