DeepCodeProbe: Towards Understanding What Models Trained on Code Learn

TL;DR

DeepCodeProbe is a probing method that analyzes how machine learning models trained on code learn syntax and representations, revealing their capabilities, limitations, and patterns, with practical recommendations for improving interpretability and performance.

Contribution

We introduce DeepCodeProbe, a novel probing approach to understand syntax learning in ML models for code, and provide insights and best practices for training more interpretable models.

Findings

Small models capture some syntactic abstractions

Larger models improve syntax learning but risk overfitting

Models learn specific code patterns from training data

Abstract

Machine learning models trained on code and related artifacts offer valuable support for software maintenance but suffer from interpretability issues due to their complex internal variables. These concerns are particularly significant in safety-critical applications where the models' decision-making processes must be reliable. The specific features and representations learned by these models remain unclear, adding to the hesitancy in adopting them widely. To address these challenges, we introduce DeepCodeProbe, a probing approach that examines the syntax and representation learning abilities of ML models designed for software maintenance tasks. Our study applies DeepCodeProbe to state-of-the-art models for code clone detection, code summarization, and comment generation. Findings reveal that while small models capture abstract syntactic representations, their ability to fully grasp programming language syntax is limited. Increasing model capacity improves syntax learning but introduces trade-offs such as increased training time and overfitting. DeepCodeProbe also identifies specific code patterns the models learn from their training data. Additionally, we provide best practices for training models on code to enhance performance and interpretability, supported by an open-source replication package for broader application of DeepCodeProbe in interpreting other code-related models.