Modeling Code: Is Text All You Need?

TL;DR

This paper explores combining text-based and structured representations in code models to enhance reasoning and generative capabilities, addressing limitations of current transformer-based models in understanding code structure.

Contribution

It introduces a novel approach that integrates text and structured data modeling for code, improving reasoning and generation over existing methods.

Findings

Enhanced code understanding and generation capabilities

Improved reasoning about control and data flow in code

Combines strengths of text-based and structured modeling

Abstract

Code LLMs have become extremely popular recently for modeling source code across a variety of tasks, such as generation, translation, and summarization. However, transformer-based models are limited in their capabilities to reason through structured, analytical properties of code, such as control and data flow. Previous work has explored the modeling of these properties with structured data and graph neural networks. However, these approaches lack the generative capabilities and scale of modern LLMs. In this work, we introduce a novel approach to combine the strengths of modeling both code as text and more structured forms.