A new approach for encoding code and assisting code understanding

TL;DR

This paper introduces a novel code encoding paradigm inspired by diffusion techniques, enabling better global understanding and zero-shot prediction for code comprehension tasks, surpassing traditional autoregressive models.

Contribution

The paper proposes a heterogeneous image-based code encoding method inspired by diffusion models, improving code understanding and enabling zero-shot predictions, addressing limitations of autoregressive paradigms.

Findings

Achieved zero-shot prediction on 456,360 text-code pairs

Demonstrated improved global understanding of code

Proposed a new paradigm for code encoding inspired by diffusion models

Abstract

Some companies (e.g., Microsoft Research and Google DeepMind) have discovered some of the limitations of GPTs' autoregressive paradigm next-word prediction, manifested in the model's lack of planning, working memory, backtracking, and reasoning skills. GPTs rely on a local and greedy process of generating the next word, without a global understanding of the task or the output. We have confirmed the above limitations through specialized empirical studies of code comprehension. Although GPT-4 is good at producing fluent and coherent text, it cannot handle complex logic and generate new code that hasn't been seen, and it relies too much on the formatting of the prompt to generate the correct code. We propose a new paradigm for code understanding that goes beyond the next-word prediction paradigm, inspired by the successful application of diffusion techniques to image generation (Dalle-2, Sora) and protein structure generation (AlphaFold-3), which have no autoregressive constraints. Instead of encoding the code in a form that mimics natural language, we encode the code as a heterogeneous image paradigm with a memory of global information that mimics both images and protein structures. We then refer to Sora's CLIP upstream text-to-image encoder model to design a text-to-code encoder model that can be applied to various downstream code understanding tasks. The model learns the global understanding of code under the new paradigm heterogeneous image, connects the encoding space of text and code, and encodes the input of text into the vector of code most similar to it. Using self-supervised comparative learning on 456,360 text-code pairs, the model achieved a zero-shot prediction of new data. This work is the basis for future work on code generation using diffusion techniques under a new paradigm to avoid autoregressive limitations.