Zero-Shot Program Representation Learning

TL;DR

Zecoler introduces a zero-shot learning method for code representations that leverages pre-trained models and prompt tuning, enabling effective code task performance in domain-specific languages with little to no training data.

Contribution

It proposes a novel zero-shot approach using prompt learning to adapt pre-trained models for code tasks in scarce-data scenarios, especially for domain-specific languages.

Findings

Outperforms baseline models in zero-shot and few-shot settings

Effective in domain-specific languages like Solidity and Go

Leverages pre-trained models with prompt tuning for code tasks

Abstract

Learning program representations has been the core prerequisite of code intelligent tasks such as code search and code clone detection. The state-of-the-art pre-trained models such as CodeBERT require the availability of large-scale code corpora. However, gathering training samples can be costly and infeasible for domain-specific languages such as Solidity for smart contracts. In this paper, we propose Zecoler, a zero-shot learning approach for code representations. Zecoler is built upon a pre-trained programming language model. In order to elicit knowledge from the pre-trained models efficiently, Zecoler casts the downstream tasks to the same form of pre-training tasks by inserting trainable prompts into the original input. Then, it employs the prompt learning technique which optimizes the pre-trained model by merely adjusting the original input. This enables the representation model to efficiently fit the scarce task-oriented data while reusing pre-trained knowledge. We evaluate Zecoler in three code intelligent tasks in two program languages that have no training samples, namely, Solidity and Go, with model trained in corpora of common languages such as Java. Experimental results show that our approach significantly outperforms baseline models in both zero-shot and few-shot settings.