CoTexT: Multi-task Learning with Code-Text Transformer

TL;DR

CoTexT is a transformer-based model trained on large code and text corpora, enabling it to excel at various programming language tasks like summarization, code generation, and defect detection, achieving state-of-the-art results.

Contribution

The paper introduces CoTexT, a novel multi-task pre-trained transformer model that effectively handles multiple NL-PL tasks with state-of-the-art performance.

Findings

Achieves SOTA results on CodeXGLUE tasks

Effective multi-task learning across diverse programming languages

Versatile performance on code summarization, generation, and debugging

Abstract

We present CoTexT, a pre-trained, transformer-based encoder-decoder model that learns the representative context between natural language (NL) and programming language (PL). Using self-supervision, CoTexT is pre-trained on large programming language corpora to learn a general understanding of language and code. CoTexT supports downstream NL-PL tasks such as code summarizing/documentation, code generation, defect detection, and code debugging. We train CoTexT on different combinations of available PL corpus including both "bimodal" and "unimodal" data. Here, bimodal data is the combination of text and corresponding code snippets, whereas unimodal data is merely code snippets. We first evaluate CoTexT with multi-task learning: we perform Code Summarization on 6 different programming languages and Code Refinement on both small and medium size featured in the CodeXGLUE dataset. We further conduct extensive experiments to investigate CoTexT on other tasks within the CodeXGlue dataset, including Code Generation and Defect Detection. We consistently achieve SOTA results in these tasks, demonstrating the versatility of our models.