Natural Language to Code Using Transformers

TL;DR

This paper demonstrates that transformer models, combined with cycle consistent training techniques, can effectively generate code snippets from natural language descriptions, outperforming previous recurrent models on the CoNaLa dataset.

Contribution

The authors introduce a transformer-based approach with cycle consistent losses for code generation from natural language, achieving state-of-the-art BLEU scores.

Findings

Transformer outperforms recurrent models in code generation.

Cycle consistent training improves model performance.

Achieved BLEU score of 16.99 on CoNaLa dataset.

Abstract

We tackle the problem of generating code snippets from natural language descriptions using the CoNaLa dataset. We use the self-attention based transformer architecture and show that it performs better than recurrent attention-based encoder decoder. Furthermore, we develop a modified form of back translation and use cycle consistent losses to train the model in an end-to-end fashion. We achieve a BLEU score of 16.99 beating the previously reported baseline of the CoNaLa challenge.