PyMT5: multi-mode translation of natural language and Python code with transformers
Colin B. Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy,, Neel Sundaresan

TL;DR
PyMT5 is a versatile transformer model trained to translate between Python code and natural language, excelling in code generation and summarization tasks with high accuracy and syntactic correctness.
Contribution
The paper introduces PyMT5, a unified model capable of translating between Python code and natural language, trained on a large-scale dataset, outperforming existing models in code and docstring generation.
Findings
PyMT5 achieves 92.1% syntactically correct code generation.
It outperforms GPT2-based models in BLEU and ROUGE scores.
The model demonstrates strong performance on the CodeSearchNet benchmark.
Abstract
Simultaneously modeling source code and natural language has many exciting applications in automated software development and understanding. Pursuant to achieving such technology, we introduce PyMT5, the Python method text-to-text transfer transformer, which is trained to translate between all pairs of Python method feature combinations: a single model that can both predict whole methods from natural language documentation strings (docstrings) and summarize code into docstrings of any common style. We present an analysis and modeling effort of a large-scale parallel corpus of 26 million Python methods and 7.7 million method-docstring pairs, demonstrating that for docstring and method generation, PyMT5 outperforms similarly-sized auto-regressive language models (GPT2) which were English pre-trained or randomly initialized. On the CodeSearchNet test set, our best model predicts 92.1%…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
