A Comparative Study on Code Generation with Transformers

TL;DR

This paper compares various transformer-based models for automated C++ code generation, evaluating their robustness and effectiveness across different problem complexities in NLP-driven code synthesis.

Contribution

It provides a systematic comparison of transformer architectures for code generation, highlighting their strengths and limitations across diverse problem types.

Findings

Transformer models vary in robustness depending on architecture complexity.

Performance differences are notable between simple and complex problem sets.

The study identifies key factors influencing code generation quality.

Abstract

In an era of widespread influence of Natural Language Processing (NLP), there have been multiple research efforts to supplant traditional manual coding techniques with automated systems capable of generating solutions autonomously. With rapid research for code generation and a sole focus on large language models, there emerges a need to compare and evaluate the performance of transformer architectures based on several complexities of the model. This paper introduces the concept of a "A Comparative Study on Code Generation with Transformers," a model based on Transformer architecture, and NLP methodologies to automatically generate C++ source code for different varieties of problems. Here, a comparative study is performed to evaluate the robustness of transformer-based models on the basis of their architecture complexities and their capability to handle diverse problem sets, from basic arithmetic to complex computations.