Low-Cost Language Models: Survey and Performance Evaluation on Python Code Generation

TL;DR

This paper surveys low-cost language models for Python code generation, evaluates their performance, introduces a new dataset, and proposes a Chain-of-Thought prompting strategy to enhance reasoning and code quality.

Contribution

It provides a comprehensive evaluation of resource-efficient models, introduces a new dataset of programming problems, and proposes a prompting strategy to improve code generation quality.

Findings

Some low-cost models achieve competitive results with larger models.

The Chain-of-Thought prompting improves model reasoning and code quality.

The new dataset extends existing benchmarks for Python code generation.

Abstract

Large Language Models (LLMs) have become a popular choice for many Natural Language Processing (NLP) tasks due to their versatility and ability to produce high-quality results. Specifically, they are increasingly used for automatic code generation to help developers tackle repetitive coding tasks. However, LLMs' substantial computational and memory requirements often make them inaccessible to users with limited resources. This paper focuses on very low-cost models which offer a more accessible alternative to resource-intensive LLMs. We notably: (1) propose a thorough semi-manual evaluation of their performance in generating Python code, (2) introduce a Chain-of-Thought (CoT) prompting strategy to improve model reasoning and code quality, and (3) propose a new dataset of 60 programming problems, with varied difficulty levels, designed to extend existing benchmarks like HumanEval and EvalPlus. Our findings show that some low-cost compatible models achieve competitive results compared to larger models like ChatGPT despite using significantly fewer resources. We will make our dataset and prompts publicly available to support further research.