Performance Review on LLM for solving leetcode problems

TL;DR

This paper evaluates the effectiveness of Large Language Models like GPT-4 and GPT-3.5-turbo in solving diverse Leetcode programming problems, analyzing correctness, efficiency, and potential for automated coding assistance.

Contribution

It provides a systematic performance assessment of LLMs on Leetcode problems, highlighting their strengths and limitations in code generation and problem-solving.

Findings

LLMs achieve varying success rates across problem difficulties

GPT-4 outperforms GPT-3.5-turbo in correctness and efficiency

Identifies areas for improvement in automated programming tools

Abstract

This paper presents a comprehensive performance evaluation of Large Language Models (LLMs) in solving programming challenges from Leetcode, a widely used platform for algorithm practice and technical interviews. We began by crawling the Leetcode website to collect a diverse set of problems encompassing various difficulty levels and topics. Using this dataset, we generated solutions with multiple LLMs, including GPT-4 and GPT-3.5-turbo (ChatGPT-turbo). The generated solutions were systematically evaluated for correctness and efficiency. We employed the pass@k metric to assess the success rates within a given number of attempts and analyzed the runtime performance of the solutions. Our results highlight the strengths and limitations of current LLMs [10] in code generation and problem-solving tasks, providing insights into their potential applications and areas for improvement in automated programming assistance.