Performance Review on LLM for solving leetcode problems
Lun Wang, Chuanqi Shi, Shaoshui Du, Yiyi Tao, Yixian Shen, Hang Zheng,, Yanxin Shen, Xinyu Qiu

TL;DR
This paper evaluates the effectiveness of Large Language Models like GPT-4 and GPT-3.5-turbo in solving diverse Leetcode programming problems, analyzing correctness, efficiency, and potential for automated coding assistance.
Contribution
It provides a systematic performance assessment of LLMs on Leetcode problems, highlighting their strengths and limitations in code generation and problem-solving.
Findings
LLMs achieve varying success rates across problem difficulties
GPT-4 outperforms GPT-3.5-turbo in correctness and efficiency
Identifies areas for improvement in automated programming tools
Abstract
This paper presents a comprehensive performance evaluation of Large Language Models (LLMs) in solving programming challenges from Leetcode, a widely used platform for algorithm practice and technical interviews. We began by crawling the Leetcode website to collect a diverse set of problems encompassing various difficulty levels and topics. Using this dataset, we generated solutions with multiple LLMs, including GPT-4 and GPT-3.5-turbo (ChatGPT-turbo). The generated solutions were systematically evaluated for correctness and efficiency. We employed the pass@k metric to assess the success rates within a given number of attempts and analyzed the runtime performance of the solutions. Our results highlight the strengths and limitations of current LLMs [10] in code generation and problem-solving tasks, providing insights into their potential applications and areas for improvement in automated…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAlgorithms and Data Compression · Data Mining Algorithms and Applications · Advanced Computational Techniques and Applications
MethodsRefunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Absolute Position Encodings · Linear Layer · Layer Normalization · Dense Connections · Attention Dropout · Residual Connection · Label Smoothing · Multi-Head Attention
