Are Large Language Models a Threat to Programming Platforms? An Exploratory Study

TL;DR

This study evaluates the problem-solving capabilities of large language models on competitive programming platforms, revealing strengths in some areas but limitations in more challenging contests, raising concerns about future impacts.

Contribution

It provides a comprehensive assessment of LLMs' performance on diverse programming challenges across multiple platforms, highlighting their current capabilities and limitations.

Findings

LLMs like ChatGPT excel in LeetCode and HackerRank certifications.

They underperform in difficult Codeforces contests.

LLMs outperform human users in some cases but face challenges in complex problems.

Abstract

Competitive programming platforms like LeetCode, Codeforces, and HackerRank evaluate programming skills, often used by recruiters for screening. With the rise of advanced Large Language Models (LLMs) such as ChatGPT, Gemini, and Meta AI, their problem-solving ability on these platforms needs assessment. This study explores LLMs' ability to tackle diverse programming challenges across platforms with varying difficulty, offering insights into their real-time and offline performance and comparing them with human programmers. We tested 98 problems from LeetCode, 126 from Codeforces, covering 15 categories. Nine online contests from Codeforces and LeetCode were conducted, along with two certification tests on HackerRank, to assess real-time performance. Prompts and feedback mechanisms were used to guide LLMs, and correlations were explored across different scenarios. LLMs, like ChatGPT (71.43% success on LeetCode), excelled in LeetCode and HackerRank certifications but struggled in virtual contests, particularly on Codeforces. They performed better than users in LeetCode archives, excelling in time and memory efficiency but underperforming in harder Codeforces contests. While not immediately threatening, LLMs performance on these platforms is concerning, and future improvements will need addressing.