Explaining Competitive-Level Programming Solutions using LLMs

TL;DR

This paper investigates how large language models can generate natural language explanations for competitive programming solutions, highlighting their strengths in explanation rather than problem-solving, and introduces a method for automatic annotation of such explanations.

Contribution

It proposes a novel approach to automatically annotate explanations for problem-solution pairs and evaluates their effectiveness in improving problem understanding and solving.

Findings

GPT-4 better understands key ideas behind solutions

LLMs are strong in describing and explaining solutions

Explanation generation aids in problem comprehension

Abstract

In this paper, we approach competitive-level programming problem-solving as a composite task of reasoning and code generation. We propose a novel method to automatically annotate natural language explanations to \textit{<problem, solution>} pairs. We show that despite poor performance in solving competitive-level programming problems, state-of-the-art LLMs exhibit a strong capacity in describing and explaining solutions. Our explanation generation methodology can generate a structured solution explanation for the problem containing descriptions and analysis. To evaluate the quality of the annotated explanations, we examine their effectiveness in two aspects: 1) satisfying the human programming expert who authored the oracle solution, and 2) aiding LLMs in solving problems more effectively. The experimental results on the CodeContests dataset demonstrate that while LLM GPT3.5's and GPT-4's abilities in describing the solution are comparable, GPT-4 shows a better understanding of the key idea behind the solution.