CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow

TL;DR

CodeInsight presents a curated dataset of 3,409 Python coding examples from Stack Overflow, including intent, code snippets, and unit tests, designed to improve code generation models and analyze their strengths and weaknesses.

Contribution

The paper introduces a new, high-quality dataset for code generation with detailed annotations and contamination reduction, enabling better model training and evaluation.

Findings

Models show varied performance across different task categories.

Refined dataset reduces data contamination, improving evaluation reliability.

GPT-4 performance analyzed on the dataset.

Abstract

We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as \texttt{Pandas}, \texttt{Numpy}, and \texttt{Regex}, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,409 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models' strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLaMa 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset. The benchmark can be accessed at \url{https://github.com/NathanaelBeau/CodeInsight}.