From Query to Usable Code: An Analysis of Stack Overflow Code Snippets

TL;DR

This paper evaluates the usability of Stack Overflow code snippets across four languages, analyzing their potential for automated code generation tools and examining how natural language context influences snippet usefulness.

Contribution

It provides a comprehensive analysis of snippet usability in four languages and explores how natural language annotations relate to snippet quality for automated tools.

Findings

Python and JavaScript have the highest usable snippet rates.

Java and C# have the lowest usability rates.

Usable snippets often have specific characteristics in answers.

Abstract

Enriched by natural language texts, Stack Overflow code snippets are an invaluable code-centric knowledge base of small units of source code. Besides being useful for software developers, these annotated snippets can potentially serve as the basis for automated tools that provide working code solutions to specific natural language queries. With the goal of developing automated tools with the Stack Overflow snippets and surrounding text, this paper investigates the following questions: (1) How usable are the Stack Overflow code snippets? and (2) When using text search engines for matching on the natural language questions and answers around the snippets, what percentage of the top results contain usable code snippets? A total of 3M code snippets are analyzed across four languages: C\#, Java, JavaScript, and Python. Python and JavaScript proved to be the languages for which the most code snippets are usable. Conversely, Java and C\# proved to be the languages with the lowest usability rate. Further qualitative analysis on usable Python snippets shows the characteristics of the answers that solve the original question. Finally, we use Google search to investigate the alignment of usability and the natural language annotations around code snippets, and explore how to make snippets in Stack Overflow an adequate base for future automatic program generation.