Executability of Python Snippets in Stack Overflow

TL;DR

This study investigates the executability of Python code snippets on Stack Overflow, revealing that about 28% are directly executable with minor adjustments, and analyzing factors influencing their usability and impact.

Contribution

The paper introduces a scalable framework for large-scale analysis of code snippet executability and provides empirical insights into the factors affecting Python snippet execution on Stack Overflow.

Findings

27.92% of snippets are executable with minor adjustments

Executability has remained stable over time

GitHub-referenced snippets are more likely to be executable

Abstract

Online resources today contain an abundant amount of code snippets for documentation, collaboration, learning, and problem-solving purposes. Their executability in a "plug and play" manner enables us to confirm their quality and use them directly in projects. But, in practice that is often not the case due to several requirements violations or incompleteness. However, it is a difficult task to investigate the executability on a large scale due to different possible errors during the execution. We have developed a scalable framework to investigate this for SOTorrent Python snippets. We found that with minor adjustments, 27.92% of snippets are executable. The executability has not changed significantly over time. The code snippets referenced in GitHub are more likely to be directly executable. But executability does not affect the chances of the answer to be selected as the accepted answer significantly. These properties help us understand and improve the interaction of users with online resources that include code snippets.