Gistable: Evaluating the Executability of Python Code Snippets on GitHub

TL;DR

This paper analyzes the executability of Python code snippets on GitHub Gists, revealing many require complex setup and introducing Gistable, a framework with over 10,000 executable snippets for reproducible research.

Contribution

It provides an empirical analysis of Python gist executability and introduces Gistable, a large database and framework for running shared code snippets reliably.

Findings

75.6% of gists need complex configuration

Developers correctly resolve resource names less than half the time

Gistable contains over 10,000 snippets, with about 5,000 executable via Docker

Abstract

Software developers create and share code online to demonstrate programming language concepts and programming tasks. Code snippets can be a useful way to explain and demonstrate a programming concept, but may not always be directly executable. A code snippet can contain parse errors, or fail to execute if the environment contains unmet dependencies. This paper presents an empirical analysis of the executable status of Python code snippets shared through the GitHub gist system, and the ability of developers familiar with software configuration to correctly configure and run them. We find that 75.6% of gists require non-trivial configuration to overcome missing dependencies, configuration files, reliance on a specific operating system, or some other environment configuration. Our study also suggests the natural assumption developers make about resource names when resolving configuration errors is correct less than half the time. We also present Gistable, a database and extensible framework built on GitHub's gist system, which provides executable code snippets to enable reproducible studies in software engineering. Gistable contains 10,259 code snippets, approximately 5,000 with a Dockerfile to configure and execute them without import error. Gistable is publicly available at https://github.com/gistable/gistable.