Syntax and Stack Overflow: A methodology for extracting a corpus of syntax errors and fixes

TL;DR

This paper presents a methodology to extract a large, natural corpus of Python syntax errors and fixes from Stack Overflow, addressing the lack of representative datasets for syntax error research.

Contribution

The authors introduce a novel approach using a Python AST parser and execution validation to create a publicly available dataset of syntax errors and corrections from Stack Overflow.

Findings

Errors from Stack Overflow differ from student errors and random mutations.

The dataset contains 62,965 Python code snippets with errors and fixes.

The approach enables the collection of natural, real-world syntax error data.

Abstract

One problem when studying how to find and fix syntax errors is how to get natural and representative examples of syntax errors. Most syntax error datasets are not free, open, and public, or they are extracted from novice programmers and do not represent syntax errors that the general population of developers would make. Programmers of all skill levels post questions and answers to Stack Overflow which may contain snippets of source code along with corresponding text and tags. Many snippets do not parse, thus they are ripe for forming a corpus of syntax errors and corrections. Our primary contribution is an approach for extracting natural syntax errors and their corresponding human made fixes to help syntax error research. A Python abstract syntax tree parser is used to determine preliminary errors and corrections on code blocks extracted from the SOTorrent data set. We further analyzed our code by executing the corrections in a Python interpreter. We applied our methodology to produce a public data set of 62,965 Python Stack Overflow code snippets with corresponding tags, errors, and stack traces. We found that errors made by Stack Overflow users do not match errors made by student developers or random mutations, implying there is a serious representativeness risk within the field. Finally we share our dataset openly so that future researchers can re-use and extend our syntax errors and fixes.

Tables

Table 1. TABLE I: Filtered Dataset Entity Counts

Filter Metric	Entity Block Count
All code snippets	82,469,989
Block matched python tag	7,260,631
Content AST parseable	3,774,225
Prior version exists	1,550,090
Prior version AST error	62,965

Table 2. TABLE II: Top Python3.6 ast.parse Errors

Error	Message	Count	%
Syntax	invalid syntax	26336	41.83
Indentation	expected an indented block	14535	23.08
Indentation	unexpected indent	11030	17.52
Syntax	Missing parentheses in call to ’print’	3063	4.86
Syntax	unexpected EOF while parsing	2732	4.34
Syntax	EOL while scanning string literal	2045	3.25
Indentation	unindent does not match	1468	2.33
	any outer indentation level
Tab	inconsistent use of tabs	326	0.52
	and spaces in indentation
Syntax	invalid character in identifier	297	0.47
Syntax	unexpected character after	183	0.29
	line continuation character
Syntax	invalid token	139	0.22
Syntax	positional argument follows	126	0.20
	keyword argument
Syntax	EOF while scanning	102	0.16
	triple-quoted string literal
Syntax	(unicode error) ’unicodeescape’	97	0.15
	codec can’t decode bytes…
Syntax	can’t assign to function call	94	0.15
Syntax	can’t assign to operator	63	0.10
Syntax	illegal target for annotation	59	0.09
Syntax	keyword can’t be an expression	54	0.09
Syntax	Generator expression must be	47	0.07
	parenthesized if not sole argument
Indentation	unexpected unindent	42	0.07
Syntax	other…	127	0.20
Total		62965	100

Table 3. TABLE III: Top Python3.6 Corrected Code Runtime Errors

Error	Count	%
NameError	24270	38.55
No Error	21332	33.88
ModuleNotFoundError	9927	15.77
EOFError	2223	3.53
FileNotFoundError	2156	3.42
SyntaxError	1013	1.61
TypeError	567	0.90
TclError	348	0.55
Execution Timeout	251	0.40
AttributeError	244	0.39
ImportError	116	0.18
other…	518	0.82
Total	62965	100

Full text

A Python abstract syntax tree parser is used to determine preliminary errors and corrections on code blocks extracted from the SOTorrent data set. We further analyzed our code by executing the corrections in a Python interpreter. We applied our methodology to produce a public data set of 62,965 Python Stack Overflow code snippets with corresponding tags, errors, and stack traces. We found that errors made by Stack Overflow users do not match errors made by student developers or random mutations, implying there is a serious representativeness risk within the field. Finally we share our dataset openly so that future researchers can re-use and extend our syntax errors and fixes.

I Introduction

Syntax errors stymie novice and expert developers alike [1, 2]. Many researchers have sought to help developers recover from syntax errors by locating syntax errors [3] and/or fixing them [4]. One impediment to this kind of research is getting access to a representative corpus of syntax errors and their subsequent corrections. Corporae are limited because syntax error commits are rare as open source developers typically do not commit syntax error ridden code to their repositories.

Many existing datasets suffer from limited access due to ethical concerns while other datasets lack representativeness as they are pulled from student developers rather than practicing software engineers [5, 6]. Some authors address the lack of access to syntax errors by synthetically creating their own syntax error datasets via mutation [7, 3]. Mutation does not fully address the representativeness of syntax errors as they can differ from the syntax errors of actual developers [8]. These shortcomings can be mitigated by using data containing natural errors created by a more general population of developers, where natural is defined as “product of human effort” [9]. Unfortunately, naturally made syntax errors are difficult to obtain as current methods require fine grain observations of programmer activity [10].

Thus the goal of this work is to reproducibly and openly produce a corpus of Python syntax errors that is both natural and representative of syntax errors made by developers as a whole. A dataset of natural human made syntax errors and human revised fixes would enable future software engineering research. Models tasked with source code completion or error detection could use this dataset to train and evaluate their performance on actual developer errors. Statistical analysis on commonly made syntax errors and their corresponding fixes could offer insight into planning future software language specifications. This dataset would enable novel approaches for measuring the similarity of source code to natural languages. Potential new studies could explore the naturalness of syntax corrections, comparing how natural language errors are fixed to how code errors are fixed [11]. Most importantly, a replicable and reproducible methodology to produce such a corpus would allow researchers to update and improve the dataset on their own and in the future.

We extend the SOTorrent dataset111SOTorrent Dataset: https://zenodo.org/record/2273117, built from Stack Overflow, to extract syntax errors and corrections [12]. Our main contribution is a methodology for extraction of human made errors and their fixes. With our methodology, it is important to understand how accurate the pairs of syntax errors and their fixes are, and what are the general characteristics of those syntax errors. We answer the following research questions:

1. RQ0)

How accurate is our approach for extracting pairs of syntax errors and their fixes? 2. RQ1)

What are Python parse errors and corresponding runtime properties of their corrections in Stack Overflow? 3. RQ2)

Are Python errors in Stack Overflow similar to errors made by student programmers? 4. RQ3)

Are Python errors in Stack Overflow similar to generated errors created by mutating valid code?

Our generated dataset is released to enable future work in source code naturalness research222Dataset of Python3.6 Natural Syntax Errors and Corrections: https://doi.org/10.6084/m9.figshare.8244686.v1.

II Related Work

Syntactically incorrect code is artificially derivable, as formal programming languages provide grammar rules which can be referred to for correctness. Random token level insertions, deletions, and replacements were performed to generate syntax errors from existing open source Java projects [13]. Campbell et al. created Python syntax errors from valid code mined from GitHub by applying mutations on tokens, characters, and lines [3]. Although generated errors are appealing due to the availability of open source code, Just et al. demonstrated limitations of using mutations for software testing research [7]. Given the task of using mutants as replacements for real faults in automated software testing research, only 73% of mutants were coupled to faults. Furthermore, when accounting for code coverage, the mutant to fault coupling effect is small [7].

Automated source code repair, like identifying and refactoring improper method names, also required a labeled dataset of valid and invalid source code [14]. Program repair is often viewed as different than syntax error correction because testing is performed which serves as a benchmark for repaired code, while syntax errors rely primarily on parseability.

Free and open datasets of naturally made errors and their fixes are more difficult to obtain. Blackbox, a data collection project within the BlueJ Java development environment, requires manual staff contact for access to data and forbids the release of the raw dataset [10]. Pritchard analyzed Python programs submitted to CS Circles, an online tool for learning Python [5]. Kelley et al. studied Python code submitted by students in an introductory programming course at MIT [6]. Gathering this data without privileged access to the provided code submissions is difficult, limiting the reproducibility of their research. Our research used Stack Overflow and is advantageous as the raw content is freely accessible to the internet, revisions and history is tracked, and contributors have a wide range of software engineering expertise and skill sets333Stack Overflow 2019 Survey https://insights.stackoverflow.com/survey/2019.

III Methodology

Our methodology for finding syntax error and fix pairs is: to leverage the historical view of Stack Overflow extracted from the SOTorrent project [12]; parse detected Python source snippet histories; extract pairs of failed and fixed revisions; validate pairs; and finally record successfully evaluated pairs.

III-A Code Snippets from SOTorrent

The SOTorrent schema contains a PostBlockVersion table which stores version history on the latest Posts, which must be either a question or answer [12]. We queried a subset of rows from the PostBlockVersion table where content type equaled CodeSnippet and the corresponding Post was tagged with a term matching python. If the corresponding Post was typed Answer, we used tags defined in the referenced Question. To limit TabError exceptions, extraneous white space is removed from the content while preserving nested indentation.

III-B Parsing Abstract Syntax Tree

To determine the presence of syntax errors, we attempted to parse abstract syntax trees (AST) given the extracted code.

We tackled this problem using Python’s built-in ast module, which compiles source code into an AST object. Valid code parses without error, while invalid code raises errors such as IndentationError, TabError, SyntaxError, or MemoryError. The specific error message, offending line number and column offset are stored for the indentation, tab, and syntax errors (memory errors offer no such metadata).

III-C Filtering Candidates using Block Versions

One issue is that many extracted code snippets containing a python matching tag are not valid Python, but instead contain stack traces, script execution commands, tabular data, markup, etc. Further candidate filtering is necessary to distinguish invalid Python code from unrelated text.

To address this issue, we queried for an existing prior version within the PostBlockVersion table for each code snippet that successfully parsed into an AST. If the prior version contained a parse error, we stored these two versions of code as our syntax error and fix. Unrelated snippets that are never fixed into valid Python source code are removed from analysis.

III-D Runtime Validation with the Interpreter

We checked for code snippet validity by running the corrected source code. Code that validly parsed into an AST may still be faulty due to various errors, like importing invalid modules or referencing undefined variables. Source code that cannot parse into an AST does not need to be run as the thrown exception will be the same as the parse error.

We used the Python built-in exec function, isolating the global and local execution scopes. To mitigate the impact of executing arbitrary source code, we ran the snippets in an isolated Linux kernel virtual machine (KVM) with no access to network. Each snippet invocation was further containerized within its own Docker image running Python 3.6 using Debian Jessie. We capped the maximum execution time to four seconds to account for long running and non-terminating programs. All encountered exceptions and their corresponding stack traces are stored in our dataset for further analysis.

IV Results

We successfully parsed and analyzed 82,469,989 code snippets from the SOTorrent dataset. Of these snippets, 7,260,631 contained a python matching tag. A subset of 3,774,225 snippets parsed correctly. Of the parseable snippets, 1,550,090 had prior versions, 62,965 of which contained a parse error. A summary of the filtering steps can be found in Tab. I. With this dataset, we answer the four research questions.

RQ0. How accurate is our approach for extracting pairs of syntax errors and their fixes?

We wanted to determine how accurate our methodology is for extracting true syntax errors and fixes. Three co-authors together inspected 100 random samples of syntax errors and corrections from the extracted dataset of 62,965 pairs. Only 2 observed cases were not entirely syntax error revisions, as they included large amounts of new code. The 95% confidence interval for our methodology is 0-5% erroneous cases.

While our methodology is accurate, future research can take care of these erroneous cases (e.g., by leveraging the code snippets version similarity provided by SOTorrent).

Stack Overflow is a good source for methodologically generating pairs of natural syntax errors and fixes.

RQ1. What are Python parse errors and corresponding runtime properties of their corrections in Stack Overflow?

Stack Overflow posts are written and parsed using a subset of Markdown. Code blocks are defined by using four space indentations, creating an indented code block, or by surrounding the text using three backtick characters (‘) to create a fenced code block. In addition, the Stack Overflow text editor does not provide code assistance features, such as syntax highlighting and code completion.

Of the 62,965 AST parse errors, there were 35,564 (56.5%) syntax errors, 27,075 (43.0%) indentation errors, and 326 (0.5%) tab errors. Over 41% of all AST parse errors were SyntaxError: invalid syntax. A detailed summary of the error types and messages are listed in Tab. II.

Over a third of all corrected snippets still throw a Name error. Roughly one third, or 21,332 corrected snippets ran in a Python interpreter with no errors. Only 0.4% of all evaluated snippets triggered an execution timeout by running for longer than four seconds. The results from running all AST fixes in a Python interpreter are summarized in Tab. III.

We extracted 62,965 parse errors and corrections from over 82 million code snippets. Of the corrections, 21,332 ran in a Python interpreter without error.

RQ2. Are Python errors in Stack Overflow similar to errors made by student programmers?

We compared the distribution of our errors with the distribution of runtime errors as presented in two prior works. If the distribution of errors are similar to student programmer errors then student errors can be viewed as representative. If they are not similar then this provide evidence to arguments regarding how representative student errors are considering their error distribution is different.

Kelley et al. collected the distribution of Python errors submitted by MIT programming students to an online tutor as part of an introductory Python programming course [6]. The collected student error distributions were TypeError: $28\%$; AttributeError: $22\%$; NameError: $19\%$; SyntaxError: $12\%$; IndexError: $6\%$ and other: $13\%$. We performed a Pearson’s Chi-squared Test for given probabilities, comparing our combined errors with the MIT student error distributions. We found that the error distribution in Stack Overflow was not statistically similar to the errors made by MIT student programmers ($\alpha=0.01$, $X^{2}=239031$, $df=5$, $p$-value $<2.2\mathrm{e}{-16}$).

Pritchard collected the student error distribution from interactive tutorials within CS Circles [5]. The paper reported distribution was SyntaxError: $48.14\%$; NameError: $15.11\%$; EOFError: $11.82\%$; IndentationError: $3.23\%$; and other: $21.7\%$. We performed a Chi-squared test for the probabilities given in CS Circles student Python error distributions and found that errors in Stack Overflow were not similarly distributed ($\alpha=0.01$, $X^{2}=154689$, $df=4$, $p$-value $<2.2\mathrm{e}{-16}$).

The distribution of errors found in Stack Overflow extracted source code snippets did not match the distribution of errors prior literature has found in student and novice developers. This empirically suggests that creating a corpus of syntax errors strictly from novice developers is insufficient to capture the full spectrum of errors made by the developer community as a whole. Additional methods of contributing to existing syntax corpus datasets are therefore necessary. We therefore emphasize the value of our corpus of data, as it provides a traceable distribution of natural programming errors and corrections.

The errors posted on Stack Overflow are not similar to the errors made by student and novice programmers.

RQ3. Are Python errors in Stack Overflow similar to generated errors created by mutating valid code?

We evaluated our errors against a distribution of syntax errors generated by random source code mutations, summarized in Fig. 1. Campbell et al. provided frequencies of Python exceptions caused by mutations of a randomly chosen token within a sequence of valid Python source code [3]. We performed a Pearson’s Chi-squared Test for the distributions of Token deletions, insertions, and replacements.

• •

Delete; $X^{2}=933962$, $df=10$, $p$-value $<2.2\mathrm{e}{-16}$

• •

Insert; $X^{2}=8424235$, $df=10$, $p$-value $<2.2\mathrm{e}{-16}$

• •

Replace; $X^{2}=652478$, $df=10$, $p$-value $<2.2\mathrm{e}{-16}$

We reject the assumption that the Stack Overflow extracted code error distributions are similar to the mutated source code using any mutation type with $\alpha=0.01$. This suggests that random mutations alone are insufficient for creating an error corpus that compares to natural faults. As the distributions of mutation-based syntax errors were also not similar to errors found in programming novices, additional tuning is necessary in order to emulate natural distributions of syntax errors.

The errors posted on Stack Overflow are not similar to generated errors from randomly mutating source code.

V Future Work

The methodology and data provided in this paper enable additional opportunities of research for software engineering.

• •

The current paper generates syntax errors and corrections only for Python3.6. A natural, incremental addition is to perform the same computation for all Python versions.

• •

Further languages can also be analyzed using our methodology. These languages will have their own static parsing and runtime execution specifications.

• •

Extra runtime error corrections can be done by installing required dependencies or initializing variables.

• •

We will investigate how this dataset enhances existing code completion and syntax error location models, as well as provide a new evaluation benchmark for these tasks.

• •

More insight on natural error corrections is crucial. Fixing an improperly indented block of code involves multiple white-space changes across many lines but can be resolved by highlighting the lines of code in an editor and indenting the selection. Character/token changes alone do not fully encapsulate developer-code interaction.

VI Threats to Validity

We relied on user submitted tags to identify Python code. Because snippets not matching the Python tag were not considered, unaccounted Python code in Stack Overflow may still exist. Unfortunately, relying only on source code parseability generated many false positives. One recurring issue was JavaScript object notation (JSON) had the same structure as Python dictionaries. Without matching the Python tag, an additional 5.4 million false positive code snippets would require analysis.

Another concern is whether or not we can rely on methodology to accurately distinguish between true syntactically invalid Python code versus syntactically invalid arbitrary text. Although we found no occurrences of blatantly invalid source code, we publicize our dataset so anyone can audit our results.

Our approach obtains pairs of snippets where one snippet is syntactically correct and the other is not. We only look at each snippet previous version at one point in time rather than looking at the entire history. This prevents the inclusion of code blocks that have evolved greatly over time.

Another hurdle is to obtain runtime faults from parseable Python code snippets, it is necessary to run the source code in a Python interpreter. Despite our attempts to eliminate variability of source code execution using KVMs and Docker, we acknowledge that more work is necessary to rigorously evaluate runtime faults of programs. Improvements involve accommodating operating system specificity, file system case sensitivity, and underlying hardware limitations.

We argue that the distribution of errors extracted from Stack Overflow are representative of developers, but not representative of software projects as a whole. We acknowledge this distinction as code snippets are a subset of working software.

Conclusion

We provide a novel methodology for automatically extracting natural source code syntax errors and their fixes from Stack Overflow, which sets the groundwork for future software naturalness and future syntax error detection and correction research. Syntax errors extracted from Stack Overflow do not match prior distributions found in novice code or randomly generated errors. We hope our data will be used for training and evaluating code completion & error detection models, analysis of common programming language pitfalls, and future source code naturalness research.