On the Complexity and Performance of Parsing with Derivatives

TL;DR

This paper reexamines parsing with derivatives, proving its worst-case complexity is cubic rather than exponential, and introduces simple modifications that improve practical performance while maintaining ease of understanding.

Contribution

It demonstrates that parsing with derivatives has cubic complexity and proposes simple implementation improvements for better practical performance.

Findings

Parsing with derivatives has cubic worst-case complexity.

Modified implementation is easier to understand and more efficient.

Original assumptions of exponential complexity are refuted.

Abstract

Current algorithms for context-free parsing inflict a trade-off between ease of understanding, ease of implementation, theoretical complexity, and practical performance. No algorithm achieves all of these properties simultaneously. Might et al. (2011) introduced parsing with derivatives, which handles arbitrary context-free grammars while being both easy to understand and simple to implement. Despite much initial enthusiasm and a multitude of independent implementations, its worst-case complexity has never been proven to be better than exponential. In fact, high-level arguments claiming it is fundamentally exponential have been advanced and even accepted as part of the folklore. Performance ended up being sluggish in practice, and this sluggishness was taken as informal evidence of exponentiality. In this paper, we reexamine the performance of parsing with derivatives. We have discovered that it is not exponential but, in fact, cubic. Moreover, simple (though perhaps not obvious) modifications to the implementation by Might et al. (2011) lead to an implementation that is not only easy to understand but also highly performant in practice.