Testing with Jupyter notebooks: NoteBook VALidation (nbval) plug-in for pytest
Hans Fangohr, Vidar Fauske, Thomas Kluyver, Maximilian Albert, Oliver, Laslett, David Cort\'es-Ortu\~no, Marijan Beg, and Min Ragan-Kelly

TL;DR
The paper introduces nbval, a pytest plugin that validates Jupyter notebooks by executing code cells and comparing outputs to ensure consistency, aiding in testing and documentation validation.
Contribution
It presents nbval, a novel plugin for pytest that automates notebook validation by executing code and verifying outputs, enhancing testing workflows.
Findings
Enables automated validation of notebooks through output comparison.
Supports use cases like documentation, tutorials, and library testing.
Integrates seamlessly with pytest for flexible testing configurations.
Abstract
The Notebook validation tool nbval allows to load and execute Python code from a Jupyter notebook file. While computing outputs from the cells in the notebook, these outputs are compared with the outputs saved in the notebook file, treating each cell as a test. Deviations are reported as test failures, with various configuration options available to control the behaviour. Application use cases include the validation of notebook-based documentation, tutorials and textbooks, as well as the use of notebooks as additional unit, integration and system tests for the libraries that are used in the notebook. Nbval is implemented as a plugin for the pytest testing software.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Software Testing and Debugging Techniques · Software Reliability and Analysis Research
