Testing with Jupyter notebooks: NoteBook VALidation (nbval) plug-in for pytest

TL;DR

The paper introduces nbval, a pytest plugin that validates Jupyter notebooks by executing code cells and comparing outputs to ensure consistency, aiding in testing and documentation validation.

Contribution

It presents nbval, a novel plugin for pytest that automates notebook validation by executing code and verifying outputs, enhancing testing workflows.

Findings

Enables automated validation of notebooks through output comparison.

Supports use cases like documentation, tutorials, and library testing.

Integrates seamlessly with pytest for flexible testing configurations.

Abstract

The Notebook validation tool nbval allows to load and execute Python code from a Jupyter notebook file. While computing outputs from the cells in the notebook, these outputs are compared with the outputs saved in the notebook file, treating each cell as a test. Deviations are reported as test failures, with various configuration options available to control the behaviour. Application use cases include the validation of notebook-based documentation, tutorials and textbooks, as well as the use of notebooks as additional unit, integration and system tests for the libraries that are used in the notebook. Nbval is implemented as a plugin for the pytest testing software.