Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks

TL;DR

This paper introduces Themisto, an AI system that supports data scientists in documenting their code in computational notebooks through automated generation, retrieval, and prompting, improving efficiency and satisfaction.

Contribution

It presents a human-centered AI system with three novel approaches for supporting code documentation in notebooks, inspired by human practices and evaluated with practitioners.

Findings

Automated documentation reduces time spent on writing documentation.

The system encourages documentation of previously ignored code.

Participants reported increased satisfaction with their notebooks.

Abstract

Computational notebooks allow data scientists to express their ideas through a combination of code and documentation. However, data scientists often pay attention only to the code, and neglect creating or updating their documentation during quick iterations. Inspired by human documentation practices learned from 80 highly-voted Kaggle notebooks, we design and implement Themisto, an automated documentation generation system to explore how human-centered AI systems can support human data scientists in the machine learning code documentation scenario. Themisto facilitates the creation of documentation via three approaches: a deep-learning-based approach to generate documentation for source code, a query-based approach to retrieve online API documentation for source code, and a user prompt approach to nudge users to write documentation. We evaluated Themisto in a within-subjects experiment with 24 data science practitioners, and found that automated documentation generation techniques reduced the time for writing documentation, reminded participants to document code they would have ignored, and improved participants' satisfaction with their computational notebook.