One Documentation Does Not Fit All: Case Study of TensorFlow Documentation

TL;DR

This study analyzes TensorFlow documentation and related developer questions, revealing that despite customization efforts, the documentation often fails to meet the needs of its diverse user base, especially those lacking software engineering experience.

Contribution

It provides a detailed analysis of TensorFlow tutorials and Stack Overflow questions, highlighting gaps in documentation effectiveness for non-expert users and proposing taxonomies for understanding user queries.

Findings

24.9% of questions concern errors and exceptions

64.3% relate to inadequate examples in documentation

Current documentation does not effectively support target users

Abstract

Software documentation guides the proper use of tools or services. With the rapid growth of machine learning libraries, individuals from various fields are incorporating machine learning into their workflows through programming. However, many of these users lack software engineering experience, affecting the usability of the documentation. Traditionally, software developers have created documentation primarily for their peers, making it challenging for others to interpret and effectively use these resources. Moreover, no study has specifically focused on machine learning software documentation or analyzing the backgrounds of developers who rely on such documentation, highlighting a critical gap in understanding how to make these resources more accessible. This study examined customization trends in TensorFlow tutorials and compared these artifacts to analyze content and design differences. We also analyzed Stack Overflow questions related to TensorFlow documentation to understand the types of questions and the backgrounds of the developers asking them. Further, we developed two taxonomies based on the nature and triggers of the questions for machine learning software. Our findings showed no significant differences in the content or the nature of the questions across different tutorials. Our results show that 24.9% of the questions concern errors and exceptions, while 64.3% relate to inadequate and non-generalizable examples in the documentation. Despite efforts to create customized documentation, our analysis indicates that current TensorFlow documentation does not effectively support its target users.