Assessing the alignment between the information needs of developers and the documentation of programming languages: A case study on Rust

TL;DR

This paper introduces an automated semi-supervised topic modeling approach to evaluate how well programming language documentation aligns with developers' actual information needs, demonstrated through a case study on Rust.

Contribution

It presents a novel method for assessing topical alignment between documentation and developer queries, aiding documentation improvement and prioritization.

Findings

High overall topical alignment in Rust documentation

Scarcity of specific niche topics in documentation and Q&A

Certain language feature topics are only present in Q&A, not in official docs

Abstract

Programming language documentation refers to the set of technical documents that provide application developers with a description of the high-level concepts of a language. Such documentation is essential to support application developers in the effective use of a programming language. One of the challenges faced by documenters (i.e., personnel that produce documentation) is to ensure that documentation has relevant information that aligns with the concrete needs of developers. In this paper, we present an automated approach to support documenters in evaluating the differences and similarities between the concrete information need of developers and the current state of documentation (a problem that we refer to as the topical alignment of a programming language documentation). Our approach leverages semi-supervised topic modelling to assess the similarities and differences between the topics of Q&A posts and the official documentation. To demonstrate the application of our approach, we perform a case study on the documentation of Rust. Our results show that there is a relatively high level of topical alignment in Rust documentation. Still, information about specific topics is scarce in both the Q&A websites and the documentation, particularly related topics with programming niches such as network, game, and database development. For other topics (e.g., related topics with language features such as structs, patterns and matchings, and foreign function interface), information is only available on Q&A websites while lacking in the official documentation. Finally, we discuss implications for programming language documenters, particularly how to leverage our approach to prioritize topics that should be added to the documentation.