Automating API Documentation from Crowdsourced Knowledge

TL;DR

AutoDoc leverages online discussions and advanced language models to generate more accurate, comprehensive, and less redundant API documentation, addressing issues of obsolescence and incompleteness in official docs.

Contribution

The paper introduces AutoDoc, a novel approach combining dense retrieval and GPT-4-based summarization to improve API documentation quality from crowdsourced knowledge.

Findings

AutoDoc produces API documents up to 77.7% more accurate.

It reduces redundancy by 9.5%.

Generated documents uncover 34.4% more knowledge than official docs.

Abstract

API documentation is crucial for developers to learn and use APIs. However, it is known that many official API documents are obsolete and incomplete. To address this challenge, we propose a new approach called AutoDoc that generates API documents with API knowledge extracted from online discussions on Stack Overflow (SO). AutoDoc leverages a fine-tuned dense retrieval model to identify seven types of API knowledge from SO posts. Then, it uses GPT-4o to summarize the API knowledge in these posts into concise text. Meanwhile, we designed two specific components to handle LLM hallucination and redundancy in generated content. We evaluated AutoDoc against five comparison baselines on 48 APIs of different popularity levels. Our results indicate that the API documents generated by AutoDoc are up to 77.7% more accurate, 9.5% less duplicated, and contain 34.4% knowledge uncovered by the official documents. We also measured the sensitivity of AutoDoc to the choice of different LLMs. We found that while larger LLMs produce higher-quality API documents, AutoDoc enables smaller open-source models (e.g., Mistral-7B-v0.3) to achieve comparable results. Finally, we conducted a user study to evaluate the usefulness of the API documents generated by AutoDoc. All participants found API documents generated by AutoDoc to be more comprehensive, concise, and helpful than the comparison baselines. This highlights the feasibility of utilizing LLMs for API documentation with careful design to counter LLM hallucination and information redundancy.