Combining Contexts from Multiple Sources for Documentation-Specific Code Example Generation

TL;DR

This paper explores automatic generation of code examples for documentation using GPT-3 based Codex, demonstrating promising passability and relevance rates, and showing that including error logs improves code execution success.

Contribution

It introduces a novel approach to generate documentation-specific code examples using Codex and evaluates the impact of error logs on code passability.

Findings

72.5% code examples executed without error

82.5% code examples relevant to documentation

Error logs improve passability to 87.5%

Abstract

Code example is a crucial part of good documentation. It helps the developers to understand the documentation easily and use the corresponding code unit (e.g., method) properly. However, many official documentation still lacks (good) code example and it is one of the common documentation issues as found by several studies. Hence in this paper, we consider automatic code example generation for documentation, a direction less explored by the existing research. We employ Codex, a GPT-3 based model, pre-trained on both natural and programming languages to generate code examples from source code and documentation given as input. Our preliminary investigation on 40 scikit-learn methods reveals that this approach is able to generate good code examples where 72.5% code examples were executed without error (passability) and 82.5% properly dealt with the target method and documentation (relevance). We also find that incorporation of error logs (produced by the compiler while executing a failed code example) in the input further improves the passability from 72.5% to 87.5%. Thus, our investigation sets the base of documentation-specific code example generation and warrants in-depth future studies.