Documenting API Input/Output Examples

TL;DR

This paper presents a prototype toolset that automatically generates dynamic I/O examples for APIs by logging actual function executions and embedding these examples into documentation, enhancing understanding for users.

Contribution

The paper introduces a novel toolset that captures real I/O examples during API testing and integrates them into documentation, addressing a gap in current static documentation tools.

Findings

Generated 400 I/O examples for three C libraries

Successfully integrated I/O examples into API documentation

Demonstrated the toolset's effectiveness in real-world libraries

Abstract

When learning to use an Application Programming Interface (API), programmers need to understand the inputs and outputs (I/O) of the API functions. Current documentation tools automatically document the static information of I/O, such as parameter types and names. What is missing from these tools is dynamic information, such as I/O examples---actual valid values of inputs that produce certain outputs. In this paper, we demonstrate a prototype toolset we built to generate I/O examples. Our tool logs I/O values when API functions are executed, for example in running test suites. Then, the tool puts I/O values into API documents as I/O examples. Our tool has three programs: 1) funcWatch, which collects I/O values when API developers run test suites, 2) ioSelect, which selects one I/O example from a set of I/O values, and 3) ioPresent, which embeds the I/O examples into documents. In a preliminary evaluation, we used our tool to generate four hundred I/O examples for three C libraries: ffmpeg, libssh, and protobuf-c.

Tables

Table 1. TABLE I: The number of functions that have I/O examples generated by our tool

API library	# of functions in source code	# of functions in api doc.	# of functions with I/O examples
ffmpeg	20,347	625	191
libssh	1,473	519	202
protobuf-c	3,171	18	9

Full text

In this paper, we demonstrate a prototype toolset we built to generate I/O examples. our tool logs I/O values when API functions are executed, for example in running test suites. Then, our tool puts I/O values into API documents as I/O examples.

our tool has three programs: 1) funcWatch, which collects I/O values when API developers run test suites, 2) ioSelect, which selects one I/O example from a set of I/O values, and 3) ioPresent, which embeds the I/O examples into documents. In a preliminary evaluation, we used our tool to generate four hundred I/O examples for three C libraries: ffmpeg, libssh, and protobuf-c.

I Introduction

Recent studies by Duala-Ekoko and Robillard [1] and by Sillito et al. [2] found that when learning Application Programming Interfaces (API), programmers look for input/output examples, actual values of a function’s input/output, in order to understand the function. In particular, the programmers asked “How does this data structure look at runtime?” [2] and “we have a newInstance(String) method that takes a String argument and I have no idea what this String is supposed to be” [1].

Programmers often refer to API documents to answer such questions [3, 4]. However, a study by Maalej and Robillard shows that most of the API documents do not contain input/output (I/O) examples [5]. One reason for the documents lacking I/O examples is that API developers have little tool support for documenting I/O examples. Current documentation tools, such as doxygen [6], JavaDoc [7], automate the process of converting comments and source code into structured html pages. If API developers write I/O examples in comments, the tools can present the I/O examples in html pages. Otherwise, the tools have no I/O examples to present.

When there is little help from API documents, programmers (the users of API libraries) often try a variety of approaches to get input/output examples [8, 9, 10]. For instance, programmers 1) search or ask a question online (e.g., stackoverflow.com), 2) look for existing code that use the API (e.g., open source repositories), or 3) write code to do experiments about the API. If the API documents have input/output examples, programmers may not need these approaches.

To help API developers add I/O examples into API documents, we built our tool, which is a prototype toolset that (1) logs actual I/O values when API functions are executed, (2) selects an example from the logged values for each API function, and (3) visualizes the examples to API document.

In the rest of the paper, we will describe what our tool adds to API documents, how API developers can use our tool, how our tool creates input/output examples, the preliminary results, the related work, and the future work.

II our tool in a Nutshell

our tool is for developers of C API libraries to create API documents with I/O examples—actual values of input/output—of API functions. our tool has three programs: funcWatch, ioSelect and ioPresent. First, funcWatch logs I/O values of a function when the function is executed (in tests or in other executable programs). Often the function is executed multiple times. Then, ioSelect selects one of the calls. Finally, ioPresent adds the I/O values of the selected call into the API document. IoPresent is a patch for the documentation tool, doxygen [6]. The patch enables doxygen to visualize I/O examples into html pages.

This website contains a virtual machine that is set up for running our tool, the videos about our tool, the installation guides, and three API documents generated by our tool.

III Motivational Example

In this section, we will show what our tool adds to API documents. Figure 1 shows a document generated by our tool and doxygen. The document is about the function av_bprint_channel_layout in ffmpeg.

The API document is composed of three parts. Part 1 is the declaration of av_bprint_channel_layout. Part 2 is a sentence describing the function. Parts 1 and 2 are generated by doxygen. Part 3 is what our tool adds to the document, which is an I/O example. In the rest of the section, we will walk through the three parts of the document from the perspective of a user who wants to learn av_bprint_channel_layout.

First, the user reads Part 1 and gets some basic information: (1) the name of the function, (2) the return type, which is void, and (3) the types and the names of the parameters. According to the previous studies [1, 2], the user may have the following questions after reading Part 1:

Q1. What does av_bprint_channel_layout do?

Q2. What is struct AVBPrint?

Q3. What is nb_channels?

Second, the user reads Part 2, which answers Q1: the function “append a description of a channel layout to a bprint buffer.” From this description, the user may expect an input representing “a description of a channel layout”, and an output representing “a bprint buffer”. The names of the parameters indicate that the parameter channel_layout is a channel layout and bp is a bprint. But the user still does not know where the description of the channel layout is, and where the buffer of the bprint is. In other words, the user may have the questions:

Q4. Where is the description of a channel layout?

Q5. Where is the buffer of the bprint?

Part 3, the I/O example, provides some hints for some unanswered questions. The I/O example shows that bp->str changed from an empty string to “stereo”. So the reader can guess that bp->str is the buffer of the bprint (answers Q5).

The I/O example also shows that nb_channels can be “-1”, which is related to Q3. For Q4, the user still does not know where the description of a channel layout is. What the reader can know from the I/O example is that when channel_layout is three, the description is “stereo”.

In summary, all the three parts have useful information that helps the user to understand av_bprint_channel_layout. Part 1 has static information extracted from source code. Part 2 has the high-level description that only API developers can provide. our tool adds Part 3, the actual, concrete input/output values of a function.

IV Usage of our tool

The usage of our tool has three steps: 1) run funcWatch, 2) run ioSelect, and 3) run ioPresent. If an API developer wants to get I/O examples for all the functions in the library, the developer needs to have a list of the function names for running funcWatch (so that funcWatch knows which function to log). To help developers with this situation, we modified doxygen so that doxygen writes all the function names in a file when doxygen is executed.

We put a demo video in our online appendix (Section II) in which we followed these steps and generated nearly two hundred I/O examples for the ffmpeg library.

V Envisioned Users

The envisioned users of our tool are the developers of API libraries. The developers can use our tool to create API documents with I/O examples. Although the developers are the envisioned users of our tool, the API users (the programmers who use and learn API libraries) are the ones that read the documents and benefit from the I/O examples.

Additionally, the programs in our tool can be used independently for other purposes. For example, funcWatch can be used to detect whether a parameter always takes the same value.

VI Design of our tool

In this section, we will describe the design of our tool. Figure 2 shows the architecture of our tool, which illustrates the relationship among the three programs: funcWatch, ioSelect, and ioPresent. In the following subsections, we will describe how we design and implement each program.

VI-A FuncWatch: Logging I/O values

The input of funcWatch is an API function name and a test. FuncWatch runs and monitors the test, and, during the execution of the test, funcWatch logs the I/O values of the given API function.

We built funcWatch based on the code of Flashback [11], which uses libdwarf [12], a library that provides access to Dwarf debugging information. By using libdwarf, funcWatch inserts breakpoints at the entry and the exits of the given function (a C function may have multiple exits).

Then, funcWatch runs the given test with the breakpoints. After a breakpoint is reached and the test paused, funcWatch collects the parameters/return values of the API function. Furthermore, funcWatch is able to dereference pointers at runtime, so funcWatch can collect not only pointers’ values, but also the values that the pointers point to. FuncWatch also calculates the addresses of a struct variable’s fields, so that funcWatch can log the values of the fields. Similarly, funcWatch also work for union and enum types.

In C, arrays are often passed without size information to functions. While a common practice is to pass the size of an array as a separate parameter, the naming conventions differ across different projects and developers. So there is no simple way to automatically detect the size parameters. In this case, funcWatch logs only the first item of an array.

In a test, the given function can be called multiple times. FuncWatch logs a call id for every parameter/return value in order to distinguish the values collected from different calls.

VI-B ioSelect: Choosing I/O examples

The input of ioSelect is a set of I/O values of an API function. Each value is labeled with a call id logged by funcWatch. IoSelect selects a call id randomly, and all the values with the same call id become the I/O example for the function. Because ioSelect selects I/O examples randomly, for the same API function and the same test, our tool may create different I/O examples. We implemented ioSelect as a standalone program so that we can easily add new algorithms for selecting I/O examples in the future.

VI-C ioPresent: Adding I/O examples to documents

The input of ioPresent is a set of I/O examples. Each API function has only one I/O example. The output of ioPresent is html API documents with the I/O examples.

The main functionality of ioPresent is to visualize an I/O example into a html table like the table in Figure 1. For example, in Figure 1, ioPresent automatically detects bp->str is a field of a struct variable that bp points to. Then, ioPresent automatically indents the row of bp->str and makes the row of bp collapsible (so that all the values that bp points to can be hidden if needed).

We implemented ioPresent as a patch of doxygen. Specifically, we modified two parts of doxygen: Data Organizer and Html Generator.

Data Organizer builds data structures to represent elements in the API libraries (that are going to be documented), such as classes and functions. First, we added a new class to represent a new type of element—the I/O examples. Second, we added I/O examples into the data structure that represents a function. We also modified Html Generator, in which we implemented the process of visualizing I/O examples in html format.

VII Preliminary Evaluation

To assess the ability of our tool to generate I/O examples for real-world libraries, we did a preliminary evaluation to answer the question: how many functions will have I/O examples by using our tool?

In this evaluation, we used our tool on three API libraries: ffmpeg, libssh, and protobuf-c. For each library, we executed funcWatch on every function (found by doxygen) and every test in the test suites. Then, we executed ioSelect and ioPresent to generate API documents with I/O examples.

In total, our tool generates I/O examples for 402 functions, 191 functions in ffmpeg, 202 in libssh, and 9 in protobuf-c. To put the numbers in context, Table I shows the number of functions in source code, the number of functions in API documents, along with the number of functions that have I/O examples. The API documents have much fewer functions than source code, because doxygen includes only the functions that are commented in a predefined format. This default setting helps doxygen exclude the internal functions, which should not be seen by API users.

our tool did not generate I/O examples for all the functions that are in API documents, because not all the functions were executed in the test suites. For example, ffmpeg has more than twenty thousand functions defined in source code. Only about six hundred functions are documented. our tool created I/O examples for 30% of the documented functions.

VIII Visualizing all I/O values

Sometimes, it is useful to see all the possible values of the parameters and return variables of an API function. So we built a prototype to visualize all the values logged by funcWatch for an API function. For each parameter in a function, we draw a bar chart of all the possible values of the parameter, where the height of each bar represents the frequency of that value. For example, in Figure 3, function av_gcd has three bar charts. Each bar chart represents a parameter or a return variable. Parameter a has five different values: 0, 1, 3, 4, 25 (logged by funcWatch). The most frequent value for a is 0, which occurs in more than 140 calls.

The bar charts are interactive to show the relationships between the parameters and return variables. For example, in Figure 3, the cursor is hovered over bar 25 in bar chart a. The other two bar charts are updated to reflect the values collected when a was 25. In other words, Figure 3 shows that there are about 30 calls where a is 25. In these calls, b is either 1 or 25. And all these calls return 1.

IX Related Work

In this section, we will present two categories of related work. The first category is about introducing different types of contents into API documents, such as information from StackOverflow posts [13], usage information [14], frequently-asked-questions from the Web [15], and code example summaries mined from the Web [16, 17]. Different from these approaches, our tool does not mine the Web, and our tool adds I/O values instead of source code examples into API documents.

The second category is about collecting and analyzing dynamic information, which can be useful in API documentation. For example, Daikon analyzes dynamic information to report likely program invariants [18]. The reports can be useful in API documents. We plan to embed this type of reports to API documents in the future in order to study whether and how the reports can help programmers use API functions.

X Future Work and Limitations

our tool is a prototype toolset, which serves as our first step towards investigating what information is useful for API users. Previous studies have shown that programmers ask questions about dynamic information [1, 2], such as I/O examples. What is not shown is whether and how this information helps programmers. First, we will conduct an empirical study to assess the usefulness of I/O examples. In this study, we plan to hire programmers to complete certain program tasks with and without I/O examples in documents. By comparing the programmers’ performance with and without I/O examples, we can see how I/O examples can help programmers.

Furthermore, we plan to design and implement different selection strategies for ioSelect. Our assumption is that some I/O examples are more useful than others. We plan to investigate some potential characteristics of useful I/O examples. For example, the most common I/O values may be more useful.

Additionally, the quality and the quantity of the I/O examples generated by our tool depend on the test suites of the API libraries. In the future work, we may consider code examples, tutorial examples, and real-world applications that use the API libraries in addition to the test suites to generate I/O examples.

Implementation-wise, our tool currently has three major limitations. First, funcWatch does not support 64-bit memory addressing so our tool can work only in 32-bit operating systems. Second, funcWatch collects only the values for statically-linked libraries. Third, funcWatch collects all the I/O values no matter whether a test passes or fails. If a test fails, the logged values may be invalid. The current workaround for this limitation is to detect the outputs of the tests in a script and discard the outputs of funcWatch for those failed tests.

XI Conclusion

To conclude, we built and demonstrated a prototype toolset, our tool. The goal of our tool is to help API developers create and add I/O examples into API documents. our tool has three programs: funcWatch, ioSelect, and ioPresent. Our preliminary evaluation shows that our tool can generate four hundred I/O examples for three real-world C libraries.

Acknowledgment

We thank Douglas Smith for his work in visualization of all I/O values. This work was partially supported by the NSF CCF-1452959 and CNS-1510329 grants, and the Office of Naval Research grant N000141410037. Any opinions, findings, and conclusions expressed herein are the authors’ and do not necessarily reflect those of the sponsors.