A Neural Question Answering System for Basic Questions about Subroutines

TL;DR

This paper introduces a neural question answering system tailored for basic questions about subroutines in software engineering, utilizing a large dataset and neural models to assist programmers.

Contribution

It presents a novel context-based neural QA system for software engineering, including dataset creation, model training, and evaluation with professional programmers.

Findings

The system effectively answers basic subroutine questions.

Large dataset of 10.9 million tuples enhances model training.

Identifies strengths and weaknesses for future improvements.

Abstract

A question answering (QA) system is a type of conversational AI that generates natural language answers to questions posed by human users. QA systems often form the backbone of interactive dialogue systems, and have been studied extensively for a wide variety of tasks ranging from restaurant recommendations to medical diagnostics. Dramatic progress has been made in recent years, especially from the use of encoder-decoder neural architectures trained with big data input. In this paper, we take initial steps to bringing state-of-the-art neural QA technologies to Software Engineering applications by designing a context-based QA system for basic questions about subroutines. We curate a training dataset of 10.9 million question/context/answer tuples based on rules we extract from recent empirical studies. Then, we train a custom neural QA model with this dataset and evaluate the model in a study with professional programmers. We demonstrate the strengths and weaknesses of the system, and lay the groundwork for its use in eventual dialogue systems for software engineering.