Using clarification questions to improve software developers' Web search

TL;DR

This paper presents a conversational technique that guides software developers in manually refining their Web search queries through clarification questions, improving search effectiveness by eliciting additional relevant query terms.

Contribution

It introduces a neural-based recommendation system for clarification questions tailored to software developers' queries, enhancing query reformulation beyond automatic methods.

Findings

The system predicts valid clarification questions 80% of the time.

It outperforms simple and state-of-the-art learning to rank baselines.

Developers find the approach useful across experience levels.

Abstract

Context: Recent research indicates that Web queries written by software developers are not very successful in retrieving relevant results, performing measurably worse compared to general purpose Web queries. Most approaches up to this point have addressed this problem with software engineering-specific automated query reformulation techniques, which work without developer involvement but are limited by the content of the original query. In other words, these techniques automatically improve the existing query but can not contribute new, previously unmentioned, concepts. Objective: In this paper, we propose a technique to guide software developers in manually improving their own Web search queries. We examine a conversational approach that follows unsuccessful queries with a clarification question aimed at eliciting additional query terms, thus providing to the developer a clear dimension along which the query could be improved. Methods: We describe a set of clarification questions derived from a corpus of software developer queries and a neural approach to recommending them for a newly issued query. Results: Our evaluation indicates that the recommendation technique is accurate, predicting a valid clarification question 80% of the time and outperforms simple baselines, as well as, state-of-the-art Learning To Rank (LTR) baselines. Conclusion: As shown in the experimental results, the described approach is capable at recommending appropriate clarification questions to software developers and considered useful by a sample of developers ranging from novices to experienced professionals.