Gold Standard for Expert Ranking: A Survey on the XWiki Dataset

TL;DR

This paper presents a survey-based approach to establish a gold standard for expert ranking in Requirements Engineering, analyzing the reliability of human-generated rankings to evaluate automated expert identification techniques.

Contribution

It introduces a methodology for creating and validating a gold standard for expert ranking using external survey data in the context of Requirements Engineering.

Findings

The gold standard is reasonably reliable despite some uncertainties.

Subjects with less ordered rankings tend to be more reliable.

The results challenge common assumptions about ranking reliability in Information Retrieval.

Abstract

We are designing an automated technique to find and recommend experts for helping in Requirements Engineering tasks, which can be done by ranking the available people by level of expertise. For evaluating the correctness of the rankings produced by the automated technique, we want to compare them to a gold standard. In this work, we ask external people to look at a set of discussions and to rank their participants, before to evaluate the reliability of these rankings to serve as a gold standard. We describe the setting and running of this survey, the method used to build the gold standard from the rankings of the subjects, and the analysis of the results to obtain and validate this gold standard. Through the analysis of the results, we conclude that we obtained a reasonable gold standard although we lack evidences to support its total correctness. We also made the interesting observation that the most reliable subjects build the least ordered rankings (i.e. has few ranks with several people per rank), which goes against the usual assumptions of Information Retrieval measures.