On the Biased Assessment of Expert Finding Systems

TL;DR

This paper examines how automated annotation biases in expert finding system evaluations can lead to overestimated performance and unfair comparisons, proposing constraints to improve evaluation fairness.

Contribution

It reveals biases caused by automated annotations and synonyms in expert retrieval benchmarks, and proposes constraints to mitigate these biases.

Findings

Automated annotations inflate performance metrics for traditional models.

Synonym augmentation introduces bias towards literal mentions.

Constraints can reduce bias while maintaining annotation utility.

Abstract

In large organisations, identifying experts on a given topic is crucial in leveraging the internal knowledge spread across teams and departments. So-called enterprise expert retrieval systems automatically discover and structure employees' expertise based on the vast amount of heterogeneous data available about them and the work they perform. Evaluating these systems requires comprehensive ground truth expert annotations, which are hard to obtain. Therefore, the annotation process typically relies on automated recommendations of knowledge areas to validate. This case study provides an analysis of how these recommendations can impact the evaluation of expert finding systems. We demonstrate on a popular benchmark that system-validated annotations lead to overestimated performance of traditional term-based retrieval models and even invalidate comparisons with more recent neural methods. We also augment knowledge areas with synonyms to uncover a strong bias towards literal mentions of their constituent words. Finally, we propose constraints to the annotation process to prevent these biased evaluations, and show that this still allows annotation suggestions of high utility. These findings should inform benchmark creation or selection for expert finding, to guarantee meaningful comparison of methods.