Automatic Construction of Evaluation Sets and Evaluation of Document Similarity Models in Large Scholarly Retrieval Systems

TL;DR

This paper introduces a novel automatic evaluation method for document similarity models in scholarly retrieval systems, using download logs to generate pseudo-relevant pairs and comparing their distribution to random pairs, achieving high correlation with traditional metrics.

Contribution

It presents a new approach that leverages user download logs to automatically evaluate document similarity models without human annotations.

Findings

High correlation with traditional evaluation metrics like MAP.

Efficient evaluation process using log data.

Applicable to models in term vector and topic spaces.

Abstract

Retrieval systems for scholarly literature offer the ability for the scientific community to search, explore and download scholarly articles across various scientific disciplines. Mostly used by the experts in the particular field, these systems contain user community logs including information on user specific downloaded articles. In this paper we present a novel approach for automatically evaluating document similarity models in large collections of scholarly publications. Unlike typical evaluation settings that use test collections consisting of query documents and human annotated relevance judgments, we use download logs to automatically generate pseudo-relevant set of similar document pairs. More specifically we show that consecutively downloaded document pairs, extracted from a scholarly information retrieval (IR) system, could be utilized as a test collection for evaluating document similarity models. Another novel aspect of our approach lies in the method that we employ for evaluating the performance of the model by comparing the distribution of consecutively downloaded document pairs and random document pairs in log space. Across two families of similarity models, that represent documents in the term vector and topic spaces, we show that our evaluation approach achieves very high correlation with traditional performance metrics such as Mean Average Precision (MAP), while being more efficient to compute.