Learning to Rank Scientific Documents from the Crowd

TL;DR

This paper introduces a crowd-sourcing method to create an expert-annotated corpus for ranking scientific articles and develops a supervised learning-to-rank model that outperforms traditional text similarity approaches.

Contribution

It presents a novel crowd-sourcing approach for expert annotation and a new supervised learning-to-rank model tailored for scientific document relatedness.

Findings

Expert rankings differ from text similarity models

Supervised learning-to-rank outperforms baselines

SVM-Rank achieves significant improvement

Abstract

Finding related published articles is an important task in any science, but with the explosion of new work in the biomedical domain it has become especially challenging. Most existing methodologies use text similarity metrics to identify whether two articles are related or not. However biomedical knowledge discovery is hypothesis-driven. The most related articles may not be ones with the highest text similarities. In this study, we first develop an innovative crowd-sourcing approach to build an expert-annotated document-ranking corpus. Using this corpus as the gold standard, we then evaluate the approaches of using text similarity to rank the relatedness of articles. Finally, we develop and evaluate a new supervised model to automatically rank related scientific articles. Our results show that authors' ranking differ significantly from rankings by text-similarity-based models. By training a learning-to-rank model on a subset of the annotated corpus, we found the best supervised learning-to-rank model (SVM-Rank) significantly surpassed state-of-the-art baseline systems.