Studies on Relevance, Ranking and Results Display

TL;DR

This paper explores how paleontologists perceive relevance in search results, developing a prototype system that clusters scholarly articles into relevance categories and evaluates user trust and preferences in display methods.

Contribution

It introduces a rules-based clustering algorithm for relevance categorization and assesses user trust and preferences in a specialized digital library system.

Findings

System matches 87% of relevance ratings

Users trust relevance labels but prefer standard layouts

Time constraints influence relevance judgments

Abstract

This study considers the extent to which users with the same query agree as to what is relevant, and how what is considered relevant may translate into a retrieval algorithm and results display. To combine user perceptions of relevance with algorithm rank and to present results, we created a prototype digital library of scholarly literature. We confine studies to one population of scientists (paleontologists), one domain of scholarly scientific articles (paleo-related), and a prototype system (PaleoLit) that we built for the purpose. Based on the principle that users do not pre-suppose answers to a given query but that they will recognize what they want when they see it, our system uses a rules-based algorithm to cluster results into fuzzy categories with three relevance levels. Our system matches at least 1/3 of our participants' relevancy ratings 87% of the time. Our subsequent usability study found that participants trusted our uncertainty labels but did not value our color-coded horizontal results layout above a standard retrieval list. We posit that users make such judgments in limited time, and that time optimization per task might help explain some of our findings.