Bullseye: Structured Passage Retrieval and Document Highlighting for Scholarly Search

TL;DR

Bullseye enhances scholarly search by accurately retrieving, structuring, and highlighting relevant passages within research papers, significantly improving user efficiency and satisfaction.

Contribution

The paper introduces Bullseye, a novel system that integrates passage retrieval, document structure detection, and highlighting to improve scholarly search effectiveness.

Findings

Modest improvement in system effectiveness over baseline

Substantial enhancement in user effectiveness and effort

Strong user demand for passage highlighting

Abstract

We present the Bullseye system for scholarly search. Given a collection of research papers, Bullseye: 1) identifies relevant passages using any on-the-shelf algorithm; 2) automatically detects document structure and restricts retrieved passages to user-specifed sections; and 3) highlights those passages for each PDF document retrieved. We evaluate Bullseye with regard to three aspects: system effectiveness, user effectiveness, and user effort. In a system-blind evaluation, users were asked to compare passage retrieval using Bullseye vs. a baseline which ignores document structure, in regard to four types of graded assessments. Results show modest improvement in system effectiveness while both user effectiveness and user effort show substantial improvement. Users also report very strong demand for passage highlighting in scholarly search across both systems considered.