Academic Search Engines: Constraints, Bugs, and Recommendation

TL;DR

This study investigates usability issues in academic search engines, identifying bugs and constraints through regression testing of 42 engines, and offers recommendations to improve their support for systematic reviews.

Contribution

It provides a systematic analysis of bugs and constraints in academic search engines and offers practical recommendations for developers and researchers to address usability gaps.

Findings

Identified 13 bugs across 8 search engines

Detected various usability constraints affecting researchers

Highlighted the gap between search engine development and evaluation

Abstract

Background: Academic search engines (i.e., digital libraries and indexers) play an increasingly important role in systematic reviews however these engines do not seem to effectively support such reviews, e.g., researchers confront usability issues with the engines when conducting their searches. Aims: To investigate whether the usability issues are bugs (i.e., faults in the search engines) or constraints, and to provide recommendations to search-engine providers and researchers on how to tackle these issues. Method: Using snowball-sampling from tertiary studies, we identify a set of 621 secondary studies in software engineering. By physically re-attempting the searches for all of these 621 studies, we effectively conduct regression testing for 42 search engines. Results: We identify 13 bugs for eight engines, and also identify other constraints. We provide recommendations for tackling these issues. Conclusions: There is still a considerable gap between the search-needs of researchers and the usability of academic search engines. It is not clear whether search-engine developers are aware of this gap. Also, the evaluation, by academics, of academic search engines has not kept pace with the development, by search-engine providers, of those search engines. Thus, the gap between evaluation and development makes it harder to properly understand the gap between the search-needs of researchers and search-features of the search engines.