Academic Search Engines: Constraints, Bugs, and Recommendation
Zheng Li, Austen Rainer

TL;DR
This study investigates usability issues in academic search engines, identifying bugs and constraints through regression testing of 42 engines, and offers recommendations to improve their support for systematic reviews.
Contribution
It provides a systematic analysis of bugs and constraints in academic search engines and offers practical recommendations for developers and researchers to address usability gaps.
Findings
Identified 13 bugs across 8 search engines
Detected various usability constraints affecting researchers
Highlighted the gap between search engine development and evaluation
Abstract
Background: Academic search engines (i.e., digital libraries and indexers) play an increasingly important role in systematic reviews however these engines do not seem to effectively support such reviews, e.g., researchers confront usability issues with the engines when conducting their searches. Aims: To investigate whether the usability issues are bugs (i.e., faults in the search engines) or constraints, and to provide recommendations to search-engine providers and researchers on how to tackle these issues. Method: Using snowball-sampling from tertiary studies, we identify a set of 621 secondary studies in software engineering. By physically re-attempting the searches for all of these 621 studies, we effectively conduct regression testing for 42 search engines. Results: We identify 13 bugs for eight engines, and also identify other constraints. We provide recommendations for tackling…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsOpen Source Software Innovations · Software Engineering Research · Information Retrieval and Search Behavior
