Does it matter which search engine is used? A user study using post-task relevance judgments
Sebastian Suenkler, Dirk Lewandowski

TL;DR
This study compares Google and Bing in a user experiment, revealing Google’s superior relevance performance and user reliance, with relevance judgments influenced by task difficulty and result quality.
Contribution
It provides empirical evidence on search engine performance and user relevance judgments in real-world tasks, highlighting differences between Google and Bing.
Findings
Google produced more relevant results than Bing
Users relied heavily on Google for search
Relevance judgments were affected by task difficulty
Abstract
The objective of this research was to find out how the two search engines Google and Bing perform when users work freely on pre-defined tasks, and judge the relevance of the results immediately after finishing their search session. In a user study, 64 participants conducted two search tasks each, and then judged the results on the following: (1) The quality of the results they selected in their search sessions, (2) The quality of the results they were presented with in their search sessions (but which they did not click on), (3) The quality of the results from the competing search engine for their queries (which they did not see in their search session). We found that users heavily relied on Google, that Google produced more relevant results than Bing, that users were well able to select relevant results from the results lists, and that users judged the relevance of results lower when…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsInformation Retrieval and Search Behavior · Expert finding and Q&A systems · Misinformation and Its Impacts
