Correlation of Expert and Search Engine Rankings

TL;DR

This study investigates whether expert rankings of real-world entities correlate with search engine rankings, finding limited significant correlations that tend to decrease as list sizes increase.

Contribution

It provides empirical evidence on the correlation between expert assessments and search engine rankings across various topics and list sizes.

Findings

Few significant correlations between expert and search engine rankings.

Correlations tend to decrease as list size increases.

Strong correlations observed only for smaller lists of 10 items.

Abstract

In previous research it has been shown that link-based web page metrics can be used to predict experts' assessment of quality. We are interested in a related question: do expert rankings of real-world entities correlate with search engine rankings of corresponding web resources? For example, each year US News & World Report publishes a list of (among others) top 50 graduate business schools. Does their expert ranking correlate with the search engine ranking of the URLs of those business schools? To answer this question we conducted 9 experiments using 8 expert rankings on a range of academic, athletic, financial and popular culture topics. We compared the expert rankings with the rankings in Google, Live Search (formerly MSN) and Yahoo (with list lengths of 10, 25, and 50). In 57 search engine vs. expert comparisons, only 1 strong and 4 moderate correlations were statistically significant. In 42 inter-search engine comparisons, only 2 strong and 4 moderate correlations were statistically significant. The correlations appeared to decrease with the size of the lists: the 3 strong correlations were for lists of 10, the 8 moderate correlations were for lists of 25, and no correlations were found for lists of 50.