Intertemporal Connections Between Query Suggestions and Search Engine Results for Politics Related Queries

TL;DR

This paper examines the relationship between search engine results and query suggestions for political terms, analyzing their stability and similarity over time using the RBO metric before the 2017 German federal election.

Contribution

It introduces a comparative analysis of search results and query suggestions for political queries, highlighting their temporal stability and interconnection.

Findings

Query suggestions and search results show varying stability over time.

Ranked lists for different political terms exhibit measurable similarity.

The study provides insights into the temporal dynamics of political search data.

Abstract

This short paper deals with the combination and comparison of two data sources: Search engine results and query suggestions for 16 terms related to political candidates and parties. The data was collected before the federal election in Germany in September 2017 for a period of two months. The rank biased overlap (RBO) statistic is used to measure the similarity of the top-weighted rankings. For each search term and for both the search results and query auto-completions we study the stability of the rankings over time.

Figures

Tables

Table 1. Table 1: Illustration of the query suggestions data-set.

source	queryterm	date	suggestterm	position
google	Alexander Gauland	2017-08-04 05:30:51	twitter	0
google	Alexander Gauland	2017-08-04 05:30:51	itate	1
google	Alexander Gauland	2017-08-04 05:30:51	kontakt	2
google	Alexander Gauland	2017-08-04 05:30:51	dorothea gauland	3
google	Alexander Gauland	2017-08-04 05:30:51	boateng	4
google	Alexander Gauland	2017-08-04 05:30:51	krawatte	5
google	Alexander Gauland	2017-08-04 05:30:51	carola hein	6
google	Alexander Gauland	2017-08-04 05:30:51	ehefrau	7
google	Alexander Gauland	2017-08-04 05:30:51	youtube	8
google	Alexander Gauland	2017-08-04T05:30:51	islam	9
$\cdots$	$\cdots$	$\cdots$	$\cdots$	$\cdots$

Full text

11institutetext: Technische Hochschule Köln, 50678 Cologne, Germany

11email: firstname.lastname@th-koeln.de

Intertemporal Connections Between Query Suggestions and Search Engine Results for Politics Related Queries

Malte Bonart 11   

Philipp Schaer 11

1 Introduction

This paper deals with the combined analysis of two data sources: First the AlgorithmWatch data-set on search results for political queries during the German Federal Elections in 2017 [5] and second data from a long-term data crawl of search engine query suggestions. This enables us to examine result lists and query suggestions for the same set of politics related queries over a 2 month period.

Query suggestions comprise a set of interactive techniques to assist users with the formulation of their query. As Kelly et al. [4] point out they seem particularly important in cases where users are searching for topics on which they have little knowledge or familiarity and in which they are in a so called “anomalous state of knowledge” [1]. In the context of political opinion formation little is known about the influence of query suggestions in driving the users information behaviour. Regarding search results, the “search engine manipulation effect” was measured in several experimental setups [3]. It claims that biased search rankings can shift the voting preferences of undecided voters. However, the paper is criticized for presenting exaggerated effect sizes [7].

In this paper we (1) present a data set with query suggestions crawled from different web search engines to complement the AlgorithmWatch data. As a first approach we (2) compare the aggregated ranking stability of search results and query suggestions for $16$ search queries over a $2$ month observation period. Here, stability refers to the degree of similarity of successive lists over time. We suspect that external events both influence the composition of query suggestions and search results and the goal is to better understand how these two search engine features interact with each other through the search behaviour of the users. 111E.g. an external event such as the marriage of a politician, may change his or her result set since websites dealing with this event become more prominent. In addition, search engine users will possibly also search for the event in connection with the politician’s name which influences the composition of the query suggestions. In a last step, we (3) shortly discuss further research topics.

2 Data Sets

Our data-set consists of query suggestions (e.g. autocompletions) to complement the search engine results from the AlgorithmWatch data. For most queries an exact match can be obtained.222The term “cdu” is missing in our data-set. The search-terms “Bündniss90/Die Grünen” and “die linke” from the AlgorithmWatch data are compared to the search suggestions for the queries “grüne” and “dielinke”. For both data-sets we consider queries submitted between August 4st and September 30th, 2017. We programmed a web client which made calls to Google’s query suggestion API 333http://clients1.google.de/complete/search. The crawling ran usually twice a day around 05:00 and 17:00 CET and produced an outcome of $401$ unique suggestion terms during the observation period. A sample of the data set can be seen in table 1. The data is published under a Creative Commons licence [2].

For the AlgorithmWatch data we restrict the analysis to organic search results for requests submitted from within Germany using a German keyboard layout. We also cleaned the data from errors as described in [5]. This results in 896 333 unique result lists from 1 441 430 single requests.

While the comparison of different rankings from query suggestions is straightforward this is not true for the search results. Although there is only a small effect of search personalization in the data the ranking usually slightly differs between search engine users. However, the public data-set does not provide any persistent user/plugin ids which prevents the comparison of result lists on an individual level over time. As a workaround, for each time-point we construct an aggregated “typical” search result list. This is done by calculating the average position of each URL which is present in more than a third of the requests for this time-point and query.

3 First insights and discussion

Typically, result and suggestion lists of search engines are incomplete, top-weighted and indefinite. The importance of the items decreases with the depth which justifies a truncation of the list. As a measure of stability we therefore rely on the so called rank-biased overlap (RBO) statistic. It measures the similarity of two indefinite possibly non-conjoint lists and properly incorporates the characteristics of indefinite rankings [6].

The RBO statistic can take values between [math] indicating no similarity and $1$ for similar lists. The degree of top-weightiness can be controlled by a parameter $p$. For $p\rightarrow 0$ the first item in both lists are assigned all the weight while for $p\rightarrow 1$ all list items become equally important. For the analysis we choose $p=0.85$ as this puts almost all the weight ($93\%$) on the first $10$ list items and results in an expected evaluation depth of $6.7$ list items. For each query and time-point we compare the ordered lists of aggregated search results and query suggestions with the preceding list (figure 1) and with the list from the earliest time-point in the sample (figure 2). To smooth the time series, a moving average with a window of three days is applied to the RBO values.

The figures show that the degree of stability for the query suggestions and the result sets mainly follow broad similar trends. Obviously, the composition of the lists changes during the two month observation period but the stability seems relatively high: Comparing the last result list with the first one yields a minimum RBO of around $0.5$ for most queries. For some queries, the query suggestions seem to be less stable.

This initial analysis only gives a broad overview over the general trend and can shed light to some interesting patterns in the data (e.g. the negative peak in figure 1 for both search results and query suggestions for the term “CSU” around the date of the federal election). To better understand the role of specific events and how search results and query suggestions influence each other a more detailed look is required. For this it is important to study the actual, possibly dynamic, semantic similarities between the data-sets over time. It may be beneficial to include the top news headlines which appear in the organic search result lists to capture the influence of specific important events on the composition of result and suggestion lists.

In a political context it can also be interesting to ask whether query suggestions do lead to new and more diverse information sources. It could be possible that search suggestions for a specific term mainly resemble the topics and trending events listed on the term’s result page. If this is the case, the effect of query suggestions on a user’s process of political opinion formation could be neglected regardless whether he or she makes use of this feature or not.

Acknowledgments

This research was supported by the Digital Society research program funded by the Ministry of Culture and Science of the German State of North Rhine-Westphalia.