It Takes So Little to Change So Much: Investigating the Robustness of a Danish Voting Advice Algorithm

TL;DR

This paper critically examines the robustness of a Danish Voting Advice Application, revealing that small algorithmic changes can significantly alter recommendations, thereby impacting voter trust and election outcomes.

Contribution

It provides an empirical audit of the Kandidattest VAA, highlighting its sensitivity to minor algorithmic modifications and raising concerns about its reliability.

Findings

Small algorithmic adjustments lead to different matching results.

The VAA's recommendations are not robust against minor changes.

Potential influence on election results due to algorithm brittleness.

Abstract

Voting Advice Applications (VAA) are tools designed to help voters compare political candidates on policy preferences prior to elections. VAAs are popular tools in European countries and in other countries with multi-party democratic systems. Through a freedom of information request we got access to the inner workings of a popular Danish VAA called the Kandidattest which is implemented by major Danish news outlet and has been used for general, municipal, and European elections. Users and politicians from every political party answer the same online questionnaire and get matched based on the agreement percentage stemming from their answers. VAAs play a significant role in elections with 45% of surveyed voters reporting they followed its recommendations in the past Danish general election, however, the inner workings of VAAs have not been thoroughly evaluated. We find that the algorithm is not robust enough for users to trust the agreement percentages in the output, as small changes to the algorithm can lead to different results, potentially affecting election results. We conduct an algorithmic audit of the Kandidattest's robustness, using simulated responses to investigate the tool's brittleness, with respect to minor adjustments of the algorithm's weight, and changes in the number of questions of the questionnaire.