Why polls fail to predict elections

TL;DR

This paper investigates why traditional polls fail to predict elections, identifies biases such as social-desirability bias, and introduces a big-data analytics method using social media and machine learning that improves prediction accuracy, exemplified by the 2019 Argentina election.

Contribution

The paper reveals biases in traditional polling methods and proposes a novel big-data analytics framework that enhances election prediction accuracy.

Findings

Traditional polls underestimated social-desirability bias.

The new model accurately predicted the 2019 Argentina election outcome.

The framework can be applied to societal trend analysis beyond elections.

Abstract

In the past decade we have witnessed the failure of traditional polls in predicting presidential election outcomes across the world. To understand the reasons behind these failures we analyze the raw data of a trusted pollster which failed to predict, along with the rest of the pollsters, the surprising 2019 presidential election in Argentina which has led to a major market collapse in that country. Analysis of the raw and re-weighted data from longitudinal surveys performed before and after the elections reveals clear biases (beyond well-known low-response rates) related to mis-representation of the population and, most importantly, to social-desirability biases, i.e., the tendency of respondents to hide their intention to vote for controversial candidates. We then propose a longitudinal opinion tracking method based on big-data analytics from social media, machine learning, and network theory that overcomes the limits of traditional polls. The model achieves accurate results in the 2019 Argentina elections predicting the overwhelming victory of the candidate Alberto Fern\'andez over the president Mauricio Macri; a result that none of the traditional pollsters in the country was able to predict. Beyond predicting political elections, the framework we propose is more general and can be used to discover trends in society; for instance, what people think about economics, education or climate change.