Correcting public opinion trends through Bayesian data assimilation

TL;DR

This paper introduces a Bayesian data assimilation method that combines survey and Twitter opinion data to improve real-time estimation of public opinion, demonstrated through Brexit referendum analysis.

Contribution

It presents a novel Bayesian data assimilation framework that effectively merges traditional survey data with social media opinion mining to enhance public opinion measurement.

Findings

Successfully identified a 16-day time gap between data sources.

Effectively captured a rising trend in Leave support before Brexit.

Provided more accurate opinion estimates than individual data sources.

Abstract

Measuring public opinion is a key focus during democratic elections, enabling candidates to gauge their popularity and alter their campaign strategies accordingly. Traditional survey polling remains the most popular estimation technique, despite its cost and time intensity, measurement errors, lack of real-time capabilities and lagged representation of public opinion. In recent years, Twitter opinion mining has attempted to combat these issues. Despite achieving promising results, it experiences its own set of shortcomings such as an unrepresentative sample population and a lack of long term stability. This paper aims to merge data from both these techniques using Bayesian data assimilation to arrive at a more accurate estimate of true public opinion for the Brexit referendum. This paper demonstrates the effectiveness of the proposed approach using Twitter opinion data and survey data from trusted pollsters. Firstly, the possible existence of a time gap of 16 days between the two data sets is identified. This gap is subsequently incorporated into a proposed assimilation architecture. This method was found to adequately incorporate information from both sources and measure a strong upward trend in Leave support leading up to the Brexit referendum. The proposed technique provides useful estimates of true opinion, which is essential to future opinion measurement and forecasting research.