Image-based Social Sensing: Combining AI and the Crowd to Mine Policy-Adherence Indicators from Twitter

TL;DR

This paper introduces VisualCit, a pipeline that leverages image recognition, geocoding, and crowdsourcing to extract policy-adherence indicators from social media images, aiding COVID-19 response efforts.

Contribution

The paper presents a novel image-based social sensing pipeline that combines AI and crowdsourcing to measure policy adherence from social media images, specifically for COVID-19.

Findings

Social media images can reliably indicate policy adherence.

VisualCit correlates well with existing behavior trackers.

Image-based sensing provides timely data for policymakers.

Abstract

Social Media provides a trove of information that, if aggregated and analysed appropriately can provide important statistical indicators to policy makers. In some situations these indicators are not available through other mechanisms. For example, given the ongoing COVID-19 outbreak, it is essential for governments to have access to reliable data on policy-adherence with regards to mask wearing, social distancing, and other hard-to-measure quantities. In this paper we investigate whether it is possible to obtain such data by aggregating information from images posted to social media. The paper presents VisualCit, a pipeline for image-based social sensing combining recent advances in image recognition technology with geocoding and crowdsourcing techniques. Our aim is to discover in which countries, and to what extent, people are following COVID-19 related policy directives. We compared the results with the indicators produced within the CovidDataHub behavior tracker initiative. Preliminary results shows that social media images can produce reliable indicators for policy makers.