Measuring the Salad Bowl: Superdiversity on Twitter

TL;DR

This paper introduces a superdiversity index derived from Twitter sentiment analysis that correlates with immigration rates and can potentially predict immigration trends in regions lacking official data.

Contribution

It develops a novel superdiversity index based on sentiment shifts in multilingual Twitter data and validates its correlation with immigration statistics.

Findings

Index correlates with official immigration data across regions.

Method works effectively for both English and Italian tweets.

Potential for nowcasting immigration in data-scarce areas.

Abstract

Superdiversity refers to large cultural diversity in a population due to immigration. In this paper, we introduce a superdiversity index based on the changes in the emotional content of words used by a multi-cultural community, compared to the standard language. To compute our index we use Twitter data and we develop an algorithm to extend a dictionary for lexicon-based sentiment analysis. We validate our index by comparing it with official immigration statistics available from the European Commission's Joint Research Center, through the D4I data challenge. We show that, in general, our measure correlates with immigration rates, at various geographical resolutions. Our method produces very good results across languages, being tested here both on English and Italian tweets. We argue that our index has predictive power in regions where exact data on immigration is not available, paving the way for a nowcasting model of immigration rates.